Radmin‎ > ‎Events‎ > ‎

Scrapers

What Are Scrapers?


Scrapers are the lines of code that crawl venue and ticketing sites around the web to pull in Events for your site. Each scraper is designed to “read” a certain site on given intervals (e.g. daily, weekly, monthly) and determine if any new events have been added.

If it detects any new events, the details are then scraped and added to your Pending queue in Radmin for you to review and approve.

Managing Scrapers


The Scrapers tab in Radmin is where you can view all the Scrapers running on your site, request new Scrapers to be built, and chat with the coders about fixes that are needed.



From the main screen, you’re able to view all your metro’s Scrapers along with relevant details and recent activity for that Scraper.

You will also see icons on the right for viewing the Scraper’s details (magnifying glass), editing the Scraper (pencil), and chatting with the coder about fixing or tweaking the Scraper (speech bubble).

By clicking on the Scraper's name or magnifying glass, you can view the details page with all the information about that Scraper along with a summary of its most recent activity. You can also press "Run" to force the Scraper to run its script and look for any new Events.

Fixing a Scraper

If you find that a certain Scraper is repeatedly making mistakes, or isn’t running at all as it should, you should edit the Scraper to mark that a fix is needed and then send the coder details about the problem

To mark that a fix is needed, open up the Scraper’s details page and click “Edit”, or just click the pencil icon to the right of the Scraper. Then, check the box that says “Needs Fix” and press “Save”.

Our coders will automatically be notified that the Scraper needs a fix, but you need to provide details before they can know what the solution is. To do this, click the speech bubble icon on the main Scraper page or on that particular Scraper’s details page. In the chat window that pops up, describe the issue and any other details that the coder may need to know to diagnose the problem.

Whenever the coder responds, you will receive an email notification to check the message inbox for that Scraper. You and the coder can continue to chat about the problem in Radmin until the Scraper is fixed.

Adding a New Scraper


The Scrapers tab in Radmin is where you can view all the Scrapers running on your site, request new Scrapers to be built, and chat with the coders about fixes that are needed.

After you clicking the “New Scraper” button on the main Scraper page, you’ll be presented with the form below. These are the details that our team of coders will use to build your new Scraper.


  1. Venue: Nearly every Scraper is associated with a single Venue. Here is where you add the name of the Venue that this Scraper will be attached to. If you don’t find the right Venue from the drop down, you can add a new Venue from this screen. If the Scraper is not associated with a single Venue, check “Multiple Venues” (see #7 below).
  2. URL to Scrape: This is the URL that the Scraper will be pulling event info from. Usually it’s the calendar page for a venue or ticketing site.
  3. Category: This is the Category that will automatically be tagged to the events that get pulled in (e.g. Music, Performing Arts, Drink Specials). If it’s anything other than Comedy or Music, you will likely need to check “Ignore Bands” (see #8 below).
  4. Script: Sometimes you will need to create a new script for for a page you’re trying to scrape, but often venues will use a third-party calendar service that we already have a script for. If you know that the page you’re scraping is running a third-party calendar, check here to see if the script already exists. If you can’t find it or know that this is a custom calendar page, check the “Script Needed” box (see #9 below).
  5. Active: Check this box to make the Scraper active right away. If you ever want to turn a Scraper off, you can go to the edit Scraper page and deselect this box.
  6. Auto Approve: If you check this box, scraped events will skip the Pending queue and automatically be added to the site without review. Only use this in rare situation where you know the content does not change and you do not need to check the event listings for quality.
  7. Multiple Venues: Check this box if the URL this Scraper is pulling from is associated with multiple Venues. This is sometimes the case if it’s a promotion company’s calendar page or a festival calendar.
  8. Ignore Bands: Performing Arts events, Drink Specials, Family events, and others do not usually have a band or performer associated with them. Check this box for those Categories of events so that the Scraper doesn’t pull in false Artists.
  9. Script Needed: If there is not an existing script that can be used, you should check this box and then provide more details on the next screen.
  10. Needs Fix: This box is grayed out for new Scrapers. Read above for fixing existing Scrapers.

After filling out all the relevant details, click "Next". From here one of two things will happen. 

If you selected an existing script in #4 above, your Scraper will be created immediately. Congratulations, you're done.

If you left #4 blank and checked "Script Needed" (#7), you will be presented with the screen below. Fill out as many of the boxes as you can for the top event on the URL you are linking to, along with any notes for the coder that will be writing the new script. This way he/she will know how to design the script to read the page accurately.

Building a new script can take a few days or more, so it will take a bit longer to get your Scraper than if you use an existing script.



Want more?


Want to learn more about how Events are pulled into your site? Click here to read about approving Events in the Pending queue, handling Duplicate Events, and managing Approved Events.
Comments