Web Fetching
Web fetching is grabbing the content of web sites to be used and combined in different ways. screen-scraper grabs data from web sites by acting as a browser and navigating through pages while fetching the data that you specify. The data you get can then be put into a database, a spreadsheet, or anything else you need.
Also consider that screen-scraper can automatically spider through thousands of pages in a fraction of the time it would take you to grab the data manually. Web fetching with screen-scraper saves time and money.
Examples of web fetching:
✓ Crawl a genealogy web site, extracting names of ancestors for an individual
✓ Collect client information from a healthcare provider web site
✓ Periodically scrape stock data for analysis and comparison
Web Scraping
A common term for screen scraping is web scraping. Web scrapers generally go to web sites and scrape data from them, either piecemeal or by entire pages. screen-scraper is a web scraping tool that can get information from web sites very selectively or simply take whole pages, depending on what you need.
Extracting information from web sites is laborious if done by copying and pasting or by writing specialized scripts. screen-scraper dramatically reduces the amount of time it takes to perform web scraping.
Examples of web scraping:
✓ Search a real estate site by zip code and put information on properties listed into a database
✓ Scrape insurance rates so they can be compared and analyzed
✓ Automatically scrape data from different car web sites so that all the listings can be searched
Web Content Mining
The web is often referred to as the world's largest database. screen-scraper allows you to mine content from the web almost as if you were searching a database.
Just like you might search a SQL Server or mySQL database from your software, likewise you can interface your software with screen-scraper so that it mines the content you're interested in, then allows you to use that data in your own application. screen-scraper can be accessed from most modern programming languages, including Java, .NET, PHP, Visual Basic, Active Server Pages (ASP), and any language that can use SOAP.
Examples of web content mining:
✓ Crawl a web site containing data such as product prices and descriptions that you want to mine
✓ Use content mined from financial web sites to make predictions about how data such as rates and quotes might change over time
✓ Automatically scrape data from various auto or car web sites so that listings can be searched from a single web site
Web Site Data Extraction
Web sites often contain similar information that they present in different formats. Because of the different presentation formats it makes it difficult to easily compare and analyze their data. screen-scraper is a web site data extraction tool that can crawl sites for common information, even if the sites use different presentation formats.
screen-scraper can capture web pages, then pin-point specific pieces of information you'd like to extract. The time required to do web site data extraction goes down considerably in comparison to manually copying and pasting or writing special scripts.
Examples of web site data extraction:
✓ Log in to a web site containing information on products you sell through your web site, extract the data, then upload that content to your web site
✓ Extract data from a web site containing store locations, then download that data to a spreadsheet
✓ Mine content related to various insurance rates so that they can be compared and analyzed
Page Scraping
Page scraping is used for gathering information from sources on the web and putting it all at one location. That way, the data can easily be sorted, filtered, and searched. Screen-scraper scrapes the data that you need from the web pages that you want so that you can take the data and put it into a spreadsheet, a database, or any format you desire. Whether you want to keep an eye on competitor prices, compare deals, or copy an online database, screen-scraper can get you what you need.
Scraping web pages is easy and fast through screen-scraper's graphical user interface and high performance extraction engine. When screen-scraper collects data from web sites it operates as if it were a web browser, making the scraping process extremely efficient.
Examples of page scraping:
✓ Crawl a shopping site to scrape data such as product prices and descriptions
✓ Monitor mortgage rates from several lending companies
✓ Perform page scraping to download images from a stock photography web site
Web Parsing
Sometimes screen scraping is referred to as web parsing. To parse basically means to split it up based on some kind of delimiter, which is something that screen-scraper does very well. Essentially screen-scraper can act as a web page parser, finding certain data based on some constraints given to it, whether the data is for gathering product or contact information, doing market research, or otherwise.
screen-scraper can be configured to capture data from web pages as it parses through them, downloading that information to a file, spreadsheet, or database. Once data has been extracted from web pages you can then have it searched, filtered, and analyzed.
Examples of web parsing:
✓ Gather contact information for potential clients
✓ Get product listings from competitors' sites to compare with your own
✓ See multiple loan offers from several different lenders
Web Site Scraping
Scraping web sites for information of value can be done automatically by screen-scraper. Once you set up screen-scraper for a specific site it can crawl through the web pages, extracting out data such as stock quotes, mortgage rates, or travel itineraries.
A common use case for screen-scraper is to set it up to scrape multiple web sites for information, insert that information into a database, then allow it to be searched via a web interface. This is commonly known as "meta-searching", and is probably the most common use for screen-scraper. screen-scraper can mine sites for data either on-demand (e.g., right when the user submits a search form) or on a scheduled basis.
Examples of web site scraping:
✓ Create a web site that allows users to search for travel rates from multiple web sites in real time
✓ Scrape web sites containing products you'd like to sell as an affiliate, then upload that information to your own web site
✓ Mine text headlines from a news web site in order to create RSS feeds