Web Scraping – Mining the Web!

Web Scraping:

is the automated collection of data from one or more websites. The data is usually stored in a database for later use.

So why would you want to do this?

Simply, data or information is valuable to a business. A web scraping bot can efficiently search the web saving the results in a database, providing updated information to meet your needs.

here are many examples of using web scraping, let’s look at a couple of them.

Market Research:

You need info from say eBay or Craigslist. You want to answer Where, how many and for how much. Using a web scrapping bot will automate your task. Providing quantitative data in a reportable format.

Guaranteed Lowest Price:

Your company wants to have a sale each week and guarantee the lowest price on selected items. However, they never want to lose money on a sales item. Solution, use a web bot to crawl the completions websites, store the info in a database. Then run a report cross referencing potential sales items, know lowest price to your company’s price break point sorted on profit margin. This report quickly allows management to make a decisions based upon fact not assumptions.

Travel Packages / Travel Club:

So you want to start a travel business and offer unique travel packages. Using web scrapping you can search all of the travel sites and identify opportunities that fit your market and have value written all over them. It’s like being able to regularly finding the needle in the haystack.

Real Estate:

So you know the market, but the information you need in in 3 different websites. You need to cross reference all three and then rank the results. But doing this manually takes too much time. So simply web scrape all 3 sites, placing the data into a database. Now you can let the database do the cross referencing, producing reports that allow you or your clients to make better decisions faster. Time is money and the first one to make an offer has the best chance to walk with a great deal!

Collecting and merging data:

So your company is growing fast and has purchase 5 companies in the past year. Each of these companies has the data and reports in different formats. Web scraping might be the best solution to quickly get the information needed by top management.

Creating a buyer’s club:

Collecting monthly fees and say 1 to 5 percent from each purchase.

So let’s say you are in the welding business (or cleaning or electrical or plumbing…) and you spend thousands each month on parts and supplies. You know there are hundreds if not thousands of small businesses just like you across the country. Would it not be great if you could sign up a hundreds of these businesses that you can guarantee to save them at least 10%. Using your web scrapping database, you could even prove it – on every invoice you could show the current list price from 2 competing vendors.

At DevAppCo, we specialize in providing cost effective Web Scrapping solutions!

© 2016 DevAppCo