Reading a price from an online shop is easy. It gets hard when the data comes from many shops, pages keep changing and you still want to offer reliable price histories. Those are the questions I dealt with at PriceFeed.
Why data quality matters
People who watch prices make decisions with them. A retailer lowers his price because the dashboard says the competition got cheaper. If that number is wrong, he gives away margin because of a bug in a parser. So I set one hard rule. Whatever the system cannot read with certainty, it does not show. The user then sees a gap in the history and can make sense of it. With a wrong price he has no such chance, and he acts on it. In a price tracker, accuracy is the product. Wrong data can steer decisions the wrong way and wears down trust in the analysis.
The sources of error are banal. One shop shows prices including tax and the next one without. Shipping is sometimes in the price and sometimes in a footnote. In the page source, the crossed-out old price often comes before the current one, and a careless crawler grabs the first match. At PriceFeed, algorithms synchronize and check the data across sources. My advice for the start is simpler. Run every new value against the history of the same product. If a television drops to a tenth of its price overnight, that is almost never a sale. Usually a decimal separator ended up in the wrong place.
Access and data sources
Many shops push back against automated requests. They throttle them or block addresses, and after the next redesign the request comes back empty. So the first successful request proves little. A source only counts as reliable once it has run under monitoring for weeks. And a request that works technically does not mean you are allowed to use the data that way. Read the source's terms of use and clear up the legal side before you build a product on it.
Failed requests are part of normal operation. Build the system so it expects them. Every error gets a message a person can understand, retries come at growing intervals, and if a source stays down for a while, someone hears about it. Do not fetch more often than you need to, either, because the shop on the other end pays for the load too. If there is an official API or a data feed, use it. It is almost always more stable than any crawler.
When shops change their pages
A crawler needs maintenance because shops keep changing their pages. Somewhere a marketing team rearranges the product page, the price moves into a different element, and your parser returns empty fields or, worse, the item number as the price. With millions of products, that happens on any ordinary Tuesday afternoon. Two habits help. First, read the structured data that shops embed in their pages for search engines, because it usually survives a redesign. Second, monitor every source on its own. If a shop delivered a price for almost every product yesterday and for only half of them today, someone on your team needs to get an alert before the first customer asks. With a crawler, maintenance is the real work.
Not every price needs the same refresh rate
For PriceFeed we chose a stack that processes millions of data points, inside a cloud architecture with caching and load balancing. Both sound like they came straight out of a textbook. With price data, caching mostly means asking honestly how fresh a value needs to be. A price that has not moved in weeks does not need a request every minute. With a fiercely contested product, it is a different story. Treat every product the same and you pay for a lot of pointless requests, and you get blocked more often on top of it.
Load balancing solves the other end of the problem. Crawlers and users hit the same data at the same time, and both have their peak hours. Separate writing from reading as early as you can. The crawlers deliver into a queue, validation pulls the data from there, and the dashboard reads from its own prepared data set. That way a crawl run can take longer than planned without a single customer noticing it in load times. Spread the load across several machines, too, before the first one hits its ceiling at the worst possible moment.
How the data becomes useful
On the technical side, capturing the data is often the most work. For users, what counts is what they can do with the information afterwards. What people pay for is the answer to a question like “which competitor has been cheaper than me since yesterday?” At PriceFeed, a broad user base needs a different slice of the same data in each case. So in the dashboard and in the notification system, users filter for themselves and set what they want to see and when they want to hear about it.
The purchasing manager wants to see one product group. Management wants a curve across the quarter. Millions of cleanly scraped prices help neither of them if they sit in a table without filters. So work on the interface early, and watch a real user operate it. You will notice that they skip half of your lovingly built analyses and hunt for one single number. That number belongs at the top. Historical data and price comparisons can wait one level down until someone actually needs them.
What I would start with today
I would start with a few sources and with monitoring that runs from day one. With five shops you find an error in an hour. With five hundred you spend a week looking. Keep the raw data, too. If you only store the extracted price and the parser was wrong for three weeks, those three weeks of history are gone. If you stored the shop's response, you simply read them out again. Storage costs little. A hole in a customer's price history costs you the conversation about renewing the contract.
Before you start, clarify which sources you are allowed to use at all. Terms of use, robots.txt and the legal situation belong on the table before the first line of code is written. Then do an honest calculation of the running costs. Servers, monitoring and maintenance cost money every month, even before the first customer pays.
