Automating product tracking: Create a product scraper and sale notification system with Node-RED and Telegram

Web interface dashboard, save products to database, get discounts notification through Telegram, check the history

Node-RED is a great tool for creating a product scraper, as you can make the web interface and connection with a database fast, easy, and reliable.

This is quite a complex project, so I’ll show you how to adapt it to your needs and put the flow at the bottom of the article, so you can import it.

Product list example from the scraper UI

It’s also great for getting a push notification or a message on Telegram, Messenger, WhatsApp, or similar services when the product you are looking for is on a discount. Check my articles for chatbots with Node-RED.

If you own a store, you can consider using this to watch for price changes between different concurrent stores.

Define the product data structure

The objective is clear: 🔍 to scrape information about product title, price, sale price, and other relevant data such as description, image, new, used, and resealed price.

I’ll pick, as an example, a popular e-commerce website from Eastern Europe, but it doesn’t matter what website you want to use as a target, as this is intended to be a universal approach that will work for most of the online stores.

Define the product data structure and focus

As you can see, I focus on the 🟨 product title, 🟧 images, 🟫 reviews, 🟩 recommended or original product price and the sale price, 🟦 the resealed price and probably the most important is 🟪 product code, as is possible to associate products in the different stores.

Identify the HTML tags

Now you know what you want to scrape, let’s identify the HTML tags which can be used in order to get the product information we want.

Using Chrome, Firefox, or your favorite browser, right-click on the element you want to extract and select “Inspect” or “Inspect element”