Scrapify

Visit here to see a live demo

Table of Contents Link to heading

Problem Formulation Link to heading

As someone with great interest in mechanical keyboards, I found that almost all shops were created and hosted using Shopify. Often times I had to look at multiple sites for checking the availability, prices and timelines.

The idea to build this originated when I found out that Shopify makes it really easy to get information about the products, by exposing a products.json file for the store (with some limitations). The problem however was that these JSON files are over 100k+ lines in length and overflowing with data. So this is esentially a parser which is super easy to use and allows flexibility to add any website to the stores being parsed, provided it’s a Shopify storefront.

Solution Link to heading

Language and tools used: Link to heading

  • Go: the app is entirely written in Go and uses standard libraries for everything
  • Docker: Easy containerization, run anywhere
  • Prettier: for formatting the JSON files

Working Link to heading

Source code can be found here

  • The list of sites along with the domain is added to a sites.csv file
  • The app parses the files and downloads the products.json files from the stores and stores them in a json/ directory
  • The app then starts a simple server, which after parsing the JSON files displays the data using a HTML template

Features: Link to heading

  • Sort data based on vendors and tags
  • Easily see stock status
  • Easy to add any sites, built with flexibility in mind

Screenshots Link to heading

Screenshots of the App Screenshots of the App