Scrapify
Visit here to see a live demo
Table of Contents Link to heading
Problem Formulation Link to heading
As someone with great interest in mechanical keyboards, I found that almost all shops were created and hosted using Shopify. Often times I had to look at multiple sites for checking the availability, prices and timelines.
The idea to build this originated when I found out that Shopify makes
it really easy to get information about the products, by exposing a
products.json
file for the store (with some limitations). The problem however
was that these JSON files are over 100k+ lines in length and overflowing with
data. So this is esentially a parser which is super easy to use and allows
flexibility to add any website to the stores being parsed, provided it’s a
Shopify storefront.
Solution Link to heading
Language and tools used: Link to heading
- Go: the app is entirely written in Go and uses standard libraries for everything
- Docker: Easy containerization, run anywhere
- Prettier: for formatting the JSON files
Working Link to heading
Source code can be found here
- The list of sites along with the domain is added to a
sites.csv
file - The app parses the files and downloads the
products.json
files from the stores and stores them in ajson/
directory - The app then starts a simple server, which after parsing the JSON files displays the data using a HTML template
Features: Link to heading
- Sort data based on vendors and tags
- Easily see stock status
- Easy to add any sites, built with flexibility in mind
Screenshots Link to heading