A web scraper for testing if a web page is secure towards scrapers and web spiders.
- The scraper is a command-line tool that first needs a .txt with all the pages that are going to be wrapped:
- Then you only need to execute the index.js with node
node src/index.js
-
This will create one folder for each web page containing the headers and the data.html
-
Amazon example
-
For cleaning the scraped folder just execute
node src/index.js --clean