Scrapy Assignment

Intro

This is a Python Scrapy project to scrap some info from https://arxiv.org. It is submitted as part of 1st Sem coursework for M.E. in Big Data & Data Analytics @MSIS, MAHE, Manipal.

Usage

To run the project:

Option 1

• Mount the entire project folder to Google Drive.
• Open PDV | Scrapy | Assignment.ipynb in Google Colab
• Execute the commands in Colab.

Option 2

• Run the project locally
• Ensure scrapy installed either using pip or conda
• From a terminal, switch to scrapy_arxiv_org folder & run scrapy crawl basic