This tool scrapes and analysis vacancies and technologies trends from the dou.ua, allowing users to specify domains and technologies to collect the desired information.
- Scrape vacancies using the Scrapy and Selenium frameworks.
- Specify domains for search (e.g., Python, Java, QA, etc.).
- Specify technologies to measure their popularity.
- Use Jupyter Notebook to analyze data and generate charts in a PDF file.
- Support automated execution via script, as well as manual launch with Scrapy and Jupyter Notebook.
You can modify the scraping configuration in the app.config section to specify the domain and technologies to scrape.
After filling out the app.config file, you can run the automated script using:
python app/main.pyAlternatively, you can manually scrape data using:
scrapy crawl vacanciesTo launch the Jupyter Notebook manually, run it yourself as needed.
- Python 3.10+
- Scrapy
- Selenium
- Jupyter Notebook
You can install all dependencies by running:
pip install -r requirements.txt