The objective of this project is to scrape a corpus of news articles from a set of web pages, pre-process the corpus, and then to apply unsupervised clustering algorithms to explore and summarise the contents of the corpus.
Part 1. Text Data Scraping This part of the project should be implemented as a Python script
- Identify the URLs for all news articles listed on the website: http://mlg.ucd.ie/modules/COMP41680/news/index.html
- Retrieve all web pages corresponding to these article URLs.
- From the web pages, extract the main body text containing the content of each news article. Save the body of each article as plain text.
Part 2. Corpus Exploration Tasks to be completed in your IPython notebook:
- Load the text corpus generated in Part 1. Apply any appropriate pre-processing steps and construct a document-term matrix representation of the corpus.
- Summarise the overall corpus by identifying the most characteristic terms and phrases in the corpus.
- Apply two alternative clustering algorithms of your choice to the document-term matrix to produce clusters of related documents. This might require applying each algorithm several times with different parameter values.
- For each clustering generated in Step 3, summarise the contents of the clusters. Based on your summary, suggest a topic/theme for each cluster.