Web Scrapping Task.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2b202d66",
   "metadata": {},
   "source": [
    "# Web Scrapping and Data Analysis\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46eda4fc",
   "metadata": {},
   "source": [
    "# Scrapping Data from Skytrax\n",
    "f you visit [https://www.airlinequality.com] you can see that there is a lot of data there. For this task, we are only interested in reviews related to British Airways and the Airline itself.\n",
    "\n",
    "If you navigate to this link: [https://www.airlinequality.com/airline-reviews/british-airways] you will see this data. Now, we can use Python and BeautifulSoup to collect all the links to the reviews and then to collect the text data on each of the individual review links."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "afebad9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Importing Liabraries\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns \n",
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "61ffa54f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scraping page 1\n",
      "   ---> 100 total reviews\n",
      "Scraping page 2\n",
      "   ---> 200 total reviews\n",
      "Scraping page 3\n",
      "   ---> 300 total reviews\n",
      "Scraping page 4\n",
      "   ---> 400 total reviews\n",
      "Scraping page 5\n",
      "   ---> 500 total reviews\n",
      "Scraping page 6\n",
      "   ---> 600 total reviews\n",
      "Scraping page 7\n",
      "   ---> 700 total reviews\n",
      "Scraping page 8\n",
      "   ---> 800 total reviews\n",
      "Scraping page 9\n",
      "   ---> 900 total reviews\n",
      "Scraping page 10\n",
      "   ---> 1000 total reviews\n",
      "Scraping page 11\n",
      "   ---> 1100 total reviews\n",
      "Scraping page 12\n",
      "   ---> 1200 total reviews\n",
      "Scraping page 13\n",
      "   ---> 1300 total reviews\n",
      "Scraping page 14\n",
      "   ---> 1400 total reviews\n",
      "Scraping page 15\n",
      "   ---> 1500 total reviews\n",
      "Scraping page 16\n",
      "   ---> 1600 total reviews\n",
      "Scraping page 17\n",
      "   ---> 1700 total reviews\n",
      "Scraping page 18\n",
      "   ---> 1800 total reviews\n",
      "Scraping page 19\n",
      "   ---> 1900 total reviews\n",
      "Scraping page 20\n",
      "   ---> 2000 total reviews\n",
      "Scraping page 21\n",
      "   ---> 2100 total reviews\n",
      "Scraping page 22\n",
      "   ---> 2200 total reviews\n",
      "Scraping page 23\n",
      "   ---> 2300 total reviews\n",
      "Scraping page 24\n",
      "   ---> 2400 total reviews\n",
      "Scraping page 25\n",
      "   ---> 2500 total reviews\n",
      "Scraping page 26\n",
      "   ---> 2600 total reviews\n",
      "Scraping page 27\n",
      "   ---> 2700 total reviews\n",
      "Scraping page 28\n",
      "   ---> 2800 total reviews\n",
      "Scraping page 29\n",
      "   ---> 2900 total reviews\n",
      "Scraping page 30\n",
      "   ---> 3000 total reviews\n"
     ]
    }
   ],
   "source": [
    "# Set the URL of the paginated webpage that you want to scrape\n",
    "url = \"https://www.airlinequality.com/airline-reviews/british-airways\"\n",
    "\n",
    "# Initialize an empty list to store the data that you scrape\n",
    "data = []\n",
    "\n",
    "# Setting the initial page number and the increment that you want to use to paginate through the webpage\n",
    "page_num = 1\n",
    "page_incr = 1\n",
    "page_size = 100\n",
    "# maximum number of pages to be scraped\n",
    "max_pages = 30\n",
    "\n",
    "# Set the URL of the webpage to be scraped \n",
    "paginated_url = f\"{url}/page/{page_num}/?sortby=post_date%3ADesc&pagesize={page_size}\"\n",
    "\n",
    "# A while loop to paginate through the webpage and scrape the data\n",
    "while page_num <= max_pages:\n",
    "\n",
    "    print(f\"Scraping page {page_num}\")\n",
    "\n",
    "    # A GET request to the paginated URL\n",
    "    response = requests.get(paginated_url)\n",
    "\n",
    "    # Parsing the response using BeautifulSoup\n",
    "    parsed_content = BeautifulSoup(response.text, \"html.parser\")\n",
    "\n",
    "    # Finding all the elements on the page that contain the data to be scraped\n",
    "    elements = parsed_content.find_all(\"div\",class_ = \"body\")\n",
    "\n",
    "    # Looping through the elements and extract the data that you want to scrape\n",
    "    for element in elements:\n",
    "        header = element.find(\"h2\",class_ = \"text_header\").text.replace(\"\\n\", \" \")\n",
    "        sub_header = element.find(\"h3\",class_ = \"text_sub_header\").text.replace(\"\\n\", \" \")\n",
    "        content = element.find(\"div\",class_ = \"text_content\").text.replace(\"\\n\", \" \")\n",
    "        \n",
    "        data.append([header,sub_header,content])\n",
    "\n",
    "    # Increasing the page number and setting the paginated URL to the new page\n",
    "    page_num += page_incr\n",
    "    paginated_url = f\"{url}/page/{page_num}/?sortby=post_date%3ADesc&pagesize={page_size}\"\n",
    "\n",
    "    print(f\"   ---> {len(data)} total reviews\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "404a91b3",
   "metadata": {},
   "source": [
    "#The loops above collected 3000 reviews by iterating through the paginated pages on the website.\n",
    "\n",
    "#The next thing that you should do is clean this data to remove any unnecessary text from each of the rows. For example, \"✅ Trip Verified\" can be removed from each row if it exists, as it's not relevant to what we want to investigate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "fa083856",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>REVIEW</th>\n",
       "      <th>PERSONAL INFO</th>\n",
       "      <th>CONTENT</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\"Customer Service does not exist\"</td>\n",
       "      <td>N Hancock (United Kingdom) 18th July 2023</td>\n",
       "      <td>✅ Trip Verified |  Customer Service does not e...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\"another great experience\"</td>\n",
       "      <td>Steven Hodgson (United Kingdom) 17th July 2023</td>\n",
       "      <td>✅ Trip Verified | Another really great pair of...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"Not recommended\"</td>\n",
       "      <td>John Grainger (United Kingdom) 17th July 2023</td>\n",
       "      <td>Not Verified |  Our A380 developed a fault tax...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\"5 hours and 30 mins but no meal\"</td>\n",
       "      <td>K Robson (United Kingdom) 16th July 2023</td>\n",
       "      <td>Not Verified | Horrible airline. Does not care...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\"Things have really deteriorated\"</td>\n",
       "      <td>Pradeep Madhavan (United Kingdom) 9th July 2023</td>\n",
       "      <td>✅ Trip Verified |  My family and I have flown ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2995</th>\n",
       "      <td>British Airways customer review</td>\n",
       "      <td>I Gowon (United Kingdom) 17th March 2015</td>\n",
       "      <td>This is not reflective of BA in general only t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2996</th>\n",
       "      <td>British Airways customer review</td>\n",
       "      <td>Panos Anastasopoulos (United Kingdom) 17th M...</td>\n",
       "      <td>BA085 LHR to YVR 100315. Narrow seats without ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2997</th>\n",
       "      <td>British Airways customer review</td>\n",
       "      <td>Angie Kirkpatrick (United Kingdom) 17th Marc...</td>\n",
       "      <td>London to Budapest and back in Club class. Fir...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2998</th>\n",
       "      <td>British Airways customer review</td>\n",
       "      <td>Clive V Drake (United Kingdom) 17th March 2015</td>\n",
       "      <td>LHR-PISA-LHR. I have a house in Tuscany so fly...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2999</th>\n",
       "      <td>British Airways customer review</td>\n",
       "      <td>A Crow (United Kingdom) 17th March 2015</td>\n",
       "      <td>Flew to Bangkok with Thai Airways who were lov...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3000 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 REVIEW  \\\n",
       "0     \"Customer Service does not exist\"   \n",
       "1            \"another great experience\"   \n",
       "2                    \"Not recommended\"    \n",
       "3     \"5 hours and 30 mins but no meal\"   \n",
       "4     \"Things have really deteriorated\"   \n",
       "...                                 ...   \n",
       "2995    British Airways customer review   \n",
       "2996    British Airways customer review   \n",
       "2997    British Airways customer review   \n",
       "2998    British Airways customer review   \n",
       "2999    British Airways customer review   \n",
       "\n",
       "                                          PERSONAL INFO  \\\n",
       "0             N Hancock (United Kingdom) 18th July 2023   \n",
       "1        Steven Hodgson (United Kingdom) 17th July 2023   \n",
       "2         John Grainger (United Kingdom) 17th July 2023   \n",
       "3              K Robson (United Kingdom) 16th July 2023   \n",
       "4       Pradeep Madhavan (United Kingdom) 9th July 2023   \n",
       "...                                                 ...   \n",
       "2995           I Gowon (United Kingdom) 17th March 2015   \n",
       "2996    Panos Anastasopoulos (United Kingdom) 17th M...   \n",
       "2997    Angie Kirkpatrick (United Kingdom) 17th Marc...   \n",
       "2998     Clive V Drake (United Kingdom) 17th March 2015   \n",
       "2999            A Crow (United Kingdom) 17th March 2015   \n",
       "\n",
       "                                                CONTENT  \n",
       "0     ✅ Trip Verified |  Customer Service does not e...  \n",
       "1     ✅ Trip Verified | Another really great pair of...  \n",
       "2     Not Verified |  Our A380 developed a fault tax...  \n",
       "3     Not Verified | Horrible airline. Does not care...  \n",
       "4     ✅ Trip Verified |  My family and I have flown ...  \n",
       "...                                                 ...  \n",
       "2995  This is not reflective of BA in general only t...  \n",
       "2996  BA085 LHR to YVR 100315. Narrow seats without ...  \n",
       "2997  London to Budapest and back in Club class. Fir...  \n",
       "2998  LHR-PISA-LHR. I have a house in Tuscany so fly...  \n",
       "2999  Flew to Bangkok with Thai Airways who were lov...  \n",
       "\n",
       "[3000 rows x 3 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Coverting the list data into a dataframe\n",
    "df = pd.DataFrame(data)\n",
    "df.columns = [\"REVIEW\",\"PERSONAL INFO\",\"CONTENT\"]\n",
    "\n",
    "#Removing unwanted text(first text preprocessing)\n",
    "df.replace(re.compile(r'/s*✅ Trip Verified /|/s*'), '', inplace=True)\n",
    "df\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e3adbdd3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Saving data into csv\n",
    "df.to_csv(\"/home/papa/Downloads/British Airline/BA_reviews.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "555060d5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CONTENT</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>✅ Trip Verified |  Customer Service does not e...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>✅ Trip Verified | Another really great pair of...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Not Verified |  Our A380 developed a fault tax...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Not Verified | Horrible airline. Does not care...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>✅ Trip Verified |  My family and I have flown ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2995</th>\n",
       "      <td>This is not reflective of BA in general only t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2996</th>\n",
       "      <td>BA085 LHR to YVR 100315. Narrow seats without ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2997</th>\n",
       "      <td>London to Budapest and back in Club class. Fir...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2998</th>\n",
       "      <td>LHR-PISA-LHR. I have a house in Tuscany so fly...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2999</th>\n",
       "      <td>Flew to Bangkok with Thai Airways who were lov...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3000 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                CONTENT\n",
       "0     ✅ Trip Verified |  Customer Service does not e...\n",
       "1     ✅ Trip Verified | Another really great pair of...\n",
       "2     Not Verified |  Our A380 developed a fault tax...\n",
       "3     Not Verified | Horrible airline. Does not care...\n",
       "4     ✅ Trip Verified |  My family and I have flown ...\n",
       "...                                                 ...\n",
       "2995  This is not reflective of BA in general only t...\n",
       "2996  BA085 LHR to YVR 100315. Narrow seats without ...\n",
       "2997  London to Budapest and back in Club class. Fir...\n",
       "2998  LHR-PISA-LHR. I have a house in Tuscany so fly...\n",
       "2999  Flew to Bangkok with Thai Airways who were lov...\n",
       "\n",
       "[3000 rows x 1 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentiment_analysis_df = df.drop([\"REVIEW\",\"PERSONAL INFO\"], axis=1)\n",
    "sentiment_analysis_df.replace(re.compile(r'/s*✅ Verified Review /|/s*'), '', inplace=True)\n",
    "sentiment_analysis_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d83d21f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the DataFrame to a CSV file\n",
    "sentiment_analysis_df.to_csv(\"sentiment_content.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aca5448e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "814e358f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}