meeting1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Meeting 1 Introduction to BeautifulSoup  \n",
    "\n",
    "Welcome to webscraping! We will start by introducting BeautifulSoup, a python package made for making webscraping easy.  \n",
    "\n",
    "\n",
    "We will be introducting some basic concepts by scraping https://quotes.toscrape.com\n",
    "\n",
    "Link to BeautifulSoup documentation:\n",
    "https://www.crummy.com/software/BeautifulSoup/bs4/doc/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "import requests\n",
    "\n",
    "\n",
    "def getSoup(url: str) -> BeautifulSoup:\n",
    "    page = requests.get(url)\n",
    "    bs = BeautifulSoup(page.content, \"html.parser\")\n",
    "    page.close()\n",
    "    \n",
    "    return bs\n",
    "\n",
    "page=\"https://quotes.toscrape.com\"\n",
    "\n",
    "soup = getSoup(page)\n",
    "print(soup.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Navigating to a specific Element\n",
    "https://www.crummy.com/software/BeautifulSoup/bs4/doc/#navigating-the-tree\n",
    "\n",
    "A beautifulSoup object contains the entire HTML tree you see when you inspect element on the browser (and maybe some extra stuff I don't even know about).\n",
    "\n",
    "#### Exercise 1: Grab the title \n",
    "Grab and print the website title \"Quotes to Scrape\" by navigating the HTML tree.\n",
    "\n",
    "\n",
    "The answer for this simple exercise should be an ugly, hardcoded list of HTML attributes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "soup.some_tags.some_more_tags.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Searching the Tree\n",
    "https://www.crummy.com/software/BeautifulSoup/bs4/doc/#searching-the-tree\n",
    "\n",
    "As you saw, that was pretty ugly. Typing those long statements to get an element won't scale when we deal with bigger, more complicated websites.\n",
    "\n",
    "#### Exercise 2: Search for the title\n",
    "Do the same as above but using a filter to grab that specific element (remove the .text from your solution to see what type it is)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# maybe there is a function for \"find\"ing things"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_quotes = []\n",
    "for quote in all_quotes:\n",
    "    print(quote)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python: Creating Better Functions  \n",
    "\n",
    "We found a pattern! Each quote has a class=\"text\" attribute. But, is scraping the quotes directly using this pattern the best thing? Well, maybe.  \n",
    "\n",
    "It might not be if we want to scrape other data from each tile on the page.\n",
    "\n",
    "#### Exercise 4: Write a function to scrape a tile  \n",
    "Each quote is contained in a tile which holds information about the author and some tags that relate to the quote.  \n",
    "Create a function "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import Tag\n",
    "def scrape_tile(tile: Tag) -> tuple[str, str, list[str]]:\n",
    "    # print(tile.prettify())\n",
    "    \n",
    "    quote = \"\"\n",
    "    author = \"\"\n",
    "    tags = []\n",
    "\n",
    "    return (quote, author, tags)\n",
    "\n",
    "all_divs = [] # find what represents the tiles\n",
    "\n",
    "for div in all_divs:\n",
    "    data = scrape_tile(div)\n",
    "    print(data)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## You did it.\n",
    "\n",
    "Now write the code for and combining the data into a pandas dataframe, or JSON object.\n",
    "\n",
    "After you're done. You've completed a baby-version of what we will be doing in this project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# something"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "myEnv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}