2024_Notebook_3.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/TurkuNLP/ATP_kurssi/blob/master/2024_Notebook_3.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Fetch data and basic statistics\n",
        "\n",
        "1. Fetch the tweet file found at http://dl.turkunlp.org/atp/tweets_en_nort.csv\n",
        "2.  How many lines does it have? What if we remove empty lines?\n",
        "3.  Direct the non-empty lines to a file.\n",
        "4.  How many words does it have?\n",
        "5.  How does the last tweet look like? (This can be suprisingly useful when working with real data)"
      ],
      "metadata": {
        "id": "cVGiNOizCxMK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Further cleaning\n",
        "Remember that the file still had a lot of other information than just the tweets.\n",
        "\n",
        "6. Create a version of the data file where you have only the tweet texts. Read some of the lines to understand which column had the actual tweets."
      ],
      "metadata": {
        "id": "V8PbWFpAGOLe"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Matching specific patterns\n",
        "There are a lot of Charlie-related tweets.\n",
        "\n",
        "7. Do all the tweets have the hashtag #JeSuisCharlie?\n",
        "8. How could you look at tweets that do not have this hashtag?\n",
        "9. How could you search any CaPS-variant of the hashtag using one single egrep-command?\n",
        "10. How are the lines that don't have any variant of the hashtag? How many are there?\n"
      ],
      "metadata": {
        "id": "21ffCS_FDMkW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### User names and their frequencies\n",
        "\n",
        "Column 5 of the original tweet file (tweets_no_empties.txt) has user names.\n",
        "\n",
        "11. Take them to a separate file\n",
        "12. What are the 5 most frequent usernames in the dataset?\n",
        "13. How many different usernames does the data have?"
      ],
      "metadata": {
        "id": "qmXz0aEbP6M6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Duplicate tweets\n",
        "\n",
        "14. Does the tweet text data include duplicates, i.e., same tweets that have multiple occurrences in the dataset?\n",
        "15. It is a bit suspicious that the dataset includes the same tweets several times. Can you think of a way to exclude those? Uniq man pages or instructions may help here.\n",
        "16. Direct the tweets that occur only once to a file of its own.\n",
        "17. How many tweets do you end up having here?"
      ],
      "metadata": {
        "id": "yt7LxhJmOlqj"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Hashtags\n",
        "The hashtags used in the tweets are available at http://dl.turkunlp.org/atp/hashtags.txt\n",
        "\n",
        "19. Which hashtags are the most frequent ones?\n",
        "20. How many different hashtags do you have?\n",
        "21. Different versions of jesuischarlie seem to be frequent. How could you exclude those? Which are the most frequent hashtags when those are excluded?\n",
        "22. We can find hashtags with different mentions of \"je suis\" (I am) that don't include \"Charlie\" but something else. How could you find those? What do they look like? These were used to express solidarity to also other\n",
        "23. Count these hashtags with \"je suis\" that don't include Charlie but something else."
      ],
      "metadata": {
        "id": "SaYej1X2XqJy"
      }
    }
  ]
}