1-Data_Processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Processing for Arabic - Data Extraction and Cleaning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "In this exercise, we present code snippets for reading files in different formats and extracting data from them.\n",
    "Specifically, we'll look at reading raw text files, tab seperated values (TSV) files, and XML files in a complex directory structure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Cleaning Process\n",
    "\n",
    "Cleaning in this case involves the following steps:\n",
    "\n",
    "1. **Normalize unicode characters:** This process converts similar characters or character sequences\n",
    "   into their canonical forms.\n",
    "2. **Remove spaces/tabs/newlines/etc at the beginning and end of the sentence**:\n",
    "   These are unnecessary for most cases and are either the result of indentation or are accidental.\n",
    "3. **Split tokens by space and punctuation**: TThis process involves first spliting space-delimitted\n",
    "   tokens and then further splitting these tokens at punctuation boundaries. Here we treat each\n",
    "   punctuation as an individual token.\n",
    "4. **Rejoin the tokens with a single space between them**: A single space is\n",
    "   sufficient if splitting a sentence is needed later on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Building Blocks for Processing Arabic Text\n",
    "Below are some building blocks that we will use throughout this exercise:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from camel_tools.tokenizers.word import simple_word_tokenize\n",
    "from camel_tools.utils.charmap import CharMapper\n",
    "from camel_tools.utils.normalize import normalize_unicode\n",
    "\n",
    "\n",
    "# This is a character mapping object provided by camel-tools that maps\n",
    "# Buckwalter encoded characters to Arabic script.\n",
    "buckwalter_to_arabic = CharMapper.builtin_mapper('bw2ar')\n",
    "\n",
    "\n",
    "# This function takes an arabic sentence as input and cleans it by doing the\n",
    "# following:\n",
    "#    - Normalize unicode characters\n",
    "#    - Remove spaces at the beginning and end of the sentence\n",
    "#    - Splitting tokens by space and punctuation\n",
    "#    - Rejoining the tokens with a single space between them\n",
    "#\n",
    "# For example, 'ما    هذا!!! عجيب' becomes 'ما هذا ! ! ! عجيب'.\n",
    "def clean_arabic_sentence(sentence):\n",
    "    normalized_sentence = normalize_unicode(sentence)\n",
    "    stripped_sentence = normalized_sentence.strip()\n",
    "    tokens = simple_word_tokenize(stripped_sentence)\n",
    "    return ' '.join(tokens)\n",
    "\n",
    "\n",
    "# This function takes a Buckwalter-encoded sentence as input and cleans it by\n",
    "# doing the following:\n",
    "#    - Normalize unicode characters\n",
    "#    - Remove spaces at the beginning and end of the sentence\n",
    "#    - Convert sentence from Buckwalter encoding to Arabic script\n",
    "#    - Split tokens by space and punctuation\n",
    "#    - Rejoin the tokens with a single space between them\n",
    "#\n",
    "# For example, 'mA    h*A!!! Ejyb' becomes 'ما هذا ! ! ! عجيب'.\n",
    "def clean_buckwalter_sentence(sentence):\n",
    "    normalized_sentence = normalize_unicode(sentence)\n",
    "    stripped_sentence = normalized_sentence.strip()\n",
    "    arabic_sentence = buckwalter_to_arabic(stripped_sentence)\n",
    "    tokens = simple_word_tokenize(arabic_sentence)\n",
    "    return ' '.join(tokens)\n",
    "\n",
    "\n",
    "# This function takes a list of sentences and a file path as input and writes\n",
    "# the sentences to the given file one line per sentence.\n",
    "def write_sentences_to_file(sentences, file_path):\n",
    "    with open(file_path, 'w', encoding='utf-8') as output_file:\n",
    "        for sentence in sentences:\n",
    "            output_file.write(sentence)\n",
    "            output_file.write('\\n')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleaning Raw Arabic Text\n",
    "\n",
    "Reading and cleaning raw Arabic text is the most basic task one might do when\n",
    "processing Arabic text.\n",
    "Using the building blocks we defined before, we define the below function.\n",
    "This function reads a given file of raw Arabic text, cleans all the input sentences,\n",
    "and writes them to a specified output file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This function reads raw Arabic text from a specified input file, cleans it,\n",
    "# and writes the output to a specified output file.\n",
    "def clean_sentences_from_arabic_raw(input_path, output_path):\n",
    "    sentences = []\n",
    "\n",
    "    # Open the file for reading, assuming it is UTF-8 encoded\n",
    "    with open(input_path, 'r', encoding='utf8') as input_file:\n",
    "\n",
    "        # Iterate through every line in the file\n",
    "        for line in input_file:\n",
    "\n",
    "            # Remove spaces/tabs/newlines at the beginning and end of the\n",
    "            # sentence\n",
    "            cleaned_sentence = clean_arabic_sentence(line)\n",
    "\n",
    "            # Add the sentence to the existing list of sentences\n",
    "            sentences.append(cleaned_sentence)\n",
    "\n",
    "    # Write cleaned sentences to output file\n",
    "    write_sentences_to_file(sentences, output_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now run this function on the provided Gigaword file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_sentences_from_arabic_raw('Data/Gigaword_AR/gigaword_tiny.txt',\n",
    "                                'Results/Gigaword_AR/gigaword_tiny_cleaned.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleaning Buckwalter-encoded Text\n",
    "\n",
    "Cleaning Buckwalter-encoded raw text is very similar to raw Arabic text but \n",
    "involves an extra step of converting the Buckwalter-encoded sentences into\n",
    "Arabic after Unicode normalization but before splitting into word tokens.\n",
    "\n",
    "The function below performs this process on a given input file and writes the\n",
    "resulting clean text into a specified output file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This function reads Buckwalter-encoded text from a specified input file,\n",
    "# cleans it and converts it to Arabic script, and writes the output to a\n",
    "# specified output file.\n",
    "def clean_sentences_from_buckwalter_raw(input_path, output_path):\n",
    "    sentences = []\n",
    "\n",
    "    # Open the file for reading, assuming it is UTF-8 encoded\n",
    "    with open(input_path, 'r', encoding='utf8') as input_file:\n",
    "\n",
    "        # Iterate through every line in the file\n",
    "        for line in input_file:\n",
    "            # Remove spaces/tabs/newlines at the beginning and end of the\n",
    "            # sentence\n",
    "            cleaned_sentence = clean_buckwalter_sentence(line)\n",
    "\n",
    "            # Add the sentence to the existing list of sentences\n",
    "            sentences.append(cleaned_sentence)\n",
    "\n",
    "    # Write cleaned sentences to output file\n",
    "    write_sentences_to_file(sentences, output_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now run this function on the provided Hindawi file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_sentences_from_buckwalter_raw('Data/Hindawi_BW/hindawi.bw.txt',\n",
    "                                    'Results/Hindawi_BW/hindawi_cleaned.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parsing TSV (Tab-seperated Values) Files\n",
    "\n",
    "TSV files are also very common but tend to be handled incorrectly most of the\n",
    "time.\n",
    "They are used to store tabular data by seperating columns using tabs.\n",
    "They are in the same family as Comma-seperated Values (CSV) files only differing\n",
    "in the delimeter used.\n",
    "It is very convenient to use a simple split function on each sentence but\n",
    "TSVs have special formatting conventions in certain cases.\n",
    "It is recommended to use a specialized library to handle these cases\n",
    "sautomatically.\n",
    "Below is a function that uses the Python Standard Library's `csv` module\n",
    "to extract sentences from TSVs in the MADAR data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "\n",
    "# This function extracts Arabic sentences from a specified TSV file,\n",
    "# cleans them, and outputs the resulots to a specified output file.\n",
    "def clean_sentences_from_tsv(input_path, output_path):\n",
    "    sentences = []\n",
    "\n",
    "    # Open the file for reading, assuming it is UTF-8 encoded\n",
    "    with open(input_path, 'r', encoding='utf-8') as tsv_file:\n",
    "\n",
    "        # Create a DictReader object that will parse standard TSV file\n",
    "        reader = csv.DictReader(tsv_file,\n",
    "                                dialect='excel-tab',\n",
    "                                fieldnames=['sentence', 'dialect'])\n",
    "\n",
    "        # Extract MSA sentences only\n",
    "        for row in reader:\n",
    "            if row.get('dialect', None) == 'MSA':\n",
    "                sentence = row.get('sentence', '')\n",
    "                cleaned_sentence = clean_arabic_sentence(sentence)\n",
    "                sentences.append(cleaned_sentence)\n",
    "\n",
    "    # Write cleaned sentences to output file\n",
    "    write_sentences_to_file(sentences, output_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now run this function on the provided MADAR file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_sentences_from_tsv('Data/MADAR_TSV/MADAR-Corpus-6-train.tsv',\n",
    "                         'Results/MADAR_TSV/madar_cleaned.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parsing XML from a Directory\n",
    "\n",
    "Extensible Markup Language (XML) allows us to store more complex data structures\n",
    "to file but this makes processing it more complex.\n",
    "Similar to TSVs and CSVs, it is recommended to use a specialized library to read\n",
    "them since they also have special formatting conventions for certain charcaters.\n",
    "\n",
    "Furthermore, data may not necessarily come from a single file but may be split\n",
    "into many files in complex directory structures.\n",
    "\n",
    "The function below goes through a given directory and all its subdirectories,\n",
    "extracts and cleans all sentences from any XML file it encounters, and writes\n",
    "all these sentences to a single file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import os.path\n",
    "import xml.etree.ElementTree as ET\n",
    "\n",
    "\n",
    "def get_sentences_from_xml_file(file_path):\n",
    "    with open(file_path, 'r', encoding='utf-8') as xml_file:\n",
    "        sentences = []\n",
    "        for event, elem in ET.iterparse(xml_file, ['end']):\n",
    "            if elem.tag == 's':\n",
    "                cleaned_sentence = clean_arabic_sentence(elem.text)\n",
    "                sentences.append(cleaned_sentence)\n",
    "    return sentences\n",
    "\n",
    "\n",
    "def clean_sentences_from_xml_directory(directory_path, output_path):\n",
    "    all_sentences = []\n",
    "\n",
    "    # Iterate through every subdirectory in the data directory\n",
    "    for root, dirs, files in os.walk(directory_path):\n",
    "        \n",
    "        # Iterate through each file in a directory\n",
    "        for file_name in files:\n",
    "\n",
    "            if file_name.endswith('.xml'):\n",
    "                # Create a full path for each file that we can pass to open()\n",
    "                file_path = os.path.join(root, file_name)\n",
    "                \n",
    "                # Get the cleaned sentences from the current file and added to our accumulator list\n",
    "                sentences = get_sentences_from_xml_file(file_path)\n",
    "                all_sentences.extend(sentences)\n",
    "\n",
    "    # Write cleaned sentences to output file\n",
    "    write_sentences_to_file(all_sentences, output_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now run this function on the provided UN directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_sentences_from_xml_directory('Data/UN_XML',\n",
    "                                   'Results/UN_XML/un_cleaned.txt')"
   ]
  }
 ],
 "metadata": {
  "file_extension": ".py",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  },
  "mimetype": "text/x-python",
  "name": "python",
  "npconvert_exporter": "python",
  "pygments_lexer": "ipython2",
  "version": 2
 },
 "nbformat": 4,
 "nbformat_minor": 4
}