2-Running_MADAMIRA.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Processing for Arabic - Using MADAMIRA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "MADAMIRA is an end-to-end morphological disambiguator for Arabic. It generates ranked morphological analyses for each word in context.\n",
    "In this tutorial, we use raw text files as input and expect structured unmarked output.\n",
    "_**Please refer to the MADAMIRA manual for to learn more the about the different features and configurations.**_\n",
    "\n",
    "MADAMIRA is written in JAVA. To call MADAMIRA use:\n",
    "```\n",
    "java -Xmx4000m -Xms4000m -XX:NewRatio=3 -jar MADAMIRA-release-20170403-2.1.jar -rawinput <input_text> -rawoutdir <output_dir> -rawconfig sampleConfigFile.xml\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each word in the sentence MADAMIRA produces an array of morphosyntactic and lexical features. Additionally MADMAIRA produces separate tokenization files in different schemes.\n",
    "\n",
    "In this exercise we will be looking at the following features and tokenization schemes:\n",
    "\n",
    "1. Diacritized word - `diac`: the diacritized form of the word.\n",
    "2. Lemma - `lex`: the citation form of the word.\n",
    "3. POS - `pos`: part-of-speech (34 pos tagset).\n",
    "4. ATB tokenization: all prepositions, articles and pronouns are split, except for the definite article 'Al'\n",
    "5. D3 tokenization: all prepositions, articles and pronouns are split.\n",
    "\n",
    "The output for this exercise will be in the specified `<output_dir>`, please open the files ending with the extensions: `.mada`, `.ATB.tok`, and `.D3.tok` to examine them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run MADAMIRA: Gigaword_AR as example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "cd MADAMIRA\n",
    "java -Xmx4000m -Xms4000m -XX:NewRatio=3 -jar MADAMIRA-release-20170403-2.1.jar -rawinput ../Results/Gigaword_AR/gigaword_tiny_cleaned.txt -rawoutdir ../Results/Gigaword_AR/ -rawconfig WIDH_2020_ConfigFile.xml "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parse .mada output file for additional representations:\n",
    "\n",
    "We will extract the following representations out of the .mada file using the parse_mada function defined below.\n",
    "1. Diacritized text.\n",
    "2. Undiacritized text.\n",
    "3. Lemmatized text.\n",
    "4. Lemma and POS pairs.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "from camel_tools.utils.dediac import dediac_ar\n",
    "_LEMMA_SPLIT_RE = re.compile(r'-|_')\n",
    "\n",
    "def parse_mada(filename):\n",
    "    sentence_count = 0 # keep a sentence counter\n",
    "\n",
    "    diac_text = [] # sentecnes of diacritized words\n",
    "    undiac_text = [] # sentences of undiacritized words\n",
    "    lemmas_text = [] # sentences of lemmas\n",
    "    lemma_and_pos = [] # list of lemmas and their corrisponding POS pairs\n",
    "    with open(filename, 'r', encoding='utf-8') as f:\n",
    "        for line in f:\n",
    "            if line.startswith(';;; SENTENCE'): # encounter a sentence\n",
    "                diac_str = []\n",
    "                undiac_str = []\n",
    "                lemma_str = []\n",
    "                lemma_pos_str = []\n",
    "            elif line.startswith('*'): # encounter the analysis line\n",
    "                pos = re.search('pos:(\\S+)', line).group(1)\n",
    "                diac = re.search('diac:(\\S+)', line).group(1)\n",
    "                undiac = dediac_ar(diac)\n",
    "                lemma = re.search('lex:(\\S+)', line).group(1)\n",
    "                lemma = _LEMMA_SPLIT_RE.split(lemma)[0] # extract the lexical part of the lemma only\n",
    "            \n",
    "                diac_str.append(diac)\n",
    "                undiac_str.append(undiac)\n",
    "                lemma_str.append(lemma)\n",
    "                lemma_pos_str.append(f'{lemma}+{pos}')\n",
    "            elif line.startswith(';;WORD'): # encounter a word\n",
    "                word = line.strip().split(' ')[1]\n",
    "            elif line.startswith('NO-ANALYSIS'): # MADAMIRA did not generate analysis for the word\n",
    "                diac_str.append(word) # use the raw word as a placeholder\n",
    "                undiac_str.append(word) # use the raw word as a placeholder\n",
    "                lemma_str.append(word) # use the raw word as a placeholder\n",
    "                lemma_pos_str.append('{}+NOAN'.format(word)) \n",
    "            elif line.startswith('SENTENCE BREAK'): # encounter the end of the sentence:\n",
    "                diac_text.append(' '.join(diac_str))\n",
    "                undiac_text.append(' '.join(undiac_str))\n",
    "                lemmas_text.append(' '.join(lemma_str))\n",
    "                lemma_and_pos.append(' '.join(lemma_pos_str))\n",
    "            else:\n",
    "                continue\n",
    "    # write to file\n",
    "    with open(filename+'.diac', 'w', encoding='utf-8') as f:\n",
    "        f.write('\\n'.join(diac_text))\n",
    "    with open(filename+'.undiac', 'w', encoding='utf-8') as f:\n",
    "        f.write('\\n'.join(undiac_text))\n",
    "    with open(filename+'.lex', 'w', encoding='utf-8') as f:\n",
    "        f.write('\\n'.join(lemmas_text))\n",
    "    with open(filename+'.lexPOS', 'w', encoding='utf-8') as f:\n",
    "        f.write('\\n'.join(lemma_and_pos))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### To call the `parse_mada()` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parse_mada('Results/Gigaword_AR/gigaword_tiny_cleaned.txt.mada')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}