Tutorial_en.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ca5c36c8-6f32-4f4c-bdf6-e2f8a49477d0",
   "metadata": {},
   "source": [
    "# Aemet OpenData\n",
    "Aemet makes available to the interested public meteorological data from some of its stations through the OpenData service (https://opendata.aemet.es/centrodedescargas/inicio).\n",
    "\n",
    "Aemet allows you to make as many data requests as you need, but there are some limitations per request.:\n",
    "* If all stations are requested, the maximum date range is 31 days.\n",
    "* If data from some stations are requested, one request per station must be made.\n",
    "* For each station, a maximum of 5 years for daily data and 36 months for monthly data can be requested.\n",
    "\n",
    "The module automates the download of meteorological time series for long time periods and for more than one station. Internally, it decomposes the request into multiple requests that satisfy Aemet constraints. For each internal request to the Aemet server, a csv file is saved. Once all the files have been downloaded, they can be stored in a Sqlite3 database and exported to a single csv file.\n",
    "\n",
    "To start using the service you have to create a free key in (https://opendata.aemet.es/centrodedescargas/inicio) and save it in the apikey.txt file. \n",
    "\n",
    "## Data and Metadata\n",
    "For each data request, you can download the data, the metadata or both (recommended). \n",
    "\n",
    "Information provided by metadata.\n",
    "* Monthly meteorological data. All columns are of type string. The data containing data of real type (float) must be deduced from the examination of the downloaded contents.\n",
    "* Daily data. Columns are of type float or string. In the float columns the separator is the comma. The daily precipitation column, in spite of being of float type, has string values to denote special values: 1) Ip = less than 0.1 mm: 2) Acum = Accumulated precipitation.\n",
    "\n",
    "Consider that the original format of the downloaded data is json. For each request, the metadata corresponds to the data available in the request. It may happen that the columns downloaded in different requests are different, as some columns are optional.\n",
    "\n",
    "## Export the downloaded files to a Sqlite database\n",
    "Once the download process is finished, you can export the data and metadata files into two tables of a Sqlite database; each of the two tables can be saved as a CSV file. If you want the daily float data to be saved with the period character as decimal separator, it is required that the metadata has been saved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "62f1f8da-6bd2-463f-8962-1ec9cbae08f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import date\n",
    "\n",
    "from aemet_open_data import AemetOpenData"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ab69dfc-86af-498b-9fd1-31655d9775dc",
   "metadata": {},
   "source": [
    "## User-supplied parameters.\n",
    "The methods used to download data require some values to be supplied by the user. Not all methods require the same parameters.\n",
    "\n",
    "* __Parameters__\n",
    "* d1 (date): Data will be requested between date d1 and date d2; d1 is the start date\n",
    "* d2 (date): Data will be requested between date d1 and date d2; d2 is the end date\n",
    "* dir_path (str): directory path where files will be saved\n",
    "* fetch (str), optional, default value 'both': Controls whether data, metadata or both will be downloaded; must be a value in ('data', 'metadata' or 'both')\n",
    "* stations (list of str): List of meteorological stations from which data is requested\n",
    "* time_step (str): Temporal resolution of meteorological data to be downloaded: 'day' or 'month'\n",
    "* use_files (bool), optional, default True: If True data saved in previously downloaded files not be requested again \n",
    "* verbose (bool), optional, default True: Controls the number of messages in the screen. If True all messages will be displayed\n",
    "\n",
    "The notebook is organized in pairs of cells:\n",
    "* In the first one the parameters are given values.\n",
    "* In the second cell the corresponding method is executed"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a44b149-9339-4ff2-bc7d-3a6c7f9e5ed2",
   "metadata": {},
   "source": [
    "## Meteorological stations in OpenData"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "28f33fde-c093-4711-bb0b-8599b3bf9d5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Required parameter\n",
    "dir_path = './download'\n",
    "# Optional parameters\n",
    "fetch = 'both'  \n",
    "use_files = True\n",
    "verbose = True  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2bf9e202-f08f-4fb0-8a82-90c1bc2bdd31",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "estaciones_open_data_data.csv: 200, OK \n",
      "estaciones_open_data_metadata.csv: 200, OK Inventario de estaciones para el apartado Valores Climatología\n"
     ]
    }
   ],
   "source": [
    "aod = AemetOpenData()\n",
    "aod.meteo_stations(dir_path)\n",
    "\n",
    "# If we specify the optional parameters, the function call will be as indicated below\n",
    "# (remove optional parameters you do not use)\n",
    "# aod.meteo_stations(dir_path, fetch=fetch, use_files=use_files, verbose=verbose)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "633d93bd-cce9-4976-b6a2-a1ec272e4d93",
   "metadata": {},
   "source": [
    "## Daily meteorological data from all stations\n",
    "When you request data from all stations, Aemet only provides daily data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "5df5fc90-0b85-49a1-be39-4c56da215d73",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Daily data from all stations between d1 and d2 \n",
    "# Required parameters\n",
    "d1 = date(2023, 1, 1)\n",
    "d2 = date(2023, 8, 1)\n",
    "dir_path = './download'\n",
    "\n",
    "# Optional parameters\n",
    "fetch ='both'  \n",
    "verbose = True  \n",
    "use_files = True  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "045295f7-71bf-43ce-8df1-987ec21f4d7e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "stations_20230101T000000UTC_20230201T235959UTC_data.csv: 200, OK \n",
      "stations_20230101T000000UTC_20230201T235959UTC_metadata.csv: 200, OK Climatologías diarias\n",
      "stations_20230202T000000UTC_20230305T235959UTC_data.csv: 200, OK \n",
      "stations_20230202T000000UTC_20230305T235959UTC_metadata.csv: 200, OK Climatologías diarias\n",
      "stations_20230306T000000UTC_20230406T235959UTC_data.csv: 200, OK \n",
      "stations_20230306T000000UTC_20230406T235959UTC_metadata.csv: 200, OK Climatologías diarias\n",
      "stations_20230407T000000UTC_20230508T235959UTC_data.csv: 200, OK \n",
      "stations_20230407T000000UTC_20230508T235959UTC_metadata.csv: 200, OK Climatologías diarias\n",
      "stations_20230509T000000UTC_20230609T235959UTC_data.csv: 200, OK \n",
      "stations_20230509T000000UTC_20230609T235959UTC_metadata.csv: 200, OK Climatologías diarias\n",
      "stations_20230610T000000UTC_20230711T235959UTC_data.csv: 200, OK \n",
      "stations_20230610T000000UTC_20230711T235959UTC_metadata.csv: 200, OK Climatologías diarias\n",
      "stations_20230712T000000UTC_20230801T235959UTC_data.csv: 200, OK \n",
      "stations_20230712T000000UTC_20230801T235959UTC_metadata.csv: 200, OK Climatologías diarias\n",
      "Downloaded files:  14\n"
     ]
    }
   ],
   "source": [
    "aod = AemetOpenData()\n",
    "downloaded_files = aod.meteo_data_all_stations(d1, d2, dir_path)\n",
    "print('Downloaded files: ', len(downloaded_files))\n",
    "\n",
    "# if you accept default values, you cau use\n",
    "# downloaded_files = aod.meteo_data_all_stations(d1, d2, dir_path, fetch, verbose, use_files)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d0c912e-c2da-4554-8e13-0c3d6f6e3de0",
   "metadata": {},
   "source": [
    "## Daily or monthly meteorological data from selected stations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "73fd5e6a-7001-468f-85fa-2a2fd58fc5a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Request of daily (time_step) data between d1 and d2\n",
    "# Required parameters\n",
    "time_step = 'day'  # for manthly data time_step = 'month' \n",
    "d1 = date(2019, 1, 1)\n",
    "d2 = date(2019, 3, 31)\n",
    "stations = ['7178I', '7031', '7031X']\n",
    "dir_path = './download'\n",
    "\n",
    "# Optional parameters\n",
    "fetch = 'both'\n",
    "verbose = True\n",
    "use_files = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "182cfed3-f47c-4789-be79-5216bf544226",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7178I_20190101T000000UTC_20190331T235959UTC_data.csv: 200, OK \n",
      "7178I_20190101T000000UTC_20190331T235959UTC_metadata.csv: 200, OK Climatologías diarias\n",
      "7031_20190101T000000UTC_20190331T235959UTC_data.csv: 200, OK \n",
      "7031_20190101T000000UTC_20190331T235959UTC_metadata.csv: 200, OK Climatologías diarias\n",
      "7031X_20190101T000000UTC_20190331T235959UTC_data.csv: 200, OK \n",
      "7031X_20190101T000000UTC_20190331T235959UTC_metadata.csv: 200, OK Climatologías diarias\n",
      "Downloaded files:  6\n"
     ]
    }
   ],
   "source": [
    "aod = AemetOpenData()\n",
    "downloaded_files = aod.meteo_data_by_station(time_step, d1, d2, stations, dir_path)\n",
    "print('Downloaded files: ', len(downloaded_files))\n",
    "\n",
    "# If we specify the optional parameters, the function call will be as indicated below\n",
    "# downloaded_files = aod.meteo_data_by_station(time_step, d1, d2, stations, dir_path, fetch=fetch, verbose=verbose, use_files=use_files)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "810027d9-15c8-42c5-9088-5aa3a86e6075",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7178I_2019_2019_data.csv: 200, OK \n",
      "7178I_2019_2019_metadata.csv: 200, OK Climatologías mensuales anuales\n",
      "7031_2019_2019_data.csv: 200, OK \n",
      "7031_2019_2019_metadata.csv: 200, OK Climatologías mensuales anuales\n",
      "7031X_2019_2019_data.csv: 200, OK \n",
      "7031X_2019_2019_metadata.csv: 200, OK Climatologías mensuales anuales\n",
      "Downloaded files:  6\n"
     ]
    }
   ],
   "source": [
    "# Now we download monthly data; we only modified the value of time_step \n",
    "time_step = 'month'\n",
    "\n",
    "aod = AemetOpenData()\n",
    "downloaded_files = aod.meteo_data_by_station(time_step, d1, d2, stations, dir_path)\n",
    "print('Downloaded files: ', len(downloaded_files))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c57390c-11a3-4fda-b046-8123c2d6ffe7",
   "metadata": {},
   "source": [
    "## Export downloaded meteorological data files to a Sqlite3 database \n",
    "If the data request to Aemet generated many files, the AOD_2db module can be used to concatenate them into one.\n",
    "The module saves the data and the metadata in a Sqlite3 database; each of the created tables can be exported to a csv file.<br>\n",
    "A specific database is created according to how the data request was made to the Aemet server: \n",
    "* Daily weather data from all stations.\n",
    "* Daily weather data from selected stations.\n",
    "* Monthly weather data from selected stations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "269f31de-450d-4fad-bd88-790a9e4c4b09",
   "metadata": {},
   "outputs": [],
   "source": [
    "from aod_2db import AOD_2db"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4431aff0-689b-498c-9df7-896382e5dd4b",
   "metadata": {},
   "source": [
    "### Export csv files to tables: one for data and another for metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "575fc322-a6b9-4668-880a-5dc6ebdf6234",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Required parameters        \n",
    "dpath = './download'  # directory with csv files\n",
    "\"\"\"\n",
    "type of request to Aemet: \n",
    "* 'stations_day'. Daily meteorological data from all stations\n",
    "* 'station1_day'. Daily meteorological data from selected stations\n",
    "* 'station1_month'. Monthly meteorological data from selected stations\n",
    "\"\"\"\n",
    "ftype = 'stations_day'  # We can export the rest od data using 'station1_day' or 'station1_month'   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "d75c2228-8021-4d7f-8aec-91ae97eb46d2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 stations_20230101T000000UTC_20230201T235959UTC_data.csv\n",
      "1 stations_20230202T000000UTC_20230305T235959UTC_data.csv\n",
      "2 stations_20230306T000000UTC_20230406T235959UTC_data.csv\n",
      "3 stations_20230407T000000UTC_20230508T235959UTC_data.csv\n",
      "4 stations_20230509T000000UTC_20230609T235959UTC_data.csv\n",
      "5 stations_20230610T000000UTC_20230711T235959UTC_data.csv\n",
      "6 stations_20230712T000000UTC_20230801T235959UTC_data.csv\n",
      "\n",
      "data has been inserted into metd\n",
      "Updated decimal separator as \".\" in columns: altitud, dir, presmax, velmedia, tmax, presmin, sol, tmed, racha, prec, tmin\n",
      "0 stations_20230101T000000UTC_20230201T235959UTC_metadata.csv\n",
      "1 stations_20230202T000000UTC_20230305T235959UTC_metadata.csv\n",
      "2 stations_20230306T000000UTC_20230406T235959UTC_metadata.csv\n",
      "3 stations_20230407T000000UTC_20230508T235959UTC_metadata.csv\n",
      "4 stations_20230509T000000UTC_20230609T235959UTC_metadata.csv\n",
      "5 stations_20230610T000000UTC_20230711T235959UTC_metadata.csv\n",
      "6 stations_20230712T000000UTC_20230801T235959UTC_metadata.csv\n",
      "\n",
      "metadata has been inserted into metd_metadata\n"
     ]
    }
   ],
   "source": [
    "a2db = AOD_2db(dpath, ftype)\n",
    "if not a2db.to_db():\n",
    "    print('Attemp failed')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d13703a-acc2-444c-bd94-722cc3168892",
   "metadata": {},
   "source": [
    "### Export tables in database to a csv file\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "9151f48a-aabb-421a-a1d2-25ff92bc593f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data dumped to\n",
      "C:\\Users\\solis\\Documents\\DEV\\python3\\meteoro\\aemet_open_data\\download\\metd_all_stations.csv\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The values of dpath and ftype have been previously assigned in this session:\n",
    "a2db.to_csv()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe2efa46-ef8d-4d50-8f25-3b2dae1b206f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}