Tweets_by_location_30sep.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# Find location and count tweets per location from tweetID\n",
    "Use cityID from Geonamescache to manage and count locations. Each cityID represents a unique city with related information(name, longitude, latitude, population, country, state ect). \n",
    "\n",
    "Find the location from tweetIDs based on two ways:\n",
    "\n",
    "1.Longitudes and latitudes\n",
    "\n",
    "2.User location\n",
    "\n",
    "Explanation:\n",
    "\n",
    "1.Reverse geocode: from coordinates to city:\n",
    "\n",
    "I read each tweetIDs to see if it has location information of longitudes and latitudes. If yes, I reversed geocode: from longitude and latitude output the cityID with related information.\n",
    "Algorithm: I used the nearest neighbor algorithm (NN), in a brute force way. It calculated distance using Euclidian distance method to to find the shortest path from one point to another in a list of points.(One point from out tweet's longlats and the other is longlats from the list of all cities in geonamescache)\n",
    "\n",
    "2. Find user's location and look it up in geonamescache\n",
    "\n",
    "Most of the tweets do not have exact location so I used user's location to determine the city.  Geonamescache will return a list of several cityIDs that matches that name. Then I used the country information and loop through that list to pick the exact city. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import necessary files and the path of our files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "import json\n",
    "import os,glob\n",
    "import geonamescache\n",
    "folder_path = 'D:\\\\COVID-19-TweetIDs\\\\2020-01\\\\New folder\\\\22'\n",
    "gc = geonamescache.GeonamesCache()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Reverse geocode\n",
    "\n",
    "findlonglat is the function that find citycode from longitude and latitude and return a dictionary with key: citycode, value: relation information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "city = gc.get_cities()\n",
    "longlat = []\n",
    "for k in city.keys():\n",
    "    lst = []\n",
    "    lst.append(city.get(k).get(\"longitude\"))\n",
    "    lst.append(city.get(k).get(\"latitude\"))\n",
    "    lst.append(city.get(k).get('geonameid'))\n",
    "    longlat.append(lst)\n",
    "tuple(longlat) #a tuple of all cities listed by geonamescache\n",
    "def get_distance(data1, data2):\n",
    "    points = zip(data1, data2)\n",
    "    diffs_squared_distance = [pow(a - b, 2) for (a, b) in points]\n",
    "    return math.sqrt(sum(diffs_squared_distance))\n",
    "def findlonglat(a,b):\n",
    "    found = []\n",
    "    smallest = get_distance(b,a[0][0:2])\n",
    "    for i in a:\n",
    "        comparelist = i[0:2]\n",
    "        if get_distance(b,comparelist) <= smallest:\n",
    "            smallest = get_distance(b,comparelist)\n",
    "            found = list(i)\n",
    "    citycode = str(found[2])\n",
    "    return {citycode:city[citycode]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Look up city in geonamescache\n",
    "\n",
    "findloc is the function that looks for the city in geonamescache data and return a dictionary with key: citycode and values: related information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def findloc(str):\n",
    "    cities = str.split(\",\")[0].title() #use.title to make cities like hong kong to Hong Kong\n",
    "    ct = gc.get_cities_by_name(cities)\n",
    "    if (len(ct) == 1):\n",
    "        return(ct[0]) #if return ct, it is a list of one member so lead to inconsistency of type\n",
    "    elif ',' in str:\n",
    "        loc2 = str.split(',')[1].strip()\n",
    "        #print(loc2)\n",
    "        for c in ct:\n",
    "            if(loc2 in json.dumps(c)) :\n",
    "                return(c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = []\n",
    "location = {}   # dictionary of location: key is cityID, value is city information coming service\n",
    "count = {}  # dictionary of count of tweet per location: key is cityID and value is number of tweet\n",
    "\n",
    "for filename in glob.glob(os.path.join(folder_path, '*.jsonl')):\n",
    "    fhand = open(filename, 'r',encoding=\"UTF-8\")\n",
    "    for line in fhand:\n",
    "        tweet = json.loads(line)\n",
    "        try:\n",
    "            geo = tweet['coordinates']\n",
    "            if geo != None: #Use coordinates to find location first (findlonglat)\n",
    "                geo = tweet['coordinates'][\"coordinates\"]\n",
    "                loc = findlonglat(longlat,geo)\n",
    "            else: #Then try to use user's location (findloc)\n",
    "                loc = findloc(tweet['user']['location'])\n",
    "            location.update(loc) #update loc info into the location dictionary\n",
    "            for key,value in loc.items():\n",
    "                if(key not in count):\n",
    "                    count.update({key:1})   #update in the count dictionary, for places with 1 tweet\n",
    "                elif(key in count):\n",
    "                    count.update({key:(count[key]+1)})     #update in the count dictionary, if cityID is repeated,just increase the value by 1       \n",
    "            \n",
    "            #this step to compare the userlocation vs location that we found. \n",
    "            \"\"\"\n",
    "            user_name = tweet['user'][\"screen_name\"]\n",
    "            user_city_name = loc[key][\"name\"]\n",
    "            user_country_name = loc[key]['countrycode']\n",
    "            print(user_name, tweet['user']['location'], user_city_name, user_country_name)\n",
    "            \"\"\"\n",
    "        \n",
    "        except:\n",
    "            continue\n",
    "            \n",
    "#output the two dictionaries if necessary\n",
    "#print(location) \n",
    "#print(count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}