INFO_DOCUMENT.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Semantic Image Search"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Goal\n",
    "\n",
    "The goal of this week's capstone project is to combine techniques from computer vision and natural language processing (NLP) to allow querying for images using keywords. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Approach\n",
    "\n",
    "We'll learn a mapping from images features (extracted using pre-trained computer vision models such as ResNet-18) to semantic features (based on word embeddings).\n",
    "\n",
    "We'll create training data consisting of triples of the form: \n",
    "\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; `(text, good_image, bad_image)`\n",
    "\n",
    "We want the similarity in semantic space between the `text` and `good_image` to be greater than the similarity between the `text` and `bad_image`, i.e.,\n",
    "\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; `sim(se(text), se(good_image)) > sim(se(text), se(bad_image))`\n",
    "\n",
    "where `se()` stands for semantic embedding. We'll use cosine similarity to measure similarity. But note that if all vectors are normalized to have unit length, then cosine similarity is equivalent to the dot product between the two vectors.\n",
    "\n",
    "We'll embed text by essentially averaging the word embeddings (e.g., GloVe embeddings) for all words in the string.\n",
    "\n",
    "We'll embed images with a linear projection from image feature space to the semantic space (same dimension as word embeddings):\n",
    "\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; `se(im) = im M`\n",
    "\n",
    "where `im` has shape (1, 512) and `M` has shape (512, 50) assuming 512-dimensional image features and 50-dimensional word embedding features. The idea (hope) is that `im` will be mapped to a good region of the semantic space.\n",
    "\n",
    "For example, if `im` contains a dog, we want multiplication by `M` to result in a vector `se(im)` that's close to where the word \"dog\" is in GloVe space (`glove[\"dog\"]`).\n",
    "\n",
    "### Ranking Loss\n",
    "\n",
    "To find a good M matrix, we'll use PyTorch's MarginRankingLoss. This loss penalizes when the ordering of the values is wrong, and stops penalizing once the order is right \"enough\" (determined by the desired margin). The reasoning is that once the ordering between values is right, we don't need to waste effort trying to make it even more right."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset\n",
    "\n",
    "We'll be using the COCO dataset. From the website (http://cocodataset.org/),\n",
    "\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; \"COCO is a large-scale object detection, segmentation, and captioning dataset.\"\n",
    "\n",
    "We've already pre-extracted image features using the resnet18 model for train2014 so you don't need to worry about this step. We've put these as well as the captions in dropbox for you to download:\n",
    "\n",
    "captions_train2014.json\n",
    "<br/>\n",
    "https://www.dropbox.com/s/h5u86wp9wfhtkz1/captions_train2014.json?dl=0\n",
    "\n",
    "resnet18_features_train.pkl.gz\n",
    "<br/>\n",
    "https://www.dropbox.com/s/2g6m70ouitxftt9/resnet18_features_train.pkl.gz?dl=0\n",
    "\n",
    "or in zip format:\n",
    "\n",
    "resnet18_features_train.pkl.zip\n",
    "<br/>\n",
    "https://www.dropbox.com/s/83skvy9bub36pkl/resnet18_features_train.pkl.zip?dl=0\n",
    "\n",
    "The fe file contains the image features (in dictionary of PyTorch tensors indexed by image_id).\n",
    "\n",
    "If during the course of the project, you find you want to do things with the raw images, you can download from the COCO dataset, but the files are pretty large. :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tasks\n",
    "\n",
    "These tasks are the minimum set of things that need to be completed as part of the capstone project. You should coordinate with your team about how to divide them up. (Note: Some might naturally be combined.)\n",
    "\n",
    "* create capability to embed captions\n",
    " * add up word embeddings for all words in caption weighted by IDF of words across all captions (to down-weight common words); then normalize\n",
    "* create training and validation sets of triples\n",
    "* train model\n",
    " * embed caption\n",
    " * embed good image\n",
    " * embed bad image\n",
    " * compute similarities from caption to good and caption to bad\n",
    " * compute loss with margin ranking loss\n",
    " * take optimization step\n",
    "* measure loss, accuracy (in terms of triples correct)\n",
    "* create image \"database\" by mapping whole set of image features to semantic features with trained model\n",
    "* create function to query database and return top k images\n",
    "* create function to display set of images (given image ids)\n",
    " * note that the image metadata (contained in `captions_train2014.json`) includes a property called \"coco_url\" that can be used download a particular image on demand for display\n",
    " * maybe display their captions, too (for debugging)\n",
    "* create function that finds top k similar images to a query image\n",
    " * give option for doing similarity search in original image feature space or learned semantic space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optional Tasks\n",
    "\n",
    "* make a simple website (with bottle or flask); more details to follow\n",
    "* embed some new images (e.g., from your phones) and add to database or create new separate database\n",
    "* measure actual rank of good image during training (ask for details)\n",
    "* try hard negative mining (ask for details)\n",
    " * e.g., see brief desciption here: http://www.robots.ox.ac.uk/~vgg/practicals/category-detection/index.html#part4\n",
    "* check out \"Hubs in Space\" paper:\n",
    " * http://www.jmlr.org/papers/v11/radovanovic10a.html (ask for details)\n",
    "* try different word embeddings\n",
    " * e.g., pre-trained word2vec embeddings that did \"phrase detection\" first\n",
    "* try different image features (will require using PyTorch trained models; details to follow)\n",
    "* try Roccio relevance feedback:\n",
    " * https://en.wikipedia.org/wiki/Rocchio_algorithm\n",
    "* try different way to embed captions (other than average)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extra Optional Tasks\n",
    "\n",
    "* image captioning (ala \"Show and Tell\" paper)\n",
    " * https://www.cv-foundation.org/openaccess/content_cvpr_2015/html/Vinyals_Show_and_Tell_2015_CVPR_paper.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}