nlp/text_search/search_article_in_medium.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "332f97b2",
   "metadata": {},
   "source": [
    "# Searh article in Medium"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8db978d0",
   "metadata": {},
   "source": [
    "## 0. Overview\n",
    "\n",
    "We'll search for text in the Medium dataset, and it will find the most similar results to the search text across all titles. Searching for articles is different from traditional keyword searches, which search for semantically relevant content. If you search for \"**funny python demo**\" it will return \"**Python Coding for Kids - Setting Up For the Adventure**\", not \"**No key words about funny python demo**\".\n",
    "\n",
    "We will use Milvus and Towhee to help searches. Towhee is used to extract the semantics of the text and return the text embedding. The Milvus vector database can store and search vectors, and return related articles. So we first need to install [Milvus](https://github.com/milvus-io/milvus) and [Towhee](https://github.com/towhee-io/towhee).\n",
    "\n",
    "Before getting started, please make sure that you have started a [Milvus service](https://milvus.io/docs/install_standalone-docker.md). This notebook uses [milvus 2.2.10](https://milvus.io/docs/v2.2.x/install_standalone-docker.md) and [pymilvus 2.2.11](https://milvus.io/docs/release_notes.md#2210)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6f85c5cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip3 install -q towhee pymilvus==2.2.11"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4aa8171",
   "metadata": {},
   "source": [
    "## 1. Data preprocessing\n",
    "\n",
    "The data is from the [Cleaned Medium Articles Dataset](https://www.kaggle.com/datasets/shiyu22chen/cleaned-medium-articles-dataset)(you can download it from Kaggle), which cleared the empty article titles in the data and conver the string title to the embeeding with Towhee [text_embedding.dpr operator](https://towhee.io/text-embedding/dpr), as you can see the `title_vector` is the embedding vectors of the title."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "cbf16ddf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download data\n",
    "! wget -q https://github.com/towhee-io/examples/releases/download/data/New_Medium_Data.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3285cd05",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>title</th>\n",
       "      <th>title_vector</th>\n",
       "      <th>link</th>\n",
       "      <th>reading_time</th>\n",
       "      <th>publication</th>\n",
       "      <th>claps</th>\n",
       "      <th>responses</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>The Reported Mortality Rate of Coronavirus Is ...</td>\n",
       "      <td>[0.041732933, 0.013779674, -0.027564144, -0.01...</td>\n",
       "      <td>https://medium.com/swlh/the-reported-mortality...</td>\n",
       "      <td>13</td>\n",
       "      <td>The Startup</td>\n",
       "      <td>1100</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Dashboards in Python: 3 Advanced Examples for ...</td>\n",
       "      <td>[0.0039737443, 0.003020432, -0.0006188639, 0.0...</td>\n",
       "      <td>https://medium.com/swlh/dashboards-in-python-3...</td>\n",
       "      <td>14</td>\n",
       "      <td>The Startup</td>\n",
       "      <td>726</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>How Can We Best Switch in Python?</td>\n",
       "      <td>[0.031961977, 0.00047043373, -0.018263113, 0.0...</td>\n",
       "      <td>https://medium.com/swlh/how-can-we-best-switch...</td>\n",
       "      <td>6</td>\n",
       "      <td>The Startup</td>\n",
       "      <td>500</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>Maternity leave shouldn’t set women back</td>\n",
       "      <td>[0.032572296, -0.011148319, -0.01688577, -0.00...</td>\n",
       "      <td>https://medium.com/swlh/maternity-leave-should...</td>\n",
       "      <td>9</td>\n",
       "      <td>The Startup</td>\n",
       "      <td>460</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>Python NLP Tutorial: Information Extraction an...</td>\n",
       "      <td>[-0.011735886, -0.016938083, -0.027233299, 0.0...</td>\n",
       "      <td>https://medium.com/swlh/python-nlp-tutorial-in...</td>\n",
       "      <td>7</td>\n",
       "      <td>The Startup</td>\n",
       "      <td>163</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id                                              title  \\\n",
       "0   0  The Reported Mortality Rate of Coronavirus Is ...   \n",
       "1   1  Dashboards in Python: 3 Advanced Examples for ...   \n",
       "2   2                  How Can We Best Switch in Python?   \n",
       "3   3           Maternity leave shouldn’t set women back   \n",
       "4   4  Python NLP Tutorial: Information Extraction an...   \n",
       "\n",
       "                                        title_vector  \\\n",
       "0  [0.041732933, 0.013779674, -0.027564144, -0.01...   \n",
       "1  [0.0039737443, 0.003020432, -0.0006188639, 0.0...   \n",
       "2  [0.031961977, 0.00047043373, -0.018263113, 0.0...   \n",
       "3  [0.032572296, -0.011148319, -0.01688577, -0.00...   \n",
       "4  [-0.011735886, -0.016938083, -0.027233299, 0.0...   \n",
       "\n",
       "                                                link  reading_time  \\\n",
       "0  https://medium.com/swlh/the-reported-mortality...            13   \n",
       "1  https://medium.com/swlh/dashboards-in-python-3...            14   \n",
       "2  https://medium.com/swlh/how-can-we-best-switch...             6   \n",
       "3  https://medium.com/swlh/maternity-leave-should...             9   \n",
       "4  https://medium.com/swlh/python-nlp-tutorial-in...             7   \n",
       "\n",
       "   publication  claps  responses  \n",
       "0  The Startup   1100         18  \n",
       "1  The Startup    726          3  \n",
       "2  The Startup    500          7  \n",
       "3  The Startup    460          1  \n",
       "4  The Startup    163          0  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('New_Medium_Data.csv', converters={'title_vector': lambda x: eval(x)})\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c89e09d",
   "metadata": {},
   "source": [
    "## 2. Load Data\n",
    "\n",
    "The next step is to get the text embedding, and then insert all the extracted embedding vectors into Milvus."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3107e24c",
   "metadata": {},
   "source": [
    "### Create Milvus Collection\n",
    "\n",
    "We need to create a collection in Milvus first, which contains multiple fields of `id`, `title`, `title_vector`, `link`, `reading_time`, `publication`, `claps` and `responses`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "17a85c11",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pymilvus import connections, FieldSchema, CollectionSchema, DataType, Collection, utility\n",
    "\n",
    "connections.connect(host='127.0.0.1', port='19530')\n",
    "\n",
    "def create_milvus_collection(collection_name, dim):\n",
    "    if utility.has_collection(collection_name):\n",
    "        utility.drop_collection(collection_name)\n",
    "    \n",
    "    fields = [\n",
    "            FieldSchema(name=\"id\", dtype=DataType.INT64, is_primary=True, auto_id=False),\n",
    "            FieldSchema(name=\"title\", dtype=DataType.VARCHAR, max_length=500),   \n",
    "            FieldSchema(name=\"title_vector\", dtype=DataType.FLOAT_VECTOR, dim=dim),\n",
    "            FieldSchema(name=\"link\", dtype=DataType.VARCHAR, max_length=500),\n",
    "            FieldSchema(name=\"reading_time\", dtype=DataType.INT64),\n",
    "            FieldSchema(name=\"publication\", dtype=DataType.VARCHAR, max_length=500),\n",
    "            FieldSchema(name=\"claps\", dtype=DataType.INT64),\n",
    "            FieldSchema(name=\"responses\", dtype=DataType.INT64)\n",
    "    ]\n",
    "    schema = CollectionSchema(fields=fields, description='search text')\n",
    "    collection = Collection(name=collection_name, schema=schema)\n",
    "    \n",
    "    index_params = {\n",
    "        'metric_type': \"L2\",\n",
    "        'index_type': \"IVF_FLAT\",\n",
    "        'params': {\"nlist\": 2048}\n",
    "    }\n",
    "    collection.create_index(field_name='title_vector', index_params=index_params)\n",
    "    return collection\n",
    "\n",
    "collection = create_milvus_collection('search_article_in_medium', 768)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5de5f0d2",
   "metadata": {},
   "source": [
    "### Data to Milvus\n",
    "\n",
    "\n",
    "Towhee supports reading df data through the `from_df` interface, and then we need to convert the `title_vector` column in the data to a two-dimensional list in float format, and then insert all the fields into Milvus, each field inserted into Milvus corresponds to one Collection fields created earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "08d52c41",
   "metadata": {},
   "outputs": [],
   "source": [
    "from towhee import ops, pipe, DataCollection\n",
    "\n",
    "insert_pipe = (pipe.input('df')\n",
    "                   .flat_map('df', 'data', lambda df: df.values.tolist())\n",
    "                   .map('data', 'res', ops.ann_insert.milvus_client(host='127.0.0.1', \n",
    "                                                                    port='19530',\n",
    "                                                                    collection_name='search_article_in_medium'))\n",
    "                   .output('res')\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c24e29c6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 11.1 s, sys: 2.38 s, total: 13.5 s\n",
      "Wall time: 2min 12s\n"
     ]
    }
   ],
   "source": [
    "%time _ = insert_pipe(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df2c149d",
   "metadata": {},
   "source": [
    "We need to call `collection.load()` to load the data after inserting the data, then run `collection.num_entities` to get the number of vectors in the collection. We will see the number of vectors is 5979, and we have successfully load the data to Milvus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "902e5924",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5979"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "collection.load()\n",
    "collection.num_entities"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea34c1f4",
   "metadata": {},
   "source": [
    "## 3. Search embedding title"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fa8662a",
   "metadata": {},
   "source": [
    "### Search one text in Milvus\n",
    "\n",
    "\n",
    "The retrieval process also to generate the text embedding of the query text, then search for similar vectors in Milvus, and finally return the result, which contains `id`(primary_key) and `score`. For example, we can search for \"funny python demo\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "1215dc8b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "search_pipe = (pipe.input('query')\n",
    "                    .map('query', 'vec', ops.text_embedding.dpr(model_name=\"facebook/dpr-ctx_encoder-single-nq-base\"))\n",
    "                    .map('vec', 'vec', lambda x: x / np.linalg.norm(x, axis=0))\n",
    "                    .flat_map('vec', ('id', 'score'), ops.ann_search.milvus_client(host='127.0.0.1', \n",
    "                                                                                   port='19530',\n",
    "                                                                                   collection_name='search_article_in_medium'))  \n",
    "                    .output('query', 'id', 'score')\n",
    "               )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "abbca936",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table style=\"border-collapse: collapse;\"><tr><th style=\"text-align: center; font-size: 130%; border: none;\">query</th> <th style=\"text-align: center; font-size: 130%; border: none;\">id</th> <th style=\"text-align: center; font-size: 130%; border: none;\">score</th></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">funny python demo</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">3897</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.3737614154815674</td></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">funny python demo</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">1342</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.4368068277835846</td></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">funny python demo</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">1832</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.45723873376846313</td></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">funny python demo</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">5671</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.45932769775390625</td></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">funny python demo</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">1752</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.4645398259162903</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "res = search_pipe('funny python demo')\n",
    "DataCollection(res).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07a5dc7f",
   "metadata": {},
   "source": [
    "### Search multi text in Milvus\n",
    "\n",
    "We can also retrieve multiple pieces of data, for example we can specify the array(['funny python demo', 'AI in data analysis']) to search in batch, which will be retrieved in Milvus:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "e5a97384",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table style=\"border-collapse: collapse;\"><tr><th style=\"text-align: center; font-size: 130%; border: none;\">query</th> <th style=\"text-align: center; font-size: 130%; border: none;\">id</th> <th style=\"text-align: center; font-size: 130%; border: none;\">score</th></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">funny python demo</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">3897</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.3737614154815674</td></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">funny python demo</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">1342</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.4368068277835846</td></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">funny python demo</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">1832</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.45723873376846313</td></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">funny python demo</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">5671</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.45932769775390625</td></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">funny python demo</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">1752</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.4645398259162903</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table style=\"border-collapse: collapse;\"><tr><th style=\"text-align: center; font-size: 130%; border: none;\">query</th> <th style=\"text-align: center; font-size: 130%; border: none;\">id</th> <th style=\"text-align: center; font-size: 130%; border: none;\">score</th></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">AI in data analysis</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">3493</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.24436673521995544</td></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">AI in data analysis</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">4542</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.24851186573505402</td></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">AI in data analysis</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">2649</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.28404244780540466</td></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">AI in data analysis</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">4539</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.31868308782577515</td></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">AI in data analysis</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">5337</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.32242846488952637</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "res = search_pipe.batch(['funny python demo', 'AI in data analysis'])\n",
    "for re in res:\n",
    "    DataCollection(re).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c835da6f",
   "metadata": {},
   "source": [
    "### Search text and return multi fields\n",
    "\n",
    "If we want to return more information when retrieving, we can set the `output_fields` parameter in [ann_search.milvus operator](https://towhee.io/ann-search/milvus). For example, in addition to `id` and `score`, we can also return `title`, `link`, `claps`, `reading_time`, `and response`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "0c736c2a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table style=\"border-collapse: collapse;\"><tr><th style=\"text-align: center; font-size: 130%; border: none;\">query</th> <th style=\"text-align: center; font-size: 130%; border: none;\">id</th> <th style=\"text-align: center; font-size: 130%; border: none;\">score</th> <th style=\"text-align: center; font-size: 130%; border: none;\">title</th></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">funny python demo</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">3897</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.3737614154815674</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">Python Coding for Kids — Setting Up For the Adventure</td></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">funny python demo</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">1342</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.4368068277835846</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">How to Design Professional Venn Diagrams in Python</td></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">funny python demo</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">1832</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.45723873376846313</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">How to mock AWS services for rapid local development.</td></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">funny python demo</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">5671</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.45932769775390625</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">Adventure into Machine Learning using Python</td></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">funny python demo</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">1752</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.4645398259162903</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">Custom neural networks in Keras: a street fighter’s guide to build a graphCNN</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "search_pipe1 = (pipe.input('query')\n",
    "                    .map('query', 'vec', ops.text_embedding.dpr(model_name=\"facebook/dpr-ctx_encoder-single-nq-base\"))\n",
    "                    .map('vec', 'vec', lambda x: x / np.linalg.norm(x, axis=0))\n",
    "                    .flat_map('vec', ('id', 'score', 'title'), ops.ann_search.milvus_client(host='127.0.0.1', \n",
    "                                                                                   port='19530',\n",
    "                                                                                   collection_name='search_article_in_medium',\n",
    "                                                                                   output_fields=['title']))  \n",
    "                    .output('query', 'id', 'score', 'title')\n",
    "               )\n",
    "\n",
    "res = search_pipe1('funny python demo')\n",
    "DataCollection(res).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "334c6fce",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table style=\"border-collapse: collapse;\"><tr><th style=\"text-align: center; font-size: 130%; border: none;\">query</th> <th style=\"text-align: center; font-size: 130%; border: none;\">id</th> <th style=\"text-align: center; font-size: 130%; border: none;\">score</th> <th style=\"text-align: center; font-size: 130%; border: none;\">title</th> <th style=\"text-align: center; font-size: 130%; border: none;\">link</th> <th style=\"text-align: center; font-size: 130%; border: none;\">reading_time</th> <th style=\"text-align: center; font-size: 130%; border: none;\">publication</th> <th style=\"text-align: center; font-size: 130%; border: none;\">claps</th> <th style=\"text-align: center; font-size: 130%; border: none;\">responses</th></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">funny python demo</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">3897</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.3737614154815674</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">Python Coding for Kids — Setting Up For the Adventure</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">https://medium.com/swlh/python-coding-for-kids-setting-up-for-the-adventure-9be4bef6b24e</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">14</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">The Startup</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">119</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">2</td></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">funny python demo</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">1342</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.4368068277835846</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">How to Design Professional Venn Diagrams in Python</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">https://towardsdatascience.com/how-to-design-professional-venn-diagrams-in-python-693c9ed2c288</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">6</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">Towards Data Science</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">97</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">1</td></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">funny python demo</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">1832</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.45723873376846313</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">How to mock AWS services for rapid local development.</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">https://medium.com/swlh/how-to-mock-aws-services-for-rapid-local-development-3d07581ffc3a</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">3</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">The Startup</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">84</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0</td></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">funny python demo</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">5671</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.45932769775390625</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">Adventure into Machine Learning using Python</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">https://towardsdatascience.com/adventure-into-machine-learning-using-python-7a85fce81b7d</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">14</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">Towards Data Science</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">25</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0</td></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">funny python demo</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">1752</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.4645398259162903</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">Custom neural networks in Keras: a street fighter’s guide to build a graphCNN</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">https://towardsdatascience.com/custom-neural-networks-in-keras-a-street-fighters-guide-to-build-a-graphcnn-e91f6b05f12e</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">7</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">Towards Data Science</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">55</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# milvus search with multi outpt fields\n",
    "search_pipe2 = (pipe.input('query')\n",
    "                    .map('query', 'vec', ops.text_embedding.dpr(model_name=\"facebook/dpr-ctx_encoder-single-nq-base\"))\n",
    "                    .map('vec', 'vec', lambda x: x / np.linalg.norm(x, axis=0))\n",
    "                    .flat_map('vec', ('id', 'score', 'title', 'link', 'reading_time', 'publication', 'claps', 'responses'), \n",
    "                                       ops.ann_search.milvus_client(host='127.0.0.1', \n",
    "                                                                    port='19530',\n",
    "                                                                    collection_name='search_article_in_medium',\n",
    "                                                                    output_fields=['title', 'link', 'reading_time', 'publication', 'claps', 'responses'], \n",
    "                                                                    limit=5))  \n",
    "                    .output('query', 'id', 'score', 'title', 'link', 'reading_time', 'publication', 'claps', 'responses')\n",
    "               )\n",
    "\n",
    "res = search_pipe2('funny python demo')\n",
    "DataCollection(res).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ff903e3",
   "metadata": {},
   "source": [
    "### Search text with some expr\n",
    "\n",
    "\n",
    "In addition, we can also set some expressions for retrieval. For example, we can specify that the beginning of the article is an article in Python by setting expr='title like \"Python%\"':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "b6b1d85d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table style=\"border-collapse: collapse;\"><tr><th style=\"text-align: center; font-size: 130%; border: none;\">query</th> <th style=\"text-align: center; font-size: 130%; border: none;\">id</th> <th style=\"text-align: center; font-size: 130%; border: none;\">score</th> <th style=\"text-align: center; font-size: 130%; border: none;\">title</th> <th style=\"text-align: center; font-size: 130%; border: none;\">link</th> <th style=\"text-align: center; font-size: 130%; border: none;\">reading_time</th> <th style=\"text-align: center; font-size: 130%; border: none;\">publication</th> <th style=\"text-align: center; font-size: 130%; border: none;\">claps</th> <th style=\"text-align: center; font-size: 130%; border: none;\">responses</th></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">funny python demo</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">3897</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.3737614154815674</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">Python Coding for Kids — Setting Up For the Adventure</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">https://medium.com/swlh/python-coding-for-kids-setting-up-for-the-adventure-9be4bef6b24e</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">14</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">The Startup</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">119</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">2</td></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">funny python demo</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">4644</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.49374908208847046</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">Python for Finance — The Complete Beginner’s Guide</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">https://towardsdatascience.com/python-for-finance-the-complete-beginners-guide-764276d74cef</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">8</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">Towards Data Science</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">292</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">5</td></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">funny python demo</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">2736</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.49569693207740784</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">Python for Beginners — Basics</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">https://towardsdatascience.com/python-for-beginners-basics-7ac6247bb4f4</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">7</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">Towards Data Science</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">11</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0</td></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">funny python demo</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">1667</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.5019434690475464</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">Python — How to measure thread execution time in multithreaded application?</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">https://medium.com/swlh/python-how-to-measure-thread-execution-time-in-multithreaded-application-f4b2e2112091</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">6</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">The Startup</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">55</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0</td></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">funny python demo</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">1298</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0.516698956489563</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">Python Testing with a mock database (SQL)</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">https://medium.com/swlh/python-testing-with-a-mock-database-sql-68f676562461</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">4</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">The Startup</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">51</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">0</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "search_pipe3 = (pipe.input('query')\n",
    "                    .map('query', 'vec', ops.text_embedding.dpr(model_name=\"facebook/dpr-ctx_encoder-single-nq-base\"))\n",
    "                    .map('vec', 'vec', lambda x: x / np.linalg.norm(x, axis=0))\n",
    "                    .flat_map('vec', ('id', 'score', 'title', 'link', 'reading_time', 'publication', 'claps', 'responses'), \n",
    "                                       ops.ann_search.milvus_client(host='127.0.0.1', \n",
    "                                                                    port='19530',\n",
    "                                                                    collection_name='search_article_in_medium',\n",
    "                                                                    expr='title like \"Python%\"',\n",
    "                                                                    output_fields=['title', 'link', 'reading_time', 'publication', 'claps', 'responses'], \n",
    "                                                                    limit=5))  \n",
    "                    .output('query', 'id', 'score', 'title', 'link', 'reading_time', 'publication', 'claps', 'responses')\n",
    "               )\n",
    "\n",
    "res = search_pipe3('funny python demo')\n",
    "DataCollection(res).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50699084",
   "metadata": {},
   "source": [
    "## 4. Query data in Milvus\n",
    "\n",
    "We have done the text retrieval process before, and we can get articles such as \"Python coding for kids - getting ready for an adventure\" by retrieving \"fun python demos\".\n",
    "\n",
    "We can also do a simple query on the data, we need to set `expr` and `output_fields` with the `collection.query` interface, for example, we can filter out articles with faults greater than 300 and reading time less than 15 minutes, and submitted to TDS :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "a35c619f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'responses': 17,\n",
       "  'id': 913,\n",
       "  'title': 'I Thought I Was Mastering Python Until I Discovered These Tricks',\n",
       "  'link': 'https://towardsdatascience.com/i-thought-i-was-mastering-python-until-i-discovered-these-tricks-e40d9c71f4e2',\n",
       "  'reading_time': 9,\n",
       "  'publication': 'Towards Data Science',\n",
       "  'claps': 5200},\n",
       " {'responses': 28,\n",
       "  'id': 1195,\n",
       "  'title': 'Do Not Use “+” to Join Strings in Python',\n",
       "  'link': 'https://towardsdatascience.com/do-not-use-to-join-strings-in-python-f89908307273',\n",
       "  'reading_time': 5,\n",
       "  'publication': 'Towards Data Science',\n",
       "  'claps': 4200},\n",
       " {'responses': 20,\n",
       "  'id': 2572,\n",
       "  'title': 'Top 3 Python Functions You Don’t Know About (Probably)',\n",
       "  'link': 'https://towardsdatascience.com/top-3-python-functions-you-dont-know-about-probably-978f4be1e6d',\n",
       "  'reading_time': 4,\n",
       "  'publication': 'Towards Data Science',\n",
       "  'claps': 4400},\n",
       " {'responses': 18,\n",
       "  'id': 2865,\n",
       "  'title': 'How to process a DataFrame with billions of rows in seconds',\n",
       "  'link': 'https://towardsdatascience.com/how-to-process-a-dataframe-with-billions-of-rows-in-seconds-c8212580f447',\n",
       "  'reading_time': 6,\n",
       "  'publication': 'Towards Data Science',\n",
       "  'claps': 3100},\n",
       " {'responses': 8,\n",
       "  'id': 4639,\n",
       "  'title': 'Do You Know Python Has A Built-In Database?',\n",
       "  'link': 'https://towardsdatascience.com/do-you-know-python-has-a-built-in-database-d553989c87bd',\n",
       "  'reading_time': 6,\n",
       "  'publication': 'Towards Data Science',\n",
       "  'claps': 3500},\n",
       " {'responses': 45,\n",
       "  'id': 4799,\n",
       "  'title': 'Sorry, Online Courses Won’t Make you a Data Scientist',\n",
       "  'link': 'https://towardsdatascience.com/sorry-online-courses-wont-make-you-a-data-scientist-8639d5f00889',\n",
       "  'reading_time': 8,\n",
       "  'publication': 'Towards Data Science',\n",
       "  'claps': 5000},\n",
       " {'responses': 73,\n",
       "  'id': 5766,\n",
       "  'title': 'Machine Learning Engineers Will Not Exist In 10 Years.',\n",
       "  'link': 'https://towardsdatascience.com/machine-learning-engineers-will-not-exist-in-10-years-c9cbbf4472f3',\n",
       "  'reading_time': 6,\n",
       "  'publication': 'Towards Data Science',\n",
       "  'claps': 4600}]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "collection.query(\n",
    "  expr = 'claps > 3000 && reading_time < 15 && publication like \"Towards Data Science%\"', \n",
    "  output_fields = ['id', 'title', 'link', 'reading_time', 'publication', 'claps', 'responses'],\n",
    "  consistency_level='Strong'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f7f67c1",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}