Notebooks/Day 1 - Text Generation with Chat Completions API.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../Assets/Images/Day 1 Header.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Welcome to Day 1\n",
    "\n",
    "Today we will \n",
    "\n",
    "- Introduce LLMs\n",
    "- Understand the available OpenAI LLMs\n",
    "- Look at the OpenAI Chat Messages (Prompt) Structure\n",
    "- Study Chat Completions API Parameters\n",
    "- Decode the Chat Completions Response Object\n",
    "- Learn how to Stream responses\n",
    "- Look at JSON Response format\n",
    "- Understand Reproducibility\n",
    "- Learn about managing Tokens\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Brief Introduction to Large Language Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"font-size: 20px; color: orange\"><b>Generative AI, and LLMs specifically, is a General Purpose Technology that is useful for a variety of applications</b></span>\n",
    "\n",
    "<span style=\"font-size: 16px;\"><i>\"LLMs can be, generally, thought of as a next word prediction model\"</i></span>\n",
    "\n",
    "<span style=\"font-size: 16px; color: blue\"><b>What is an LLM?</b></span>\n",
    "\n",
    "- LLMs are __machine learning models__ that have learned from __massive datasets__ of human-generated content, finding statistical patterns to replicate human-like abilities.\n",
    "\n",
    "- __Foundation models__, also known as base models, have been trained on trillions of words for weeks or months using extensive compute power. These models have __billions of parameters__, which represent their memory and enable sophisticated tasks.\n",
    "\n",
    "- __Interacting with LLMs differs from traditional programming paradigms. Instead of formalized code syntax, you provide natural language prompts to the models__.\n",
    "\n",
    "- When you pass a __prompt__ to the model, it predicts the next words and generates a __completion__. This process is known as __inference__.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"font-size: 16px; color: blue\"> <b>Prompts, Completions and Inference!</b></span>\n",
    "\n",
    "<img src=\"../Assets/Images/LLM Inference.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Recommendation__ : An excellent course \"[Generative AI using Large Language Models](https://www.deeplearning.ai/courses/generative-ai-with-llms/)\" is offered by DeepLearning.ai and AWS via Coursera.\n",
    "\n",
    "I've prepared notes on this course that you can download \n",
    "\n",
    "<a href=\"https://abhinavkimothi.gumroad.com/l/GenAILLM\"><img src=\"../Assets/Images/Coursera Course Notes.png\" width=600></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Available OpenAI LLMs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"font-size: 16px; color: blue\"> <b>GPT-4</b></span>\n",
    "\n",
    "__<u>(Production)</u>__\n",
    "\n",
    "<u>Name        | Context Window    | Cut-off date      | Snapshot</u>\n",
    "\n",
    "__gpt-4__       | 8,192 tokens      | Up to Sep 2021    | gpt-4-0613\n",
    "\n",
    "__gpt-4-32k__   | 32,768 tokens     | Up to Sep 2021    | gpt-4-32k-0613\n",
    "\n",
    "<b><u>(Preview)</b></u>\n",
    "\n",
    "__gpt-4-turbo-preview__     | 128,000 tokens | Up to Dec 2023    | gpt-4-1106-preview\n",
    "\n",
    "__gpt-4-vision-preview__    | 128,000 tokens | Up to Apr 2023    | gpt-4-1106-vision-preview\n",
    "\n",
    "_Note : gpt-4-1106-vision-preview is a multi-modal model that works on both text and images_\n",
    "\n",
    "---\n",
    "\n",
    "<span style=\"font-size: 16px; color: blue\"> <b>GPT-3.5</b></span>\n",
    "\n",
    "\n",
    "__gpt-3.5-turbo__ | 16,385 tokens | Up to Sep 2021 | gpt-3.5-turbo-1106\n",
    "\n",
    "__gpt-3.5-turbo-instruct__ | 4,096 tokens | Up to Sep 2021\n",
    "\n",
    "---\n",
    "\n",
    "For more details, visit the official documentation -> https://platform.openai.com/docs/models\n",
    "\n",
    "---\n",
    "\n",
    "<span style=\"font-size: 14px; color: orange\">__IMP__ : __\"model\"__ is passed as a parameter in the chat completions API</span>\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## OpenAI Chat Messages (Prompt) Structure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"font-size: 16px;\"><u>Role</u></span> : OpenAI allows for *three* roles/personas - \n",
    "1. __System__ : The overarching constraints/definitions/intructions of the system that the LLM should \"remember\"\n",
    "2. __User__ : Any instruction a user wants to pass to the LLM\n",
    "3. __Assistant__ : The response from the LLM\n",
    "\n",
    "<span style=\"font-size: 16px;\"><u>Content</u></span> : Any message or \"prompt\" of these personas are passed as \"Content\"\n",
    "\n",
    "\n",
    "\n",
    "__Why is this important?__ : Makes it easier to adapt an LLM to store conversation history.\n",
    "\n",
    "---\n",
    "\n",
    "<span style=\"font-size: 14px; color: orange\">__IMP__ : __\"role\"__ and __\"content\"__ are passed as a dictionary in the __\"messages\"__ parameter in the chat completions API </span>\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's try!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recall from Day 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Have you installed required Python packages:\n",
    "    \n",
    "_pip install -r requirements.txt_\n",
    "\n",
    "If not, run the cell below after removing the \"''' (triple single quotes)\" from the beginning and the end\n",
    "\n",
    "Alternatively, please go to {Home}/OpenAI-API-Explorer/Readme.md and follow the instructions\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'import os\\n\\n# Get the current working directory\\ncurrent_directory = os.getcwd()\\n\\n# Extract the directory name from the path\\ncurrent_directory_name = os.path.basename(current_directory)\\n\\n# Check if the directory name is \\'Notebooks\\'\\nif current_directory_name != \\'Notebooks\\':\\n    notebooks_directory = os.path.join(current_directory, \\'Notebooks\\')\\n    \\n    # Check if \\'Notebooks\\' directory exists\\n    if os.path.exists(notebooks_directory) and os.path.isdir(notebooks_directory):\\n        # Change the current working directory to \\'Notebooks\\'\\n        os.chdir(notebooks_directory)\\n        print(\"Changed to \\'Notebooks\\' directory.\")\\n        %pip install -r ../requirements.txt --quiet\\n        print(\"Requirements Installed\")\\n    else:\\n        print(\"Please go to {Home}/OpenAI-API-Explorer/Readme.md and follow the instructions\")\\nelse:\\n    print(\"Already in \\'Notebooks\\' directory.\")\\n    %pip install -r ../requirements.txt --quiet\\n    print(\"Requirements Installed\")\\n'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''import os\n",
    "\n",
    "# Get the current working directory\n",
    "current_directory = os.getcwd()\n",
    "\n",
    "# Extract the directory name from the path\n",
    "current_directory_name = os.path.basename(current_directory)\n",
    "\n",
    "# Check if the directory name is 'Notebooks'\n",
    "if current_directory_name != 'Notebooks':\n",
    "    notebooks_directory = os.path.join(current_directory, 'Notebooks')\n",
    "    \n",
    "    # Check if 'Notebooks' directory exists\n",
    "    if os.path.exists(notebooks_directory) and os.path.isdir(notebooks_directory):\n",
    "        # Change the current working directory to 'Notebooks'\n",
    "        os.chdir(notebooks_directory)\n",
    "        print(\"Changed to 'Notebooks' directory.\")\n",
    "        %pip install -r ../requirements.txt --quiet\n",
    "        print(\"Requirements Installed\")\n",
    "    else:\n",
    "        print(\"Please go to {Home}/OpenAI-API-Explorer/Readme.md and follow the instructions\")\n",
    "else:\n",
    "    print(\"Already in 'Notebooks' directory.\")\n",
    "    %pip install -r ../requirements.txt --quiet\n",
    "    print(\"Requirements Installed\")\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#### Import Libraries ####\n",
    "from openai import OpenAI #OpenAI Client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dotenv import load_dotenv\n",
    "import os\n",
    "\n",
    "load_dotenv()\n",
    "\n",
    "openai_api_key=os.getenv(\"OPENAI_API_KEY\")\n",
    "\n",
    "client = OpenAI(api_key=openai_api_key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### API Call"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's call the Chat Completions API. \n",
    "\n",
    "Let's talk to the LLM about the beautiful game of Cricket.\n",
    "\n",
    "Remember \"System\" is the overarching instruction that you can give to the LLM. Here let's ask the LLM to be a helpful assistant who has a knowledge of cricket.\n",
    "\n",
    "Then will ask it a question about cricket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = client.chat.completions.create(\n",
    "  model=\"gpt-3.5-turbo\",\n",
    "  messages=[\n",
    "    {\"role\": \"system\", \"content\": \"You are a helpful assistant knowledgeable in the field of Cricket.\"}, ### The overarching instruction\n",
    "    {\"role\": \"user\", \"content\": \"When did Australia win their first Cricket World Cup?\"} ### Our Question\n",
    "  ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we used the gpt-3.5-turbo model. This is the same model used in the free version of ChatGPT.\n",
    "\n",
    "Let's look at the response that the LLM generated. We'll look at the response object later in detail. The response text is found under choices->message->content in the response object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Australia won their first Cricket World Cup in 1987 when they co-hosted the tournament with India and Pakistan. They defeated England in the final to\n",
      "claim their maiden title.\n"
     ]
    }
   ],
   "source": [
    "import textwrap\n",
    "print(textwrap.fill(response.choices[0].message.content, 150))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! We got a response from the LLM. The response also seems fairly accurate. \n",
    "\n",
    "Remember that there is no guarantee that the response will be factually correct"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's ask a follow up question - \"How much did they score?\"\n",
    "\n",
    "This question implies that we are chatting with the LLM and the LLM has the context of the previously asked questions and responses. To the API we will have to pass all of the previous prompts and responses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In the final of the 1987 Cricket World Cup, Australia scored 253/5 in their allotted 50 overs. England, in response, managed to score 246/8, resulting\n",
      "in Australia winning the match by 7 runs and clinching their first World Cup title.\n"
     ]
    }
   ],
   "source": [
    "\n",
    "\n",
    "response = client.chat.completions.create(\n",
    "  model=\"gpt-3.5-turbo\",\n",
    "  messages=[\n",
    "    {\"role\": \"system\", \"content\": \"You are a helpful assistant knowledgeable in the field of Cricket.\"}, ### The overarching instruction\n",
    "    {\"role\": \"user\", \"content\": \"When did Australia win their first Cricket World Cup?\"}, ### Our Question\n",
    "    {\"role\": \"assistant\", \"content\": \"Australia won their first Cricket World Cup in the year 1987. They defeated England in the final to clinch their maiden title in the tournament.\"}, ### LLM Response\n",
    "    {\"role\": \"user\", \"content\": \"How much did they score?\"} ### Our followup question\n",
    "  ]\n",
    ")\n",
    "\n",
    "\n",
    "print(textwrap.fill(response.choices[0].message.content,150))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, we can see that model considers all the previous context while answering the latest question.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we understand how prompts and responses work, let's deep dive into the API parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chat Completion API Parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"font-size: 20px; color: orange\"><b>\"model\"</b> and <b>\"messages\"</b> are the two required API parameters</span>\n",
    "\n",
    "<span style=\"font-size: 16px; color: blue\"> There are several other optional parameters that help configure the response</span>\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__n__ : Number of responses you want the LLM to generate for the instruction\n",
    "\n",
    "__max_tokens__ : Maximum number of tokens you want to restrict the Inference to (This includes both the prompt/messages and the completion)\n",
    "\n",
    "__temperature__ : Temperature controls the \"randomness\" of the responses. Higher value increases the randomness; lower value makes the output deterministic *(value between 0 and 2)*\n",
    "\n",
    "__top_p__ : The model considers the results of the tokens with top_p probability mass *(value between 0 and 1)*\n",
    "\n",
    "\n",
    "<img src=\"../Assets/Images/Temperature - Top P.png\">\n",
    "\n",
    "<span style=\"font-size: 14px; color: orange\"> __IMP__ : It is recommended to configure either one of \"temperature\" and \"top_p\" but not both</span>\n",
    "\n",
    "\n",
    "__frequency_penalty__ : Penalize new tokens based on their existing frequency in the text so far *(Value between -2 and 2)*\n",
    "\n",
    "__presence_penalty__ : Penalize new tokens based on whether they appear in the text so far *(Value between -2 and 2)*\n",
    "\n",
    "__logprobs__ : Flag to return log probability of the generated tokens *(True/False)*\n",
    "\n",
    "__logit_bias__ : Parameter to control the presence of particular tokens in the output *(Value between -100 and 100)*\n",
    "\n",
    "__response_format__ : Response of the model can be requested in a particular format *(Currently : JSON and Text)*\n",
    "\n",
    "__seed__ : Beta feature for reproducible outputs (setting a seed value may produce the same output repeatedly)\n",
    "\n",
    "__stop__ : End of Sequence tokens that will stop the generation\n",
    "\n",
    "__stream__ : To receive partial message deltas *(True/False)*\n",
    "\n",
    "__user__ : ID representing end user (This helps OpenAI detect abuse. May be mandatory for higher rate limits)\n",
    "\n",
    "__tools__ : used in function calling\n",
    "\n",
    "__tool_choice__ : used in function calling\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a function for calling the API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gpt_call(model:str=\"gpt-3.5-turbo\",prompt:str=\"Have I provided any input\",n:int=1,max_tokens:int=100,temperature:float=0.5,presence_penalty:float=0):\n",
    "\n",
    "\n",
    "    response = client.chat.completions.create(\n",
    "    model=model,\n",
    "    messages=[\n",
    "       {\"role\": \"user\", \"content\": prompt}\n",
    "    ],\n",
    "    max_tokens=max_tokens,\n",
    "    temperature=temperature,\n",
    "    presence_penalty=presence_penalty,\n",
    "    n=n\n",
    "    )\n",
    "    \n",
    "\n",
    "\n",
    "    if len(response.choices)>1:\n",
    "        output=''\n",
    "        for i in range(0,len(response.choices)):\n",
    "            output+='\\n\\n-- n = '+str(i+1)+' ------\\n\\n'+str(response.choices[i].message.content)\n",
    "    else:\n",
    "        output=response.choices[0].message.content\n",
    "        \n",
    "    \n",
    "\n",
    "    return output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "-- n = 1 ------\n",
      "\n",
      "\"Unlocking the Potential of AI: A Deep Dive into the OpenAI API\"\n",
      "\n",
      "-- n = 2 ------\n",
      "\n",
      "\"Exploring the Potential of OpenAI: A Hands-On Workshop on Harnessing the Power of AI Technology\"\n"
     ]
    }
   ],
   "source": [
    "print(gpt_call(prompt=\"Write a title for a workshop on openai API\",n=2,temperature=1,model=\"gpt-3.5-turbo\",max_tokens=100,presence_penalty=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Notice what happens when the temperature value is set to zero__\n",
    "\n",
    "__Try changing different parameter values__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "-- n = 1 ------\n",
      "\n",
      "\"Unlocking the Power of OpenAI: A Hands-On Workshop on Harnessing the OpenAI API\"\n",
      "\n",
      "-- n = 2 ------\n",
      "\n",
      "\"Unlocking the Power of OpenAI: A Hands-On Workshop on Harnessing the OpenAI API\"\n",
      "\n",
      "-- n = 3 ------\n",
      "\n",
      "\"Unlocking the Power of OpenAI: A Hands-On Workshop on Harnessing the OpenAI API\"\n"
     ]
    }
   ],
   "source": [
    "print(gpt_call(prompt=\"Write a title for a workshop on openai API\",n=3,temperature=0,model=\"gpt-3.5-turbo\",max_tokens=100,presence_penalty=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "Below is a simple __Gradio__ interface to try different prompts and settings. You can skip this part if you want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/kim/Desktop/Github/OpenAI-API-Explorer/.myenv/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running on local URL:  http://127.0.0.1:7860\n",
      "\n",
      "To create a public link, set `share=True` in `launch()`.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div><iframe src=\"http://127.0.0.1:7860/\" width=\"100%\" height=\"500\" allow=\"autoplay; camera; microphone; clipboard-read; clipboard-write;\" frameborder=\"0\" allowfullscreen></iframe></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import gradio as gr\n",
    "def text_to_uppercase(model,n,max_tokens,temperature,presence_penalty,prompt):\n",
    "    return str(type(n)) + str(type(temperature))\n",
    "\n",
    "model=gr.Radio([\"gpt-4\",\"gpt-3.5-turbo\"], label=\"Select Model\")\n",
    "n=gr.Radio([1,2,3], label=\"Number of Responses\")\n",
    "max_tokens=gr.Slider(minimum=10, maximum=500, label=\"Maximum Tokens\")\n",
    "temperature=gr.Slider(minimum=0.0, maximum=1.0, label=\"Temperature\")\n",
    "prompt=gr.Text(label=\"Prompt\")\n",
    "\n",
    "\n",
    "iface = gr.Interface(fn=gpt_call, inputs=[model,prompt,n,max_tokens,temperature], outputs=\"text\")\n",
    "iface.launch()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Closing server running on port: 7860\n"
     ]
    }
   ],
   "source": [
    "iface.close() # Close the gradio interface"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decoding the Chat Completions Response Object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may have noticed that we used __\"response.choices[0].message.content\"__ to get the text that the LLM responded with. However, the Chat Completions Response Object has much more information that is useful from a development standpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = client.chat.completions.create(\n",
    "  model=\"gpt-3.5-turbo\",\n",
    "  messages=[\n",
    "    {\"role\": \"system\", \"content\": \"You are a helpful assistant knowledgeable in the field of Cricket.\"},\n",
    "    {\"role\": \"user\", \"content\": \"When did Australia win their first Cricket World Cup?\"},\n",
    "    {\"role\": \"assistant\", \"content\": \"Australia won their first Cricket World Cup in the year 1987. They defeated England in the final to clinch their maiden title in the tournament.\"},\n",
    "    {\"role\": \"user\", \"content\": \"How much did they score?\"}\n",
    "  ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "     \"id\": \"chatcmpl-9IiBFB7wmCrGo4RZguowGSWjDr2L5\",\n",
      "     \"choices\": [\n",
      "          {\n",
      "               \"finish_reason\": \"stop\",\n",
      "               \"index\": 0,\n",
      "               \"logprobs\": null,\n",
      "               \"message\": {\n",
      "                    \"content\": \"In the final of the 1987 Cricket World Cup, Australia scored 253 runs for the loss of 5 wickets in their allocated 50 overs. England, in response, could only manage 246 runs, thus Australia won the match by 7 runs.\",\n",
      "                    \"role\": \"assistant\",\n",
      "                    \"function_call\": null,\n",
      "                    \"tool_calls\": null\n",
      "               }\n",
      "          }\n",
      "     ],\n",
      "     \"created\": 1714246761,\n",
      "     \"model\": \"gpt-3.5-turbo-0125\",\n",
      "     \"object\": \"chat.completion\",\n",
      "     \"system_fingerprint\": \"fp_3b956da36b\",\n",
      "     \"usage\": {\n",
      "          \"completion_tokens\": 54,\n",
      "          \"prompt_tokens\": 77,\n",
      "          \"total_tokens\": 131\n",
      "     }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "print((response.model_dump_json(indent=5)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../Assets/Images/Chat Completion Object.png\" >\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Streaming Responses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LLM responses take time to generate. This may lead to sub-optimal user experience where time to response is crucial (like Live Chat, QnA, Conversation etc.)\n",
    "\n",
    "One way to address it is not to wait for the entire response but get partial responses and stream that to the user. You will observe that ChatGPT also streams the response rather than waiting for the entire response.\n",
    "\n",
    "This is acoomplished by setting the __stream__ parameter to `True` and reading the __delta.content__ instead of __message.content__ from the response object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "In\n",
      " the\n",
      " final\n",
      " of\n",
      " the\n",
      " \n",
      "198\n",
      "7\n",
      " Cricket\n",
      " World\n",
      " Cup\n",
      ",\n",
      " Australia\n",
      " scored\n",
      " \n",
      "253\n",
      "/\n",
      "5\n",
      " in\n",
      " their\n",
      " \n",
      "50\n",
      " overs\n",
      ".\n",
      " England\n",
      ",\n",
      " in\n",
      " response\n",
      ",\n",
      " could\n",
      " only\n",
      " manage\n",
      " \n",
      "246\n",
      "/\n",
      "8\n",
      ",\n",
      " thereby\n",
      " giving\n",
      " Australia\n",
      " a\n",
      " comfortable\n",
      " victory\n",
      " in\n",
      " the\n",
      " match\n",
      ".\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "response = client.chat.completions.create(\n",
    "  model=\"gpt-3.5-turbo\",\n",
    "  messages=[\n",
    "    {\"role\": \"system\", \"content\": \"You are a helpful assistant knowledgeable in the field of Cricket.\"},\n",
    "    {\"role\": \"user\", \"content\": \"When did Australia win their first Cricket World Cup?\"},\n",
    "    {\"role\": \"assistant\", \"content\": \"Australia won their first Cricket World Cup in the year 1987. They defeated England in the final to clinch their maiden title in the tournament.\"},\n",
    "    {\"role\": \"user\", \"content\": \"How much did they score?\"}\n",
    "  ],\n",
    "  stream=True\n",
    ")\n",
    "\n",
    "for chunk in response:\n",
    "  print(chunk.choices[0].delta.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## JSON Mode"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you're developing LLM Applications, often times you'd want the responses in JSON format so that they can be used by downstream applications and services. \n",
    "\n",
    "While you can mention this instruction in the prompt itself, sometimes the results are not in a valid json format. \n",
    "\n",
    "With the __response_format__ parameter set to `True`, OpenAI makes sure that the responses will always be in JSON format.  This is very helpful in app development.\n",
    "\n",
    "You just have to make sure that the instruction to respond as JSON is present in the prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt=\"generate the entire text for a blog on a cricket match. \\\"Title\\\" is a catchy and attractive title of the blog. The \\\"Heading\\\" is the heading for each point in the blog and the \\\"Body\\\" is the text for that heading.Output in a json structure\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"Title\": \"Thrilling Cricket Match: A Recap of the Exciting Game\",\n",
      "  \"Heading\": [\n",
      "    {\n",
      "      \"Heading\": \"Introduction\",\n",
      "      \"Body\": \"The cricket match between Team A and Team B was nothing short of a rollercoaster ride for both the players and the fans. The game was filled with excitement, suspense, and unforgettable moments that kept everyone on the edge of their seats throughout.\"\n",
      "    },\n",
      "    {\n",
      "      \"Heading\": \"Innings Recap\",\n",
      "      \"Body\": \"Team A won the toss and elected to bat first. They got off to a great start with their openers scoring quick runs. However, Team B fought back with some excellent bowling and fielding, restricting Team A to a decent total. In response, Team B got off to a shaky start but managed to steady their innings through some solid partnerships in the middle order.\"\n",
      "    },\n",
      "    {\n",
      "      \"Heading\": \"Key Performances\",\n",
      "      \"Body\": \"One of the standout performances of the match was the century scored by Team A's captain, who played a captain's knock under pressure. Team B's bowlers also shone bright, taking crucial wickets at regular intervals. The fielding from both teams was exceptional, with some stunning catches and run-outs that added to the thrill of the game.\"\n",
      "    },\n",
      "    {\n",
      "      \"Heading\": \"Thrilling Finish\",\n",
      "      \"Body\": \"As the game entered its final overs, the tension in the air was palpable. Team A needed a few runs to win, while Team B needed a couple of wickets to secure victory. In the end, it all came down to the wire, with the match being decided on the last ball, making it a nail-biting finish that left everyone speechless.\"\n",
      "    },\n",
      "    {\n",
      "      \"Heading\": \"Conclusion\",\n",
      "      \"Body\": \"The cricket match between Team A and Team B will be remembered as one of the most thrilling encounters in recent times. Both teams displayed great skill, determination, and sportsmanship, making it a truly memorable game for all those who witnessed it. It was a reminder of why cricket is called the 'gentleman's game' and why it continues to captivate audiences around the world.\"\n",
      "    }\n",
      "  ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "response = client.chat.completions.create(\n",
    "  model=\"gpt-3.5-turbo\",\n",
    "  messages=[\n",
    "        {\"role\": \"user\", \"content\": prompt}\n",
    "  ],\n",
    "  response_format={ \"type\": \"json_object\" }\n",
    ")\n",
    "\n",
    "print((response.choices[0].message.content))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reproducibility of Outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even for the same prompts and parameter values, the LLM will respond differently. This becomes challenging in cases where you're looking to reproduce the same text for a give set of prompts and parameters.\n",
    "\n",
    "To address this issue, a __seed__ parameter is used. When a seed is set to a value, the outputs are the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Australia won their first Cricket World Cup in 1987. They defeated England in the final held at Eden Gardens in Kolkata, India.\n"
     ]
    }
   ],
   "source": [
    "response = client.chat.completions.create(\n",
    "  model=\"gpt-3.5-turbo\",\n",
    "  messages=[\n",
    "    {\"role\": \"system\", \"content\": \"You are a helpful assistant knowledgeable in the field of Cricket.\"},\n",
    "    {\"role\": \"user\", \"content\": \"When did Australia win their first Cricket World Cup?\"}\n",
    "  ],\n",
    "  seed=42\n",
    ")\n",
    "\n",
    "print(response.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"font-size: 20px; color: orange\"><b>Tokens are the fundamental units of NLP</b></span>\n",
    "\n",
    "<span style=\"font-size: 16px; color: blue\"><b>These units are typically words, punctuation marks, or other meaningful substrings that make up the text</b></span>\n",
    "\n",
    "Counting the number of tokens becomes important because - \n",
    "- Number of Tokens determine the amount of computation required and hence the cost you incur both in terms of money and the latency\n",
    "- Context Window or the maximum number of tokens an LLM can process in one go is limited\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Tiktoken__ is a library by OpenAI used to count tokens for LLMs and Embedding models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tiktoken"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "####num_tokens_from_string function to count number of tokens in a text string\n",
    "####uses tiktoken to count number of tokens in a text string\n",
    "####parameters: \"string\" is the text string, \"encoding_name\" is the encoding name to be used by tiktoken\n",
    "####returns: num_tokens->number of tokens in the text string\n",
    "####This function is used within extract_data, extract_page, extract_YT, extract_audio, extract_image functions\n",
    "def num_tokens_from_string(string: str, encoding_name=\"cl100k_base\") -> int: #### Function to count number of tokens in a text string ####\n",
    "    encoding = tiktoken.get_encoding(encoding_name) #### Initialize encoding ####\n",
    "    return len(encoding.encode(string)) #### Return number of tokens in the text string ####"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The total number of tokens in 'Hello! how are you?' are : 6\n"
     ]
    }
   ],
   "source": [
    "input_text=\"Hello! how are you?\"\n",
    "\n",
    "print(f\"The total number of tokens in '{input_text}' are : {num_tokens_from_string(input_text)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The total number of tokens in Alice In Wonderland are : 38680\n"
     ]
    }
   ],
   "source": [
    "with open(\"../Assets/Data/alice_in_wonderland.txt\") as f:\n",
    "    AliceInWonderland = f.read()\n",
    "\n",
    "print(f\"The total number of tokens in Alice In Wonderland are : {num_tokens_from_string(AliceInWonderland)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we were discussing, tokens become important because that determines the cost of invoking the LLMs. The pricing per million tokens for text generation models is given below. It is different for the embeddings models which we will discuss later\n",
    "\n",
    "__PRICING__\n",
    "\n",
    "__gpt-3.5-turbo-0125__\t    |  PROMPT - $0.50 / 1M tokens   |   RESPONSE - $1.50 / 1M tokens\n",
    "\n",
    "__gpt-4__\t                |   PROMPT - $30.00 / 1M tokens\t|   RESPONSE - $60.00 / 1M tokens\n",
    "\n",
    "__gpt-4-turbo__             |\tPROMPT - $10.00 / 1M tokens\t|   RESPONSE - $30.00 / 1M tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations! We're at the end of Day 1!\n",
    "\n",
    "Hopefully, now we are fairly confident around using the Chat Completions API and generating text using OpenAI models. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../Assets/Images/That’s all for the day.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# About\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../Assets/Images/Profile_AK.png\" width=100> \n",
    "\n",
    "#### Hi! I'm Abhinav! A data science and AI professional with over 15 years in the industry. Passionate about AI advancements, I constantly explore emerging technologies to push the boundaries and create positive impacts in the world. Let’s build the future, together!\n",
    "\n",
    "<span style=\"font-size: 20px; color: orange\"><b>Connect with me!</b></span>\n",
    "\n",
    "\n",
    "[![GitHub followers](https://img.shields.io/badge/Github-000000?style=for-the-badge&logo=github&logoColor=black&color=orange)](https://github.com/abhinav-kimothi)\n",
    "[![LinkedIn](https://img.shields.io/badge/LinkedIn-000000?style=for-the-badge&logo=linkedin&logoColor=orange&color=black)](https://www.linkedin.com/comm/mynetwork/discovery-see-all?usecase=PEOPLE_FOLLOWS&followMember=abhinav-kimothi)\n",
    "[![Medium](https://img.shields.io/badge/Medium-000000?style=for-the-badge&logo=medium&logoColor=black&color=orange)](https://medium.com/@abhinavkimothi)\n",
    "[![Insta](https://img.shields.io/badge/Instagram-000000?style=for-the-badge&logo=instagram&logoColor=orange&color=black)](https://www.instagram.com/akaiworks/)\n",
    "[![Mail](https://img.shields.io/badge/email-000000?style=for-the-badge&logo=gmail&logoColor=black&color=orange)](mailto:abhinav.kimothi.ds@gmail.com)\n",
    "[![X](https://img.shields.io/badge/Follow-000000?style=for-the-badge&logo=X&logoColor=orange&color=black)](https://twitter.com/abhinav_kimothi)\n",
    "[![Linktree](https://img.shields.io/badge/Linktree-000000?style=for-the-badge&logo=linktree&logoColor=black&color=orange)](https://linktr.ee/abhinavkimothi)\n",
    "[![Gumroad](https://img.shields.io/badge/Gumroad-000000?style=for-the-badge&logo=gumroad&logoColor=orange&color=black)](https://abhinavkimothi.gumroad.com/)\n",
    "\n",
    "\n",
    "<span style=\"font-size: 20px; color: orange\"><b>You can also book a time-slot with me</b></span>\n",
    "\n",
    "[![Static Badge](https://img.shields.io/badge/Resume%20Review%20(DS/AI/ML)%20(30%20min)-000000?style=for-the-badge&logo=googlecalendar&logoColor=blue&color=black)](https://topmate.io/abhinav_kimothi/544382)\n",
    "[![Static Badge](https://img.shields.io/badge/AIML%20Learning%20Path%20(30%20min)-000000?style=for-the-badge&logo=googlecalendar&logoColor=black&color=blue)](https://topmate.io/abhinav_kimothi/544380)\n",
    "[![Static Badge](https://img.shields.io/badge/Generative%20AI%20Consulting%20(60%20min)-000000?style=for-the-badge&logo=googlecalendar&logoColor=blue&color=black)](https://topmate.io/abhinav_kimothi/544379)\n",
    "\n",
    "\n",
    "<span style=\"font-size: 20px; color: orange\"><b>Also, read my ebooks for more on Generative AI!</b></span>\n",
    "\n",
    "\n",
    "\n",
    "<a href=\"https://abhinavkimothi.gumroad.com/l/GenAILLM\">\n",
    "    <img src=\"https://public-files.gumroad.com/jsdnnne2gnhu61f6hrdprwx2255i\" width=150>\n",
    "</a><a href=\"abhinavkimothi.gumroad.com/l/RAG\">\n",
    "    <img src=\"https://public-files.gumroad.com/v17k9tp2fnbbtg8iwoxt4m3xgivq\" width=150>\n",
    "</a><a href=\"abhinavkimothi.gumroad.com/l/GenAITaxonomy\">\n",
    "    <img src=\"https://public-files.gumroad.com/a730ysxb7a928bb5xkz6fuqabaqp\" width=150>\n",
    "</a>\n",
    "\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}