thought_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# smart LLM imports\n",
    "from langchain.llms import VertexAI\n",
    "from langchain import PromptTemplate, LLMChain\n",
    "from langchain.prompts import PromptTemplate\n",
    "from langchain.chat_models import ChatOpenAI\n",
    "from langchain_experimental.smart_llm import SmartLLMChain\n",
    "# tree of thoughts imports\n",
    "from typing import Tuple\n",
    "from langchain_experimental.tot.base import ToTChain\n",
    "from langchain_experimental.tot.checker import ToTChecker\n",
    "from langchain_experimental.tot.thought import ThoughtValidity\n",
    "\n",
    "import load_dotenv\n",
    "load_dotenv.load_dotenv()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A tricky question to start with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "hard_question = \"\"\"\n",
    "Bob is in the living room.\n",
    "He walks to the kitchen, carrying a cup.\n",
    "He puts a ball in the cup and carries the cup to the bedroom.\n",
    "He turns the cup upside down, then walks to the garden.\n",
    "He puts the cup down in the garden, then walks to the garage.\n",
    "Where is the ball?\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normal prompting technique"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "np = PromptTemplate.from_template(\"Answer user's question. {question}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### answer from GPT 3.5 turbo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The ball is in the cup, which Bob left in the garden.\n"
     ]
    }
   ],
   "source": [
    "llm = ChatOpenAI(model_name = 'gpt-3.5-turbo-0613', temperature = '1')\n",
    "llm_chain = LLMChain(prompt=np, llm=llm)\n",
    "print(llm_chain(hard_question)['text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### answer from Vertex AI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " The ball is in the garden.\n"
     ]
    }
   ],
   "source": [
    "llm = VertexAI(model_name = 'text-bison-32k', temperature = '0.2')\n",
    "llm_chain = LLMChain(prompt=np, llm=llm)\n",
    "print(llm_chain(hard_question)['text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### answer from GPT 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The ball is in the bedroom.\n"
     ]
    }
   ],
   "source": [
    "llm = ChatOpenAI(model_name = 'gpt-4', temperature = '1')\n",
    "llm_chain = LLMChain(prompt=np, llm=llm)\n",
    "print(llm_chain(hard_question)['text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ideation -> Critique -> Resolve technique"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new SmartLLMChain chain...\u001b[0m\n",
      "Prompt after formatting:\n",
      "\u001b[32;1m\u001b[1;3mDont jump to the conclusion immediately, analyze step by step, apply different reasoning strategy: \"\"\"\n",
      "Bob is in the living room.\n",
      "He walks to the kitchen, carrying a cup.\n",
      "He puts a ball in the cup and carries the cup to the bedroom.\n",
      "He turns the cup upside down, then walks to the garden.\n",
      "He puts the cup down in the garden, then walks to the garage.\n",
      "Where is the ball?\n",
      "?\"\"\"\u001b[0m\n",
      "Idea 1:\n",
      "\u001b[36;1m\u001b[1;3m1. Bob starts in the living room.\n",
      "2. He walks to the kitchen, carrying a cup.\n",
      "3. He puts a ball in the cup.\n",
      "4. He carries the cup with the ball to the bedroom.\n",
      "5. He turns the cup upside down.\n",
      "6. He walks to the garden.\n",
      "7. He puts the cup down in the garden.\n",
      "8. He walks to the garage.\n",
      "\n",
      "Based on the given information, we can conclude that the ball is still in the bedroom. The ball was placed in the cup in the kitchen, but Bob never took the ball out of the cup before putting it down in the garden. Therefore, the ball must still be in the cup in the bedroom.\u001b[0m\n",
      "Idea 2:\n",
      "\u001b[36;1m\u001b[1;3m1. Bob is in the living room.\n",
      "2. Bob walks to the kitchen, carrying a cup.\n",
      "3. Bob puts a ball in the cup.\n",
      "4. Bob carries the cup to the bedroom.\n",
      "5. Bob turns the cup upside down in the bedroom.\n",
      "6. Bob walks to the garden.\n",
      "7. Bob puts the cup down in the garden.\n",
      "8. Bob walks to the garage.\n",
      "\n",
      "Based on the given information, we can conclude that the ball is in the bedroom because Bob puts the ball in the cup in the kitchen, carries the cup to the bedroom, and then turns the cup upside down in the bedroom.\u001b[0m\n",
      "Idea 3:\n",
      "\u001b[36;1m\u001b[1;3m1. Bob is in the living room.\n",
      "2. He walks to the kitchen, carrying a cup.\n",
      "3. He puts a ball in the cup and carries the cup to the bedroom.\n",
      "4. He turns the cup upside down, then walks to the garden.\n",
      "5. He puts the cup down in the garden, then walks to the garage.\n",
      "\n",
      "Based on the given information, we can conclude that the ball is in the bedroom. This is because Bob puts the ball in the cup in the kitchen, and carries the cup to the bedroom. He then turns the cup upside down in the garden, meaning that the ball is no longer inside the cup. So, the ball must still be in the place where he last had the cup, which is the bedroom.\u001b[0m\n",
      "Critique:\n",
      "\u001b[33;1m\u001b[1;3mFlaws and faulty logic in each answer option:\n",
      "\n",
      "Idea 1:\n",
      "The flaw in this answer option is that it assumes Bob never took the ball out of the cup before putting it down in the garden. However, the given information does not explicitly state whether Bob took the ball out or left it in the cup. It is possible that he removed the ball from the cup before putting it down in the garden.\n",
      "\n",
      "Idea 2:\n",
      "The flaw in this answer option is that it assumes Bob turned the cup upside down in the bedroom. However, the given information does not state that Bob turned the cup upside down in the bedroom. It only mentions that he turned the cup upside down, but not where he did it. It is possible that he turned the cup upside down in the garden.\n",
      "\n",
      "Idea 3:\n",
      "The flaw in this answer option is that it assumes the ball is in the bedroom. However, the given information does not explicitly state that Bob left the ball in the cup when he turned it upside down. It is possible that he removed the ball from the cup before turning it upside down in the garden.\n",
      "\n",
      "In conclusion, none of the answer options can be determined as correct or flaw-free based on the given information.\u001b[0m\n",
      "Resolution:\n",
      "\u001b[32;1m\u001b[1;3mIdea 3: 1. Bob is in the living room.\n",
      "2. He walks to the kitchen, carrying a cup.\n",
      "3. He puts a ball in the cup and carries the cup to the bedroom.\n",
      "4. He turns the cup upside down, then walks to the garden.\n",
      "5. He puts the cup down in the garden, then walks to the garage.\n",
      "\n",
      "Based on the given information, we can conclude that the ball is in the bedroom. This is because Bob puts the ball in the cup in the kitchen, carries the cup to the bedroom, and then turns the cup upside down in the garden. Therefore, the ball must still be in the place where he last had the cup, which is the bedroom.\n",
      "\n",
      "Improved Answer: The ball is in the bedroom, as Bob puts the ball in the cup in the kitchen, carries the cup to the bedroom, and then turns the cup upside down in the garden. Therefore, the ball must still be in the bedroom.\n",
      "\n",
      "Full Answer: The ball is in the bedroom. Bob puts the ball in the cup in the kitchen, carries the cup to the bedroom, and then turns the cup upside down in the garden. Therefore, the ball must still be in the bedroom.\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Idea 3: 1. Bob is in the living room.\\n2. He walks to the kitchen, carrying a cup.\\n3. He puts a ball in the cup and carries the cup to the bedroom.\\n4. He turns the cup upside down, then walks to the garden.\\n5. He puts the cup down in the garden, then walks to the garage.\\n\\nBased on the given information, we can conclude that the ball is in the bedroom. This is because Bob puts the ball in the cup in the kitchen, carries the cup to the bedroom, and then turns the cup upside down in the garden. Therefore, the ball must still be in the place where he last had the cup, which is the bedroom.\\n\\nImproved Answer: The ball is in the bedroom, as Bob puts the ball in the cup in the kitchen, carries the cup to the bedroom, and then turns the cup upside down in the garden. Therefore, the ball must still be in the bedroom.\\n\\nFull Answer: The ball is in the bedroom. Bob puts the ball in the cup in the kitchen, carries the cup to the bedroom, and then turns the cup upside down in the garden. Therefore, the ball must still be in the bedroom.'"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prompt_template = 'Don''t jump to the conclusion immediately, analyze step by step, apply different reasoning strategy: \"\"\"{question}?\"\"\"'\n",
    "prompt_template = PromptTemplate.from_template(prompt_template)\n",
    "llm = VertexAI(model_name = 'text-bison', temperature = '0.7')\n",
    "smart_chain = SmartLLMChain(ideation_llm = ChatOpenAI(model_name = 'gpt-3.5-turbo-0613', temperature = '1'),\n",
    "                    critique_llm = ChatOpenAI(model_name = 'gpt-3.5-turbo-0613', temperature = '0.7'),\n",
    "                    resolver_llm = ChatOpenAI(model_name = 'gpt-3.5-turbo-0613', temperature = '0'),\n",
    "                    prompt=prompt_template, n_ideas=3, verbose=True)\n",
    "smart_chain.run(hard_question)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tree of Thought technique"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new ToTChain chain...\u001b[0m\n",
      "Starting the ToT solve procedure.\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new ProposePromptStrategy chain...\u001b[0m\n",
      "Prompt after formatting:\n",
      "\u001b[32;1m\u001b[1;3mYou are an intelligent agent that is generating thoughts in a tree of\n",
      "thoughts setting.\n",
      "\n",
      "The output should be a markdown code snippet formatted as a JSON list of\n",
      "strings, including the leading and trailing \"```json\" and \"```\":\n",
      "\n",
      "```json\n",
      "[\n",
      "    \"<thought-1>\",\n",
      "    \"<thought-2>\",\n",
      "    \"<thought-3>\"\n",
      "]\n",
      "```\n",
      "\n",
      "PROBLEM\n",
      "\n",
      "\n",
      "Bob is in the living room.\n",
      "He walks to the kitchen, carrying a cup.\n",
      "He puts a ball in the cup and carries the cup to the bedroom.\n",
      "He turns the cup upside down, then walks to the garden.\n",
      "He puts the cup down in the garden, then walks to the garage.\n",
      "Where is the ball?\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Possible next 3 valid thoughts based on the PROBLEM:\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/_blitzzz/Documents/GitHub/llm-meetup-demo/llm_venv/lib/python3.9/site-packages/langchain/chains/llm.py:278: UserWarning: The predict_and_parse method is deprecated, instead pass an output parser directly to LLMChain.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "\u001b[31;1m\u001b[1;3mThought: The ball is in the cup.\n",
      "\u001b[0m\u001b[32;1m\u001b[1;3mThought: The ball is in the bedroom.\n",
      "\u001b[0m\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "class check_answer(ToTChecker):\n",
    "    def evaluate(self, problem_description: str, thoughts: Tuple[str, ...] = ()) -> ThoughtValidity:\n",
    "        last_thought = thoughts[-1]\n",
    "        if 'bedroom' in last_thought:\n",
    "            return ThoughtValidity.VALID_FINAL\n",
    "        else:\n",
    "            return ThoughtValidity.INVALID\n",
    "            \n",
    "llm = VertexAI(model_name = 'text-bison', temperature = '0.7')\n",
    "tot_chain = ToTChain(llm=llm, checker=check_answer(), k=10, c=3, verbose=True, verbose_llm=True)\n",
    "tot_answer = tot_chain.run(problem_description=hard_question)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llm_venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}