README.md

Human in the Loop

Here is a simple example of how to use websockets to create a human in the loop (HITL) system. This demo uses ChatOpenAI in streaming mode to send responses to the client as soon as they are available. The client has the context available to it. When human action is needed, client asks the user for the input and sends it back to the server. The server then uses the context to continue the conversation.

Setup

This directory contains 3 files

hitl.py

1. Defines a function hitl decorated with @serving with websocket=True.

   langchain-serve/examples/websockets/hitl/hitl.py

   Lines 11 to 12 in c0af0de

   	@serving(websocket=True)
   	def hitl(question: str, **kwargs) -> str:

2. Gets streaming_handler from kwargs and passes it to ChatOpenAI and OpenAI callback managers. This handler is responsible to stream the response to the client.

   langchain-serve/examples/websockets/hitl/hitl.py

   Lines 19 to 22 in c0af0de

   	streaming=True, # Pass `streaming=True` to make sure the client receives the data.
   	callback_manager=CallbackManager(
   	[streaming_handler]
   	), # Pass the callback handler

   langchain-serve/examples/websockets/hitl/hitl.py

   Lines 27 to 30 in c0af0de

   	streaming=True, # Pass `streaming=True` to make sure the client receives the data.
   	callback_manager=CallbackManager(
   	[streaming_handler]
   	), # Pass the callback handler

3. Returns agent.run output which is finally streamed to the client.

   langchain-serve/examples/websockets/hitl/hitl.py

   Line 43 in c0af0de

   return agent_chain.run(question)

---

requirements.txt

Contains the dependencies for the hitl endpoint.

---

Deploy using lc-serve

lc-serve deploy jcloud hitl

╭──────────────┬──────────────────────────────────────────────────────────────────╮
│ AppID        │                       langchain-1da55ad36a                       │
├──────────────┼──────────────────────────────────────────────────────────────────┤
│ Phase        │                             Serving                              │
├──────────────┼──────────────────────────────────────────────────────────────────┤
│ Endpoint     │        wss://langchain-1da55ad36a-websocket.wolf.jina.ai         │
├──────────────┼──────────────────────────────────────────────────────────────────┤
│ Swagger UI   │     https://langchain-1da55ad36a-websocket.wolf.jina.ai/docs     │
├──────────────┼──────────────────────────────────────────────────────────────────┤
│ OpenAPI JSON │ https://langchain-1da55ad36a-websocket.wolf.jina.ai/openapi.json │
╰──────────────┴──────────────────────────────────────────────────────────────────╯

---

hitl_client.py

A simple python client that connects to the websocket server and sends the question to the hitl endpoint. It then listens to the stream of responses and prints it to the console. When it receives a response in the following format, it asks the prompt to the user using the client and waits for the user to input the answer. (This is how human is brought into the loop). Next, this answer is then sent to the server.

This can be implemented in any language that supports websockets.

1. Connects to the websocket server and sends the following to the hitl endpoint.

   {
       "question": "${question}", 
       "envs": {
           "OPENAI_API_KEY": "${OPENAI_API_KEY}"
       }
   }

   langchain-serve/examples/websockets/hitl/hitl_client.py

   Lines 24 to 29 in c0af0de

   	await ws.send_json(
   	{
   	"question": question,
   	"envs": envs if envs else {},
   	}
   	)

2. Listens to the stream of responses and prints it to the console

   langchain-serve/examples/websockets/hitl/hitl_client.py

   Lines 31 to 39 in c0af0de

   	async for msg in ws:
   	if msg.type == aiohttp.WSMsgType.TEXT:
   	if msg.data == 'close cmd':
   	await ws.close()
   	break
   	else:
   	try:
   	response = Response.parse_raw(msg.data)
   	print(response.result, end='')

3. When it receives a response in the following format, it asks the prompt to the user using the client and waits for the user to input the answer. (This is how human is brought into the loop). Next, this answer is then sent to the server.

   {
       "prompt": "$prompt"
   }

   langchain-serve/examples/websockets/hitl/hitl_client.py

   Lines 42 to 44 in c0af0de

   	prompt = HumanPrompt.parse_raw(msg.data)
   	answer = input(prompt.prompt + '\n')
   	await ws.send_str(answer)

4. Finally, the client is disconnected from the server automatically when the hitl function is done executing.

---

Example run on localhost

python hitl_client.py

Connected to ws://localhost:8080/hitl.
I don't know Eric Zhu's birthday, so I need to ask a human.
Action: Human
Action Input: "Do you know Eric Zhu and his birthday?"
Yes
Great, now I can ask for Eric Zhu's birthday.
Action: Human
Action Input: "What is Eric Zhu's birthday?"
29th Feb
I need to make sure this is a valid date.
Action: Calculator
Action Input: Check if 29th Feb is a valid date

import datetime

try:
    datetime.datetime(2020, 2, 29)
    print("Valid date")
except ValueError:
    print("Invalid date")

I now have a valid birth date, but I need to know the year for Eric's age.
Action: Human
Action Input: "Do you know Eric Zhu's birth year?"
1990
Now I can calculate Eric Zhu's age.
Action: Calculator
Action Input: Current year minus 1990

import datetime
print(datetime.datetime.now().year - 1990)

I now know Eric Zhu's age.
Final Answer: Eric Zhu's birthday is February 29th, 1990 and he is currently 33 years old.Eric Zhu's birthday is February 29th, 1990 and he is currently 33 years old.%