[ContextAugmentedFunctionAgent] multi tool function call with retrieval #148

w5688414 · 2023-12-12T15:41:38Z

遇到的问题：
- 构造retrieval的fake function call，会导致编排失效（planning是在第一次触发function call的时候）
解决方法：把检索的结果作为上下文
- Context-Augmented retrieval Agent, refer to: https://docs.llamaindex.ai/en/latest/examples/agent/openai_agent_context_retrieval.html
  - Add Query rewrite
- direct retrieval prompt agent
- 单测

TODO:

PR会进行拆分：
- FunctionAgentWithMultiRetrievalTools : summary fulltext tool [FunctionAgentWithMultiRetrievalTools]Add summary fulltext multi retrieval strategy #303
  - Planning and execute as post processing for solving unlimited loop
- FunctionAgentWithMultiRetrievalTools: one document as a tool
  - Automatic prompt engineering 优化tool description和few shot

erniebot-agent/erniebot_agent/agents/functional_agent_with_retrieval.py

sijunhe · 2023-12-18T13:36:43Z

erniebot-agent/erniebot_agent/agents/functional_agent_with_retrieval.py

+            # return response
+            tool_ret_json = json.dumps({"documents": outputs}, ensure_ascii=False)
+            # Direct Prompt
+            next_step_input = HumanMessage(content=f"问题：{prompt}，要求：请在第一步执行检索的操作,并且检索只允许调用一次")


FunctionalAgentWithRetrievalScoreTool和FunctionalAgentWithRetrievalTool 完全分不出来了

FunctionAgentWithRetrievalTool是模拟一次function call的调用
FunctionAgentWithRetrievalScoreTool是通过prompt的方式来引导function call调用检索

sijunhe · 2023-12-18T13:37:30Z

erniebot-agent/erniebot_agent/agents/functional_agent_with_retrieval.py

@@ -298,3 +385,36 @@ async def _maybe_retrieval(
        results = {}
        results["documents"] = documents
        return results
+
+
+class FunctionalAgentWithQueryPlanning(FunctionalAgent):


这是个啥..

这个是把retrieval 当成tool的类，planning部分用的是functional agent默认的，可能后面会加入一些后处理

sijunhe · 2023-12-18T13:38:50Z

erniebot-agent/erniebot_agent/tools/openai_search_tool.py

+    documents: List[SearchResponseDocument] = Field(description="检索结果，内容和用户输入query相关的段落")
+
+
+class OpenAISearchTool(Tool):


这个类和OpenAI有啥关系呀?

这里面的db用的是langchain 的faiss，faiss的search接口和返回的参数跟欧若拉不一样，所以重新写了一个tool

sijunhe · 2023-12-18T13:40:01Z

erniebot-agent/examples/functional_agent_with_retrieval_example.py

                llm=llm,
                knowledge_base=baizhong_db,
                top_k=3,
-                tools=toolkit.get_tools() + [retrieval_tool],
+                tools=toolkit.get_tools() + [NotesTool(), retrieval_tool],


为啥这里还需要传入retrieval_tool这个工具呢？

example文件删除

sijunhe · 2023-12-18T13:41:02Z

erniebot-agent/examples/functional_agent_with_retrieval_example.py

+                tools=[NotesTool(), retrieval_tool],
+                memory=memory,
+            )
+        elif args.retrieval_type == "context_aug":
+            agent = ContextAugmentedFunctionalAgent(  # type: ignore
+                llm=llm,
+                knowledge_base=baizhong_db,
+                top_k=3,
+                threshold=0.1,
+                tools=[NotesTool(), retrieval_tool],


这些agent的tools哪些应该传入retrieval_tool, 哪些不应该传入，需要想清楚

已删除functional_agent_with_retrieval_example.py

…rieval.py Co-authored-by: Sijun He <[email protected]>

… into eb10

CLAassistant · 2024-10-14T07:09:55Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

w5688414 added 4 commits December 12, 2023 23:30

Fix conflicts

7e700d4

multi tool function call with retrieval

62cab2c

Update to _async_run_llm_without_hooks

9a63a99

Add direct prompts

039f53b

w5688414 changed the title ~~multi tool function call with retrieval~~ [enhance] multi tool function call with retrieval Dec 13, 2023

w5688414 self-assigned this Dec 13, 2023

Update prompts

bc07951

w5688414 changed the title ~~[enhance] multi tool function call with retrieval~~ [AutoRetrievel] multi tool function call with retrieval Dec 15, 2023

w5688414 changed the title ~~[AutoRetrievel] multi tool function call with retrieval~~ [AutoRetrievel-Context-Augmented Agent] multi tool function call with retrieval Dec 15, 2023

w5688414 changed the title ~~[AutoRetrievel-Context-Augmented Agent] multi tool function call with retrieval~~ [AutoRetrievel] multi tool function call with retrieval Dec 15, 2023

w5688414 added 7 commits December 15, 2023 09:53

Add context augmented retrieval agent

4b14f85

Remove type error

a3ca306

Add direct prompt

1f4674f

Add knowledge base tools impl

775c363

Add retriever config

f028c84

Update tool retrieval

b3b1a99

Update tool retrieval

67066da

sijunhe reviewed Dec 18, 2023

View reviewed changes

w5688414 and others added 12 commits December 19, 2023 00:41

Add system prompt

51c0137

Add faiss indexing

3320e7b

Update erniebot-agent/erniebot_agent/agents/functional_agent_with_ret…

76604f8

…rieval.py Co-authored-by: Sijun He <[email protected]>

Add planning and execute rules

e28cf0b

Update planning logic

a6d331a

Add automatic prompt engineer

a78fa66

Update prompt engineer

c25e6aa

Add prompt agent

ff980ba

Fix conflicts

cc37313

Resolve conflicts

f6ba05b

Fix conflicts

e37f2bf

Update retrieval tools

6b95d87

w5688414 added 15 commits January 16, 2024 11:57

Update name

254df32

Update format

1b059d6

Update unitests

add83cd

remove functional_agent_with_retrieval_example.py

b915176

Update function_agent_with_retrieval.py

d4a034d

Update ContextAugmentedFunctionalAgent

30d8081

Update prompt agent

d97ee9d

Update prompt agent

2c39536

remove prompt agent

4534b1d

delete faiss_util.py

ce74d54

Update

f8b0450

Merge branch 'develop' of https://github.com/PaddlePaddle/ERNIE-Bot-SDK…

a59f84e

… into eb10

delete knowledge_tools_example

c104795

Update format

c85c7c4

remove FunctionAgentWithQueryPlanning

6740056

w5688414 changed the title ~~[AutoRetrievel] multi tool function call with retrieval~~ [ContextAugmentedFunctionAgent] multi tool function call with retrieval Jan 19, 2024

w5688414 added 6 commits January 19, 2024 03:40

Update retrieval agents

ccea25b

Add unitests

4551c7c

Update steps_taken

f17e225

reformat

b24e685

Update erniebot

a860def

Update unitest

e3dd45f

w5688414 closed this Oct 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ContextAugmentedFunctionAgent] multi tool function call with retrieval #148

[ContextAugmentedFunctionAgent] multi tool function call with retrieval #148

w5688414 commented Dec 12, 2023 •

edited

Loading

sijunhe Dec 18, 2023

w5688414 Jan 19, 2024

sijunhe Dec 18, 2023

w5688414 Dec 19, 2023

sijunhe Dec 18, 2023

w5688414 Dec 19, 2023

sijunhe Dec 18, 2023

w5688414 Jan 17, 2024

sijunhe Dec 18, 2023

w5688414 Jan 17, 2024

CLAassistant commented Oct 14, 2024

		documents: List[SearchResponseDocument] = Field(description="检索结果，内容和用户输入query相关的段落")


		class OpenAISearchTool(Tool):

[ContextAugmentedFunctionAgent] multi tool function call with retrieval #148

[ContextAugmentedFunctionAgent] multi tool function call with retrieval #148

Conversation

w5688414 commented Dec 12, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CLAassistant commented Oct 14, 2024

w5688414 commented Dec 12, 2023 •

edited

Loading