Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Separate Knowledge system + Multi-Agent RAG Optimization #1620

Merged

Conversation

azep-ninja
Copy link
Contributor

Relates to:

Risks

Low Risk, as the new RAG system is completely separate, and must be enabled by the user to use as of now.

Background

What does this PR do?

What kind of change is this?

Improvements (misc. changes to existing features)

This improves the knowledge system by separating it from the memories table, while allowing users to better manager their knowledge and have true separation of actions.

Additionally, allowing multiple agents to "share" knowledge to avoid duplicating entries for better optimizations.

All database adapters were updated to have the new functions to create/get/search/delete knowledge.

Why are we doing this? Any context or related work?

This change is a critical change as it helps the agent scale, reduce database overlap, and separates the agents actions, to ensure proper segregation.

Additionally, this also allows users to create a knowledge folder of txt/md files and reference them in their character knowledge, without having to paste the full contents in character file.

Documentation changes needed?

My changes require a change to the project documentation.

Minimal changes to exiting knowledge docs, to outline how to enable the new rag knowledge system.

Testing

Where should a reviewer start?

Detailed testing steps

  • Enable the new rag knowledge
    image

  • Create a "knowledge" folder within your characters folder
    image

  • Add files in the root of knowledge, or create additional subfolders for organization

  • Set your knowledge items in your character file directly, or objects specifying the path, and if it should be a shared item or not, depending on multi characters or not.
    image

  • Start agent(s) as normal, and watch knowledge get processed!

Screenshots

Before

After

Knowledge table with information and split into chunks, along with main document, shared documents, or agent specific documents in the database.
image

Deploy Notes

Database changes

  • New knowledge table is created

Deployment instructions

Discord username

ninja_dev

@LinuxIsCool
Copy link

Trying to run this locally. Getting:

 ["⛔ Error parsing character from /characters/character.character.json: Error: Character configuration validation failed:\nknowledge.0: Expected string, received object\nknowledge.1: Expected string, received object"]

@azep-ninja
Copy link
Contributor Author

Error parsing character from

@LinuxIsCool , good catch. Updated the character validation now. Can you pull the latest version and try.

Updated "package/core/src/environment.ts" file within CharacterSchema, replacing "knowledge:"

With

knowledge: z.array( z.union([ z.string(), z.object({ path: z.string(), shared: z.boolean().optional() }) ]) ).optional(),

@azep-ninja azep-ninja requested a review from thisisomar January 2, 2025 21:50
@LinuxIsCool
Copy link

@azep-ninja

I'm running it now without errors on the latest version, however, it's not working as expected. The agent is running but there is no prompt in the console.

I am seeing:

["✓ REST API bound to 0.0.0.0:3003. If running locally, access it at http://localhost:3003."]

However, I'm not sure how to access the REST API.

And when I try the client with pnpm start:client

Then, I see my agent in the dashboard, but there is not response when I post a message, and I see in the client console:

> eliza@ start:client /home/ygg/Workspace/Eliza/akemi_rag
> pnpm --dir client dev


> [email protected] dev /home/ygg/Workspace/Eliza/akemi_rag/client
> vite


  VITE v5.4.11  ready in 158 ms

  ➜  Local:   http://localhost:5173/
  ➜  Network: use --host to expose
  ➜  press h + enter to show help
(node:118597) ExperimentalWarning: CommonJS module /home/ygg/Workspace/Eliza/akemi_rag/node_modules/tailwindcss/lib/lib/load-config.js is loading ES Module /home/ygg/Workspace/Eliza/akemi_rag/client/tailwind.config.js using require().
Support for loading ES Module in require() is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
6:35:05 PM [vite] http proxy error: /agents
AggregateError [ECONNREFUSED]:
    at internalConnectMultiple (node:net:1128:18)
    at afterConnectMultiple (node:net:1693:7)
6:35:06 PM [vite] http proxy error: /agents
AggregateError [ECONNREFUSED]:
    at internalConnectMultiple (node:net:1128:18)
    at afterConnectMultiple (node:net:1693:7) (x2)
6:35:08 PM [vite] http proxy error: /agents
AggregateError [ECONNREFUSED]:
    at internalConnectMultiple (node:net:1128:18)
    at afterConnectMultiple (node:net:1693:7) (x3)
    ```

@LinuxIsCool
Copy link

The client is now working properly after nvm use latest and re-install and re-build. So I will now test out the rag functionality.

@LinuxIsCool
Copy link

The RAG seems to be working!

Although now I have added a 5MB text file and the RAG system is currently hanging after about 5 minutes. I'm hoping it finishes processing soon and is not hanging indefinitely.

@azep-ninja
Copy link
Contributor Author

The RAG seems to be working!

Although now I have added a 5MB text file and the RAG system is currently hanging after about 5 minutes. I'm hoping it finishes processing soon and is not hanging indefinitely.

Hmm, a 5MB file isn't that huge.

Let me know if it ever succeeds or doesn't and I can replicate to see if it's a bug.

In my tests I didn't have any humongous files so could be some need for optimizations of course.

But glad to hear it did work!

@LinuxIsCool
Copy link

I think testing with big files is a good idea. It does seem to hang. But when I restart the agent then the knowledge is loaded. Are there ways that it could be sped up?

Comment on lines -27 to -29
-- Then check for GAIANET
ELSIF current_setting('app.use_gaianet_embedding', TRUE) = 'true' THEN
RETURN 768; -- Gaianet nomic-embed dimension
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any specific reason why this is removed?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved!

@shakkernerd shakkernerd merged commit 24a38ee into elizaOS:develop Jan 8, 2025
3 checks passed
@augchan42
Copy link
Contributor

augchan42 commented Jan 9, 2025

Note that only the following file formats are supported:

 await this.ragKnowledgeManager.processFile({
                            path: contentItem,
                            content: content,
                            type: fileExtension as 'pdf' | 'md' | 'txt',
                            isShared: isShared
                        });

If a different file format is specified (e.g., .json) then only the filename is vectorized and stored as knowledge

@iwasfirsthere
Copy link

Could you please let me know how should I add links to my knowledge file and make sure agent reads them correctly, send them when asked?

It was working, but I adjusted style of my character (did all the data deletion before running again and so) but it 50% of the time now gives random links that does not exists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants