diff --git a/.semversioner/next-release/patch-20241126215650769602.json b/.semversioner/next-release/patch-20241126215650769602.json new file mode 100644 index 0000000000..bd384d0f4e --- /dev/null +++ b/.semversioner/next-release/patch-20241126215650769602.json @@ -0,0 +1,4 @@ +{ + "type": "patch", + "description": "Fix question gen." +} diff --git a/dictionary.txt b/dictionary.txt index 0288faac9a..e2ea99f021 100644 --- a/dictionary.txt +++ b/dictionary.txt @@ -24,6 +24,7 @@ getcwd fillna noqa dtypes +ints # Azure abfs @@ -167,6 +168,7 @@ FIRUZABAD Krohaara KROHAARA POKRALLY +René Tazbah TIRUZIA Tiruzia diff --git a/docs/blog_posts.md b/docs/blog_posts.md index b2467a1371..2ff64abcea 100644 --- a/docs/blog_posts.md +++ b/docs/blog_posts.md @@ -38,4 +38,10 @@ By Bryan Li, Research Intern; [Ha Trinh](https://www.microsoft.com/en-us/research/people/trinhha/), Senior Data Scientist; [Darren Edge](https://www.microsoft.com/en-us/research/people/daedge/), Senior Director; [Jonathan Larson](https://www.microsoft.com/en-us/research/people/jolarso/), Senior Principal Data Architect +- [:octicons-arrow-right-24: __LazyGraphRAG: Setting a new standard for quality and cost__](https://www.microsoft.com/en-us/research/blog/lazygraphrag-setting-a-new-standard-for-quality-and-cost/) + + --- +
Published November 25, 2024 + + By [Darren Edge](https://www.microsoft.com/en-us/research/people/daedge/), Senior Director; [Ha Trinh](https://www.microsoft.com/en-us/research/people/trinhha/), Senior Data Scientist; [Jonathan Larson](https://www.microsoft.com/en-us/research/people/jolarso/), Senior Principal Data Architect
\ No newline at end of file diff --git a/docs/examples_notebooks/global_search.ipynb b/docs/examples_notebooks/global_search.ipynb index 02f8ba63d3..14a188e7c8 100644 --- a/docs/examples_notebooks/global_search.ipynb +++ b/docs/examples_notebooks/global_search.ipynb @@ -75,9 +75,9 @@ "source": [ "### Load community reports as context for global search\n", "\n", - "- Load all community reports in the `create_final_community_reports` table from the ire-indexing engine, to be used as context data for global search.\n", - "- Load entities from the `create_final_nodes` and `create_final_entities` tables from the ire-indexing engine, to be used for calculating community weights for context ranking. Note that this is optional (if no entities are provided, we will not calculate community weights and only use the rank attribute in the community reports table for context ranking)\n", - "- Load all communities in the `create_final_communites` table from the ire-indexing engine, to be used to reconstruct the community graph hierarchy for dynamic community selection." + "- Load all community reports in the `create_final_community_reports` table from the GraphRAG, to be used as context data for global search.\n", + "- Load entities from the `create_final_nodes` and `create_final_entities` tables from the GraphRAG, to be used for calculating community weights for context ranking. Note that this is optional (if no entities are provided, we will not calculate community weights and only use the rank attribute in the community reports table for context ranking)\n", + "- Load all communities in the `create_final_communites` table from the GraphRAG, to be used to reconstruct the community graph hierarchy for dynamic community selection." ] }, { @@ -379,21 +379,23 @@ "text": [ "### Overview of Cosmic Vocalization\n", "\n", - "Cosmic Vocalization is a phenomenon that has garnered significant attention from various individuals and groups. It is perceived as a cosmic event with potential implications for security and interstellar communication. The Paranormal Military Squad is actively engaged with Cosmic Vocalization, indicating its strategic importance in security measures [Data: Reports (6)].\n", + "Cosmic Vocalization is a phenomenon that has garnered significant attention within the community, involving various individuals and groups. It is perceived as an interstellar event with potential implications for both communication and security.\n", "\n", - "### Key Perspectives and Concerns\n", + "### Key Perspectives\n", "\n", - "1. **Strategic Engagement**: The Paranormal Military Squad's involvement suggests that Cosmic Vocalization is not only a subject of interest but also a matter of strategic importance. This engagement highlights the potential security implications of these cosmic phenomena [Data: Reports (6)].\n", + "**Alex Mercer's Viewpoint** \n", + "Alex Mercer perceives Cosmic Vocalization as part of an interstellar duet, suggesting that it may be a responsive or communicative event. This perspective highlights the potential for Cosmic Vocalization to be part of a larger cosmic interaction or dialogue [Data: Reports (6)].\n", "\n", - "2. **Community Interest**: Within the community, Cosmic Vocalization is a focal point of interest. Alex Mercer, for instance, perceives it as part of an interstellar duet, which suggests a responsive and perhaps communicative approach to these cosmic events [Data: Reports (6)].\n", + "**Taylor Cruz's Concerns** \n", + "Taylor Cruz raises concerns about the nature of Cosmic Vocalization, fearing it might be a homing tune. This adds a layer of urgency and potential threat, as it suggests that the vocalization could be attracting attention from unknown entities or forces [Data: Reports (6)].\n", "\n", - "3. **Potential Threats**: Concerns have been raised by individuals like Taylor Cruz, who fears that Cosmic Vocalization might be a homing tune. This perspective adds a layer of urgency and suggests that there may be potential threats associated with these cosmic sounds [Data: Reports (6)].\n", + "### Involvement of the Paranormal Military Squad\n", "\n", - "### Metaphorical Interpretation\n", + "The Paranormal Military Squad is actively engaged with Cosmic Vocalization, indicating its significance in security measures. Their involvement suggests that the phenomenon is not only of scientific interest but also of strategic importance, potentially impacting national or global security [Data: Reports (6)].\n", "\n", - "The Universe is metaphorically treated as a concert hall by the Paranormal Military Squad, which suggests a broader perspective on how cosmic events are interpreted and responded to by human entities. This metaphorical view may influence how strategies and responses are formulated in relation to Cosmic Vocalization [Data: Reports (6)].\n", + "### Conclusion\n", "\n", - "In summary, Cosmic Vocalization is a complex phenomenon involving strategic, communicative, and potentially threatening elements. The involvement of the Paranormal Military Squad and the concerns raised by community members underscore its significance and the need for careful consideration of its implications.\n" + "Cosmic Vocalization is a complex and multifaceted phenomenon that involves various stakeholders, each with their own perspectives and concerns. The involvement of both individuals like Alex Mercer and Taylor Cruz, as well as organized groups like the Paranormal Military Squad, underscores its importance and the need for further investigation and understanding.\n" ] } ], @@ -638,7 +640,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "LLM calls: 2. Prompt tokens: 11292. Output tokens: 606.\n" + "LLM calls: 2. Prompt tokens: 11237. Output tokens: 483.\n" ] } ], @@ -652,7 +654,7 @@ ], "metadata": { "kernelspec": { - "display_name": "graphrag", + "display_name": ".venv", "language": "python", "name": "python3" }, @@ -666,7 +668,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.12.5" + "version": "3.11.9" } }, "nbformat": 4, diff --git a/docs/examples_notebooks/local_search.ipynb b/docs/examples_notebooks/local_search.ipynb index 8f9afb3504..8d0e852772 100644 --- a/docs/examples_notebooks/local_search.ipynb +++ b/docs/examples_notebooks/local_search.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "code", - "execution_count": null, + "execution_count": 1, "metadata": {}, "outputs": [], "source": [ @@ -12,7 +12,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 2, "metadata": {}, "outputs": [], "source": [ @@ -70,7 +70,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 3, "metadata": {}, "outputs": [], "source": [ @@ -95,9 +95,185 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 4, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Entity count: 178\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
leveltitletypedescriptionsource_idcommunitydegreehuman_readable_ididsizegraph_embeddingtop_level_node_idxy
00ALEX MERCERPERSONAlex Mercer is a commanding and strategic figu...06bdff339c02ab35c80fa49320d5da66,1f2c6c263f168...3570b45241d70f0e43fca764df95b2b81f7757Noneb45241d70f0e43fca764df95b2b81f7700
10TAYLOR CRUZPERSONTaylor Cruz is a central and commanding figure...06bdff339c02ab35c80fa49320d5da66,1f2c6c263f168...35214119fd06010c494caa07f439b333f4c552None4119fd06010c494caa07f439b333f4c500
20JORDAN HAYESPERSONDr. Jordan Hayes is a central figure at Dulce ...06bdff339c02ab35c80fa49320d5da66,1f2c6c263f168...3482d3835bf3dda84ead99deadbeac5d0d7d48Noned3835bf3dda84ead99deadbeac5d0d7d00
30SAM RIVERAPERSONSam Rivera is a key operator and technologist ...06bdff339c02ab35c80fa49320d5da66,1b603cdb97651...3483077d2820ae1845bcbb1803379a3d1eae48None077d2820ae1845bcbb1803379a3d1eae00
40PARANORMAL MILITARY SQUADORGANIZATIONThe Paranormal Military Squad is a specialized...06bdff339c02ab35c80fa49320d5da66,2db9206de77cf...34843671ea0dd4e84c1a9b02c5ab2c8f4bac48None3671ea0dd4e84c1a9b02c5ab2c8f4bac00
\n", + "
" + ], + "text/plain": [ + " level title type \\\n", + "0 0 ALEX MERCER PERSON \n", + "1 0 TAYLOR CRUZ PERSON \n", + "2 0 JORDAN HAYES PERSON \n", + "3 0 SAM RIVERA PERSON \n", + "4 0 PARANORMAL MILITARY SQUAD ORGANIZATION \n", + "\n", + " description \\\n", + "0 Alex Mercer is a commanding and strategic figu... \n", + "1 Taylor Cruz is a central and commanding figure... \n", + "2 Dr. Jordan Hayes is a central figure at Dulce ... \n", + "3 Sam Rivera is a key operator and technologist ... \n", + "4 The Paranormal Military Squad is a specialized... \n", + "\n", + " source_id community degree \\\n", + "0 06bdff339c02ab35c80fa49320d5da66,1f2c6c263f168... 3 57 \n", + "1 06bdff339c02ab35c80fa49320d5da66,1f2c6c263f168... 3 52 \n", + "2 06bdff339c02ab35c80fa49320d5da66,1f2c6c263f168... 3 48 \n", + "3 06bdff339c02ab35c80fa49320d5da66,1b603cdb97651... 3 48 \n", + "4 06bdff339c02ab35c80fa49320d5da66,2db9206de77cf... 3 48 \n", + "\n", + " human_readable_id id size graph_embedding \\\n", + "0 0 b45241d70f0e43fca764df95b2b81f77 57 None \n", + "1 1 4119fd06010c494caa07f439b333f4c5 52 None \n", + "2 2 d3835bf3dda84ead99deadbeac5d0d7d 48 None \n", + "3 3 077d2820ae1845bcbb1803379a3d1eae 48 None \n", + "4 4 3671ea0dd4e84c1a9b02c5ab2c8f4bac 48 None \n", + "\n", + " top_level_node_id x y \n", + "0 b45241d70f0e43fca764df95b2b81f77 0 0 \n", + "1 4119fd06010c494caa07f439b333f4c5 0 0 \n", + "2 d3835bf3dda84ead99deadbeac5d0d7d 0 0 \n", + "3 077d2820ae1845bcbb1803379a3d1eae 0 0 \n", + "4 3671ea0dd4e84c1a9b02c5ab2c8f4bac 0 0 " + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "# read nodes table to get community and degree data\n", "entity_df = pd.read_parquet(f\"{INPUT_DIR}/{ENTITY_TABLE}.parquet\")\n", @@ -128,9 +304,161 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 5, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Relationship count: 373\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
sourcetargetweightdescriptiontext_unit_idsidhuman_readable_idsource_degreetarget_degreerank
0ALEX MERCERTAYLOR CRUZ21.0Alex Mercer and Taylor Cruz are integral membe...[06bdff339c02ab35c80fa49320d5da66, 1f2c6c263f1...148fffeb994541b2b4b6dcefda7001a805752109
1ALEX MERCERJORDAN HAYES25.0Alex Mercer and Jordan Hayes are integral team...[06bdff339c02ab35c80fa49320d5da66, 1f2c6c263f1...89c08e793298442686292454a1abff3115748105
2ALEX MERCERSAM RIVERA20.0Alex Mercer and Sam Rivera are integral member...[06bdff339c02ab35c80fa49320d5da66, 1f2c6c263f1...0467928aa65e4a4fba62bdb1467e3a5425748105
3ALEX MERCEROPERATION: DULCE BRIEFING ROOM1.0Alex Mercer was present in the Operation: Dulc...[06bdff339c02ab35c80fa49320d5da66]43c3390303c6476cb65f584e37c3e81c357461
4ALEX MERCERPARANORMAL MILITARY SQUAD20.0Alex Mercer is a prominent and influential lea...[06bdff339c02ab35c80fa49320d5da66, 2db9206de77...fa14b16c17e3417dba5a4b473ea5b18d45748105
\n", + "
" + ], + "text/plain": [ + " source target weight \\\n", + "0 ALEX MERCER TAYLOR CRUZ 21.0 \n", + "1 ALEX MERCER JORDAN HAYES 25.0 \n", + "2 ALEX MERCER SAM RIVERA 20.0 \n", + "3 ALEX MERCER OPERATION: DULCE BRIEFING ROOM 1.0 \n", + "4 ALEX MERCER PARANORMAL MILITARY SQUAD 20.0 \n", + "\n", + " description \\\n", + "0 Alex Mercer and Taylor Cruz are integral membe... \n", + "1 Alex Mercer and Jordan Hayes are integral team... \n", + "2 Alex Mercer and Sam Rivera are integral member... \n", + "3 Alex Mercer was present in the Operation: Dulc... \n", + "4 Alex Mercer is a prominent and influential lea... \n", + "\n", + " text_unit_ids \\\n", + "0 [06bdff339c02ab35c80fa49320d5da66, 1f2c6c263f1... \n", + "1 [06bdff339c02ab35c80fa49320d5da66, 1f2c6c263f1... \n", + "2 [06bdff339c02ab35c80fa49320d5da66, 1f2c6c263f1... \n", + "3 [06bdff339c02ab35c80fa49320d5da66] \n", + "4 [06bdff339c02ab35c80fa49320d5da66, 2db9206de77... \n", + "\n", + " id human_readable_id source_degree \\\n", + "0 148fffeb994541b2b4b6dcefda7001a8 0 57 \n", + "1 89c08e793298442686292454a1abff31 1 57 \n", + "2 0467928aa65e4a4fba62bdb1467e3a54 2 57 \n", + "3 43c3390303c6476cb65f584e37c3e81c 3 57 \n", + "4 fa14b16c17e3417dba5a4b473ea5b18d 4 57 \n", + "\n", + " target_degree rank \n", + "0 52 109 \n", + "1 48 105 \n", + "2 48 105 \n", + "3 4 61 \n", + "4 48 105 " + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "relationship_df = pd.read_parquet(f\"{INPUT_DIR}/{RELATIONSHIP_TABLE}.parquet\")\n", "relationships = read_indexer_relationships(relationship_df)\n", @@ -141,9 +469,17 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 6, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Claim records: 156\n" + ] + } + ], "source": [ "# NOTE: covariates are turned off by default, because they generally need prompt tuning to be valuable\n", "# Please see the GRAPHRAG_CLAIM_* settings\n", @@ -164,9 +500,175 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 7, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Report records: 20\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
communityfull_contentlevelranktitlerank_explanationsummaryfindingsfull_content_jsonid
010# Paranormal Military Squad at Dulce Base: Dec...18.5Paranormal Military Squad at Dulce Base: Decod...The impact severity rating is high due to the ...The Paranormal Military Squad, stationed at Du...[{'explanation': 'Jordan is a central figure i...{\\n \"title\": \"Paranormal Military Squad at ...1ba2d200-dd26-4693-affe-a5539d0a0e0d
111# Dulce and Paranormal Military Squad Operatio...18.5Dulce and Paranormal Military Squad OperationsThe impact severity rating is high due to the ...The community centers around Dulce, a secretiv...[{'explanation': 'Dulce is described as a top-...{\\n \"title\": \"Dulce and Paranormal Military...a8a530b0-ae6b-44ea-b11c-9f70d138298d
212# Paranormal Military Squad and Dulce Base Ope...17.5Paranormal Military Squad and Dulce Base Opera...The impact severity rating is relatively high ...The community centers around the Paranormal Mi...[{'explanation': 'Taylor is a central figure w...{\\n \"title\": \"Paranormal Military Squad and...0478975b-c805-4cc1-b746-82f3e689e2f3
313# Mission Dynamics and Leadership: Cruz and Wa...17.5Mission Dynamics and Leadership: Cruz and Wash...The impact severity rating is relatively high ...This report explores the intricate dynamics of...[{'explanation': 'Cruz is a central figure in ...{\\n \"title\": \"Mission Dynamics and Leadersh...b56f6e68-3951-4f07-8760-63700944a375
414# Dulce Base and Paranormal Military Squad: Br...18.5Dulce Base and Paranormal Military Squad: Brid...The impact severity rating is high due to the ...The community centers around the Dulce Base, a...[{'explanation': 'Sam Rivera, a member of the ...{\\n \"title\": \"Dulce Base and Paranormal Mil...736e7006-d050-4abb-a122-00febf3f540f
\n", + "
" + ], + "text/plain": [ + " community full_content level rank \\\n", + "0 10 # Paranormal Military Squad at Dulce Base: Dec... 1 8.5 \n", + "1 11 # Dulce and Paranormal Military Squad Operatio... 1 8.5 \n", + "2 12 # Paranormal Military Squad and Dulce Base Ope... 1 7.5 \n", + "3 13 # Mission Dynamics and Leadership: Cruz and Wa... 1 7.5 \n", + "4 14 # Dulce Base and Paranormal Military Squad: Br... 1 8.5 \n", + "\n", + " title \\\n", + "0 Paranormal Military Squad at Dulce Base: Decod... \n", + "1 Dulce and Paranormal Military Squad Operations \n", + "2 Paranormal Military Squad and Dulce Base Opera... \n", + "3 Mission Dynamics and Leadership: Cruz and Wash... \n", + "4 Dulce Base and Paranormal Military Squad: Brid... \n", + "\n", + " rank_explanation \\\n", + "0 The impact severity rating is high due to the ... \n", + "1 The impact severity rating is high due to the ... \n", + "2 The impact severity rating is relatively high ... \n", + "3 The impact severity rating is relatively high ... \n", + "4 The impact severity rating is high due to the ... \n", + "\n", + " summary \\\n", + "0 The Paranormal Military Squad, stationed at Du... \n", + "1 The community centers around Dulce, a secretiv... \n", + "2 The community centers around the Paranormal Mi... \n", + "3 This report explores the intricate dynamics of... \n", + "4 The community centers around the Dulce Base, a... \n", + "\n", + " findings \\\n", + "0 [{'explanation': 'Jordan is a central figure i... \n", + "1 [{'explanation': 'Dulce is described as a top-... \n", + "2 [{'explanation': 'Taylor is a central figure w... \n", + "3 [{'explanation': 'Cruz is a central figure in ... \n", + "4 [{'explanation': 'Sam Rivera, a member of the ... \n", + "\n", + " full_content_json \\\n", + "0 {\\n \"title\": \"Paranormal Military Squad at ... \n", + "1 {\\n \"title\": \"Dulce and Paranormal Military... \n", + "2 {\\n \"title\": \"Paranormal Military Squad and... \n", + "3 {\\n \"title\": \"Mission Dynamics and Leadersh... \n", + "4 {\\n \"title\": \"Dulce Base and Paranormal Mil... \n", + "\n", + " id \n", + "0 1ba2d200-dd26-4693-affe-a5539d0a0e0d \n", + "1 a8a530b0-ae6b-44ea-b11c-9f70d138298d \n", + "2 0478975b-c805-4cc1-b746-82f3e689e2f3 \n", + "3 b56f6e68-3951-4f07-8760-63700944a375 \n", + "4 736e7006-d050-4abb-a122-00febf3f540f " + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "report_df = pd.read_parquet(f\"{INPUT_DIR}/{COMMUNITY_REPORT_TABLE}.parquet\")\n", "reports = read_indexer_reports(report_df, entity_df, COMMUNITY_LEVEL)\n", @@ -184,9 +686,150 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 8, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Text unit records: 50\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
idtextn_tokensdocument_idsentity_idsrelationship_idscovariate_ids
006bdff339c02ab35c80fa49320d5da66# Operation: Dulce\\n\\n## Chapter 1\\n\\nThe thru...600[958fdd043f17ade63cb13570b59df295][b45241d70f0e43fca764df95b2b81f77, 4119fd06010...[148fffeb994541b2b4b6dcefda7001a8, 89c08e79329...[439081a3-bfeb-4693-968a-0f6189d8fa50, 3399e3d...
128dc4aa41a3e99deb7f354682f3e9904's authoritarian performance. _Protocols_, Jor...600[958fdd043f17ade63cb13570b59df295][4119fd06010c494caa07f439b333f4c5, 077d2820ae1...[9a6f414210e14841a5b0e661aedc898d, db541b72609...[b09594cb-d4b7-4de4-a1af-97778300eb1b, f7c5ea4...
2813db3138ef511c34be86f841f68aa8f, rehearsing the speech for their subordinates...600[958fdd043f17ade63cb13570b59df295][b45241d70f0e43fca764df95b2b81f77, 077d2820ae1...[0467928aa65e4a4fba62bdb1467e3a54, 7cc3356d38d...[7642fe21-7cb9-4428-848b-d2e3f5ab10ca, 6fe3d6c...
3d0afd106abf3a2966ff88a99eef710db, weighing his words carefully. \"Our tech is t...600[958fdd043f17ade63cb13570b59df295][b45241d70f0e43fca764df95b2b81f77, 4119fd06010...[148fffeb994541b2b4b6dcefda7001a8, 89c08e79329...[956e4c22-e343-4b5d-ad3c-d44ca3ce5fb5]
4f4c7c95d7bf621c6eb73d331cbf0b608stepping into their exchange. The authority i...600[958fdd043f17ade63cb13570b59df295][077d2820ae1845bcbb1803379a3d1eae, 1fd3fa8bb5a...[478e4c72d8fb46dd8cc9f0691c9878fd, 82b0446e7c9...[a54eda4d-b5bf-471d-989f-370eb9aff961, 9f85274...
\n", + "
" + ], + "text/plain": [ + " id \\\n", + "0 06bdff339c02ab35c80fa49320d5da66 \n", + "1 28dc4aa41a3e99deb7f354682f3e9904 \n", + "2 813db3138ef511c34be86f841f68aa8f \n", + "3 d0afd106abf3a2966ff88a99eef710db \n", + "4 f4c7c95d7bf621c6eb73d331cbf0b608 \n", + "\n", + " text n_tokens \\\n", + "0 # Operation: Dulce\\n\\n## Chapter 1\\n\\nThe thru... 600 \n", + "1 's authoritarian performance. _Protocols_, Jor... 600 \n", + "2 , rehearsing the speech for their subordinates... 600 \n", + "3 , weighing his words carefully. \"Our tech is t... 600 \n", + "4 stepping into their exchange. The authority i... 600 \n", + "\n", + " document_ids \\\n", + "0 [958fdd043f17ade63cb13570b59df295] \n", + "1 [958fdd043f17ade63cb13570b59df295] \n", + "2 [958fdd043f17ade63cb13570b59df295] \n", + "3 [958fdd043f17ade63cb13570b59df295] \n", + "4 [958fdd043f17ade63cb13570b59df295] \n", + "\n", + " entity_ids \\\n", + "0 [b45241d70f0e43fca764df95b2b81f77, 4119fd06010... \n", + "1 [4119fd06010c494caa07f439b333f4c5, 077d2820ae1... \n", + "2 [b45241d70f0e43fca764df95b2b81f77, 077d2820ae1... \n", + "3 [b45241d70f0e43fca764df95b2b81f77, 4119fd06010... \n", + "4 [077d2820ae1845bcbb1803379a3d1eae, 1fd3fa8bb5a... \n", + "\n", + " relationship_ids \\\n", + "0 [148fffeb994541b2b4b6dcefda7001a8, 89c08e79329... \n", + "1 [9a6f414210e14841a5b0e661aedc898d, db541b72609... \n", + "2 [0467928aa65e4a4fba62bdb1467e3a54, 7cc3356d38d... \n", + "3 [148fffeb994541b2b4b6dcefda7001a8, 89c08e79329... \n", + "4 [478e4c72d8fb46dd8cc9f0691c9878fd, 82b0446e7c9... \n", + "\n", + " covariate_ids \n", + "0 [439081a3-bfeb-4693-968a-0f6189d8fa50, 3399e3d... \n", + "1 [b09594cb-d4b7-4de4-a1af-97778300eb1b, f7c5ea4... \n", + "2 [7642fe21-7cb9-4428-848b-d2e3f5ab10ca, 6fe3d6c... \n", + "3 [956e4c22-e343-4b5d-ad3c-d44ca3ce5fb5] \n", + "4 [a54eda4d-b5bf-471d-989f-370eb9aff961, 9f85274... " + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "text_unit_df = pd.read_parquet(f\"{INPUT_DIR}/{TEXT_UNIT_TABLE}.parquet\")\n", "text_units = read_indexer_text_units(text_unit_df)\n", @@ -197,7 +840,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 9, "metadata": {}, "outputs": [], "source": [ @@ -233,7 +876,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 10, "metadata": {}, "outputs": [], "source": [ @@ -260,7 +903,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 11, "metadata": {}, "outputs": [], "source": [ @@ -303,7 +946,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 12, "metadata": {}, "outputs": [], "source": [ @@ -326,9 +969,33 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 13, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "### Overview of Agent Mercer\n", + "\n", + "Agent Alex Mercer is a prominent figure within the Paranormal Military Squad, playing a crucial role in Operation: Dulce at the Dulce Base. He is recognized for his commanding and strategic presence, particularly in overseeing operations related to communication with extraterrestrial intelligence. His responsibilities include initiating broadcasts to communicate with extraterrestrial beings, decoding alien messages, and leading the team in understanding and responding to these communications [Data: Entities (0)].\n", + "\n", + "### Role and Responsibilities\n", + "\n", + "Mercer is deeply involved in the philosophical and strategic aspects of interstellar communication, viewing these interactions as a form of cosmic dialogue. His leadership style is characterized by a blend of determination, compliance with mission protocols, and a protective approach towards his team. He collaborates with team members like Jordan Hayes, exploring secured areas and engaging in high-stakes, secretive operations. Mercer emphasizes the importance of intuition and trust beyond protocol, particularly in his mentorship of Sam Rivera [Data: Entities (0)].\n", + "\n", + "### Relationships and Influence\n", + "\n", + "Agent Mercer is known for his intellectual curiosity and deep involvement in the mission's strategic aspects. He is depicted as a thoughtful mentor, particularly to Sam Rivera, and his experiences during encounters with alien signals have led to profound changes, reinforcing his role as a key decision-maker and guardian in missions that transcend traditional boundaries [Data: Entities (0); Relationships (167)].\n", + "\n", + "### Involvement in Operation: Dulce\n", + "\n", + "Operation: Dulce was a significant mission undertaken by the Paranormal Military Squad, with Mercer being a key participant. The operation involved exploring paranormal anomalies and assessing potential threats, marking the beginning of an interstellar odyssey and leading to the first contact with an alien race. Mercer's involvement in this operation highlights his critical role in expanding the boundaries of human understanding of the universe [Data: Entities (5); Relationships (245)].\n", + "\n", + "In summary, Agent Alex Mercer is a pivotal figure in the Paranormal Military Squad, known for his strategic leadership and deep involvement in extraterrestrial communication efforts. His role in Operation: Dulce underscores his importance in the mission to uncover the secrets of the Dulce Base and engage with alien intelligence.\n" + ] + } + ], "source": [ "result = await search_engine.asearch(\"Tell me about Agent Mercer\")\n", "print(result.response)" @@ -336,9 +1003,33 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 14, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "## Overview of Dr. Jordan Hayes\n", + "\n", + "Dr. Jordan Hayes is a prominent scientist at Dulce Base, playing a crucial role in the Paranormal Military Squad's efforts to understand and communicate with extraterrestrial entities. Dr. Hayes's work is primarily focused on decoding and analyzing alien signals and codes, which are essential for interstellar communication and potentially interspecies interaction. This involves decrypting algorithms, interpreting cosmic signals, and analyzing the implications of alien society [Data: Entities (2, 17, 47); Relationships (201, 322, 283)].\n", + "\n", + "## Key Contributions and Expertise\n", + "\n", + "Dr. Hayes is noted for their adaptability and skepticism, qualities that are essential given the uncertainties and unknown challenges of their mission. They are deeply involved in the scientific exploration aspects of Operation: Dulce, where their efforts are on the verge of a significant scientific breakthrough. Dr. Hayes leads efforts in isolating and understanding complex alien signals that resemble human cognition, suggesting that these signals are artificial and patterned, indicating a tandem evolution with humanity [Data: Entities (2, 17); Claims (60, 83, 91, 134)].\n", + "\n", + "## Collaborative Efforts and Leadership\n", + "\n", + "Dr. Hayes works closely with colleagues like Alex Mercer and Taylor Cruz, engaging in thoughtful dialogue and showing analytical thinking about the mission's uncertainties. Despite some tension with Taylor Cruz due to differing leadership styles, Dr. Hayes continues to collaborate effectively with the team, contributing to the strategic discussions and decisions made in the briefing room. Their role in the command center at Dulce Base involves setting up lab stations, operating the mainframe, and playing a crucial role in the command center [Data: Entities (2, 17); Relationships (26, 82, 175, 322); Claims (2, 13)].\n", + "\n", + "## Scientific Breakthroughs and Challenges\n", + "\n", + "Dr. Hayes's work suggests that the alien signals they are studying are not just random but are structured and intentional, potentially indicating a form of extraterrestrial communication. This has led to the identification of new inviting signal patterns suggesting an intelligent confluence. Dr. Hayes's efforts are crucial in crafting humanity's responses to cosmic alignments with stars and responsive galactic signals, navigating the dark corridors of Dulce with a focus on the unknown variables and challenges beyond established protocols [Data: Entities (2, 17); Claims (60, 83, 91, 134, 153)].\n", + "\n", + "In summary, Dr. Jordan Hayes is a central figure in the efforts to understand and communicate with extraterrestrial entities at Dulce Base. Their work is characterized by a blend of scientific rigor, adaptability, and collaboration, making them a vital asset to the Paranormal Military Squad's mission.\n" + ] + } + ], "source": [ "question = \"Tell me about Dr. Jordan Hayes\"\n", "result = await search_engine.asearch(question)\n", @@ -354,18 +1045,218 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 15, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
identitydescriptionnumber of relationshipsin_context
017DR. JORDAN HAYESDr. Jordan Hayes is a key scientist at Dulce B...18True
17JORDANJordan Hayes is a key member of the Paranormal...22True
22JORDAN HAYESDr. Jordan Hayes is a central figure at Dulce ...48True
347DR. HAYESDr. Hayes is a scientist working on understand...1True
410AGENT HAYESAgent Hayes is known for emphasizing empowerme...4True
\n", + "
" + ], + "text/plain": [ + " id entity description \\\n", + "0 17 DR. JORDAN HAYES Dr. Jordan Hayes is a key scientist at Dulce B... \n", + "1 7 JORDAN Jordan Hayes is a key member of the Paranormal... \n", + "2 2 JORDAN HAYES Dr. Jordan Hayes is a central figure at Dulce ... \n", + "3 47 DR. HAYES Dr. Hayes is a scientist working on understand... \n", + "4 10 AGENT HAYES Agent Hayes is known for emphasizing empowerme... \n", + "\n", + " number of relationships in_context \n", + "0 18 True \n", + "1 22 True \n", + "2 48 True \n", + "3 1 True \n", + "4 4 True " + ] + }, + "execution_count": 15, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "result.context_data[\"entities\"].head()" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 16, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
idsourcetargetdescriptionweightlinksin_context
0260JORDANDR. JORDAN HAYESDr. Jordan Hayes and Jordan refer to the same ...1.01True
150ALEX MERCERJORDANAlex Mercer values Jordan's analytical approac...1.02True
226ALEX MERCERDR. JORDAN HAYESAlex Mercer and Dr. Jordan Hayes are collabora...5.02True
377TAYLOR CRUZJORDANJordan and Taylor Cruz are team members workin...4.02True
482TAYLOR CRUZDR. JORDAN HAYESDr. Jordan Hayes and Taylor Cruz are colleague...4.02True
\n", + "
" + ], + "text/plain": [ + " id source target \\\n", + "0 260 JORDAN DR. JORDAN HAYES \n", + "1 50 ALEX MERCER JORDAN \n", + "2 26 ALEX MERCER DR. JORDAN HAYES \n", + "3 77 TAYLOR CRUZ JORDAN \n", + "4 82 TAYLOR CRUZ DR. JORDAN HAYES \n", + "\n", + " description weight links in_context \n", + "0 Dr. Jordan Hayes and Jordan refer to the same ... 1.0 1 True \n", + "1 Alex Mercer values Jordan's analytical approac... 1.0 2 True \n", + "2 Alex Mercer and Dr. Jordan Hayes are collabora... 5.0 2 True \n", + "3 Jordan and Taylor Cruz are team members workin... 4.0 2 True \n", + "4 Dr. Jordan Hayes and Taylor Cruz are colleague... 4.0 2 True " + ] + }, + "execution_count": 16, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "result.context_data[\"relationships\"].head()" ] @@ -374,9 +1265,22 @@ "cell_type": "code", "execution_count": null, "metadata": {}, - "outputs": [], + "outputs": [ + { + "ename": "KeyError", + "evalue": "'reports'", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mKeyError\u001b[0m Traceback (most recent call last)", + "Cell \u001b[0;32mIn[17], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43mresult\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcontext_data\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mreports\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m]\u001b[49m\u001b[38;5;241m.\u001b[39mhead()\n", + "\u001b[0;31mKeyError\u001b[0m: 'reports'" + ] + } + ], "source": [ - "result.context_data[\"reports\"].head()" + "if \"reports\" in result.context_data:\n", + " result.context_data[\"reports\"].head()" ] }, { @@ -446,7 +1350,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3", + "display_name": ".venv", "language": "python", "name": "python3" }, @@ -460,7 +1364,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.10.12" + "version": "3.11.9" } }, "nbformat": 4, diff --git a/docs/get_started.md b/docs/get_started.md index 4eb2f2887f..ef6d144078 100644 --- a/docs/get_started.md +++ b/docs/get_started.md @@ -14,11 +14,6 @@ To get started with the GraphRAG system, you have a few options: To get started with the GraphRAG system we recommend trying the [Solution Accelerator](https://github.com/Azure-Samples/graphrag-accelerator) package. This provides a user-friendly end-to-end experience with Azure resources. -# Top-Level Modules - -* [Indexing Pipeline Overview](index/overview.md) -* [Query Engine Overview](query/overview.md) - # Overview The following is a simple end-to-end example for using the GraphRAG system. @@ -34,26 +29,20 @@ The graphrag library includes a CLI for a no-code approach to getting started. P # Running the Indexer -Now we need to set up a data project and some initial configuration. Let's set that up. We're using the [default configuration mode](config/overview.md), which you can customize as needed using a [config file](config/yaml.md), which we recommend, or [environment variables](config/env_vars.md). - -First let's get a sample dataset ready: +We need to set up a data project and some initial configuration. First let's get a sample dataset ready: ```sh mkdir -p ./ragtest/input ``` -Now let's get a copy of A Christmas Carol by Charles Dickens from a trusted source +Get a copy of A Christmas Carol by Charles Dickens from a trusted source: ```sh curl https://www.gutenberg.org/cache/epub/24022/pg24022.txt -o ./ragtest/input/book.txt ``` -Next we'll inject some required config variables: - ## Set Up Your Workspace Variables -First let's make sure to setup the required environment variables. For details on these environment variables, and what environment variables are available, see the [variables documentation](config/overview.md). - To initialize your workspace, first run the `graphrag init` command. Since we have already configured a directory named `./ragtest` in the previous step, run the following command: diff --git a/graphrag/query/question_gen/local_gen.py b/graphrag/query/question_gen/local_gen.py index 5372d4cf43..7ba7bb1f0a 100644 --- a/graphrag/query/question_gen/local_gen.py +++ b/graphrag/query/question_gen/local_gen.py @@ -5,12 +5,15 @@ import logging import time -from typing import Any +from typing import Any, cast import tiktoken from graphrag.prompts.query.question_gen_system_prompt import QUESTION_SYSTEM_PROMPT -from graphrag.query.context_builder.builders import LocalContextBuilder +from graphrag.query.context_builder.builders import ( + ContextBuilderResult, + LocalContextBuilder, +) from graphrag.query.context_builder.conversation_history import ( ConversationHistory, ) @@ -71,12 +74,17 @@ async def agenerate( if context_data is None: # generate context data based on the question history - context_data, context_records = self.context_builder.build_context( - query=question_text, - conversation_history=conversation_history, - **kwargs, - **self.context_builder_params, - ) # type: ignore + result = cast( + ContextBuilderResult, + self.context_builder.build_context( + query=question_text, + conversation_history=conversation_history, + **kwargs, + **self.context_builder_params, + ), + ) + context_data = cast(str, result.context_chunks) + context_records = result.context_records else: context_records = {"context_data": context_data} log.info("GENERATE QUESTION: %s. LAST QUESTION: %s", start_time, question_text) @@ -144,12 +152,17 @@ def generate( if context_data is None: # generate context data based on the question history - context_data, context_records = self.context_builder.build_context( - query=question_text, - conversation_history=conversation_history, - **kwargs, - **self.context_builder_params, - ) # type: ignore + result = cast( + ContextBuilderResult, + self.context_builder.build_context( + query=question_text, + conversation_history=conversation_history, + **kwargs, + **self.context_builder_params, + ), + ) + context_data = cast(str, result.context_chunks) + context_records = result.context_records else: context_records = {"context_data": context_data} log.info( diff --git a/v1-breaking-changes.md b/v1-breaking-changes.md index a2ec67a5c7..1905980cc2 100644 --- a/v1-breaking-changes.md +++ b/v1-breaking-changes.md @@ -1,4 +1,50 @@ -# Config Breaking Changes +# GraphRAG Data Model and Config Breaking Changes + +As we worked toward a cleaner codebase, data model, and configuration for the v1 release, we made a few changes that can break older indexes. During the development process we left shims in place to account for these changes, so that all old indexes will work up until v1.0. However, with the release of 1.0 we are removing these shims to allow the codebase to move forward without the legacy code elements. This should be a fairly painless process for most users: because we aggressively use a cache for LLM calls, re-running an index over the top of a previous one should be very low (or no) cost. Therefore, our standard migration recommendation is as follows: + +1. Rename or move your settings.yml file to back it up. +2. Re-run `graphrag init` to generate a new default settings.yml. +3. Open your old settings.yml and copy any critical settings that you changed. For most people this is likely only the LLM and embedding config. +4. Re-run `graphrag index`. This will re-execute the standard pipeline, using the cache for any LLM calls that it can. The output parquet tables will be in the latest format. + +Note that one of the new requirements is that we write embeddings to a vector store during indexing. By default, this uses a local lancedb instance. When you re-generate the default config, a block will be added to reflect this. If you need to write to Azure AI Search instead, we recommend updating these settings before you index, so you don't need to do a separate vector ingest. + +All of the breaking changes listed below are accounted for in the four steps above. + +## What if I don't have a cache available? + +If you no longer have your original GraphRAG cache, you can manually update your index. The most important aspect is ensuring that you have the required embeddings stored. If you already have your embeddings in a vector store, much of this can be avoided. + +Parquet changes: +- The `create_final_entities.name` field has been renamed to `create_final_entities.title` for consistency with the other tables. Use your parquet editor of choice to fix this. +- The `create_final_communities.id` field has been renamed to `create_final_communities.community` so that `id` can be repurposed for a UUID like the other tables. Use your parquet editor of choice to copy and rename this. You can copy it to leave the `id` field in place, or use a tool such as pandas to give each community a new UUID in the `id` field. (We join on the `community` field internally, so `id` can be effectively ignored). + +Embeddings changes: +- For Local Search, you need to have the entity.description embeddings in a vector store +- For DRIFT Search, you need the community.full_content embeddings in a vector store +- If you are only using Global search, you do not need any embeddings + +The easiest way to get both of those is to run the pipeline with all workflows skipped except for `generate_embeddings`, which will embed those fields and write them to a vector store directly. Using a newer config file that has the embeddings.vector_store block: + +- Set the `skip_workflows` value to [create_base_entity_graph, create_base_text_units, create_final_text_units, create_final_community_reports, create_final_nodes, create_final_relationships, create_final_documents, create_final_covariates, create_final_entities, create_final_communities] +- Re-run `graphrag index` + +What this does is run the pipeline, but skip over all of the usual artifact generation - the only workflow that is not skipped is the one that generates all default (or otherwise configured) embeddings. + +## Updated data model + +- We have streamlined the data model of the index in a few small ways to align tables more consistently and remove redundant content. Notably: + - Consistent use of `id` and `human_readable_id` across all tables; this also insures all int IDs are actually saved as ints and never strings + - Alignment of fields from `create_final_entities` (such as name -> title) with `create_final_nodes`, and removal of redundant content across these tables + - Rename of `document.raw_content` to `document.text` + - Rename of `entity.name` to `entity.title` + - Rename `rank` to `combined_degree` in `create_final_relationships` and removal of `source_degree` and `target_degree`fields + - Fixed community tables to use a proper UUID for the `id` field, and retain `community` and `human_readable_id` for the short IDs + - Removal of all embeddings columns from parquet files in favor of direct vector store writes + +### Migration + +- Run a new index, leveraging existing cache. ## New required Embeddings