Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AI/first-ord-extraction #234

Merged
merged 2 commits into from
Feb 4, 2024
Merged

AI/first-ord-extraction #234

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
999a6ea
extract last 15 mins of transcript
ayyubibrahimi Feb 1, 2024
5d24f06
compared various whisper models
ayyubibrahimi Feb 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion packages/backend/src/cache/faiss_index_in_depth_cj.dvc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
outs:
- md5: ccd643270cd24af47d96dfb198d468ae.dir
- md5: 9c2090ec6ee84ddf3792866364b35b0f.dir
size: 23116757
nfiles: 2
hash: md5
Expand Down
2 changes: 1 addition & 1 deletion packages/backend/src/cache/faiss_index_in_depth_fc.dvc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
outs:
- md5: a06190b8eee319f9083f642bb844d6a4.dir
- md5: ad7c688f8fff83d65f8728a403c864dd.dir
size: 63799934
nfiles: 2
hash: md5
Expand Down
4 changes: 2 additions & 2 deletions packages/backend/src/cache/faiss_index_in_depth_news.dvc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
outs:
- md5: 89036cfaca111f3d2fa33ed70003aa46.dir
size: 242119
- md5: d08e6dfb04e5afcff49028b2bffed7ad.dir
size: 526418
nfiles: 2
hash: md5
path: faiss_index_in_depth_news
4 changes: 2 additions & 2 deletions packages/backend/src/cache/faiss_index_in_depth_pc.dvc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
outs:
- md5: 7645c2b96dcaf4936ee31103ac0c0968.dir
size: 2028257
- md5: 75d23bd607ac91d67d994f624fb2e4c2.dir
size: 2097501
nfiles: 2
hash: md5
path: faiss_index_in_depth_pc
4 changes: 2 additions & 2 deletions packages/backend/src/cache/faiss_index_in_depth_pdf.dvc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
outs:
- md5: 47ce778b85c5553106c9f4da5f45ac28.dir
size: 54406163
- md5: 28bfb00c7d3ebfbb4de620e9915e75b3.dir
size: 53418928
nfiles: 2
hash: md5
path: faiss_index_in_depth_pdf
12 changes: 6 additions & 6 deletions packages/backend/src/preprocessor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

from langchain.chains import LLMChain, HypotheticalDocumentEmbedder
from langchain.prompts import PromptTemplate
from langchain.vectorstores.faiss import FAISS
from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAI
from pathlib import Path
import shutil
Expand Down Expand Up @@ -73,7 +73,7 @@ def create_db_from_minutes_and_agendas(doc_directory):

data = loader.load()
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=2000, chunk_overlap=1000
chunk_size=2000, chunk_overlap=100
)
docs = text_splitter.split_documents(data)
all_docs.extend(docs)
Expand Down Expand Up @@ -103,7 +103,7 @@ def create_db_from_news_transcripts(news_json_directory):

data = loader.load()
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=10000, chunk_overlap=5000
chunk_size=2000, chunk_overlap=100
)
docs = text_splitter.split_documents(data)
all_docs.extend(docs)
Expand Down Expand Up @@ -136,7 +136,7 @@ def create_db_from_cj_transcripts(cj_json_directory):

data = loader.load()
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=2000, chunk_overlap=1000
chunk_size=2000, chunk_overlap=100
)
docs = text_splitter.split_documents(data)

Expand Down Expand Up @@ -169,7 +169,7 @@ def create_db_from_fc_transcripts(fc_json_directory):

data = loader.load()
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=2000, chunk_overlap=1000
chunk_size=2000, chunk_overlap=100
)
docs = text_splitter.split_documents(data)
# Append the publish date to the end of page_content
Expand Down Expand Up @@ -199,7 +199,7 @@ def create_db_from_public_comments(pc_json_directory):

data = loader.load()
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=10000, chunk_overlap=5000
chunk_size=2000, chunk_overlap=100
)
docs = text_splitter.split_documents(data)
all_docs.extend(docs)
Expand Down
2 changes: 1 addition & 1 deletion packages/googlecloud/functions/getanswer/cache/faiss_index_in_depth_cj.dvc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
outs:
- md5: ccd643270cd24af47d96dfb198d468ae.dir
- md5: 9c2090ec6ee84ddf3792866364b35b0f.dir
size: 23116757
nfiles: 2
hash: md5
Expand Down
2 changes: 1 addition & 1 deletion packages/googlecloud/functions/getanswer/cache/faiss_index_in_depth_fc.dvc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
outs:
- md5: a06190b8eee319f9083f642bb844d6a4.dir
- md5: ad7c688f8fff83d65f8728a403c864dd.dir
size: 63799934
nfiles: 2
hash: md5
Expand Down
4 changes: 2 additions & 2 deletions packages/googlecloud/functions/getanswer/cache/faiss_index_in_depth_news.dvc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
outs:
- md5: 89036cfaca111f3d2fa33ed70003aa46.dir
size: 242119
- md5: d08e6dfb04e5afcff49028b2bffed7ad.dir
size: 526418
nfiles: 2
hash: md5
path: faiss_index_in_depth_news
4 changes: 2 additions & 2 deletions packages/googlecloud/functions/getanswer/cache/faiss_index_in_depth_pc.dvc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
outs:
- md5: 7645c2b96dcaf4936ee31103ac0c0968.dir
size: 2028257
- md5: 75d23bd607ac91d67d994f624fb2e4c2.dir
size: 2097501
nfiles: 2
hash: md5
path: faiss_index_in_depth_pc
4 changes: 2 additions & 2 deletions packages/googlecloud/functions/getanswer/cache/faiss_index_in_depth_pdf.dvc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
outs:
- md5: 47ce778b85c5553106c9f4da5f45ac28.dir
size: 54406163
- md5: 28bfb00c7d3ebfbb4de620e9915e75b3.dir
size: 53418928
nfiles: 2
hash: md5
path: faiss_index_in_depth_pdf
Loading