Code

Feature/background prune 3#2608

Code

Closed

rkuo-danswer wants to merge 126 commits intomainonyx-dot-app/onyx:mainfrom feature/background_prune_3onyx-dot-app/onyx:feature/background_prune_3Copy head branch name to clipboard

+685-408

Commits

Commits on Aug 23, 2024

• first cut at redis

  rkuo-danswer

  committed
  10330fb

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/redis

  rkuo-danswer

  committed
  7442957

• some new helper functions for the db

  rkuo-danswer

  committed
  09e95fe

• Merge from feature/redis

  rkuo-danswer

  committed
  7244bae

Commits on Aug 29, 2024

• ignore kombu tables in alembic migrations (used by celery)

  rkuo-danswer

  committed
  25f4b08

Commits on Aug 30, 2024

• multiline commands for readability, add vespa_metadata_sync queue to worker

  rkuo-danswer

  committed
  d4c06b3

• typo fix

  rkuo-danswer

  committed
  5100065

• fix returning tuple fields

  rkuo-danswer

  committed
  34a8bde

• add constants

  rkuo-danswer

  committed
  d856297

• fix _get_access_for_document

  rkuo-danswer

  committed
  93fb8ef

• docstrings!

  rkuo-danswer

  committed
  8d09dcf

• fix double function declaration and typing

  rkuo-danswer

  committed
  6d05e29

• fix type hinting

  rkuo-danswer

  committed
  024d4dd

• add a global redis pool

  rkuo-danswer

  committed
  186d4f1

• Add get_document function

  rkuo-danswer

  committed
  5f7ca15

• use task_logger in various celery tasks

  rkuo-danswer

  committed
  66fb6b1

• add celeryconfig.py to simplify configuration. Will be used in a subsequent commit

  rkuo-danswer

  committed
  f06efb9

• Add celery redis helper. used in a subsequent PR

  rkuo-danswer

  committed
  e36fd3c

• kombu warning getting spammy since celery is not self managing its queue in Postgres any more

  rkuo-danswer

  committed
  ab9cf9a

• add last_modified and last_synced to documents

  rkuo-danswer

  committed
  48963ed

• fix task naming convention

  rkuo-danswer

  committed
  b4b7801

• use celeryconfig.py

  rkuo-danswer

  committed
  1cfbfe1

• the big one. adds queues and tasks, updates functions to use the queues with priorities, etc

  rkuo-danswer

  committed
  2144617

• change vespa index log line to debug

  rkuo-danswer

  committed
  eea178f

• mypy fixes

  rkuo-danswer

  committed
  27eae3d

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/background_processing

  Show description for db252a6
  rkuo-danswer

  committed
  db252a6

• update alembic migration

  rkuo-danswer

  committed
  c174d00

Commits on Aug 31, 2024

• fix fence ordering, rename to "monitor", fix fetch_versioned_implementation call

  rkuo-danswer

  committed
  c0a7c7c

Commits on Sep 1, 2024

• mypy

  rkuo-danswer

  committed
  d17f49f

Commits on Sep 2, 2024

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/background_processing

  rkuo-danswer

  committed
  838a7fb

• switch to monotonic time

  rkuo-danswer

  committed
  0652698

• fix startup dependencies on redis

  LostVector

  committed
  9dbd29d

• Merge branch 'feature/redis' of https://github.com/danswer-ai/danswer

  LostVector

  committed
  575db62

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/background_processing

  LostVector

  committed
  015fab9

• rebase alembic migration

  LostVector

  committed
  bdcd553

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/redis

  LostVector

  committed
  e731a33

• kombu cleanup - fail silently

  LostVector

  committed
  c388a81

• Merge branch 'feature/redis' of https://github.com/danswer-ai/danswer into feature/background_processing

  Show description for a9aca68
  LostVector

  committed
  a9aca68

• mypy

  LostVector

  committed
  f0bdb39

• Merge branch 'feature/redis' of https://github.com/danswer-ai/danswer into feature/background_processing

  LostVector

  committed
  e53ff25

Commits on Sep 3, 2024

• add redis_host environment override

  LostVector

  committed
  076ed18

• update REDIS_HOST env var in docker-compose.dev.yml

  LostVector

  committed
  aa1eec8

• update the rest of the docker files

  LostVector

  committed
  f358acf

• in flight

  LostVector

  committed
  1d4b8cf

Commits on Sep 4, 2024

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/redis

  rkuo-danswer

  committed
  6724a13

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/redis

  rkuo-danswer

  committed
  69d762e

• Merge branch 'feature/redis' of https://github.com/danswer-ai/danswer into feature/background_processing

  Show description for a56e3d7
  rkuo-danswer

  committed
  a56e3d7

• harden indexing-status endpoint against db changes happening in the background. Needs further improvement but OK for now.

  rkuo-danswer

  committed
  d6d3d24

• allow no task syncs to run because we create certain objects with no entries but initially marked as out of date

  rkuo-danswer

  committed
  6335f5b

• add back writing to vespa on indexing

  rkuo-danswer

  committed
  4c5c4a2

• Merge branch 'feature/background_processing' of https://github.com/danswer-ai/danswer into feature/background_deletion

  rkuo-danswer

  committed
  957bd55

Commits on Sep 5, 2024

• actually working connector deletion

  rkuo-danswer

  committed
  7133450

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/redis

  rkuo-danswer

  committed
  590bd0f

• update contributing guide

  rkuo-danswer

  committed
  98ec0b7

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/redis

  rkuo-danswer

  committed
  ab097ce

• Merge branch 'feature/redis' of https://github.com/danswer-ai/danswer into feature/background_processing

  rkuo-danswer

  committed
  f363c6b

• backporting fixes from background_deletion

  rkuo-danswer

  committed
  daf46b3

• renaming cache to cache_volume

  rkuo-danswer

  committed
  dafbe13

• add redis password to various deployments

  rkuo-danswer

  committed
  bd07908

• try setting up pr testing for helm

  rkuo-danswer

  committed
  0bb2e57

• fix indent

  rkuo-danswer

  committed
  0b293dd

• hopefully this release version actually exists

  rkuo-danswer

  committed
  82bd7f4

• fix command line option to --chart-dirs

  rkuo-danswer

  committed
  0988947

• fetch-depth 0

  rkuo-danswer

  committed
  c20d921

• edit values.yaml

  rkuo-danswer

  committed
  d6c3bd3

• try setting ct working directory

  rkuo-danswer

  committed
  b8d9205

• bypass testing only on change for now

  rkuo-danswer

  committed
  f90cfa7

Commits on Sep 6, 2024

• move files and lint them

  rkuo-danswer

  committed
  af34d19

• update helm testing

  rkuo-danswer

  committed
  5d6fe92

• some issues suggest using --config works

  rkuo-danswer

  committed
  8cf4098

• add vespa repo

  rkuo-danswer

  committed
  bc5cb2d

• add postgresql repo

  rkuo-danswer

  committed
  ea35879

• increase timeout

  rkuo-danswer

  committed
  b427439

• try amd64 runner

  rkuo-danswer

  committed
  97d3671

• fix redis password reference

  rkuo-danswer

  committed
  3574c23

• add comment to helm chart testing workflow

  rkuo-danswer

  committed
  f02a231

• rename helm testing workflow to disable it

  rkuo-danswer

  committed
  3323d4c

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/redis

  rkuo-danswer

  committed
  3afa07a

• Merge branch 'feature/redis' of https://github.com/danswer-ai/danswer into feature/background_processing

  rkuo-danswer

  committed
  2af4926

• Merge branch 'feature/background_processing' of https://github.com/danswer-ai/danswer into feature/background_deletion

  Show description for 7512d68
  rkuo-danswer

  committed
  7512d68

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/background_processing

  Show description for 29860f8
  rkuo-danswer

  committed
  29860f8

• Merge branch 'feature/background_processing' of https://github.com/danswer-ai/danswer into feature/background_deletion

  rkuo-danswer

  committed
  f5cbb6d

• adding clarifying comments

  rkuo-danswer

  committed
  89619ce

Commits on Sep 7, 2024

• address code review

  rkuo-danswer

  committed
  69983ab

• missed a file

  rkuo-danswer

  committed
  138866a

• remove commented warning ... just not needed

  LostVector

  committed
  84ec5f4

Commits on Sep 9, 2024

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/background_processing

  rkuo-danswer

  committed
  ab77692

• Merge branch 'feature/background_processing' of https://github.com/danswer-ai/danswer into feature/background_deletion

  Show description for ca6858b
  rkuo-danswer

  committed
  ca6858b

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/background_processing

  Show description for fe9bb06
  rkuo-danswer

  committed
  fe9bb06

• Merge branch 'feature/background_processing' of https://github.com/danswer-ai/danswer into feature/background_deletion

  rkuo-danswer

  committed
  ba58e7d

• fix imports

  rkuo-danswer

  committed
  8aef77a

• refactor to use update_single

  rkuo-danswer

  committed
  30f6463

• mypy fixes

  rkuo-danswer

  committed
  092e190

Commits on Sep 10, 2024

• add vespa test

  rkuo-danswer

  committed
  be61371

• merge

  rkuo-danswer

  committed
  026bca4

Commits on Sep 13, 2024

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/background_deletion

  Show description for 5f85632
  rkuo-danswer

  committed
  5f85632

• multiple celery workers

  rkuo-danswer

  committed
  20ba19b

• update logs as well and set prefetch multipliers appropriate to the worker intent

  rkuo-danswer

  committed
  9b99c14

Commits on Sep 14, 2024

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/background_deletion

  Show description for b5528e3
  LostVector

  committed
  b5528e3

• add db refresh to connector deletion

  LostVector

  committed
  aca5204

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/background_deletion

  LostVector

  committed
  fc3af78

Commits on Sep 15, 2024

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/background_deletion

  LostVector

  committed
  a4d36b9

Commits on Sep 16, 2024

• add some preliminary locking

  rkuo-danswer

  committed
  9ac8a10

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/celery_multi

  rkuo-danswer

  committed
  76b0996

• organize tasks into separate files

  rkuo-danswer

  committed
  90cc753

Commits on Sep 17, 2024

• celery auto associates tasks created inside another task, which bloats the result metadata considerably. trail=False prevents this.

  rkuo-danswer

  committed
  faaccc7

• code review fixes

  LostVector

  committed
  c032861

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/background_deletion

  rkuo-danswer

  committed
  e2f3363

• move monitor_usergroup_taskset to ee, improve logging

  rkuo-danswer

  committed
  905f104

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/background_deletion

  rkuo-danswer

  committed
  6a8e4ef

Commits on Sep 18, 2024

• Merge branch 'feature/background_deletion' of https://github.com/danswer-ai/danswer into feature/celery_multi

  Show description for 166b405
  rkuo-danswer

  committed
  166b405

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/celery_multi

  Show description for 3a76733
  rkuo-danswer

  committed
  3a76733

• add multi workers to dev_run_background_jobs.py

  rkuo-danswer

  committed
  bffdf94

• update supervisord with some recommended settings for celery

  rkuo-danswer

  committed
  2f6b5f4

Commits on Sep 19, 2024

• name celery workers and shorten dev script prefixing

  rkuo-danswer

  committed
  bb4ee6b

• add configurable sql alchemy engine settings on startup (needed for various intents like API server, different celery workers and tasks, etc)

  rkuo-danswer

  committed
  4f7e5cb

• fix comments

  rkuo-danswer

  committed
  6ebc623

Commits on Sep 20, 2024

• merge main

  rkuo-danswer

  committed
  6433d47

• autoscale sqlalchemy pool size to celery concurrency (allow override later?)

  rkuo-danswer

  committed
  12219b2

• supervisord needs the percent symbols escaped

  rkuo-danswer

  committed
  e0829a9

• use name as primary check, some minor refactoring and type hinting too.

  rkuo-danswer

  committed
  84e3fbd

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/celery_multi

  rkuo-danswer

  committed
  236b1e0

Commits on Sep 24, 2024

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/celery_multi

  rkuo-danswer

  committed
  ece7d64

Commits on Sep 25, 2024

• stash merge (may not function yet)

  rkuo-danswer

  committed
  1a41cf8

Commits on Sep 27, 2024

• Merge branch 'main' of https://github.com/danswer-ai/danswer into feature/background_prune_2

  Show description for 427c86e
  rkuo-danswer

  committed
  427c86e

Commits on Sep 30, 2024

• cherry pick

  LostVector

  committed
  2c61f91