Code

feat: make it work for serp#73

Code

Open

JossWhittle wants to merge 152 commits intomainSwanseaUniversityMedical/DARE-Airflow:mainfrom feat/make-it-work-for-serpSwanseaUniversityMedical/DARE-Airflow:feat/make-it-work-for-serpCopy head branch name to clipboard

+2,201-905

Commits

Commits on Apr 10, 2024

• working version - Fast DuckDB

  tekkisse

  committed
  8bde32c

• bit of tidying up

  tekkisse

  committed
  2b4e4c8

• auto run DAG & UTF-8 testing

  tekkisse

  committed
  3fa3709

• check in

  tekkisse

  committed
  07c1263

• FIX: install minio to base image

  tekkisse

  committed
  5d0d76f

• multi part download

  tekkisse

  committed
  934d2ca

Commits on Apr 11, 2024

• working version

  tekkisse

  committed
  1e816b1

• use updated docker container

  tekkisse

  committed
  d65fe71

• tracking

  tekkisse

  committed
  24201f2

Commits on Apr 12, 2024

• parquete file size tracking

  tekkisse

  committed
  1eb3973

Commits on Apr 13, 2024

• MVP

  tekkisse

  committed
  5fdca92

Commits on Apr 16, 2024

• FEAT: add jinja to docker image

  tekkisse

  committed
  18eb439

• compute ledger working

  tekkisse

  committed
  b17010e

Commits on Apr 17, 2024

• tweaking

  tekkisse

  committed
  42896ad

• turn debug off

  tekkisse

  committed
  35ae96b

• track end iceberg table

  tekkisse

  committed
  5079d8d

Commits on Apr 21, 2024

• add notifitcation

  tekkisse

  committed
  19fda0d

Commits on Apr 23, 2024

• make build

  tekkisse

  committed
  08d8393

• .

  tekkisse

  committed
  d78c907

• made cooler

  tekkisse

  committed
  d0ae2a4

Commits on Apr 25, 2024

• sort out docker compose start order

  tekkisse

  committed
  e2e7d96

Commits on Apr 28, 2024

• regex single for list

  tekkisse

  committed
  8b5fb30

• master switch

  tekkisse

  committed
  c225b44

Commits on Apr 29, 2024

• working version hitting assetv3

  tekkisse

  committed
  08f68a2

Commits on Apr 30, 2024

• few error fixed

  tekkisse

  committed
  db63ec8

• fix

  tekkisse

  committed
  e5e3ec0

• added action

  tekkisse

  committed
  4f3ac35

• create, append, replace options

  tekkisse

  committed
  f36a61a

• MVP1

  tekkisse

  committed
  8f466bc

• use proxy

  tekkisse

  committed
  39a6b02

• bug fixes

  tekkisse

  committed
  9c49204

Commits on May 1, 2024

• bug fixing

  tekkisse

  committed
  fe30b55

• tidy up

  tekkisse

  committed
  b71fbde

Commits on May 2, 2024

• turn proxy server on

  tekkisse

  committed
  4f6af6a

• change default

  tekkisse

  committed
  5001c7c

• change logging

  tekkisse

  committed
  2cc7c8d

• .

  tekkisse

  committed
  246aedb

Commits on May 7, 2024

• refactor functions

  tekkisse

  committed
  e516ad0

Commits on May 8, 2024

• refactored into modules

  tekkisse

  committed
  c023310

• full refactor

  tekkisse

  committed
  e2a6de0

• separate DAG's

  tekkisse

  committed
  b01c54d

Commits on May 9, 2024

• extra logging

  tekkisse

  committed
  e314b8a

• ISSUE WITH SERIALISING DICT OBJECTS

  tekkisse

  committed
  44e9d4c

• tweaking

  tekkisse

  committed
  6185b0c

• add manual SED

  tekkisse

  committed
  8845877

Commits on May 10, 2024

• turn proxy on

  tekkisse

  committed
  5e4514c

• deployable version

  tekkisse

  committed
  4c7ff41

• a

  tekkisse

  committed
  ce997a4

• cope with daft hard coded values

  tekkisse

  committed
  5e29347

• dam need to get more setting into config files

  tekkisse

  committed
  9561e33

Commits on May 12, 2024

• bucket default configurations if not dataset configuration

  tekkisse

  committed
  d371af7

Commits on May 13, 2024

• allow for manual overide

  tekkisse

  committed
  cb689a4

Commits on May 28, 2024

• fix(container): use base container with our root ca baked in

  alee-x

  committed
  ba70254

• update gitignore

  alee-x

  committed
  3306be8

• fix(container): add chi root ca to trust store

  alee-x

  committed
  16f8b44

• small changes

  tekkisse

  committed
  c6fc7cb

• Merge branch 'feat/make-it-work-for-serp' of https://github.com/SwanseaUniversityMedical/DARE-Airflow into feat/make-it-work-for-serp

  tekkisse

  committed
  c8bac8c

Commits on May 29, 2024

• switch to usding redis settings from Airflow connection

  tekkisse

  committed
  812a204

• remove db number (redis) from connection

  tekkisse

  committed
  d48b618

Commits on Jun 3, 2024

• fix: when getting the redis connection, if auth is enabled then use the auth for the redis client

  alee-x

  committed
  29f55d8

Commits on Jun 9, 2024

• change tracking info

  tekkisse

  committed
  23a98ea

• correct logic

  tekkisse

  committed
  77efc18

• correct regex

  tekkisse

  committed
  2265d77

Commits on Jun 10, 2024

• increase power of SED

  tekkisse

  committed
  7fc8829

• SED validation

  tekkisse

  committed
  f8abb94

• add code to pull schema and max out

  tekkisse

  committed
  f15e6c9

• fix stupid error

  tekkisse

  committed
  9a58c83

• change way table refernced

  tekkisse

  committed
  6014996

• .

  tekkisse

  committed
  666c6be

• remove schema name from table name

  tekkisse

  committed
  f94b955

• trying to make this work

  tekkisse

  committed
  03523bc

• try again

  tekkisse

  committed
  baa9327

• correction

  tekkisse

  committed
  96365e3

• try

  tekkisse

  committed
  d882d6a

• .

  tekkisse

  committed
  b66fd5e

• deal with varchar

  tekkisse

  committed
  4eccdeb

• correct if statement

  tekkisse

  committed
  be3ec1e

• better solution

  tekkisse

  committed
  353a83f

• python error

  tekkisse

  committed
  ae253ea

• whoops - missed that

  tekkisse

  committed
  6021796

• move marker

  tekkisse

  committed
  771d101

• fix/turn debug off

  tekkisse

  committed
  967a8c3

Commits on Jun 11, 2024

• turn debug back on

  tekkisse

  committed
  f8110d1

• correct error

  tekkisse

  committed
  c4bbf66

• add time datatype into mapping

  tekkisse

  committed
  43cc96b

• try time stamp

  tekkisse

  committed
  6de1e4e

• leave file for debigging

  tekkisse

  committed
  947e828

• rename DAGS

  tekkisse

  committed
  3638818

• switch duckdb to disk instead of memory

  tekkisse

  committed
  8d7c27c

• tweaking

  tekkisse

  committed
  34c5703

• lower case for file extension

  tekkisse

  committed
  0c88e90

Commits on Jun 12, 2024

• feat(dags-container): upgrade duckdb to v1.0.0, upgrade base container to use the same as DLM-dags (#76)

  alee-x

  authored
  6b0c604

• Merge branch 'main' into feat/make-it-work-for-serp

  alee-x

  authored
  f640682

Commits on Jun 14, 2024

• fix: use airflow connection schema keys for the aws connection where possible

  alee-x

  committed
  d9a18a7

• fix: don't hardcode minio secure/ssl to false because then it doesn't work when we have a https minio

  alee-x

  committed
  7586eeb

• fix: correctly detect the connection protocol for minio and trino based on the connection details

  alee-x

  committed
  c108861

• fix: don't hardcode the trino connection as though there no bloody authentication on it!

  alee-x

  committed
  4c4873f

• fix: we can't have a module with the same name as a package we need to import, also use trino.auth BasicAuthentication

  alee-x

  committed
  8f8a8bb

Commits on Jun 15, 2024

• cope with different etag

  tekkisse

  committed
  beb411f

• Merge branch 'feat/make-it-work-for-serp' of https://github.com/SwanseaUniversityMedical/DARE-Airflow into feat/make-it-work-for-serp

  tekkisse

  committed
  33c54d4

Commits on Jun 26, 2024

• Update docker-compose.yml

  tekkisse

  committed
  f072adc

• increase max connections

  tekkisse

  committed
  0040bc8

• remove auth as does not work with http

  tekkisse

  committed
  2baf179

• options auth for trino

  tekkisse

  committed
  0de5efd

• correct code

  tekkisse

  committed
  745a1a8

• up max connections

  tekkisse

  committed
  4646f7f

• change port for mariadb

  tekkisse

  committed
  8fd0a8f

• switch ports

  tekkisse

  committed
  6761328

• .

  tekkisse

  committed
  f91a4e0

• .

  tekkisse

  committed
  a62c6cc

• switch hive server

  tekkisse

  committed
  51f8053

• add second postgres

  tekkisse

  committed
  8b2ce46

• .

  tekkisse

  committed
  aacedc2

Commits on Jul 5, 2024

• change portal tables

  tekkisse

  committed
  29e8979

• corrections

  tekkisse

  committed
  c26e6cc

• .

  tekkisse

  committed
  a0af32e

• new table for portal

  tekkisse

  committed
  f504676

Commits on Aug 2, 2024

• send schema request to rabbitmq

  tekkisse

  committed
  4abcc89

Commits on Aug 23, 2024

• use external DAGS

  tekkisse

  committed
  0bee53a

• correction

  tekkisse

  committed
  e2d6d8f

• bump airflow build up

  tekkisse

  committed
  551a65e

• real correction :-)

  tekkisse

  committed
  a0c7f29

Commits on Aug 27, 2024

• extra trino worker

  tekkisse

  committed
  b416746

• upgrade trino

  tekkisse

  committed
  8df9ad2

• change airflow image

  tekkisse

  committed
  8a12026

• extra max threads

  tekkisse

  committed
  676b535

Commits on Aug 28, 2024

• up container version

  tekkisse

  committed
  c8dbe31

Commits on Sep 2, 2024

• changing rabbit

  tekkisse

  committed
  51e4b0a

• use latest PR

  tekkisse

  committed
  cafde4b

• shell scripts

  tekkisse

  committed
  bec25db

• correct spelling

  tekkisse

  committed
  7be9e0a

• add portal details

  tekkisse

  committed
  0ba1348

• add restart script

  tekkisse

  committed
  130944c

• whoops - fix

  tekkisse

  committed
  3ad1bc0

• instructions

  tekkisse

  committed
  2222d2b

• add start firehose

  tekkisse

  committed
  faca597

• wire up firehose

  tekkisse

  committed
  ac9503c

Commits on Sep 3, 2024

• correct rabbitmq def

  tekkisse

  committed
  c7fa94b

• use correct DLM PR

  tekkisse

  committed
  5b447cc

• use local DAG's

  tekkisse

  committed
  d0a1e30

Commits on Sep 5, 2024

• a poke

  John-Vaughan

  committed
  c9e5a4b

Commits on Sep 25, 2024

• added request_schema_save

  John-Vaughan

  committed
  b7d623c

Commits on Oct 1, 2024

• update

  John-Vaughan

  committed
  156c6cb

Commits on Oct 8, 2024

• add postgres for nrdav2

  tekkisse

  committed
  c873662

• Merge branch 'feat/make-it-work-for-serp' of https://github.com/SwanseaUniversityMedical/DARE-Airflow into feat/make-it-work-for-serp

  tekkisse

  committed
  1a4f488

Commits on Oct 10, 2024

• fix

  John-Vaughan

  committed
  db38ef7

Commits on Oct 16, 2024

• added ledger

  John-Vaughan

  committed
  a3b01f0

Commits on Oct 21, 2024

• PG_MAX_CONNECTIONS?

  John-Vaughan

  committed
  50c59eb

• joss stuff

  John-Vaughan

  committed
  8171b49

Commits on Oct 23, 2024

• update

  John-Vaughan

  committed
  046592c

Commits on Nov 11, 2024

• add

  John-Vaughan

  committed
  f733ef1

Commits on Nov 12, 2024

• update

  John-Vaughan

  committed
  fe921c7