Code

support Dockerized s3 tests#337

Code

Merged

donsizemore merged 120 commits intodevelopgdcc/dataverse-ansible:developfrom 327_multiple_s3_storesgdcc/dataverse-ansible:327_multiple_s3_storesCopy head branch name to clipboardDec 7, 2023

+585-164

Commits

Commits on Oct 18, 2023

• #327 initial jenkins.yml changes for phil's review

  

  Don Sizemore

  committed
  4f8d5c9

• #327 only one storage_driver_id, remains file

  

  Don Sizemore

  committed
  57cf564

• #327 remove config deletion from dataverse-storage.yml

  

  Don Sizemore

  committed
  04c8a47

Commits on Oct 30, 2023

• Merge remote-tracking branch 'origin/develop' into 327_multiple_s3_stores

  

  Don Sizemore

  committed
  b2eeb54

Commits on Oct 31, 2023

• Merge remote-tracking branch 'origin/develop' into 327_multiple_s3_stores

  

  Don Sizemore

  committed
  83632c0

• #327 stub out docker/localstack

  

  Don Sizemore

  committed
  3e5dda7

• Merge remote-tracking branch 'origin/develop' into 327_multiple_s3_stores

  

  Don Sizemore

  committed
  547771f

• #327 tweak lookup loop

  

  Don Sizemore

  committed
  6bfe02d

Commits on Nov 6, 2023

• #327 add localstack_create.yml

  

  Don Sizemore

  committed
  e9c6adb

• #327 add localstack_create_bucket.yml

  

  Don Sizemore

  committed
  7137898

• #327 add CORS config

  

  Don Sizemore

  committed
  7f783ca

Commits on Nov 7, 2023

• #327 add cors.json for S3 testing

  

  Don Sizemore

  committed
  8646de4

• #327 remove errant localstack variable name prefix

  

  Don Sizemore

  committed
  531cfc6

• #327 correct docker repo installation

  

  Don Sizemore

  committed
  d9fca9c

• #327 remove second errant localstack variable name prefix

  

  Don Sizemore

  committed
  2314dd0

• #327 ansible_user is now ansible_ssh_user

  

  Don Sizemore

  committed
  6b53d93

• #327 infer docker user

  

  Don Sizemore

  committed
  c4b2918

• #327 do we even need raw/endraw

  

  Don Sizemore

  committed
  a520250

• #327 add necessary localstack group_vars

  

  Don Sizemore

  committed
  48d99f9

• #327 add missing single ticks

  

  Don Sizemore

  committed
  985a22b

• #327 container should be container_name

  

  Don Sizemore

  committed
  33789ca

• #327 switch to docker compose v2

  

  Don Sizemore

  committed
  09c8d33

• #327 disable most of S3 configuration in favor of supporting multiple S3-based datastores

  

  Don Sizemore

  committed
  e56b879

• #327 uncomment S3 in favor of s3.enabled false in group_vars

  

  Don Sizemore

  committed
  857316f

• #327 let's pass AWS keys to localstack

  

  Don Sizemore

  committed
  7ebc99c

• #327 too many items spoil the broth

  

  Don Sizemore

  committed
  6579842

• #327 include_tasks with loop

  

  Don Sizemore

  committed
  5f80f14

Commits on Nov 8, 2023

• #327 loop with block

  

  Don Sizemore

  committed
  0c595be

• #327 attempt include_tasks with_items

  

  Don Sizemore

  committed
  5ddd7e6

• #327 now it IS sending item properly

  

  Don Sizemore

  committed
  456dd2e

• #327 it sees item now

  

  Don Sizemore

  committed
  2b5e701

• #327 remove errant item prefix

  

  Don Sizemore

  committed
  19ec247

• #327 add AWS binary where needed

  

  Don Sizemore

  committed
  8529717

• #327 just use OS-included binary for now

  

  Don Sizemore

  committed
  b168fe8

• #327 awscli package name may be specific to RedHat systems

  

  Don Sizemore

  committed
  388c6fa

• #327 correct Red Hat check

  

  Don Sizemore

  committed
  22aa1fd

• #327 place AWS config in dataverse user homedir

  

  Don Sizemore

  committed
  3d585af

• #327 use dataverse homedir for localstack compose

  

  Don Sizemore

  committed
  fc7a7c1

Commits on Nov 9, 2023

• #327 don't kill the container

  

  Don Sizemore

  committed
  c3f16a0

• #327 perform s3 commands as service account

  

  Don Sizemore

  committed
  d043403

Commits on Nov 10, 2023

• #327 add dataverse user to docker group

  

  Don Sizemore

  committed
  fd560aa

• #327 localstack is trying to nuke its own data_dir

  

  Don Sizemore

  committed
  d5b0b66

• #327 Phil's localstack config is much simpler

  

  Don Sizemore

  committed
  28c1daf

• #327 correct container launch user

  

  Don Sizemore

  committed
  d91a83d

• #327 localstack claims it doesn't need to be background-ed

  

  Don Sizemore

  committed
  26c572d

• #327 container stays up when not background-ed

  

  Don Sizemore

  committed
  54bda38

• #327 correct conditional evaluation

  

  Don Sizemore

  committed
  59f25ed

• #327 custom_endpoint_region should just be region

  

  Don Sizemore

  committed
  13ac628

• #327 label should be item.label

  

  Don Sizemore

  committed
  cfe62fa

• #327 loop_control

  

  Don Sizemore

  committed
  63c64db

• #327 use loop_control in outer loop

  

  Don Sizemore

  committed
  8bc009e

• #327 item should be outer

  

  Don Sizemore

  committed
  59939ba

• #327 correct key/value definitions

  

  Don Sizemore

  committed
  5de7941

• #327 replace key ticks, whoops

  

  Don Sizemore

  committed
  d9cf884

Commits on Nov 13, 2023

• #327 let's define our own dict

  

  Don Sizemore

  committed
  cf23d82

Commits on Nov 14, 2023

• #327 debug outer list

  

  Don Sizemore

  committed
  2f79dcd

• #327 quotes and ticks, quotes and ticks

  

  Don Sizemore

  committed
  09e19ca

• #327 debug jvm-options

  

  Don Sizemore

  committed
  13c5e95

Commits on Nov 15, 2023

• #327 let's set ingestsizelimit to 2000000000

  

  Don Sizemore

  committed
  eeb3cb3

• #327 correct key/value syntax

  

  Don Sizemore

  committed
  f70f37c

• #327 jvm-options should be outer

  

  Don Sizemore

  committed
  abd0e34

• #327 jvm-options must be jvm_options

  

  Don Sizemore

  committed
  c84b509

• #327 let's debug bucketlist again

  

  Don Sizemore

  committed
  8a2334b

• #327 Ansible handles bucket and CORS

  

  Don Sizemore

  committed
  df0b84b

• #327 swap inner and outer loop_vars

  

  Don Sizemore

  committed
  9ae290a

• #327 do we need to escape custom_endpoint_url?

  

  Don Sizemore

  committed
  45c585d

Commits on Nov 17, 2023

• #327 correct unescaped custom endpoint url

  

  Don Sizemore

  committed
  e0401c0

• #327 I was using ticks instead of quotes

  

  Don Sizemore

  committed
  1a4c641

• #327 we bothered to escape the var, let's use it

  

  Don Sizemore

  committed
  73d9607

Commits on Nov 20, 2023

• #327 forgot to escape a dash

  

  Don Sizemore

  committed
  4441add

• #327 don't explicitly read in bucket_options when populating jvm_options

  

  Don Sizemore

  committed
  ff12dc2

• #327 correct cors.json syntax

  

  Don Sizemore

  committed
  1e5db65

• #327 whoops, cors settings requires custom_endpoint_escaped

  

  Don Sizemore

  committed
  6f00946

• #327 let's do MinIO as well, all in

  

  Don Sizemore

  committed
  01b0b7a

• #327 stub out MinIO settings

  

  Don Sizemore

  committed
  a7a4b86

Commits on Nov 21, 2023

• #327 replace original S3 group_vars, use localstack-specific

  

  Don Sizemore

  committed
  f9973bf

• #327 untested stab at minio task

  

  Don Sizemore

  committed
  5ffba8b

• #327 correct spacing, add some DV-specific jvm-options

  

  Don Sizemore

  committed
  8cd0dea

• #327 correct group_vars syntax

  

  Don Sizemore

  committed
  498075c

• #327 correct minio root setting tests

  

  Don Sizemore

  committed
  dbf37bc

• #327 docker-compose.yml should be minio_compose.yml

  

  Don Sizemore

  committed
  bcccf09

Commits on Nov 27, 2023

• #327 correct minio template src filename

  

  Don Sizemore

  committed
  4034d0b

• #327 correct variable typo in minio_compose.j2

  

  Don Sizemore

  committed
  2912023

• #327 correct variable typos in minio template

  

  Don Sizemore

  committed
  4031296

Commits on Nov 28, 2023

• #327 one final typo in MinIO jinja template

  

  Don Sizemore

  committed
  7a199c6

• #327 Ansible Docker module requires python-docker

  

  Don Sizemore

  committed
  2f38d4c

• #327 make that python3-docker

  

  Don Sizemore

  committed
  eafe3b9

• #327 Ansible needs docker-compose python package

  

  Don Sizemore

  committed
  3c6289d

• #327 what a difference a colon makes

  

  Don Sizemore

  committed
  3589b92

• #327 now pip errors. sigh.

  

  Don Sizemore

  committed
  1edfef3

• #327 specify minio_compose.yml

  

  Don Sizemore

  committed
  073ff0f

• #327 project_src is required, files conflicts with definition instead

  

  Don Sizemore

  committed
  bba73ad

• #327 don't use special docker network unless we have to

  

  Don Sizemore

  committed
  15011d9

Commits on Nov 29, 2023

• #327 launch minio as dataverse service account

  

  Don Sizemore

  committed
  4156091

• #327 do we need minio docker user/group?

  

  Don Sizemore

  committed
  f82e48a

• #327 variables must exist, length 0

  

  Don Sizemore

  committed
  5d9dff3

• #327 keep variables, comment out in case

  

  Don Sizemore

  committed
  62c4a22

• #327 adjust commented spacing

  

  Don Sizemore

  committed
  f04faaf

• #327 adjust template indentation

  

  Don Sizemore

  committed
  db5f630

• #327 if blocks causing trouble

  

  Don Sizemore

  committed
  5e74320

Commits on Nov 30, 2023

• #327 set MinIO jvm-options

  

  Don Sizemore

  committed
  34cd897

• #327 quote MinIO CEU value

  

  Don Sizemore

  committed
  a497741

• #327 escape custom-endpoint-url colons

  

  Don Sizemore

  committed
  0fa758a

• #327 underscores only

  

  Don Sizemore

  committed
  18aa4b2

Commits on Dec 1, 2023

• #327 update minio variables

  

  Don Sizemore

  committed
  e00dae3

• #327 localstack1, mybucket

  

  Don Sizemore

  committed
  e9a271a

• #327 localstack label, caps count

  

  Don Sizemore

  committed
  01dc08f

• #327 LocalStack: match Phil's options exactly

  

  Don Sizemore

  committed
  fd34213

• #327 touch up variables post-Zoom

  

  Don Sizemore

  committed
  e00d8d5

Commits on Dec 4, 2023

• #327 run tests and reports as Dataverse service account

  

  Don Sizemore

  committed
  801c6ee

• #327 disable global s3 creds/config as a test

  

  Don Sizemore

  committed
  463763a

• #327 correct localstack variables

  

  Don Sizemore

  committed
  76a21e6

• #327 correct identifier group_var syntax

  

  Don Sizemore

  committed
  4a3731f

• #327 use ibiblio mirror for now, minio wants to match AWS credentials

  

  Don Sizemore

  committed
  7c1844e

• #327 correct minio secret-access-key

  

  Don Sizemore

  committed
  d2ec1d8

Commits on Dec 5, 2023

• #327 add localstack/minio group_vars to defaults/vagrant yml

  

  Don Sizemore

  committed
  48594cf

• #327 duplicate Docker CIDR to default/vagrant

  

  Don Sizemore

  committed
  5b458dc

Commits on Dec 6, 2023

• #327 hopefully final touch-ups

  

  Don Sizemore

  committed
  d66f79f

• #327 keep aws_config.yml for now

  

  Don Sizemore

  committed
  2efaafa

• #327 correct minio secret_key

  

  Don Sizemore

  committed
  37bae1f