Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch from localstack to minIO #2640

Open
wants to merge 2 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 17 additions & 10 deletions docker-compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,20 +23,27 @@ services:
ports:
- "6380:6379"

localstack:
image: localstack/localstack:latest
minio:
image: minio/minio:latest
restart: unless-stopped
environment:
- AWS_DEFAULT_REGION=ap-south-1
- EDGE_PORT=4566
- SERVICES=s3
- EXTRA_CORS_ALLOWED_ORIGINS=*
- EXTRA_CORS_ALLOWED_HEADERS=*
MINIO_ROOT_USER: ${MINIO_ACCESS_KEY:-minioadmin}
MINIO_ROOT_PASSWORD: ${MINIO_SECRET_KEY:-minioadmin}
AWS_DEFAULT_REGION: ap-south-1 # To maintain compatibility with existing apps
volumes:
- "${TEMPDIR:-./care/media/localstack}:/var/lib/localstack"
- "./docker/awslocal:/etc/localstack/init/ready.d/"
- "./care/media/minio:/data"
- "./docker/minio/init-script.sh:/init-script.sh:ro" # Mount the init script
- "./docker/minio/entrypoint.sh:/entrypoint.sh:ro" # Mount the entrypoint script
Comment on lines +34 to +36
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

The mysterious case of the missing volume definition

While the volume mounts look correct, you might want to define a named volume for the MinIO data directory in the volumes section. You know, for persistence and all that.

volumes:
  postgres-data:
  redis-data:
+ minio-data:

services:
  minio:
    volumes:
-     - "./care/media/minio:/data"
+     - "minio-data:/data"
      - "./docker/minio/init-script.sh:/init-script.sh:ro"
      - "./docker/minio/entrypoint.sh:/entrypoint.sh:ro"
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- "./care/media/minio:/data"
- "./docker/minio/init-script.sh:/init-script.sh:ro" # Mount the init script
- "./docker/minio/entrypoint.sh:/entrypoint.sh:ro" # Mount the entrypoint script
- "minio-data:/data"
- "./docker/minio/init-script.sh:/init-script.sh:ro" # Mount the init script
- "./docker/minio/entrypoint.sh:/entrypoint.sh:ro" # Mount the entrypoint script

ports:
- "4566:4566"
- "9100:9000" # S3 API
- "9001:9001" # Web Console
entrypoint: ["/entrypoint.sh"]
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/ready"]
interval: 10s
retries: 5
start_period: 10s
timeout: 10s

volumes:
postgres-data:
Expand Down
8 changes: 4 additions & 4 deletions docker/.local.env
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,10 @@ CELERY_BROKER_URL=redis://redis:6379/0
DJANGO_DEBUG=False

BUCKET_REGION=ap-south-1
BUCKET_KEY=key
BUCKET_SECRET=secret
BUCKET_ENDPOINT=http://localstack:4566
BUCKET_EXTERNAL_ENDPOINT=http://localhost:4566
BUCKET_KEY=${MINIO_ACCESS_KEY:-minioadmin}
BUCKET_SECRET=${MINIO_SECRET_KEY:-minioadmin}
BUCKET_ENDPOINT=http://minio:9000
BUCKET_EXTERNAL_ENDPOINT=http://localhost:9100
FILE_UPLOAD_BUCKET=patient-bucket
FACILITY_S3_BUCKET=facility-bucket

Expand Down
9 changes: 5 additions & 4 deletions docker/.prebuilt.env
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,11 @@ DJANGO_SETTINGS_MODULE=config.settings.deployment
DJANGO_DEBUG=False

BUCKET_REGION=ap-south-1
BUCKET_KEY=key
BUCKET_SECRET=secret
BUCKET_ENDPOINT=http://localstack:4566
BUCKET_EXTERNAL_ENDPOINT=http://localhost:4566
# WARNING: These are default MinIO credentials. Ensure to change these in production environments
BUCKET_KEY=minioadmin
BUCKET_SECRET=minioadmin
Comment on lines +14 to +15
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Default credentials in a prebuilt environment? How... convenient

Using default MinIO credentials (minioadmin) in a prebuilt environment might not be the best security practice. While it's technically functional, we should probably add a comment suggesting users to change these in production.

+# WARNING: These are default MinIO credentials. Ensure to change these in production environments
 BUCKET_KEY=minioadmin
 BUCKET_SECRET=minioadmin
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
BUCKET_KEY=minioadmin
BUCKET_SECRET=minioadmin
# WARNING: These are default MinIO credentials. Ensure to change these in production environments
BUCKET_KEY=minioadmin
BUCKET_SECRET=minioadmin

BUCKET_ENDPOINT=http://minio:9000
BUCKET_EXTERNAL_ENDPOINT=http://localhost:9100
FILE_UPLOAD_BUCKET=patient-bucket
FACILITY_S3_BUCKET=facility-bucket

Expand Down
6 changes: 0 additions & 6 deletions docker/awslocal/bucket-setup.sh

This file was deleted.

24 changes: 24 additions & 0 deletions docker/minio/entrypoint.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
#!/bin/sh

# Start MinIO in the background
minio server /data --console-address ":9001" &

# Wait for MinIO to be ready before running the initialization script
TIMEOUT=300 # 5 minutes
start_time=$(date +%s)
until curl -s http://localhost:9000/minio/health/ready; do
current_time=$(date +%s)
elapsed=$((current_time - start_time))
if [ $elapsed -gt $TIMEOUT ]; then
echo "MinIO failed to start after ${TIMEOUT} seconds. But I'm sure you knew that could happen."
exit 1
fi
echo "Waiting for MinIO to be ready..."
sleep 5
done

# Run the bucket setup script
sh /init-script.sh
Comment on lines +20 to +21
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Assuming init-script.sh will always be there, are we?

It might be worth checking if the initialization script exists and is executable before blindly running it.

 # Run the bucket setup script
+if [ ! -f /init-script.sh ]; then
+    echo "Init script not found. I'm sure this is fine."
+    exit 1
+fi
+if [ ! -x /init-script.sh ]; then
+    echo "Init script not executable. How mysterious."
+    exit 1
+fi
 sh /init-script.sh
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# Run the bucket setup script
sh /init-script.sh
# Run the bucket setup script
if [ ! -f /init-script.sh ]; then
echo "Init script not found. I'm sure this is fine."
exit 1
fi
if [ ! -x /init-script.sh ]; then
echo "Init script not executable. How mysterious."
exit 1
fi
sh /init-script.sh


# Keep the container running
wait $!
62 changes: 62 additions & 0 deletions docker/minio/init-script.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
#!/bin/sh

set -e

# MinIO configuration
MINIO_HOST=${MINIO_HOST:-"http://localhost:9000"}
MINIO_ACCESS_KEY=${MINIO_ACCESS_KEY:-"minioadmin"}
MINIO_SECRET_KEY=${MINIO_SECRET_KEY:-"minioadmin"}

# Max retries and delay
MAX_RETRIES=10
RETRY_COUNT=0
RETRY_DELAY=5 # 5 seconds delay between retries

# Function to retry a command
retry_command() {
cmd=$1
until $cmd; do
RETRY_COUNT=$((RETRY_COUNT + 1))
if [ $RETRY_COUNT -ge $MAX_RETRIES ]; then
echo "Command failed after $MAX_RETRIES attempts. Exiting..."
exit 1
fi
echo "Command failed. Retrying ($RETRY_COUNT/$MAX_RETRIES)..."
sleep $RETRY_DELAY
done
}

# Function to create a bucket if it doesn't exist
create_bucket_if_not_exists() {
BUCKET_NAME=$1
echo "Checking if bucket $BUCKET_NAME exists..."
if mc ls local/$BUCKET_NAME > /dev/null 2>&1; then
echo "Bucket $BUCKET_NAME already exists. Skipping creation."
else
echo "Creating bucket $BUCKET_NAME..."
mc mb local/$BUCKET_NAME
fi
}

# Function to set a bucket public
set_bucket_public() {
BUCKET_NAME=$1
# WARNING: This bucket is intentionally set to public access as MinIO doesn't support ACLs
# Ensure only non-sensitive data is stored in this bucket
echo "Setting bucket $BUCKET_NAME as public..."
mc anonymous set public local/$BUCKET_NAME
}

# Retry MinIO Client alias setup
retry_command "mc alias set local $MINIO_HOST $MINIO_ACCESS_KEY $MINIO_SECRET_KEY"

# Create the necessary buckets
create_bucket_if_not_exists "patient-bucket"
create_bucket_if_not_exists "facility-bucket"

# Set only facility-bucket as public
set_bucket_public "facility-bucket"

# Graceful exit
echo "Bucket setup completed successfully."
exit 0
2 changes: 1 addition & 1 deletion docs/local-setup/configuration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Using Docker Compose
- care (main repo)
- redis (in-memory cache)
- celery (task queue)
- localstack (to mimic AWS services locally)
- minio (to mimic AWS services locally)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

The documentation seems to be missing some... details

While you've updated the service list to include MinIO, it would be absolutely wonderful if we could add some MinIO-specific setup instructions, such as:

  • How to access the MinIO console (typically at http://localhost:9001)
  • Default credentials for the MinIO console
  • Any specific requirements or considerations for the MinIO setup
     - minio (to mimic AWS services locally)
+        The MinIO console is available at http://localhost:9001
+        Default credentials: minioadmin/minioadmin
+        Note: The facility bucket is set to public access
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- minio (to mimic AWS services locally)
- minio (to mimic AWS services locally)
The MinIO console is available at http://localhost:9001
Default credentials: minioadmin/minioadmin
Note: The facility bucket is set to public access


This is the most recommended way of setting up care locally,
as it installs appropriate dependencies in containers so there
Expand Down