Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch from localstack to minIO #2640

Open
wants to merge 2 commits into
base: develop
Choose a base branch
from
Open

Switch from localstack to minIO #2640

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2f14d3d
Switch from localstack to minio
amjithtitus09 Dec 8, 2024
2a23839
Added relevant comments & suggested code changes
amjithtitus09 Dec 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 17 additions & 10 deletions docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,20 +23,27 @@ services:
ports:
- "6380:6379"

localstack:
image: localstack/localstack:latest
minio:
image: minio/minio:latest
restart: unless-stopped
environment:
- AWS_DEFAULT_REGION=ap-south-1
- EDGE_PORT=4566
- SERVICES=s3
- EXTRA_CORS_ALLOWED_ORIGINS=*
- EXTRA_CORS_ALLOWED_HEADERS=*
MINIO_ROOT_USER: minioadmin
MINIO_ROOT_PASSWORD: minioadmin
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Credentials in plain text. Classic.

While having credentials directly in the compose file is... charming, it might be better to use environment variables.

-      MINIO_ROOT_USER: minioadmin
-      MINIO_ROOT_PASSWORD: minioadmin
+      MINIO_ROOT_USER: ${MINIO_ACCESS_KEY:-minioadmin}
+      MINIO_ROOT_PASSWORD: ${MINIO_SECRET_KEY:-minioadmin}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
MINIO_ROOT_USER: minioadmin
MINIO_ROOT_PASSWORD: minioadmin
MINIO_ROOT_USER: ${MINIO_ACCESS_KEY:-minioadmin}
MINIO_ROOT_PASSWORD: ${MINIO_SECRET_KEY:-minioadmin}

AWS_DEFAULT_REGION: ap-south-1 # To maintain compatibility with existing apps
volumes:
- "${TEMPDIR:-./care/media/localstack}:/var/lib/localstack"
- "./docker/awslocal:/etc/localstack/init/ready.d/"
- "./care/media/minio:/data"
- "./docker/minio/init-script.sh:/init-script.sh:ro" # Mount the init script
- "./docker/minio/entrypoint.sh:/entrypoint.sh:ro" # Mount the entrypoint script
Comment on lines +34 to +36
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

The mysterious case of the missing volume definition

While the volume mounts look correct, you might want to define a named volume for the MinIO data directory in the volumes section. You know, for persistence and all that.

volumes:
  postgres-data:
  redis-data:
+ minio-data:

services:
  minio:
    volumes:
-     - "./care/media/minio:/data"
+     - "minio-data:/data"
      - "./docker/minio/init-script.sh:/init-script.sh:ro"
      - "./docker/minio/entrypoint.sh:/entrypoint.sh:ro"
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- "./care/media/minio:/data"
- "./docker/minio/init-script.sh:/init-script.sh:ro" # Mount the init script
- "./docker/minio/entrypoint.sh:/entrypoint.sh:ro" # Mount the entrypoint script
- "minio-data:/data"
- "./docker/minio/init-script.sh:/init-script.sh:ro" # Mount the init script
- "./docker/minio/entrypoint.sh:/entrypoint.sh:ro" # Mount the entrypoint script

ports:
- "4566:4566"
- "9100:9000" # S3 API
- "9001:9001" # Web Console
entrypoint: ["/entrypoint.sh"]
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/ready"]
interval: 10s
retries: 5
start_period: 10s
timeout: 10s

volumes:
postgres-data:
Expand Down
8 changes: 4 additions & 4 deletions docker/.local.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,10 @@ CELERY_BROKER_URL=redis://redis:6379/0
DJANGO_DEBUG=False

BUCKET_REGION=ap-south-1
BUCKET_KEY=key
BUCKET_SECRET=secret
BUCKET_ENDPOINT=http://localstack:4566
BUCKET_EXTERNAL_ENDPOINT=http://localhost:4566
BUCKET_KEY=minioadmin
BUCKET_SECRET=minioadmin
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Oh, we're using default credentials. How... convenient.

Using minioadmin as both username and password might not be the most secure approach. I'm sure you have your reasons, but perhaps we could make it slightly less obvious?

Consider using environment-specific credentials:

-BUCKET_KEY=minioadmin
-BUCKET_SECRET=minioadmin
+BUCKET_KEY=${MINIO_ACCESS_KEY:-minioadmin}
+BUCKET_SECRET=${MINIO_SECRET_KEY:-minioadmin}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
BUCKET_KEY=minioadmin
BUCKET_SECRET=minioadmin
BUCKET_KEY=${MINIO_ACCESS_KEY:-minioadmin}
BUCKET_SECRET=${MINIO_SECRET_KEY:-minioadmin}

BUCKET_ENDPOINT=http://minio:9000
BUCKET_EXTERNAL_ENDPOINT=http://localhost:9100
FILE_UPLOAD_BUCKET=patient-bucket
FACILITY_S3_BUCKET=facility-bucket

Expand Down
8 changes: 4 additions & 4 deletions docker/.prebuilt.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,10 @@ DJANGO_SETTINGS_MODULE=config.settings.deployment
DJANGO_DEBUG=False

BUCKET_REGION=ap-south-1
BUCKET_KEY=key
BUCKET_SECRET=secret
BUCKET_ENDPOINT=http://localstack:4566
BUCKET_EXTERNAL_ENDPOINT=http://localhost:4566
BUCKET_KEY=minioadmin
BUCKET_SECRET=minioadmin
Comment on lines +14 to +15
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Default credentials in a prebuilt environment? How... convenient

Using default MinIO credentials (minioadmin) in a prebuilt environment might not be the best security practice. While it's technically functional, we should probably add a comment suggesting users to change these in production.

+# WARNING: These are default MinIO credentials. Ensure to change these in production environments
 BUCKET_KEY=minioadmin
 BUCKET_SECRET=minioadmin
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
BUCKET_KEY=minioadmin
BUCKET_SECRET=minioadmin
# WARNING: These are default MinIO credentials. Ensure to change these in production environments
BUCKET_KEY=minioadmin
BUCKET_SECRET=minioadmin

BUCKET_ENDPOINT=http://minio:9000
BUCKET_EXTERNAL_ENDPOINT=http://localhost:9100
FILE_UPLOAD_BUCKET=patient-bucket
FACILITY_S3_BUCKET=facility-bucket

Expand Down
6 changes: 0 additions & 6 deletions docker/awslocal/bucket-setup.sh
View file
Open in desktop

This file was deleted.

16 changes: 16 additions & 0 deletions docker/minio/entrypoint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
#!/bin/sh

# Start MinIO in the background
minio server /data --console-address ":9001" &

# Wait for MinIO to be ready before running the initialization script
until curl -s http://localhost:9000/minio/health/ready; do
echo "Waiting for MinIO to be ready..."
sleep 5
done
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

I see we're feeling optimistic about the health check

While the health check is... functional, it might be nice to add a timeout to prevent infinite waiting. You know, just in case something goes terribly wrong.

Here's a slightly more robust version:

+TIMEOUT=300  # 5 minutes should be enough, right?
+start_time=$(date +%s)
 until curl -s http://localhost:9000/minio/health/ready; do
+    current_time=$(date +%s)
+    elapsed=$((current_time - start_time))
+    if [ $elapsed -gt $TIMEOUT ]; then
+        echo "MinIO failed to start after ${TIMEOUT} seconds. But I'm sure you knew that could happen."
+        exit 1
+    fi
     echo "Waiting for MinIO to be ready..."
     sleep 5
 done
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
until curl -s http://localhost:9000/minio/health/ready; do
echo "Waiting for MinIO to be ready..."
sleep 5
done
TIMEOUT=300 # 5 minutes should be enough, right?
start_time=$(date +%s)
until curl -s http://localhost:9000/minio/health/ready; do
current_time=$(date +%s)
elapsed=$((current_time - start_time))
if [ $elapsed -gt $TIMEOUT ]; then
echo "MinIO failed to start after ${TIMEOUT} seconds. But I'm sure you knew that could happen."
exit 1
fi
echo "Waiting for MinIO to be ready..."
sleep 5
done


# Run the bucket setup script
sh /init-script.sh
Comment on lines +20 to +21
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Assuming init-script.sh will always be there, are we?

It might be worth checking if the initialization script exists and is executable before blindly running it.

 # Run the bucket setup script
+if [ ! -f /init-script.sh ]; then
+    echo "Init script not found. I'm sure this is fine."
+    exit 1
+fi
+if [ ! -x /init-script.sh ]; then
+    echo "Init script not executable. How mysterious."
+    exit 1
+fi
 sh /init-script.sh
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# Run the bucket setup script
sh /init-script.sh
# Run the bucket setup script
if [ ! -f /init-script.sh ]; then
echo "Init script not found. I'm sure this is fine."
exit 1
fi
if [ ! -x /init-script.sh ]; then
echo "Init script not executable. How mysterious."
exit 1
fi
sh /init-script.sh


# Keep the container running
wait $!
60 changes: 60 additions & 0 deletions docker/minio/init-script.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
#!/bin/sh

set -e

# MinIO configuration
MINIO_HOST=${MINIO_HOST:-"http://localhost:9000"}
MINIO_ACCESS_KEY=${MINIO_ACCESS_KEY:-"minioadmin"}
MINIO_SECRET_KEY=${MINIO_SECRET_KEY:-"minioadmin"}

# Max retries and delay
MAX_RETRIES=10
RETRY_COUNT=0
RETRY_DELAY=5 # 5 seconds delay between retries

# Function to retry a command
retry_command() {
local cmd=$1
until $cmd; do
RETRY_COUNT=$((RETRY_COUNT + 1))
if [ $RETRY_COUNT -ge $MAX_RETRIES ]; then
echo "Command failed after $MAX_RETRIES attempts. Exiting..."
exit 1
fi
echo "Command failed. Retrying ($RETRY_COUNT/$MAX_RETRIES)..."
sleep $RETRY_DELAY
done
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

That's a nice retry mechanism you've got there... would be even better with proper POSIX compliance

The retry logic is well-implemented, but the local keyword isn't POSIX-compliant. I'm sure you meant to use a more portable approach.

-    local cmd=$1
+    cmd=$1
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
retry_command() {
local cmd=$1
until $cmd; do
RETRY_COUNT=$((RETRY_COUNT + 1))
if [ $RETRY_COUNT -ge $MAX_RETRIES ]; then
echo "Command failed after $MAX_RETRIES attempts. Exiting..."
exit 1
fi
echo "Command failed. Retrying ($RETRY_COUNT/$MAX_RETRIES)..."
sleep $RETRY_DELAY
done
}
retry_command() {
cmd=$1
until $cmd; do
RETRY_COUNT=$((RETRY_COUNT + 1))
if [ $RETRY_COUNT -ge $MAX_RETRIES ]; then
echo "Command failed after $MAX_RETRIES attempts. Exiting..."
exit 1
fi
echo "Command failed. Retrying ($RETRY_COUNT/$MAX_RETRIES)..."
sleep $RETRY_DELAY
done
}
🧰 Tools
🪛 Shellcheck (0.10.0)

[warning] 17-17: In POSIX sh, 'local' is undefined.

(SC3043)


# Function to create a bucket if it doesn't exist
create_bucket_if_not_exists() {
BUCKET_NAME=$1
echo "Checking if bucket $BUCKET_NAME exists..."
if mc ls local/$BUCKET_NAME > /dev/null 2>&1; then
echo "Bucket $BUCKET_NAME already exists. Skipping creation."
else
echo "Creating bucket $BUCKET_NAME..."
mc mb local/$BUCKET_NAME
fi
}

# Function to set a bucket public
set_bucket_public() {
BUCKET_NAME=$1
echo "Setting bucket $BUCKET_NAME as public..."
mc anonymous set public local/$BUCKET_NAME
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Public bucket access? How... brave of you

Setting a bucket to public access requires careful consideration of security implications. While I notice this aligns with the PR objective mentioning "facility bucket is converted to public access", we should probably add some warning comments for future maintainers.

 # Function to set a bucket public
 set_bucket_public() {
     BUCKET_NAME=$1
+    # WARNING: This bucket is intentionally set to public access as MinIO doesn't support ACLs
+    # Ensure only non-sensitive data is stored in this bucket
     echo "Setting bucket $BUCKET_NAME as public..."
     mc anonymous set public local/$BUCKET_NAME
 }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# Function to set a bucket public
set_bucket_public() {
BUCKET_NAME=$1
echo "Setting bucket $BUCKET_NAME as public..."
mc anonymous set public local/$BUCKET_NAME
}
# Function to set a bucket public
set_bucket_public() {
BUCKET_NAME=$1
# WARNING: This bucket is intentionally set to public access as MinIO doesn't support ACLs
# Ensure only non-sensitive data is stored in this bucket
echo "Setting bucket $BUCKET_NAME as public..."
mc anonymous set public local/$BUCKET_NAME
}


# Retry MinIO Client alias setup
retry_command "mc alias set local $MINIO_HOST $MINIO_ACCESS_KEY $MINIO_SECRET_KEY"

# Create the necessary buckets
create_bucket_if_not_exists "patient-bucket"
create_bucket_if_not_exists "facility-bucket"

# Set only facility-bucket as public
set_bucket_public "facility-bucket"

# Graceful exit
echo "Bucket setup completed successfully."
exit 0
2 changes: 1 addition & 1 deletion docs/local-setup/configuration.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Using Docker Compose
- care (main repo)
- redis (in-memory cache)
- celery (task queue)
- localstack (to mimic AWS services locally)
- minio (to mimic AWS services locally)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

The documentation seems to be missing some... details

While you've updated the service list to include MinIO, it would be absolutely wonderful if we could add some MinIO-specific setup instructions, such as:

  • How to access the MinIO console (typically at http://localhost:9001)
  • Default credentials for the MinIO console
  • Any specific requirements or considerations for the MinIO setup
     - minio (to mimic AWS services locally)
+        The MinIO console is available at http://localhost:9001
+        Default credentials: minioadmin/minioadmin
+        Note: The facility bucket is set to public access
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- minio (to mimic AWS services locally)
- minio (to mimic AWS services locally)
The MinIO console is available at http://localhost:9001
Default credentials: minioadmin/minioadmin
Note: The facility bucket is set to public access


This is the most recommended way of setting up care locally,
as it installs appropriate dependencies in containers so there
Expand Down