Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add file-specific logging to MinIO, S3 endpoints #37

Open
lbeckman314 opened this issue Aug 23, 2023 · 2 comments
Open

Add file-specific logging to MinIO, S3 endpoints #37

lbeckman314 opened this issue Aug 23, 2023 · 2 comments
Assignees
Labels
enhancement New feature or request

Comments

@lbeckman314
Copy link
Collaborator

lbeckman314 commented Aug 23, 2023

Ideally the logging capabilities for the storage backends should include information for individual files in order to help debug any issues with downloads. It might be worth checking if the existing logs are available to all ACED developers and analysts.

In the long term this logging will be required for MinIO, Wasabi, and Ceph storage endpoints.

Setting up one common logging target such as Splunk for the entire Gen3 system is worth looking into as well as that would create a single source for access logs.

@lbeckman314 lbeckman314 added the enhancement New feature or request label Aug 23, 2023
@lbeckman314 lbeckman314 changed the title Add file-specific logging to MinIO, S3 buckets Add file-specific logging to MinIO, S3 endpoints Aug 23, 2023
@lbeckman314
Copy link
Collaborator Author

Following the lead on this with @burcheja to extract logs from the k8s pods on AWS and forward them to the internal Splunk endpoint. This will go a long way with improving overall status monitoring as well!

@burcheja
Copy link
Collaborator

burcheja commented Jan 2, 2024

Configured the system and audit loggers on the MinIO server to send data to Splunk. Looks like we need some ACLs updated to allow the logging data to flow as expected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants