log file size is different when downloaded using console and cli

[YashCharpe]

Describe the issue

I am trying to download log file of writer instance using aws cli. So for that I am using 'download-db-log-file-portion' command like this:

aws rds download-db-log-file-portion \
--db-instance-identifier instance_name \
--log-file-name error/postgresql.log.2023-09-26-0300.gz \
--starting-token 0 \
--output text >> directory_path

This rds instance has .gz extension log files. On console, I can see the log file size as 25mb but when I download it via the above command, the downloaded file size is of 223mb. Does anyone know what could be the reason behind this and how do I download the log file of 25mb?

Additional Information/Context

No response

CLI version used

1.22.34

Environment details (OS name and version, etc.)

WSL2

[tim-finnigan]

Hi @YashCharpe thanks for reaching out. That version of the AWS CLI is very old. As shown in the v1 CHANGELOG it is quite a few versions behind the latest. We also recommend using v2 if possible — here are the instructions for installing that: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

Have you tried looking through related issues referencing the aws rds download-db-log-file-portion command? I found a very old one (aws/aws-cli#1613) describing how this command could return slightly larger files due to the addition of new lines. Although I'm not sure if that would explain the file size difference that you referenced.

I think we need more information in order to investigate this further. After updating your version of the AWS CLI and using v2 if possible, could you share your debug logs by adding --debug to the command? Please redact any sensitive information.

[YashCharpe]

Hi @tim-finnigan, thanks for looking out on this issue.

I worked on this a bit and found out that with the mentioned command, the logs are downloaded in a unzipped format only whereas when downloaded through aws console, the files downloaded are of .gz type (compressed file) and that is the reason behind different file size. Do you know how we can download the compressed file (.gz) using the aws cli command?

[tim-finnigan]

Hi @YashCharpe thanks for following up. The download-db-log-file-portion command corresponds to this RDS API: https://docs.aws.amazon.com/AmazonRDS/latest/APIReference/API_DownloadDBLogFilePortion.html. Therefore feature requests involving the API behavior must get forwarded to the RDS team. (If you have a support plan then we recommend reaching out through AWS Support with these requests, otherwise we can forward them on your behalf.)

Can you just zip the file upon downloading it? Or have you explored other alternatives such as publishing database logs to CloudWatch as documented here: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.Procedural.UploadtoCloudWatch.html?

[YashCharpe]

Hi @tim-finnigan, Thank you for checking in. If possible, please forward the request on my behalf as I am not really familiar with it.

As of now, the requirement doesn't allow me to zip the file after downloading and also the CloudWatch cannot be used. Somehow I need to download those archive files only.

[tim-finnigan]

Sure - thanks for following up. I'll reach out to the RDS team for feedback and transfer this to our cross-SDK repository for further tracking (because APIs like this are used across AWS SDKs). [ref: P101840457]

[tim-finnigan]

Thanks for your patience — we heard back from someone on the RDS team who said that you could use the DownloadCompleteDBLogFile API for this use case: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/DownloadCompleteDBLogFile.html. (Note that this is only available via REST API and not in the SDKs.) Please let us know if you have any follow up questions.

[github-actions]

This issue is now closed.

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.