Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance optimizations for backups check #240

Merged

Conversation

pelmered
Copy link
Contributor

@pelmered pelmered commented Aug 2, 2024

We are using the BackupsCheck for Database snapshots from your laravel-db-snapshots) package. We have over 200 snapshots in are storing them in a S3 bucket. Thanks for accepting our previous PR to allow checking external disks. However, this introduced some performance problems. The way the current BackupsCheck work if first checks the file size of all backup files and then checks the modified date. For each check the check needs to reach out to the external file and fetch that information. This takes somewhere around 0.05 - 0.1 seconds per check per file. With 200 backup files this takes up to about 200 * 2 * 0.1 = 40 seconds. That is obviously not OK.

This PR addresses that in two ways:

  • We added the option to parse the modified date from the file name instead of reaching out to the file and check the meta data. This can be enabled with ->parseModifiedFormat('Y-m-d_H-i-s')
  • We also added the option to only check the size of the youngest and the oldest backup with ->onlyCheckSizeOnFirstAndLast()

Those two things reduced the execution time for the check from about 30 seconds to somewhere between 0.6 and 0.8 seconds and it should scale well with even larger amounts of files.

We also added some metadata to the BackupsCheck and a helper function on the ´Result` class for appending metadata. This is handy for checks when you are accumulating the information as you perform the checks and don't have all the information in the beginning of the check.
It is used like this:

$result = Result::make()->meta([
    'some_metadata' => 'interesting metric',
]);
//...
$result->appendMeta([
    'some_other_metadata' => 'Another interesting metric',
]);

This merges the appended metadata to the existing metadata instead of overwriting.

@freekmurze freekmurze merged commit 98e91b8 into spatie:main Aug 2, 2024
11 checks passed
@freekmurze
Copy link
Member

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants