Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(fs): introduce chunk fs cache cleanup worker #226

Draft
wants to merge 2 commits into
base: develop
Choose a base branch
from
Draft

feat(fs): introduce chunk fs cache cleanup worker #226

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2f8a00e
feat(fs): introduce chunk fs cache cleanup worker
Oct 22, 2024
0b094db
chore(logs): add `basePath` to `FsCleanupWorker` logs
Oct 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,8 @@ services:
- CHUNK_POST_URLS=${CHUNK_POST_URLS:-}
- CHUNK_POST_RESPONSE_TIMEOUT_MS=${CHUNK_POST_RESPONSE_TIMEOUT_MS:-}
- CHUNK_POST_ABORT_TIMEOUT_MS=${CHUNK_POST_ABORT_TIMEOUT_MS:-}
- ENABLE_FS_CHUNK_CACHE_CLEANUP=${ENABLE_FS_CHUNK_CACHE_CLEANUP:-}
- CHUNK_DATA_CACHE_CLEANUP_THRESHOLD=${CHUNK_DATA_CACHE_CLEANUP_THRESHOLD:-}
- AO_CU_URL=${AO_CU_URL:-}
- AO_MU_URL=${AO_MU_URL:-}
- AO_GATEWAY_URL=${AO_GATEWAY_URL:-}
Expand Down
2 changes: 2 additions & 0 deletions src/app.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ system.headerFsCacheCleanupWorker?.start();

system.contiguousDataFsCacheCleanupWorker?.start();

system.chunkDataFsCacheCleanupWorker?.start();

// Allow starting without writers to support SQLite replication
if (config.START_WRITERS) {
system.blockImporter.start();
Expand Down
9 changes: 9 additions & 0 deletions src/config.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,15 @@ export const CONTIGUOUS_DATA_CACHE_CLEANUP_THRESHOLD = env.varOrDefault(
'',
);

export const ENABLE_FS_CHUNK_CACHE_CLEANUP =
env.varOrDefault('ENABLE_FS_CHUNK_CACHE_CLEANUP', 'false') === 'true';

// The threshold in seconds to cleanup the filesystem chunk data cache
export const CHUNK_DATA_CACHE_CLEANUP_THRESHOLD = +env.varOrDefault(
'CHUNK_DATA_CACHE_CLEANUP_THRESHOLD',
`${60 * 60 * 24}`, // 1 day
);

//
// Webhooks
//
Expand Down
29 changes: 29 additions & 0 deletions src/system.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,34 @@ export const contiguousDataFsCacheCleanupWorker = !isNaN(
})
: undefined;

export const chunkDataFsCacheCleanupWorker =
config.ENABLE_FS_CHUNK_CACHE_CLEANUP &&
!isNaN(config.CHUNK_DATA_CACHE_CLEANUP_THRESHOLD)
? new FsCleanupWorker({
log,
basePath: 'data/chunks',
shouldDelete: async (path) => {
try {
const stats = await fs.promises.stat(path);
const mostRecentTime =
stats.atime > stats.mtime ? stats.atime : stats.mtime;

const currentTimestamp = Date.now();

const thresholdDate = new Date(
currentTimestamp -
config.CHUNK_DATA_CACHE_CLEANUP_THRESHOLD * 1000,
);

return mostRecentTime <= thresholdDate;
} catch (err) {
log.error(`Error getting file stats for ${path}`, err);
return false;
}
},
})
: undefined;

Comment on lines +227 to +254
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This block is almost identical to the contiguousDataFsCacheCleanupWorker.

Consider abstracting the shared logic into a reusable function, for example:

function createFsCacheCleanupWorker({ basePath, thresholdInSeconds, isEnabled = true }) {
  if (!isEnabled || isNaN(thresholdInSeconds)) {
    return undefined;
  }

  return new FsCleanupWorker({
    log,
    basePath,
    shouldDelete: async (path) => {
      try {
        const stats = await fs.promises.stat(path);
        const mostRecentTime =
          stats.atime > stats.mtime ? stats.atime : stats.mtime;
        const currentTimestamp = Date.now();
        const thresholdDate = new Date(
          currentTimestamp - thresholdInSeconds * 1000,
        );
        return mostRecentTime <= thresholdDate;
      } catch (err) {
        log.error(`Error getting file stats for ${path}`, err);
        return false;
      }
    },
  });
}

export const contiguousDataFsCacheCleanupWorker = createFsCacheCleanupWorker({
  basePath: 'data/contiguous',
  thresholdInSeconds: contiguousDataCacheCleanupThresholdInSeconds,
});

export const chunkDataFsCacheCleanupWorker = createFsCacheCleanupWorker({
  basePath: 'data/chunks',
  thresholdInSeconds: config.CHUNK_DATA_CACHE_CLEANUP_THRESHOLD,
  isEnabled: config.ENABLE_FS_CHUNK_CACHE_CLEANUP,
});

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indeed it is - good rec! will pull in

const ans104TxMatcher = new MatchTags([
{ name: 'Bundle-Format', value: 'binary' },
{ name: 'Bundle-Version', valueStartsWith: '2.' },
Expand Down Expand Up @@ -671,6 +699,7 @@ export const shutdown = async (express: Server) => {
await db.stop();
await headerFsCacheCleanupWorker?.stop();
await contiguousDataFsCacheCleanupWorker?.stop();
await chunkDataFsCacheCleanupWorker?.stop();

process.exit(0);
});
Expand Down
2 changes: 1 addition & 1 deletion src/workers/fs-cleanup-worker.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ export class FsCleanupWorker {
pauseDuration?: number;
restartPauseDuration?: number;
}) {
this.log = log.child({ class: this.constructor.name });
this.log = log.child({ class: this.constructor.name, basePath });
this.shouldDelete = shouldDelete ?? (() => Promise.resolve(true));
this.deleteCallback =
deleteCallback ?? ((file: string) => fs.promises.unlink(file));
Expand Down