Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(fs): introduce chunk fs cache cleanup worker #226

Draft
wants to merge 2 commits into
base: develop
Choose a base branch
from
Draft
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docker-compose.yaml
Original file line number Diff line number Diff line change
@@ -92,6 +92,8 @@ services:
- CHUNK_POST_URLS=${CHUNK_POST_URLS:-}
- CHUNK_POST_RESPONSE_TIMEOUT_MS=${CHUNK_POST_RESPONSE_TIMEOUT_MS:-}
- CHUNK_POST_ABORT_TIMEOUT_MS=${CHUNK_POST_ABORT_TIMEOUT_MS:-}
- ENABLE_FS_CHUNK_CACHE_CLEANUP=${ENABLE_FS_CHUNK_CACHE_CLEANUP:-}
- CHUNK_DATA_CACHE_CLEANUP_THRESHOLD=${CHUNK_DATA_CACHE_CLEANUP_THRESHOLD:-}
- AO_CU_URL=${AO_CU_URL:-}
- AO_MU_URL=${AO_MU_URL:-}
- AO_GATEWAY_URL=${AO_GATEWAY_URL:-}
2 changes: 2 additions & 0 deletions src/app.ts
Original file line number Diff line number Diff line change
@@ -36,6 +36,8 @@ system.headerFsCacheCleanupWorker?.start();

system.contiguousDataFsCacheCleanupWorker?.start();

system.chunkDataFsCacheCleanupWorker?.start();

// Allow starting without writers to support SQLite replication
if (config.START_WRITERS) {
system.blockImporter.start();
9 changes: 9 additions & 0 deletions src/config.ts
Original file line number Diff line number Diff line change
@@ -233,6 +233,15 @@ export const CONTIGUOUS_DATA_CACHE_CLEANUP_THRESHOLD = env.varOrDefault(
'',
);

export const ENABLE_FS_CHUNK_CACHE_CLEANUP =
env.varOrDefault('ENABLE_FS_CHUNK_CACHE_CLEANUP', 'false') === 'true';

// The threshold in seconds to cleanup the filesystem chunk data cache
export const CHUNK_DATA_CACHE_CLEANUP_THRESHOLD = +env.varOrDefault(
'CHUNK_DATA_CACHE_CLEANUP_THRESHOLD',
`${60 * 60 * 24}`, // 1 day
);

//
// Webhooks
//
29 changes: 29 additions & 0 deletions src/system.ts
Original file line number Diff line number Diff line change
@@ -224,6 +224,34 @@ export const contiguousDataFsCacheCleanupWorker = !isNaN(
})
: undefined;

export const chunkDataFsCacheCleanupWorker =
config.ENABLE_FS_CHUNK_CACHE_CLEANUP &&
!isNaN(config.CHUNK_DATA_CACHE_CLEANUP_THRESHOLD)
? new FsCleanupWorker({
log,
basePath: 'data/chunks',
shouldDelete: async (path) => {
try {
const stats = await fs.promises.stat(path);
const mostRecentTime =
stats.atime > stats.mtime ? stats.atime : stats.mtime;

const currentTimestamp = Date.now();

const thresholdDate = new Date(
currentTimestamp -
config.CHUNK_DATA_CACHE_CLEANUP_THRESHOLD * 1000,
);

return mostRecentTime <= thresholdDate;
} catch (err) {
log.error(`Error getting file stats for ${path}`, err);
return false;
}
},
})
: undefined;

Comment on lines +227 to +254

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This block is almost identical to the contiguousDataFsCacheCleanupWorker.

Consider abstracting the shared logic into a reusable function, for example:

function createFsCacheCleanupWorker({ basePath, thresholdInSeconds, isEnabled = true }) {
  if (!isEnabled || isNaN(thresholdInSeconds)) {
    return undefined;
  }

  return new FsCleanupWorker({
    log,
    basePath,
    shouldDelete: async (path) => {
      try {
        const stats = await fs.promises.stat(path);
        const mostRecentTime =
          stats.atime > stats.mtime ? stats.atime : stats.mtime;
        const currentTimestamp = Date.now();
        const thresholdDate = new Date(
          currentTimestamp - thresholdInSeconds * 1000,
        );
        return mostRecentTime <= thresholdDate;
      } catch (err) {
        log.error(`Error getting file stats for ${path}`, err);
        return false;
      }
    },
  });
}

export const contiguousDataFsCacheCleanupWorker = createFsCacheCleanupWorker({
  basePath: 'data/contiguous',
  thresholdInSeconds: contiguousDataCacheCleanupThresholdInSeconds,
});

export const chunkDataFsCacheCleanupWorker = createFsCacheCleanupWorker({
  basePath: 'data/chunks',
  thresholdInSeconds: config.CHUNK_DATA_CACHE_CLEANUP_THRESHOLD,
  isEnabled: config.ENABLE_FS_CHUNK_CACHE_CLEANUP,
});

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indeed it is - good rec! will pull in

const ans104TxMatcher = new MatchTags([
{ name: 'Bundle-Format', value: 'binary' },
{ name: 'Bundle-Version', valueStartsWith: '2.' },
@@ -671,6 +699,7 @@ export const shutdown = async (express: Server) => {
await db.stop();
await headerFsCacheCleanupWorker?.stop();
await contiguousDataFsCacheCleanupWorker?.stop();
await chunkDataFsCacheCleanupWorker?.stop();

process.exit(0);
});
2 changes: 1 addition & 1 deletion src/workers/fs-cleanup-worker.ts
Original file line number Diff line number Diff line change
@@ -54,7 +54,7 @@ export class FsCleanupWorker {
pauseDuration?: number;
restartPauseDuration?: number;
}) {
this.log = log.child({ class: this.constructor.name });
this.log = log.child({ class: this.constructor.name, basePath });
this.shouldDelete = shouldDelete ?? (() => Promise.resolve(true));
this.deleteCallback =
deleteCallback ?? ((file: string) => fs.promises.unlink(file));