Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

solve disk usage #79

Merged
merged 7 commits into from
Jan 26, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 22 additions & 9 deletions packages/synthetic-chain/cli.ts
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
#!/usr/bin/env tsx

import { parseArgs } from 'node:util';
import path from 'node:path';
import { execSync } from 'node:child_process';
import path from 'node:path';
import { parseArgs } from 'node:util';
import {
bakeTarget,
buildProposalSubmissions,
bakeImages,
readBuildConfig,
} from './src/cli/build.js';
import {
writeBakefileProposals,
writeDockerfile,
} from './src/cli/dockerfileGen.js';
import { matchOneProposal, readProposals } from './src/cli/proposals.js';
import { debugTestImage, runTestImages } from './src/cli/run.js';
import { runDoctor } from './src/cli/doctor.js';
import { imageNameForProposal, matchOneProposal, readProposals } from './src/cli/proposals.js';
import { debugTestImage, runTestImage } from './src/cli/run.js';

const { positionals, values } = parseArgs({
options: {
Expand Down Expand Up @@ -62,18 +62,31 @@ const prepareDockerBuild = () => {
switch (cmd) {
case 'build': {
prepareDockerBuild();
bakeImages('use', values.dry);
bakeTarget('use', values.dry);
break;
}
case 'test':
// Always rebuild all test images to keep it simple. With the "use" stages
// cached, these are pretty fast building doesn't run agd.
prepareDockerBuild();
bakeImages('test', values.dry);

if (values.debug) {
debugTestImage(matchOneProposal(proposals, match!));
const proposal = matchOneProposal(proposals, match!);
bakeTarget(imageNameForProposal(proposal, 'test').target, values.dry);
debugTestImage(proposal);
// don't bother to delete the test image because there's just one
// and the user probably wants to run it again.
} else {
runTestImages(proposals);
for (const proposal of proposals) {
const image = imageNameForProposal(proposal, 'test');
bakeTarget(image.target, values.dry);
runTestImage(proposal);
// delete the image to reclaim disk space. The next build
// will use the build cache.
execSync('docker system df', { stdio: 'inherit' });
execSync(`docker rmi ${image.name}`, { stdio: 'inherit' });
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I gather this docker rmi ... is the gist of it.

execSync('docker system df', { stdio: 'inherit' });
}
}
break;
case 'doctor':
Expand Down
11 changes: 7 additions & 4 deletions packages/synthetic-chain/src/cli/build.ts
Original file line number Diff line number Diff line change
Expand Up @@ -64,13 +64,16 @@ export const buildProposalSubmissions = (proposals: ProposalInfo[]) => {

/**
* Bake images using the docker buildx bake command.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not familiar with docker buildx bake. But I gather it's working...

*
* Note this uses `--load` which pushes the completed images to the builder,
* consuming 2-3 GB per image.
* @see {@link https://docs.docker.com/engine/reference/commandline/buildx_build/#load}
*
* @param matrix - The group target
* @param target - The image or group target
* @param [dry] - Whether to skip building and just print the build config.
*/
export const bakeImages = (matrix: 'test' | 'use', dry = false) => {
// https://docs.docker.com/engine/reference/commandline/buildx_build/#load
const cmd = `docker buildx bake --load ${matrix} ${dry ? '--print' : ''}`;
export const bakeTarget = (target: string, dry = false) => {
const cmd = `docker buildx bake --load ${target} ${dry ? '--print' : ''}`;
console.log(cmd);
execSync(cmd, { stdio: 'inherit' });
};
14 changes: 6 additions & 8 deletions packages/synthetic-chain/src/cli/run.ts
Original file line number Diff line number Diff line change
@@ -1,14 +1,12 @@
import { execSync } from 'node:child_process';
import { ProposalInfo, imageNameForProposal } from './proposals.js';

export const runTestImages = (proposals: ProposalInfo[]) => {
for (const proposal of proposals) {
console.log(`Running test image for proposal ${proposal.proposalName}`);
const { name } = imageNameForProposal(proposal, 'test');
// 'rm' to remove the container when it exits
const cmd = `docker run --rm ${name}`;
execSync(cmd, { stdio: 'inherit' });
}
export const runTestImage = (proposal: ProposalInfo) => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the API change from runTestImages to runTestImage sets off my backward-compatibility sensor.
But I don't suppose that's all that relevant in this context.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the CLI's modules aren't part of the package API.

The package doesn't yet define "exports" so they technically could be imported somewhere, but I think it's a reasonable expectation that cli not be.

That said, this package doesn't claim backwards compatibility because, following semver, it's not yet 1.0.

console.log(`Running test image for proposal ${proposal.proposalName}`);
const { name } = imageNameForProposal(proposal, 'test');
// 'rm' to remove the container when it exits
const cmd = `docker run --rm ${name}`;
execSync(cmd, { stdio: 'inherit' });
};

export const debugTestImage = (proposal: ProposalInfo) => {
Expand Down