Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Tweak swing-store export example code to show process synchronization #8174

Merged
merged 1 commit into from
Oct 5, 2023
Merged

docs: Tweak swing-store export example code to show process synchronization #8174

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 78 additions & 51 deletions packages/swing-store/docs/data-export.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,30 +39,41 @@ The exporter is created by calling `makeSwingStoreExporter(dirpath)`, passing it
After calling `hostStorage.commit()`, the host application can extract the first-stage export data, and then the second-stage export artifacts:

```js
import { buffer } from 'node:stream/consumers';

const dirPath = '.../swing-store';
const swingStore = openSwingStore(dirPath);
...
await controller.run();
hostStorage.commit();
// spawn a child process

// child process does:
const exporter = makeSwingStoreExporter(dirPath);
// exporter now has a txn, parent process is free to proceed forward
const exportData = new Map();
for (const [key, value] of exporter.getExportData()) {
exportData.set(key, value);
}
const exportArtifacts = new Map();
for (const name of exporter.getArtifactNames()) {
const reader = exporter.getArtifact(name);
// reader is an async iterable of Uint8Array, e.g. a stream
const data = await buffer(reader);
exportArtifacts.set(name, data);
const dirPath = '.../swing-store';
const swingStore = openSwingStore(dirPath);
...
await controller.run();
hostStorage.commit();
// spawn a child process and wait for it to open a transaction
const started = makePromiseKit();
const child = fork(path, args);
child.on('error', started.reject);
child.on('exit', started.reject);
child.on('message', msg => {
if (msg?.type === 'started') {
started.resolve();
}
// export is 'exportData' and 'exportArtifacts'
});
await started.promise;
...

// child process does:
import { buffer } from 'node:stream/consumers';
const exporter = makeSwingStoreExporter(dirPath);
// exporter now has a txn, so parent process is free to proceed forward
process.send({ type: 'started' });
const exportData = new Map();
for (const [key, value] of exporter.getExportData()) {
exportData.set(key, value);
}
const exportArtifacts = new Map();
for (const name of exporter.getArtifactNames()) {
const reader = exporter.getArtifact(name);
// reader is an async iterable of Uint8Array, e.g. a stream
const data = await buffer(reader);
exportArtifacts.set(name, data);
}
// export is the combination of 'exportData' and 'exportArtifacts'
```

![image 3](./images/data-export-3.jpg)
Expand Down Expand Up @@ -92,35 +103,51 @@ Then, on the few occasions when the application needs to build a full state-sync
![image 4](./images/data-export-4.jpg)

```js
const dirPath = '.../swing-store';
const iavl = ...;
function exportCallback(key, value) {
const iavlKey = `ssed.${key}`; // 'ssed' is short for SwingStoreExportData
if (value) {
iavl.set(iavlKey, value);
} else {
iavl.delete(iavlKey); // value===undefined means delete
}
}
const swingStore = openSwingStore(dirPath, { exportCallback });
...
await controller.run();
hostStorage.commit();

// now, if the validator is configured to publish state-sync snapshots,
// and if this block height is one of the publishing points,
// do the following:

// spawn a child process

// child process does:
const exporter = makeSwingStoreExporter(dirPath);
// note: no exporter.getExportData(), the first-stage data is already in IAVL
const artifacts = new Map();
for (const name of exporter.getArtifactNames()) {
artifacts.set(name, exporter.getArtifact(name));
const dirPath = '.../swing-store';
const iavl = ...;
function exportCallback(key, value) {
const iavlKey = `ssed.${key}`; // 'ssed' is short for SwingStoreExportData
if (value === undefined) {
Comment on lines -99 to +110
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh good call!

iavl.delete(iavlKey);
} else {
iavl.set(iavlKey, value);
}
}
const swingStore = openSwingStore(dirPath, { exportCallback });
...
await controller.run();
hostStorage.commit();

// now, if the validator is configured to publish state-sync snapshots,
// and if this block height is one of the publishing points,
// do the following:

// spawn a child process and wait for it to open a transaction
const started = makePromiseKit();
const child = fork(path, args);
child.on('error', started.reject);
child.on('exit', started.reject);
child.on('message', msg => {
if (msg?.type === 'started') {
started.resolve();
}
// instruct cosmos-sdk to include 'artifacts' in the state-sync snapshot
});
try {
await started.promise;
} catch (err) {
...
}
...

// child process does:
const exporter = makeSwingStoreExporter(dirPath);
process.send({ type: 'started' });
// note: no exporter.getExportData(), the first-stage data is already in IAVL
const artifacts = new Map();
for (const name of exporter.getArtifactNames()) {
artifacts.set(name, exporter.getArtifact(name));
}
// instruct cosmos-sdk to include 'artifacts' in the state-sync snapshot
```

## Import
Expand Down
Loading