WASM-based MRF (#490)

* add skeleton * use wasmtime * disable unused features * add wit and codegen * add initial component compilation * update authors name * rebuild if wit changed * add config options and docs * split into files * add metadata functions * stuff * add test binary, add explainer * progress * enable async support * fmt * fix * remove unwrap * embed mrf into kitsune for incoming activities * rename dir * updates * fmt * disallow any network access * up * add manifest for mrf * add manifest parser/serializer * AAAAAAAAAAAAAAAAAAAA * fix test, better diagnostics * restrict version field to semver * make schemars optional * derive default * convert `activityTypes` to set * add `configSchema` field * move mrf-manifest crate * add license * add mrf-tool to tree * fix comment * remove cfg flags * add encode function * add docs * use simdutf8 where possible * add description * restructure commands * restructure cli layout * split up functions * to_owned for manifest * up * rename to mrf, version in source * load mrf module config and pass to the module * log that config has been found * add span instead of log level fields * ensure span is used properly * filter by activity types * fix url encoding for international actors * stuff * fix * add submodules * add wit deps * add wit readme * exclude wit deps * add logging to example * add logging facade to wasm modules * add tracing subscriber to wasm mrf test * remove wasi-keyvalue * create own keyvalue def * add kv example * add bucket example * add storage backend support * finish kv storage impl for fs * up * up yarn * flake.lock: Update Flake lock file updates: • Updated input 'devenv': 'github:cachix/devenv/5a30b9e5ac7c6167e61b1f4193d5130bb9f8defa' (2024-02-13) → 'github:cachix/devenv/4eccee9a19ad9be42a7859211b456b281d704313' (2024-03-05) • Updated input 'flake-utils': 'github:numtide/flake-utils/1ef2e671c3b0c19053962c07dbda38332dcebf26' (2024-01-15) → 'github:numtide/flake-utils/d465f4819400de7c8d874d50b982301f28a84605' (2024-02-28) • Updated input 'nixpkgs': 'github:nixos/nixpkgs/a4d4fe8c5002202493e87ec8dbc91335ff55552c' (2024-02-15) → 'github:nixos/nixpkgs/b8697e57f10292a6165a20f03d2f42920dfaf973' (2024-03-03) • Updated input 'rust-overlay': 'github:oxalica/rust-overlay/4ee92bf124fbc4e157cbce1bc2a35499866989fc' (2024-02-16) → 'github:oxalica/rust-overlay/e86c0fb5d3a22a5f30d7f64ecad88643fe26449d' (2024-03-05) * up fuzz * add licenses * get rid of futures-retry-policies * up * up * fix warning * update ci * add redis skeleton * impl redis backend * introduce module name to backend * add configurable redis backend * rename * progress * remove from fetcher
kitsune-soc · Mar 9, 2024 · d24354f · d24354f
1 parent cfd8a76
commit d24354f
Show file tree

Hide file tree

Showing 122 changed files with 4,852 additions and 1,437 deletions.
diff --git a/.github/workflows/audit.yml b/.github/workflows/audit.yml
@@ -14,7 +14,7 @@ jobs:
   security_audit:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - uses: rustsec/[email protected]
         with:
           token: ${{ secrets.GITHUB_TOKEN }}
diff --git a/.github/workflows/cargo-machete.yml b/.github/workflows/cargo-machete.yml
@@ -15,6 +15,6 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
       - name: Machete
         uses: bnjbvr/cargo-machete@main
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
@@ -62,23 +62,23 @@ jobs:
         # we specify bash to get pipefail; it guards against the `curl` command
         # failing. otherwise `sh` won't catch that `curl` returned non-0
         shell: bash
-        run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.7.2/cargo-dist-installer.sh | sh"
+        run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.11.1/cargo-dist-installer.sh | sh"
       # sure would be cool if github gave us proper conditionals...
       # so here's a doubly-nested ternary-via-truthiness to try to provide the best possible
       # functionality based on whether this is a pull_request, and whether it's from a fork.
       # (PRs run on the *source* but secrets are usually on the *target* -- that's *good*
       # but also really annoying to build CI around when it needs secrets to work right.)
       - id: plan
         run: |
-          cargo dist ${{ !github.event.pull_request && format('host --steps=create --tag={0}', github.ref_name) || (github.event.pull_request.head.repo.fork && 'plan' || 'host --steps=check') }} --output-format=json > dist-manifest.json
+          cargo dist ${{ (!github.event.pull_request && format('host --steps=create --tag={0}', github.ref_name)) || 'plan' }} --output-format=json > plan-dist-manifest.json
           echo "cargo dist ran successfully"
-          cat dist-manifest.json
-          echo "manifest=$(jq -c "." dist-manifest.json)" >> "$GITHUB_OUTPUT"
+          cat plan-dist-manifest.json
+          echo "manifest=$(jq -c "." plan-dist-manifest.json)" >> "$GITHUB_OUTPUT"
       - name: "Upload dist-manifest.json"
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         with:
-          name: artifacts
-          path: dist-manifest.json
+          name: artifacts-plan-dist-manifest
+          path: plan-dist-manifest.json
 
   # Build and packages all the platform-specific things
   build-local-artifacts:
@@ -113,10 +113,11 @@ jobs:
         run: ${{ matrix.install_dist }}
       # Get the dist-manifest
       - name: Fetch local artifacts
-        uses: actions/download-artifact@v3
+        uses: actions/download-artifact@v4
         with:
-          name: artifacts
+          pattern: artifacts-*
           path: target/distrib/
+          merge-multiple: true
       - name: Install dependencies
         run: |
           ${{ matrix.packages_install }}
@@ -139,9 +140,9 @@ jobs:
 
           cp dist-manifest.json "$BUILD_MANIFEST_NAME"
       - name: "Upload artifacts"
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         with:
-          name: artifacts
+          name: artifacts-build-local-${{ join(matrix.targets, '_') }}
           path: |
             ${{ steps.cargo-dist.outputs.paths }}
             ${{ env.BUILD_MANIFEST_NAME }}
@@ -160,13 +161,15 @@ jobs:
         with:
           submodules: recursive
       - name: Install cargo-dist
-        run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.7.2/cargo-dist-installer.sh | sh"
+        shell: bash
+        run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.11.1/cargo-dist-installer.sh | sh"
       # Get all the local artifacts for the global tasks to use (for e.g. checksums)
       - name: Fetch local artifacts
-        uses: actions/download-artifact@v3
+        uses: actions/download-artifact@v4
         with:
-          name: artifacts
+          pattern: artifacts-*
           path: target/distrib/
+          merge-multiple: true
       - id: cargo-dist
         shell: bash
         run: |
@@ -180,9 +183,9 @@ jobs:
 
           cp dist-manifest.json "$BUILD_MANIFEST_NAME"
       - name: "Upload artifacts"
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         with:
-          name: artifacts
+          name: artifacts-build-global
           path: |
             ${{ steps.cargo-dist.outputs.paths }}
             ${{ env.BUILD_MANIFEST_NAME }}
@@ -204,13 +207,14 @@ jobs:
         with:
           submodules: recursive
       - name: Install cargo-dist
-        run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.7.2/cargo-dist-installer.sh | sh"
+        run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.11.1/cargo-dist-installer.sh | sh"
       # Fetch artifacts from scratch-storage
       - name: Fetch artifacts
-        uses: actions/download-artifact@v3
+        uses: actions/download-artifact@v4
         with:
-          name: artifacts
+          pattern: artifacts-*
           path: target/distrib/
+          merge-multiple: true
       # This is a harmless no-op for Github Releases, hosting for that happens in "announce"
       - id: host
         shell: bash
@@ -220,9 +224,10 @@ jobs:
           cat dist-manifest.json
           echo "manifest=$(jq -c "." dist-manifest.json)" >> "$GITHUB_OUTPUT"
       - name: "Upload dist-manifest.json"
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         with:
-          name: artifacts
+          # Overwrite the previous copy
+          name: artifacts-dist-manifest
           path: dist-manifest.json
 
   # Create a Github Release while uploading all files to it
@@ -242,10 +247,11 @@ jobs:
         with:
           submodules: recursive
       - name: "Download Github Artifacts"
-        uses: actions/download-artifact@v3
+        uses: actions/download-artifact@v4
         with:
-          name: artifacts
+          pattern: artifacts-*
           path: artifacts
+          merge-multiple: true
       - name: Cleanup
         run: |
           # Remove the granular manifests

diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml
@@ -17,7 +17,7 @@ jobs:
     name: Clippy
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - uses: DeterminateSystems/nix-installer-action@v4
       - uses: DeterminateSystems/magic-nix-cache-action@main
       - uses: taiki-e/install-action@cargo-hack
@@ -29,11 +29,11 @@ jobs:
     name: Formatting
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - uses: DeterminateSystems/nix-installer-action@v4
       - uses: DeterminateSystems/magic-nix-cache-action@main
       - run: nix develop --impure -c cargo fmt --all -- --check
-  
+
   test:
     name: Test
     runs-on: ubuntu-latest
@@ -61,7 +61,7 @@ jobs:
           --health-timeout 5s
           --health-retries 5
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - uses: DeterminateSystems/nix-installer-action@v4
       - uses: DeterminateSystems/magic-nix-cache-action@main
       - uses: Swatinem/rust-cache@v2

diff --git a/.github/workflows/web.yml b/.github/workflows/web.yml
@@ -54,7 +54,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       # Setup
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
         with:
           fetch-depth: 0
       - uses: dtolnay/rust-toolchain@stable

diff --git a/.gitignore b/.gitignore
@@ -18,6 +18,9 @@ target-analyzer
 # Production configuration file
 /config.toml
 
+# MRF directory
+/mrf-modules
+
 # Devenv stuff
 /result
 /.devenv

diff --git a/.gitmodules b/.gitmodules
@@ -0,0 +1,3 @@
+[submodule "crates/kitsune-wasm-mrf/wit/wasi-logging"]
+	path = crates/kitsune-wasm-mrf/wit/wasi-logging
+	url = https://github.com/WebAssembly/wasi-logging.git