Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Broadcast vanish requests #14

Merged
merged 19 commits into from
Oct 9, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,13 @@ ENV PATH="/root/.cargo/bin:${PATH}"

RUN rustc --version

COPY ./spam_filter/Cargo.toml ./spam_filter/Cargo.lock /build/spam_filter/
COPY ./event_deleter/Cargo.toml ./event_deleter/Cargo.lock /build/event_deleter/

WORKDIR /build/spam_filter
WORKDIR /build/event_deleter

RUN cargo fetch

COPY ./spam_filter/src /build/spam_filter/src
COPY ./event_deleter/src /build/event_deleter/src

RUN cargo build --release

Expand Down Expand Up @@ -54,7 +54,7 @@ WORKDIR /app

COPY --from=build /build/strfry/strfry strfry

COPY --from=build /build/spam_filter/target/release/spam_cleaner /usr/local/bin/spam_cleaner
COPY --from=build /build/event_deleter/target/release/spam_cleaner /usr/local/bin/spam_cleaner

RUN chmod +x /usr/local/bin/spam_cleaner

Expand Down
13 changes: 13 additions & 0 deletions compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,17 @@ services:
build: .
ports:
- "7777:7777"
environment:
- RELAY_URL=wss://example.com
- REDIS_URL=redis://redis:6379

redis:
image: redis:7.2.4
ports:
- "6379:6379"
command: redis-server --loglevel notice
volumes:
- redis_data:/data

volumes:
redis_data:
File renamed without changes.
36 changes: 18 additions & 18 deletions spam_filter/Cargo.lock → event_deleter/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion spam_filter/Cargo.toml → event_deleter/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[package]
name = "spam_filter"
name = "event_deleter"
version = "0.1.0"
edition = "2021"

Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
use clap::Parser;
use nostr_sdk::Event;
use serde_json::Deserializer;
use spam_filter::{
use event_deleter::{
analyzer_worker::ValidationWorker,
deletion_task::spawn_deletion_task,
event_analyzer::{RejectReason, Validator},
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
61 changes: 61 additions & 0 deletions strfry/plugins/broadcast_vanish_requests.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
import type {
Policy,
OutputMessage,
} from "https://raw.githubusercontent.com/planetary-social/strfry-policies/refs/heads/nos-changes/mod.ts";
import { log } from "https://raw.githubusercontent.com/planetary-social/strfry-policies/refs/heads/nos-changes/mod.ts";

const REQUEST_TO_VANISH_KIND = 62;
const STREAM_KEY = "vanish_requests";

function createBroadcastVanishRequests(
redis: any,
relay_url: string
): Policy<void> {
if (!redis) {
throw new Error("REDIS_URL environment variable is not set.");
}

if (!relay_url) {
throw new Error("RELAY_URL environment variable is not set.");
}

return async (msg) => {
const event = msg.event;
const accept: OutputMessage = {
id: event.id,
action: "accept",
msg: "",
};

if (event.kind !== REQUEST_TO_VANISH_KIND) {
return accept;
}

const match = event.tags
.filter((tag) => tag["0"].toLowerCase().trim() === "relay")
.map((tag) => tag["1"].toLowerCase().trim())
.find((relay) => relay === "all_relays" || relay === relay_url);

if (!match) {
return accept;
}

await broadcastVanishRequest(event, redis);

return accept;
};
}

async function broadcastVanishRequest(event: any, redis: any) {
log(
`Pushing vanish request: id: ${event.id}, pubkey: ${event.pubkey}, tags: ${event.tags}, content: ${event.content}`
);

try {
await redis.xadd(STREAM_KEY, "*", event);
} catch (error) {
log(`Failed to push request ${event.id} to Redis Stream: ${error}`);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how can we make sure these sorts of errors blow up in our face? Should we write a Grafana alert? Wire this up to Sentry? Get a log container to watch the logs?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I remember we had talked about some ideas to have detailed metrics for the relay. I'd first ask if @gergelypolonkai knows about this because I recall some comment related to a service that inspects logs, that could be useful for other things.

One alternative we have is leveraging the sidecar service that will run next to the relay that I have almost ready to go. Right now it's just a service that listens to the Redis stream and performs the deletions. Checking this stream is an indirect way of knowing if all is good. We could run an http service inside it and expose a /metrics endpoint. I assume we could even periodically tail the strfry logs and grep whatever we want but for that the first alternative may be better

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could prioritize having a Loki instance on the metrics server that collects logs; after that we can indeed have a Grafana alert that inspects such logs.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I think it's time. I will file a ticket to tag this onto the end of the account deletion work.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

}
}

export { createBroadcastVanishRequests };
7 changes: 6 additions & 1 deletion strfry/plugins/nos_policy.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ const ALLOWED = {
5, // Event deletion
6, // Repost
7, // Reaction
62, // Request to Vanish
1059, // Gift wrap messages
1984, // Reports
10000, // Mute list
Expand All @@ -29,7 +30,11 @@ const DISALLOWED = {
const nosPolicy: Policy<void> = (msg) => {
const event = msg.event;
const content = event.content;
let res = { id: event.id, action: "reject", msg: "blocked: not authorized" };
let res = {
id: event.id,
action: "reject",
msg: "blocked: not authorized",
};

const isAllowedPub = ALLOWED.pubs.hasOwnProperty(event.pubkey);
const isAllowedEventKind = ALLOWED.eventKinds.includes(event.kind);
Expand Down
12 changes: 12 additions & 0 deletions strfry/plugins/policies.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ import {
writeStdout,
} from "https://raw.githubusercontent.com/planetary-social/strfry-policies/refs/heads/nos-changes/mod.ts";
import nosPolicy from "./nos_policy.ts";
import { createBroadcastVanishRequests } from "./broadcast_vanish_requests.ts";
import { connect, parseURL } from "https://deno.land/x/redis/mod.ts";

const localhost = "127.0.0.1";
const eventsIp = await getEventsIp();
Expand All @@ -18,6 +20,13 @@ const one_hour = 60 * one_minute;
const one_day = 24 * one_hour;
const two_days = 2 * one_day;

const redis_url = Deno.env.get("REDIS_URL");
const redis_connect_options = parseURL(redis_url);
const redis = await connect(redis_connect_options);

const relay_url = Deno.env.get("RELAY_URL");
const broadcastVanishRequests = createBroadcastVanishRequests(redis, relay_url);

// Policies that reject faster should be at the top. So synchronous policies should be at the top.
const policies = [
nosPolicy,
Expand Down Expand Up @@ -53,6 +62,9 @@ const policies = [
whitelist: [localhost, eventsIp],
},
],

// Broadcast vanish requests to Redis
broadcastVanishRequests,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be great if we could somehow write a test to verify that this is run after signature verification.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mplorentz the verification happens in the strfry ingester, then the plugins are run before passing to the writer. I'm asking in the telegram channel just to be sure. But all we see in any plugin is already valid and their signatures are verified.

Doing a manual check to protect against any weird bug or future changes wouldn't hurt though but I'd be very surprised if that happens and it would be a major bug

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, sounds like we don't need an automated test. It seemed possible to me that some folks might be running some spam filters before signature verification as an optimization. But if that's not even possible in strfry with the plugin system then that's good enough for me 👍

];

for await (const msg of readStdin()) {
Expand Down
112 changes: 112 additions & 0 deletions strfry/plugins/tests/broadcast_vanish_requests.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
import { assertEquals } from "https://deno.land/[email protected]/testing/asserts.ts";
import { buildEvent, buildInputMessage } from "./test.ts";
import { createBroadcastVanishRequests } from "../broadcast_vanish_requests.ts";
import type { Event } from "https://raw.githubusercontent.com/planetary-social/strfry-policies/refs/heads/nos-changes/mod.ts";

class RedisMock {
called: boolean = false;

async xadd(streamKey: string, id: string, event: Event): Promise<void> {
this.called = true;
}
}

Deno.test({
name: "pushes a vanish request with global relay filter",
fn: async () => {
const msg = buildInputMessage({
sourceType: "IP4",
sourceInfo: "1.1.1.1",
event: buildEvent({
kind: 62,
tags: [
["relay", "ALL_RELAYS"],
["relay", "notexample.com"],
],
}),
});

const redisMock = new RedisMock();
const broadcastVanishRequests = createBroadcastVanishRequests(
redisMock,
"example.com"
);

assertEquals((await broadcastVanishRequests(msg)).action, "accept");
assertEquals(redisMock.called, true);
},
sanitizeResources: false,
});

Deno.test({
name: "pushes a vanish request with specific relay filter",
fn: async () => {
const msg = buildInputMessage({
sourceType: "IP4",
sourceInfo: "1.1.1.1",
event: buildEvent({
kind: 62,
tags: [
["relay", "example.com"],
["relay", "notexample.com"],
],
}),
});

const redisMock = new RedisMock();
const broadcastVanishRequests = createBroadcastVanishRequests(
redisMock,
"example.com"
);

assertEquals((await broadcastVanishRequests(msg)).action, "accept");
assertEquals(redisMock.called, true);
},
sanitizeResources: false,
});

Deno.test({
name: "doesn't push a vanish request with no matching relay filter",
fn: async () => {
const msg = buildInputMessage({
sourceType: "IP4",
sourceInfo: "1.1.1.1",
event: buildEvent({
kind: 62,
tags: [["relay", "notexample.com"]],
}),
});

const redisMock = new RedisMock();
const broadcastVanishRequests = createBroadcastVanishRequests(
redisMock,
"example.com"
);

assertEquals((await broadcastVanishRequests(msg)).action, "accept");
assertEquals(redisMock.called, false);
},
sanitizeResources: false,
});

Deno.test({
name: "doesn't push when kind is not a vanish request",
fn: async () => {
const msg = buildInputMessage({
sourceType: "IP4",
sourceInfo: "1.1.1.1",
event: buildEvent({
kind: 1,
}),
});

const redisMock = new RedisMock();
const broadcastVanishRequests = createBroadcastVanishRequests(
redisMock,
"example.com"
);
assertEquals((await broadcastVanishRequests(msg)).action, "accept");
assertEquals(redisMock.called, false);
},
sanitizeResources: false,
});
2 changes: 2 additions & 0 deletions strfry/plugins/tests/run_tests.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
#!/bin/bash
deno test --watch --allow-read --allow-write --allow-env --log-level=info
34 changes: 34 additions & 0 deletions strfry/plugins/tests/test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
import type {
Event,
InputMessage,
} from "https://raw.githubusercontent.com/planetary-social/strfry-policies/refs/heads/nos-changes/mod.ts";

/** Constructs a fake event for tests. */
function buildEvent(attrs: Partial<Event> = {}): Event {
const event: Event = {
kind: 1,
id: "",
content: "",
created_at: 0,
pubkey: "",
sig: "",
tags: [],
};

return Object.assign(event, attrs);
}

/** Constructs a fake strfry input message for tests. */
function buildInputMessage(attrs: Partial<InputMessage> = {}): InputMessage {
const msg = {
event: buildEvent(),
receivedAt: 0,
sourceInfo: "127.0.0.1",
sourceType: "IP4",
type: "new",
};

return Object.assign(msg, attrs);
}

export { buildEvent, buildInputMessage };