-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Broadcast vanish requests #14
Changes from 2 commits
5eb9afe
f264a4c
8391733
14d039f
1c4f259
a28048a
454ce21
9d37c9d
4623b92
00c2393
e74b6c2
20b0789
758d975
105b1b4
30fbb2b
b1c34f5
d672464
dd621dd
9d4fc22
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,5 @@ | ||
[package] | ||
name = "spam_filter" | ||
name = "event_deleter" | ||
version = "0.1.0" | ||
edition = "2021" | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,61 @@ | ||
import type { | ||
Policy, | ||
OutputMessage, | ||
} from "https://raw.githubusercontent.com/planetary-social/strfry-policies/refs/heads/nos-changes/mod.ts"; | ||
import { log } from "https://raw.githubusercontent.com/planetary-social/strfry-policies/refs/heads/nos-changes/mod.ts"; | ||
|
||
const REQUEST_TO_VANISH_KIND = 62; | ||
const STREAM_KEY = "vanish_requests"; | ||
|
||
function createBroadcastVanishRequests( | ||
redis: any, | ||
relay_url: string | ||
): Policy<void> { | ||
if (!redis) { | ||
throw new Error("REDIS_URL environment variable is not set."); | ||
} | ||
|
||
if (!relay_url) { | ||
throw new Error("RELAY_URL environment variable is not set."); | ||
} | ||
|
||
return async (msg) => { | ||
const event = msg.event; | ||
const accept: OutputMessage = { | ||
id: event.id, | ||
action: "accept", | ||
msg: "", | ||
}; | ||
|
||
if (event.kind !== REQUEST_TO_VANISH_KIND) { | ||
return accept; | ||
} | ||
|
||
const match = event.tags | ||
.filter((tag) => tag["0"].toLowerCase().trim() === "relay") | ||
.map((tag) => tag["1"].toLowerCase().trim()) | ||
.find((relay) => relay === "all_relays" || relay === relay_url); | ||
|
||
if (!match) { | ||
return accept; | ||
} | ||
|
||
await broadcastVanishRequest(event, redis); | ||
|
||
return accept; | ||
}; | ||
} | ||
|
||
async function broadcastVanishRequest(event: any, redis: any) { | ||
log( | ||
`Pushing vanish request: id: ${event.id}, pubkey: ${event.pubkey}, tags: ${event.tags}, content: ${event.content}` | ||
); | ||
|
||
try { | ||
await redis.xadd(STREAM_KEY, "*", event); | ||
} catch (error) { | ||
log(`Failed to push request ${event.id} to Redis Stream: ${error}`); | ||
} | ||
} | ||
|
||
export { createBroadcastVanishRequests }; |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -9,6 +9,8 @@ import { | |
writeStdout, | ||
} from "https://raw.githubusercontent.com/planetary-social/strfry-policies/refs/heads/nos-changes/mod.ts"; | ||
import nosPolicy from "./nos_policy.ts"; | ||
import { createBroadcastVanishRequests } from "./broadcast_vanish_requests.ts"; | ||
import { connect, parseURL } from "https://deno.land/x/redis/mod.ts"; | ||
|
||
const localhost = "127.0.0.1"; | ||
const eventsIp = await getEventsIp(); | ||
|
@@ -18,6 +20,13 @@ const one_hour = 60 * one_minute; | |
const one_day = 24 * one_hour; | ||
const two_days = 2 * one_day; | ||
|
||
const redis_url = Deno.env.get("REDIS_URL"); | ||
const redis_connect_options = parseURL(redis_url); | ||
const redis = await connect(redis_connect_options); | ||
|
||
const relay_url = Deno.env.get("RELAY_URL"); | ||
const broadcastVanishRequests = createBroadcastVanishRequests(redis, relay_url); | ||
|
||
// Policies that reject faster should be at the top. So synchronous policies should be at the top. | ||
const policies = [ | ||
nosPolicy, | ||
|
@@ -53,6 +62,9 @@ const policies = [ | |
whitelist: [localhost, eventsIp], | ||
}, | ||
], | ||
|
||
// Broadcast vanish requests to Redis | ||
broadcastVanishRequests, | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It would be great if we could somehow write a test to verify that this is run after signature verification. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @mplorentz the verification happens in the strfry ingester, then the plugins are run before passing to the writer. I'm asking in the telegram channel just to be sure. But all we see in any plugin is already valid and their signatures are verified. Doing a manual check to protect against any weird bug or future changes wouldn't hurt though but I'd be very surprised if that happens and it would be a major bug There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ok, sounds like we don't need an automated test. It seemed possible to me that some folks might be running some spam filters before signature verification as an optimization. But if that's not even possible in strfry with the plugin system then that's good enough for me 👍 |
||
]; | ||
|
||
for await (const msg of readStdin()) { | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,112 @@ | ||
import { assertEquals } from "https://deno.land/[email protected]/testing/asserts.ts"; | ||
import { buildEvent, buildInputMessage } from "./test.ts"; | ||
import { createBroadcastVanishRequests } from "../broadcast_vanish_requests.ts"; | ||
import type { Event } from "https://raw.githubusercontent.com/planetary-social/strfry-policies/refs/heads/nos-changes/mod.ts"; | ||
|
||
class RedisMock { | ||
called: boolean = false; | ||
|
||
async xadd(streamKey: string, id: string, event: Event): Promise<void> { | ||
this.called = true; | ||
} | ||
} | ||
|
||
Deno.test({ | ||
name: "pushes a vanish request with global relay filter", | ||
fn: async () => { | ||
const msg = buildInputMessage({ | ||
sourceType: "IP4", | ||
sourceInfo: "1.1.1.1", | ||
event: buildEvent({ | ||
kind: 62, | ||
tags: [ | ||
["relay", "ALL_RELAYS"], | ||
["relay", "notexample.com"], | ||
], | ||
}), | ||
}); | ||
|
||
const redisMock = new RedisMock(); | ||
const broadcastVanishRequests = createBroadcastVanishRequests( | ||
redisMock, | ||
"example.com" | ||
); | ||
|
||
assertEquals((await broadcastVanishRequests(msg)).action, "accept"); | ||
assertEquals(redisMock.called, true); | ||
}, | ||
sanitizeResources: false, | ||
}); | ||
|
||
Deno.test({ | ||
name: "pushes a vanish request with specific relay filter", | ||
fn: async () => { | ||
const msg = buildInputMessage({ | ||
sourceType: "IP4", | ||
sourceInfo: "1.1.1.1", | ||
event: buildEvent({ | ||
kind: 62, | ||
tags: [ | ||
["relay", "example.com"], | ||
["relay", "notexample.com"], | ||
], | ||
}), | ||
}); | ||
|
||
const redisMock = new RedisMock(); | ||
const broadcastVanishRequests = createBroadcastVanishRequests( | ||
redisMock, | ||
"example.com" | ||
); | ||
|
||
assertEquals((await broadcastVanishRequests(msg)).action, "accept"); | ||
assertEquals(redisMock.called, true); | ||
}, | ||
sanitizeResources: false, | ||
}); | ||
|
||
Deno.test({ | ||
name: "doesn't push a vanish request with no matching relay filter", | ||
fn: async () => { | ||
const msg = buildInputMessage({ | ||
sourceType: "IP4", | ||
sourceInfo: "1.1.1.1", | ||
event: buildEvent({ | ||
kind: 62, | ||
tags: [["relay", "notexample.com"]], | ||
}), | ||
}); | ||
|
||
const redisMock = new RedisMock(); | ||
const broadcastVanishRequests = createBroadcastVanishRequests( | ||
redisMock, | ||
"example.com" | ||
); | ||
|
||
assertEquals((await broadcastVanishRequests(msg)).action, "accept"); | ||
assertEquals(redisMock.called, false); | ||
}, | ||
sanitizeResources: false, | ||
}); | ||
|
||
Deno.test({ | ||
name: "doesn't push when kind is not a vanish request", | ||
fn: async () => { | ||
const msg = buildInputMessage({ | ||
sourceType: "IP4", | ||
sourceInfo: "1.1.1.1", | ||
event: buildEvent({ | ||
kind: 1, | ||
}), | ||
}); | ||
|
||
const redisMock = new RedisMock(); | ||
const broadcastVanishRequests = createBroadcastVanishRequests( | ||
redisMock, | ||
"example.com" | ||
); | ||
assertEquals((await broadcastVanishRequests(msg)).action, "accept"); | ||
assertEquals(redisMock.called, false); | ||
}, | ||
sanitizeResources: false, | ||
}); |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
#!/bin/bash | ||
deno test --watch --allow-read --allow-write --allow-env --log-level=info |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,34 @@ | ||
import type { | ||
Event, | ||
InputMessage, | ||
} from "https://raw.githubusercontent.com/planetary-social/strfry-policies/refs/heads/nos-changes/mod.ts"; | ||
|
||
/** Constructs a fake event for tests. */ | ||
function buildEvent(attrs: Partial<Event> = {}): Event { | ||
const event: Event = { | ||
kind: 1, | ||
id: "", | ||
content: "", | ||
created_at: 0, | ||
pubkey: "", | ||
sig: "", | ||
tags: [], | ||
}; | ||
|
||
return Object.assign(event, attrs); | ||
} | ||
|
||
/** Constructs a fake strfry input message for tests. */ | ||
function buildInputMessage(attrs: Partial<InputMessage> = {}): InputMessage { | ||
const msg = { | ||
event: buildEvent(), | ||
receivedAt: 0, | ||
sourceInfo: "127.0.0.1", | ||
sourceType: "IP4", | ||
type: "new", | ||
}; | ||
|
||
return Object.assign(msg, attrs); | ||
} | ||
|
||
export { buildEvent, buildInputMessage }; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
how can we make sure these sorts of errors blow up in our face? Should we write a Grafana alert? Wire this up to Sentry? Get a log container to watch the logs?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I remember we had talked about some ideas to have detailed metrics for the relay. I'd first ask if @gergelypolonkai knows about this because I recall some comment related to a service that inspects logs, that could be useful for other things.
One alternative we have is leveraging the sidecar service that will run next to the relay that I have almost ready to go. Right now it's just a service that listens to the Redis stream and performs the deletions. Checking this stream is an indirect way of knowing if all is good. We could run an http service inside it and expose a /metrics endpoint. I assume we could even periodically tail the strfry logs and grep whatever we want but for that the first alternative may be better
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could prioritize having a Loki instance on the metrics server that collects logs; after that we can indeed have a Grafana alert that inspects such logs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I think it's time. I will file a ticket to tag this onto the end of the account deletion work.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
https://github.com/planetary-social/infrastructure/issues/142