SJIP: Update Judging Process

[Evert0x]

Description

Update Judging Process

Judging Guidelines PR

sherlock-protocol/sherlock-v2-docs#46
sherlock-protocol/sherlock-v2-docs#47

Rationale

1. Real-time judging provides a fast and accurate outcome for most issues. However, due to the subjective nature of judging, achieving full consensus with the community is not possible. To address this, we need to introduce a Lead Judge for each contest.

2. We've pushed for more objectiveness in the last few months, although helpful and accurate in the majority of judgments. It get's in the way of achieving the correct outcome in some cases. For that reason we will emphasize that the guidelines are guidelines and can be overwritten if it makes sense in the context of the audit.

Relevant Issue Discussions

Update: The PR has been merged but the discussion stays open

Update 2: Added PR 47 to provide more context on deviations

[jacksanford1]

Summary from the docs of how the new process will go:

Judging Process

Phase 1: Community Judging

Why? Reduces workload for Lead Judge

• Happens right after the audit contest ends
• Lasts a fixed number of days (# of issues / 100) days (for example 500 issues would be 5 days), with a minimum of 2 days.
• Dedicated Judge signups give some degree of certainty that all issues will be de-duplicated and initially judged by the end of this period

The dedicated judges need to judge all issues during the fixed number of days. See Dedicated Judge for more details.

Phase 2: Lead Judge

Why? If we go straight to preliminary results, security researchers may get frustrated by the inaccurate judging done by the community

• Lead Judge will know his start date as soon as the contest ends (if 500 issues then he starts 5 days after the contest ends)
• Lead Judge gets (# of issues / 200) days to do this preliminary judging (so 500 issues would round up to 3 days)
• Lead Judge only needs to focus on the controversial issues during this phase
• Lead Judge must provide an explanation for every judgment

Checkpoint: Preliminary Results Announced

Phase 3: Escalation Period

• After the preliminary results are announced, security researchers get 24 hours to review the judgments and comment on judgments they disagree with

Phase 4: Final Lead Judge Judgments

• Lead Judge gets (# of controversial issues / 20) days to complete Phase 4
• Lead Judge will review all controversial issues (new ones from the Escalation Period and old ones that he already judged)
• Lead Judge will make final decisions on all controversial issues
• Checkpoint: Final Results Announced

The Sherlock Judge will closely monitor the decisions during Phase 4, be available for any questions, ask questions if something is unclear, and only overturn a decision if it’s clearly wrong.

[pronobis4]

Who pays the Lead Judge?

[Evert0x]

Lead Judge is paid by Sherlock

[Tobi0x18]

Total rewards will include the contest pool, LSW reward, judging pool, and lead judge reward, as before.
The key difference now is the order and the weight of the community judging :

• Community judging was previously conducted in parallel with the lead judge, but it is now done sequentially.
• The community judging pool has been increased.
• Lead judges can save energy and focus on the main tasks.

I believe this is a significant improvement.
(btw, very curious about how much the lead judge earns.)

Can the lead judge participate in the community judging?

[midori-fuse]

Can we min-cap the Lead Judge's Phase 2 and Phase 4 to 2 days (or simply add 1 day into the current days count)?

Even if assuming the LJ is expected (and has been given ample time) to study the codebase, there might just be issues that requires some additional investigation (e.g. external researching, usually these are really good and deep-diving issues), or some missed points during prelim judging are brought up and taken for reconsideration (minor human mistakes), or simply takes time to work out what potential risks and impacts there are/how much of a threat the issues are. One day might just not cut it for some of these issues alone.

[Evert0x]

Makes sense, I will update it to a minimum of 2 days

[Evert0x]

PR 47 has been added to provide more context on deviations