[WIP] RFC 1: Legacy Leaderboard

[skippi]

I'm proposing to reinstate the legacy DB ruleset as our first leaderboard.

Rationale

• The legacy DB is a working formula for driving code quality and encouraging gcsim usage.
• Many users are already familiar with the legacy DB's ruleset. Having this as a first leaderboard gives us a "stable" leaderboard to work with until we can build new ones.
• Less work to put together since we already know the general rules for the legacy DB.

[1] Details

[1.1] Potential Names

Note that we aren't limited to a single name, these can be aliases of each other if necessary.

• lb.gcsim.app/legacy
• lb.gcsim.app/kc_gst: KurtC Generic Single Target
• lb.gcsim.app/kc_cst: KurtC Continuous Single Target

[1.2] Management

I can volunteer to maintain this leaderboard for now.

• [1.2.1] All DB sims will qualify for submission if any of the following are true:
  • [1.2.1.1] The config contains a comment of form #:lb-legacy.
    • Submissions made this way will list the author(s) on UI.
  • [1.2.1.2] The config automatically satisfies the rules stated under [1.3].
    • Submissions made this way will not list the author(s) on UI unless [1.2.1.1] is true.
    • Submissions made this way will not need review and will automatically be approved.
• [1.2.2] If a sim qualifies for submission and needs manual review, it will be approved via a separate chat bot in discord channel gcsim#leaderboard-management.
  Whether this is modified to support multiple leaderboards is an implementation detail.
  Commands are listed as follows:
  • [1.2.2.1] /list: list all submissions requiring manual review.
  • [1.2.2.3] /approve [db_id]: approve submission with id [db_id].
  • [1.2.2.3] /reject [db_id] [reason]: reject submission with id [db_id]. Send DM to submitter listing failure and [reason].
  • [1.2.2.4] /info [db_id]: list reasons for manual review. Should be generated from linter.
• [1.2.3] If a sim does not automatically satisfy [1.2.1.2], a list of failed rules will be attached to the submission.
  • I plan on writing a linter for this eventually.

[1.3] Modified Ruleset

New additions to the ruleset are in bold italics.

• [1.3.1] KurtC Weapons (R5 3*, R1 4* limited, R3 4* gacha/starglitter, R5 4* craftable/event)
• [1.3.2] KurtC Artifacts (20 fixed 20 liquid subs, less subs for 4* sets)
• [1.3.3] KurtC Characters (C0 5*, C6 4* limited, C2 4* gacha)
• [1.3.4] KurtC Talents (9/9/9)
• [1.3.5] All submissions must conform to either of the following config structures:
  • [1.3.5.1] Fixed Rotation
    • Must perform a full repeating rotation for at least 100s but no more than 200s.
    • Adjustments to first rotation is allowed.
    • [NEW] Must use TTK-style sim structure to properly track durations. Please see [ttk-def] for glossary definition.
  • [1.3.5.2] APL
    • Must perform a repeating priority-based action list for a reasonable amount of time (Legacy often saw +250s).
  • [1.3.5.3] [NEW] Hybrid
    • Sims may conform to a mixture of both APL and Fixed Rotation. These will be subject to the rules of Fixed Rotation.
• [1.3.6] All submissions must use 1000 iteration count.
• [1.3.7] All submissions must use swap delay of 12.
• [1.3.8] All submissions must have exactly one target.
• [1.3.9] [NEW] All submissions must have reasonable DPS stddev.
  • [1.3.9.1] [NEW] If the sim uses TTK-style, it must have reasonable duration stddev. *
  • The goal of this is to enforce stable configs. There might be a better way to handle this aside from reading stddev.
• [1.3.10] [NEW] All submissions must enable hitlag.
• [1.3.11] [NEW] Only one version of each team will be kept according to DPS and SD.
• [1.3.12] All submissions must passively generate 1 energy particle per random duration between 8-12 seconds.
• [1.3.13] Character Params
  • [1.3.13.1] Only the starting character is allowed to have any stacks of serpentspine.
  • [1.3.13.2] Only non-starting characters are allowed to have any stacks of huskofopulentdreams.
  • [1.3.13.3] yelan is always allowed to have a stack of barb.
  • [1.3.13.4] klee skill cannot attack more than 3 times.
  • [1.3.13.5] ningguang burst cannot have more than 6 self-jades and 6 screen-jades.
  • [1.3.13.6] ayato burst radius cannot exceed 2.
  • [1.3.13.7] ganyu burst radius cannot exceed 2.
  • [1.3.13.8] All characters may start with any percent of their HP from 0-100%.
  • [1.3.13.9] Use of travel less than 10f will be manually reviewed.
  • [1.3.13.10] Use of weakspot=1 will be manually reviewed.
• [1.3.14] Syntax
  • [1.3.14.1] Use of set_particle_delay() will be manually reviewed.
  • [1.3.14.2] Use of wait() will be manually reviewed.
  • [1.3.14.3] Use of if () {}, else () {}, else if {}, or switch {} will be manually reviewed.
  • [1.3.14.4] Use of set_target_pos() will be manually reviewed.
  • [1.3.14.5] Use of set_player_pos() will be manually reviewed.
• [1.3.15] All submissions must be built with a valid git hash.

[1.4] REST Implementation

• [1.4.1] Add new List Leaderboard endpoint for querying data: GET /lb/[name]
  • Example: GET /lb/[name]?limit=50&after=100
    • Fetch pre-sorted configs [101, 150].
  • [1.4.1.1] HTTP Query String Params
    • limit: max number of configs to send (default: 1)
    • after: the last config count in the previous page (default: 0)
    • query: query string to filter configs (default: "")
      • See [1.4.1.2] for syntax.
  • [1.4.1.2] Query String Syntax
    • Consists of a string of commands or words separated by whitespace.
    • Commands are evaluated by left-association.
    • char:[name]: filter configs with character [name].
    • -char:[name]: filter configs without character [name].
    • weapon:[name]: filter configs with weapon [name].
    • -weapon:[name]: filter configs without weapon [name].
    • Unrecognized tokens will be used as a fuzzy search string.

[1.5] DB Implementation

Pending Design until we hash out rest of work.

Glossary

• ttk-def: TTK fixed rotation sims are high HP sims that auto-calculate duration based off the action list. They are usually configured with something like:

  options swap_delay=12 debug=true iteration=1000; # no manual duration
  target lvl=100 resist=.1 hp=10000000; # high HP to prevent death
  
  ...
  
  for let rotation = 0; rotation < 5; rotation = rotation + 1 {
    # do actions
  }
  

[srliao]

create table lb_legacy (
  team_id int primary key,
  db_id serial references db_simulations(db_id) not null unique,
  dps number,
  fuzzy_search_ordinal varchar(255)
);

Main concern on this is that this only allows for one leaderboards. The original concept is that there should be multiple leaderboards. The solution I had in mind was just adding another field to track which leaderboard it is, so something like

create table lb (
  lb_id serial primary key
  -- whatever other info is required
);

create table lb_simulations (
  -- same stuff
  , lb_id integer references lb(lb_id) not null unique
);

Will probably want a couple other helper tables like ones to track which user have maintain role/access to which leaderboard

create table characters (
  id int primary key,
  name varchar(30) not null,
  unique (name)
)

This already exists actually. It's just called avatars (don't ask why).

The modified ruleset I'm not concerned about because, like I originally wrote on discord, my intention was to let whoever is the maintainer decide the rules for each of the leaderboard they are responsible for.

Outside of that I guess most of this is already in line with what's in place. The challenge is really putting in the time to write out the backend that can handle all those REST end points.

An unfortunate side effect with server side filtering though is that it's not something I can (easily) cache away on cloudflare to reduce server workload. This is something I want to think about still. Perhaps I can do the filtering on cloudflare instead (more scalable) or alternatively force the client to do the filtering. If forcing client to do the filtering, will just have to do it in such a way that it doesn't require the user to sit there for a couple second on the initial load.

[soloxcx]

[1.3.11.4] klee skill cannot attack more than 3 times.
[1.3.11.5] ningguang burst cannot have more than 6 self-jades and 6 screen-jades.

I'm not sure how many params are unbounded but I think it would be a pain to find them all and list them in the ruleset. Maybe just a general rule can cover them all like "[1.3.11.4] All params must be bounded to what is possible in game i.e you cannot use klee skill[bounce=50, mine=50], at most klee skill[bounce=3, mine=8]."

edit:

[1.3.9] [NEW] All submissions must have reasonable DPS stddev.
[1.3.9.1] [NEW] If the sim uses TTK-style, it must have reasonable duration stddev. *

Something to consider is that WoW's SimC calculates a "dps variance percentage" which is stddev/mean dps. So a team with 50k dps and 1500 stddev would evaluate to 3% "dps variance percentage." I'm not educated in statistics to know if this value has meaning beyond an arbitrary comparison to other dps variance percentages but it may be worth being part of the rule, like "your stddev/mean cannot exceed x%" -- we'd still have to determine what % is deemed reasonable. This may help narrow down what configs to reject based on dps variance alone

[skippi]

I'm listing out the rules for transparency. We need them if we want to deterministically figure out if a sim is faulty (EG, automatic checking).

I agree with looking at DPS variance percentage for a potential check though. Will report back when I learn more about it.

[Charlie-Zheng]

Can you also add a definition for what the Sara burst hits and cluster buffs should look like?