Data Model

[alexsnaps]

The current data model

The data model resides in the limitador crate source and is broken down into these few structs: Namespace, Limit and finally Counter.

• Namespace is just a type alias for String and mostly represents a hostname for which limits will be applied
• Limit is the representation of the definition of a limit: its targeted Namespace, the maximum value for a given time window and a set of conditions. Additionally counters can be qualified using “variables”, e.g. a counter per user identifier. A Limit can be “named” mostly for tracing/debugging purposes.
• Counters are the actual counters for a limit, e.g. for a given user identifier. It also holds the time-to-live (TTL) for the counter, e.g. the counter should be reset in 32 seconds as the time window would have expired by then.

Interestingly these types fulfill different roles and are closer to Data Transfer Objects (DTOs) than an actual data model. Counters are mostly used as a lookup mechanism to their actual values. In the memory store, the actual values are held in the InMemoryStorage.counters’ TtlCache, that cache is also responsible for applying the correct TTL. Redis is responsible for the same TTL enforcement and stores the values as plain integers in a Hash, keyed by Counter. Limits are too mostly used to look Counters up. They are also used as DTOs for the REST API exposed by the server, e.g. when querying for known counters.

Note: The RocksDbStorage is slightly different as the underlying RocksDB doesn’t have built-in TTL support. That feature was built in an ExpiringValue type that handles the expiry explicitly, while holding the current counter’s value.

Things to address

• Avoid doing too many hash lookups with for a single counter #183
• Fix the memory leak InMemoryStore grows unbounded #71
• Come up with a thread-safe, yet more parallel data access pattern to counters
• [cleanup] Introduce proper key'ing type for our Limit & Counter  #328
• Decentralized counter increments when in cross-cluster mode #349
• Freeze data at rest format #365

[eguzki]

@alexsnaps is this still in the scope of v1?