[Milestone] Waku Network can Support 1 Million Users

[fryorcraken]

2023 Key Milestones: https://notes.status.im/s/iylE6wdli#

Milestone: https://github.com/waku-org/pm/milestone/4

Priority Tracks: Secure Scalability

Summary

Enable Status Communities to onboard 1 million users (100 Communities, 10k users per communities).

The completion of this issue assume the usage of static sharding.
This is a continuation of #12, 2 main items are to be delivered:

1. Vac/DST Simulations

Current simulations of the Waku Relay network with 1000 nodes are positives and confirm that Waku with static sharding is likely to support 10k users (10 shards, 1k users per shards).

To confirm the support of a million user, the simulations need to include 10k nodes (10k active users per shards).

Similarly to the simulations done so far, the Waku relay protocol is the priority.
Once done, simulation with Store (using PostgreSQL) and light/filter are needed to confirm that these protocols and their implementation can scale to the need of Status.

This milestone also include any work to be done as a result of the simulation if any of improvement is needed.

2. PostgreSQL Optimization

As part of #12, a PostgreSQL implementation of Waku archive was done in nwaku and local stress testing is tracked to confirm the gain over SQLite.
As for any SQL DB usage, further optimization are possible and tracked as part of this milestone.
This may include feedback from Status community dogfooding or DST simulations.

Justification

Status Communities uses Waku for communication purposes and needs the re-assurance that Waku can scale up to 1 million users.

Risk

Note that a number of optimisation of the usage of Waku by Status Community protocol have been identified: vacp2p/research#177

While this milestone scopes the delivery of Waku's scalability, the resolution of the these items above will likely be needed to enable Status Communities to scale up.

[fryorcraken]

Weekly Update

• achieved: See 10k milestone update for PostgreSQL status.
• risks:
  • Dependency on Vac/DST to run 10k nodes simulations. Tracked under
    vac:dst:eng-10ktool.
  • Wakutorsis tool is being dropped, meaning new tooling needs to be developed for 10k nodes simulations. It is currently uncertain whether such tool can be developed.
  • Large scale simulations done by Vac/DST only covered nwaku relay. go-waku, status-go simulations are not planned short term (theoretical review of Status Communities messages is), nor are simulations including request-response protocols such as store and filter.

[fryorcraken]

Weekly Update

• achieved:
  • See 10k milestone update for PostgreSQL status.
  • First version of the 10k-tool by DST is ready and is being tested with simulation running a small nim-libp2p/gossipsub binary.
• risks:
  • Dependency on Vac/DST to run 10k nodes simulations. Tracked under
    vac:dst:eng-10ktool.
  • Wakutorsis tool is being dropped, meaning new tooling needs to be developed for 10k nodes simulations. It is currently uncertain whether such tool can be developed.
  • Large scale simulations done by Vac/DST only covered nwaku relay. go-waku, status-go simulations are not planned short term (theoretical review of Status Communities messages is), nor are simulations including request-response protocols such as store and filter.
  • lack of real world feedback/dogfooding: the complete static sharding solution involves significant changes to the Waku protocol and tech stack. Although each element is unit tested, dogfooding may hit corner cases in the integrated solution that cannot be foreseen/recreated in lab conditions.

[jm-clius]

Weekly Update

• achieved:
  • Basic Postgresql optimizations completed and benchmarks published. See 10k milestone.
  • First version of the 10k-tool by DST is ready and is being tested with simulation running a small nim-libp2p/gossipsub binary.
• risks:
  • Dependency on Vac/DST to run 10k nodes simulations. Tracked under
    vac:dst:eng-10ktool.
  • Wakutorsis tool is being dropped, meaning new tooling needs to be developed for 10k nodes simulations. It is currently uncertain whether such tool can be developed.
  • Large scale simulations done by Vac/DST only covered nwaku relay. go-waku, status-go simulations are not planned short term (theoretical review of Status Communities messages is), nor are simulations including request-response protocols such as store and filter.
  • lack of real world feedback/dogfooding: the complete static sharding solution involves significant changes to the Waku protocol and tech stack. Although each element is unit tested, dogfooding may hit corner cases in the integrated solution that cannot be foreseen/recreated in lab conditions.

[jm-clius]

Weekly Update

• achieved:
  • Closed last Postgresql issue for basic Store scalability. See 10k milestone.
  • Assisted DST in setting up initial tests with the ~1K tool. Currently still fine-tuning parameters, ensuring results are consistent, etc. for smaller configurations.
• risks:
  • Dependency on Vac/DST to run 10k nodes simulations. Tracked under
    vac:dst:eng-10ktool.
  • Wakutorsis tool is being dropped, meaning new tooling needs to be developed for 10k nodes simulations. It is currently uncertain whether such tool can be developed.
  • Large scale simulations done by Vac/DST only covered nwaku relay. go-waku, status-go simulations are not planned short term (theoretical review of Status Communities messages is), nor are simulations including request-response protocols such as store and filter.
  • lack of real world feedback/dogfooding: the complete static sharding solution involves significant changes to the Waku protocol and tech stack. Although each element is unit tested, dogfooding may hit corner cases in the integrated solution that cannot be foreseen/recreated in lab conditions.

[fryorcraken]

Weekly Update

• next:
  • Pending DST simulations of 10k nodes gossipsub network.
• risks:
  • Dependency on Vac/DST to run 10k nodes simulations. Tracked under
    vac:dst:eng-10ktool.
  • Wakutorsis tool is being dropped, meaning new tooling needs to be developed for 10k nodes simulations. It is currently uncertain whether such tool can be developed.
  • Large scale simulations done by Vac/DST only covered nwaku relay. go-waku, status-go simulations are not planned short term (theoretical review of Status Communities messages is), nor are simulations including request-response protocols such as store and filter.
  • lack of real world feedback/dogfooding: the complete static sharding solution involves significant changes to the Waku protocol and tech stack. Although each element is unit tested, dogfooding may hit corner cases in the integrated solution that cannot be foreseen/recreated in lab conditions.

[jm-clius]

Weekly Update

• achieved:
  • Researched various ways of deploying shared postgresql instances for large deployments: store-db add more appropriate db settings for current db hw status-im/infra-status#37
• next:
  • Pending DST simulations of 10k nodes gossipsub network.
• risks:
  • Dependency on Vac/DST to run 10k nodes simulations. Tracked under
    vac:dst:eng-10ktool.
  • Wakutorsis tool is being dropped, meaning new tooling needs to be developed for 10k nodes simulations. It is currently uncertain whether such tool can be developed.
  • Large scale simulations done by Vac/DST only covered nwaku relay. go-waku, status-go simulations are not planned short term (theoretical review of Status Communities messages is), nor are simulations including request-response protocols such as store and filter.
  • lack of real world feedback/dogfooding: the complete static sharding solution involves significant changes to the Waku protocol and tech stack. Although each element is unit tested, dogfooding may hit corner cases in the integrated solution that cannot be foreseen/recreated in lab conditions.