refactor(core-forger): memory leak, log spam, cleanup

[spkjp]

Proposed changes

Refactors the forger and addresses the following points:

• during startup, don't block when no host is responsive
• host selection now tries all configured hosts before throwing an error
• host selection no longer uses recursion
• no longer spams the log each possible event cycle when no host is available
• uses a timeout to schedule the next __monitor() call instead of promise recursion
• removes obsolete code

Resolves #2326

The memory leak was caused by two things:
(nodejs/node#6673)

1. promise recursion in __monitor()
2. promise recursion in __chooseHost()

The former is the lesser evil and only creeps up slowly over time, but once the forger relay becomes unresponsive the latter makes the memory leak really fast.

(It may be noted that the memory leak as described in #2326 seems to be almost non-present with Node.js 11 in the case of Core despite the provided script leaking in both versions)

Types of changes

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Refactoring (improve a current implementation without adding a new feature or fixing a bug)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Build (changes that affect the build system)
• Docs (documentation only changes)
• Test (adding missing tests or fixing existing tests)
• Other... Please describe:

Checklist

• I have read the CONTRIBUTING documentation
• Lint and unit tests pass locally with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

[codecov-io]

Codecov Report

Merging #2341 into develop will increase coverage by 0.12%.
The diff coverage is 64.38%.

@@             Coverage Diff             @@
##           develop    #2341      +/-   ##
===========================================
+ Coverage    66.01%   66.14%   +0.12%     
===========================================
  Files          397      398       +1     
  Lines         8578     8562      -16     
  Branches       414      412       -2     
===========================================
  Hits          5663     5663              
+ Misses        2870     2855      -15     
+ Partials        45       44       -1

Impacted Files	Coverage Δ
packages/core-utils/src/httpie.ts	0% <ø> (ø)	⬆️
.../core-database-postgres/src/repositories/rounds.ts	75% <ø> (ø)	⬆️
...p2p/src/server/versions/internal/handlers/utils.ts	0% <ø> (ø)	⬆️
...es/core-api/src/versions/1/delegates/controller.ts	42.85% <0%> (+1.19%)	⬆️
packages/core-database/src/wallet-manager.ts	74.17% <100%> (ø)	⬆️
packages/core-database/src/database-service.ts	31.57% <33.33%> (-0.16%)	⬇️
packages/core-forger/src/manager.ts	69.41% <46.87%> (-0.59%)	⬇️
packages/core-forger/src/client.ts	86.84% <84.61%> (+11.33%)	⬆️
packages/core-forger/src/errors.ts	88.88% <88.88%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ee0398b...2d1e6ef. Read the comment docs.

[alessiodf]

@supaiku0

(It may be noted that the memory leak as described in #2326 seems to be almost non-present with Node.js 11 despite the provided script saying otherwise)

I suggest you take a snapshot with heapdump and check it because promise recursion leaks are definitely still present in Node.js v11 and the script is accurate. Even if it doesn't leak as much, it's still misleading to say it's (almost) non-present and may give others looking at this PR the wrong impression that promise recursion is no longer a problem in Node.js v11, because it is:

[spkjp]

I suggest you take a snapshot with heapdump and check it because promise recursion leaks are definitely still present in Node.js v11 and the script is accurate.

Yeah, you are right. The promise leak is still present in Node 11. But in the case of Core I can only reproduce the leak on Node 10. My wording was a bit misleading.

[ghost]

The ci/circleci: test-node11-integration-1 job is failing as of 03b9b5dd5fba0861c6ecc4e0b789a130c1174125. Please review the logs for more information.

Once you've pushed the fixes, the build will automatically re-run. Thanks!

[kristjank]

Referencing this: #2326

[kristjank]

👏