prevent/handle panicked thread/actor

[iwanbk]

During development, sometimes i got this kind of message

thread 'main' panicked at zstor/src/actors/....

and then all the things will be failed with this error

Zstor error: error during waiting for async task completion: Mailbox has closed

Currently it is my code that still under development, but i remember that i got it before, which unfortunately i didn't check it deeper.

Considering Murphy's Law

Anything that can go wrong will go wrong."

Looks like we could improve it a bit by either prevent it or handle it.

• Prevent it: unwrap is one of the main source, and it is a lot in 0-stor-v2 code
• Handle it: i found Actix Supervisor https://docs.rs/actix/latest/actix/struct.Supervisor.html need to check it deeper.
• restart 0-stor

[scottyeager]

I think it would be good to recover in as many cases as possible, using whatever strategy makes the most sense.

But if I understand correctly zstor continues to run after a panic on the main thread? Even if we maximized our chances of recovering, I think it's better to exit than to stay alive in a non functional state. Then at least the process manager can restart the process and work can continue.

[iwanbk]

But if I understand correctly zstor continues to run after a panic on the main thread?

Yes, should be, but the main thread should be very hard to crash. Because the work is relatively simple than the worker thread.

I think it's better to exit than to stay alive in a non functional state. Then at least the process manager can restart the process and work can continue.

agree with this.

[iwanbk]

I've checked the Actix supervisor https://docs.rs/actix/latest/actix/struct.Supervisor.html
Some thoughts:

• There are a lot of learns/researchs to do, because the example in the docs is only simple while our Actor is quite complex with dependencies and restart capabilities
• With that much works, there is still possibility of bug because of the complexity
• there is no really real gain compared to simply abort the whole program and restart it using zinit or systemd

So, i think the best way for now is abort the whole program.
The drawback of aborting is unclean exit, should be fine for now because:

• the user could retry
• open connections to zdb could be cleaned up eventually

[scottyeager]

This sounds good to me. In the context of qsfs, I've also added a script to periodically check for success and retry any failed store operations. Zstor isn't expected to guarantee retries in the face of general failures, like sudden power loss, either.

[iwanbk]

I've also added a script to periodically check for success and retry any failed store operations

how you do this? by checking the logs periodically?

[scottyeager]

how you do this? by checking the logs periodically?

You can see the approach here: https://github.com/threefoldtech/quantum-storage/blob/master/lib/retry-uploads.sh

It's using the check command and comparing the returned hash against a locally computed hash.