ergoCub 1.3 S/N:002 – Weird smell coming from the robot and errors on the right leg

[S-Dafarra]

Robot Name 🤖

ergoCub 1.3 S/N:002

Request/Failure description

A few minutes after startup we started smelling something weird coming from the robot. From the motorgui I noticed that both right ankle roll and pitch were in hardware fault.

Here the full log

log_yarprobotinterface_weird_smell.txt

Detailed context

The robot was just started and in the classical calibration pose, hanging, doing nothing for about 5 minutes.

The first relevant issue seems

[ERROR] from BOARD 10.0.1.7 (right_leg-eb7-j4_5) time=498s 869m 103u :  SYS: EOtheCANservice could not tx frames on CAN bus. CanPort=CAN1. Size of fifo is 1
[WARNING] from BOARD 10.0.1.7 (right_leg-eb7-j4_5) time=498s 869m 231u :  SYS: the TX phase of the control loop has last more than wanted. TX execution time 2244[usec]. Latest previous execution times[usec] (RX=N/A, DO=N/A, TX=118); (Rx=92, DO=222, ...). Num of CAN frames[CAN1,CAN2]=[1 0]
[ERROR] from BOARD 10.0.1.7 (right_leg-eb7-j4_5) time=498s 872m 102u :  SYS: EOtheCANservice could not tx frames on CAN bus. CanPort=CAN1. Size of fifo is 2
[WARNING] from BOARD 10.0.1.7 (right_leg-eb7-j4_5) time=498s 872m 230u :  SYS: the TX phase of the control loop has last more than wanted. TX execution time 2244[usec]. Latest previous execution times[usec] (RX=N/A, DO=N/A, TX=2244); (Rx=308, DO=223, ...). Num of CAN frames[CAN1,CAN2]=[2 0]
[ERROR] from BOARD 10.0.1.7 (right_leg-eb7-j4_5) time=498s 875m 107u :  SYS: EOtheCANservice could not tx frames on CAN bus. CanPort=CAN1. Size of fifo is 3
[WARNING] from BOARD 10.0.1.7 (right_leg-eb7-j4_5) time=498s 875m 235u :  SYS: the TX phase of the control loop has last more than wanted. TX execution time 2248[usec]. Latest previous execution times[usec] (RX=N/A, DO=N/A, TX=2244); (Rx=242, DO=222, ...). Num of CAN frames[CAN1,CAN2]=[3 0]
[WARNING] from BOARD 10.0.1.7 (right_leg-eb7-j4_5) time=498s 875m 765u :  SYS: the RX phase of the control loop has last more than wanted. RX execution time 411[usec]. Latest previous execution times[usec] (..., Tx=0);(RX=0, DO=0, TX=0);

which seems to indicate that there is some issue with the ankle 2foc.

Nonetheless, we could not understand if the smell was coming from the torso or from the leg

Additional context

Again, the smell could have been coming from the torso too.

cc @filippoborgogni @Gandoo @maggia80 @Fabrizio69

How does it affect you?

No response

[valegagge]

I just want to highlight these errors in the log:

• [ERROR] BOARD right_arm-eb3-j2_3 (IP 10.0.1.3) At timestamp 1.73341e+09 In motor 0 cannot read Temperature from I2C. There might be cabling problems, TDB cable might be broken or sensor unreachable [ERROR] BOARD right_arm-eb3-j2_3 (IP 10.0.1.3) At timestamp 1.73341e+09 In motor 1 cannot read Temperature from I2C. There might be cabling problems, TDB cable might be broken or sensor unreachable

• [ERROR] BOARD left_leg-eb8-j0_3 (IP 10.0.1.8) At timestamp 1.73341e+09 In motor 2 cannot read Temperature from I2C. There might be cabling problems, TDB cable might be broken or sensor unreachable

• [ERROR] BOARD left_arm-eb4-j2_3 (IP 10.0.1.4) At timestamp 1.73341e+09 In motor 0 cannot read Temperature from I2C. There might be cabling problems, TDB cable might be broken or sensor unreachable

• [ERROR] BOARD left_leg-eb9-j4_5 (IP 10.0.1.9) At timestamp 1.73341e+09 In motor 0 cannot read Temperature from I2C. There might be cabling problems, TDB cable might be broken or sensor unreachable

• [ERROR] BOARD right_leg-eb6-j0_3 (IP 10.0.1.6) At timestamp 1.73341e+09 In motor 2 cannot read Temperature from I2C. There might be cabling problems, TDB cable might be broken or sensor unreachable

• [ERROR] BOARD left_arm-eb2-j0_1 (IP 10.0.1.2) At timestamp 1.73341e+09 In motor 0 cannot read Temperature from I2C. There might be cabling problems, TDB cable might be broken or sensor unreachable

• [ERROR] BOARD right_arm-eb1-j0_1 (IP 10.0.1.1) At timestamp 1.73341e+09 In motor 0 cannot read Temperature from I2C. There might be cabling problems, TDB cable might be broken or sensor unreachable

• [ERROR] BOARD left_leg-eb8-j0_3 (IP 10.0.1.8) At timestamp 1.73341e+09 In motor 3 cannot read Temperature from I2C. There might be cabling problems, TDB cable might be broken or sensor unreachable

• [ERROR] BOARD right_arm-eb1-j0_1 (IP 10.0.1.1) At timestamp 1.73341e+09 In motor 1 cannot read Temperature from I2C. There might be cabling problems, TDB cable might be broken or sensor unreachable

• etc

Then we have errors like the following one only on motor 0 of eb2 and on motor 0 and 1 of eb3:
[ERROR] BOARD left_arm-eb2-j0_1 (IP 10.0.1.2) Motor 0 failed to read 60001 temperature readings for 87.5094 seconds.

The first type of error stopped when the calibration ended, and start the second type until the eb7 got in trouble with its 2foc.

So, I suppose that the first errors could be false positives, but we have real errors on the above-mentioned motors.

In addition the diagnostic report always the same timestamp. (this sounds like a bug in the diagnostic).

@MSECode what do you think?

log_yarprobotinterface_weird_smell_onlt_temp_error.txt

[MSECode]

In addition the diagnostic report always the same timestamp. (this sounds like a bug in the diagnostic).

That is most likely. We need to check how the timestamp is added to the string.
Other than that bug, which we are gonna solve, I've looked at the txt file with the logs and it is not totally clear to me how the 2foc stopped streaming at a certain point because that is clearly happened. That is visible from the fact that the size of the queue of the CAN messages increased regularly until its maximum, i.e. 64. The cause of this need to be investigated since I did not see any errors on encoders or motors.

[S-Dafarra]

It seems that 2FOC was the source of the weird smell. @filippoborgogni and @edersonferrante are changing it now.

Also, for completeness, the ankle pitch does not have the i2c extender, but the roll yes. Yet, in #1986 it was the pitch going in HF

[Gandoo]

ciao @AntonioConsilvio @S-Dafarra We replaced the 2Foc 7B0 board, cleaned the LRM4 optical disk drive, removed the belt and tested rotation with the box, reconnected everything.
@S-Dafarra checked ankle limits
@AntonioConsilvio The 2Foc cod.3336.K S/N1526 board taken from the eCub toolbox da @filippoborgogni

cc @Fabrizio69 @maggia80 @DanielePucci @edersonferrante

[S-Dafarra]

I left it on for a while and seems to be fine! Thanks a lot!

[S-Dafarra]

Now the ankle pitch goes often in overheating. I wonder if this is related to #1990

[S-Dafarra]

We can still smell a bit of burnt. I think the ankle pitch motor might have burnt

[S-Dafarra]

We can still smell a bit of burnt. I think the ankle pitch motor might have burnt

It was very burnt

[AntonioConsilvio]

Hi @fbiggi and @Gandoo replaced the burnt motor. I calibrated and tested the robot and it works properly now! See the commit:

• icub-tech-iit/robots-configuration@7a0d5b6