[P078] - Eastron - SoftwareSerial - ESP8266_4M1M - failing

[LeeNX]

Using ESP_Easy_mega_20230623_energy_ESP8266_4M1M.bin on two older NodeMCUs with Software Serial is returning nan, where rolling back to ESP_Easy_mega_20221224_energy_ESP8266_4M1M.bin restores working operation.

Under device, task using plugin P078, I am seeing Checksum (pass/fail): 1/5193. I have disabled all controller and plugins and still not getting the newer of the firmware working.

I have setup, but disable for testing openhab mqtt controller. Both NodeMCUs have Sysinfo. One device has BMx280 on I2C and the other DHT22 temp sensors.

Not sure what else is useful info. Not urgent, as I have rolled back, but have a testing unit that I can/will test newer firmware on.

Checklist

• The title of this issue is "to the point" and descriptive.
• This issue describes what is happening.
• This issue describes what components are affected (e.g. name of plugin/controller)
• This issue describes how to reproduce it.
• This issue describes when it was introduced (when known) and what version is now showing the problem.

I have...

• searched the issue tracker or the forum for a similar issue. (include links when applicable)
• entered a system description using "Copy info to clipboard" on the sysinfo page. (when possible)
• entered the full filename of the used version (e.g. ESP_Easy_mega-20181001_test_ESP8266_4096_VCC.bin )
• given a list of active plugins or controllers when applicable.
• filled out all applicable fields below.

Steps already tried...

• Tried a clean install (empty .bin files are included in the ZIP)
• Tested previous/other build (mention which one already tested)
• Tested on other node to make sure hardware isn't defective.
• Verified if the problem is limited to a single plugin/controller

If you self compile, please state this and PLEASE try to ONLY REPORT ISSUES WITH OFFICIAL BUILDS!

Summarize of the problem/feature request

YOUR TEXT GOES HERE

Expected behavior

YOUR TEXT GOES HERE

Actual behavior

YOUR TEXT GOES HERE

Steps to reproduce

System configuration

Hardware:

ESP Easy version:

ESP Easy settings/screenshots:

Rules or log data

[TD-er]

What is the 'success rate' on that setup using the "working" older firmware? (good/fail ratio of packets)
What baudrate are you using?
Which pins are being used for SW serial?
Is "Use Serial Console" enabled on the tools->Advanced page on the newer firmware and if so, what serial port is selected there?

[LeeNX]

Thanks @TD-er , them are all good questions. I did not want to flood the issue with info that might not be useful.

Both devices are running five task, three which are P078

Test - test unit has an SDM120C - 2400 baud - Checksum (pass/fail): 3936/0

Edit	1	✔	Generic - System Info	SysInfo		❶		Uptime: 160.00 RSSI: -74.00 SystemLoad: 23.93 WebActivity: 117.18
Edit	2	✔	Environment - BMx280	BMx280	I2C	❶	SDA: GPIO-4 (D2)SCL: GPIO-5 (D1)	Temperature: 16.58 Humidity: 0.00 Pressure: 1027.29
Edit	3	✔	Energy (AC) - Eastron SDM120C/220T/230/630	SDM120C	SoftwareSerial	❶	RX: GPIO-13 (D7)TX: GPIO-12 (D6)	V: 236.50 A: 0.72 W: 163.50 cos_phi: 0.98
Edit	4	✔	Energy (AC) - Eastron SDM120C/220T/230/630	SDM120P	SoftwareSerial	❶	RX: GPIO-13 (D7)TX: GPIO-12 (D6)	Hz: 50.04 Wh_imp: 4728.78 Wh_exp: 0.00 Wh_tot: 4728.78
Edit	5	✔	Energy (AC) - Eastron SDM120C/220T/230/630	SDM120O	SoftwareSerial	❶	RX: GPIO-13 (D7)TX: GPIO-12 (D6)	VA: 167.55 VAr: -35.30 W: 163.70 Wh_tot: 4728.78

Live unit has an SDM220C - 9600 baud - Checksum (pass/fail): 5736/0

Edit	1	✔	Generic - System Info	SysInfo		❶		Uptime: 268.00 RSSI: -59.00 SystemLoad: 17.48 WebActivity: 499.08
Edit	2	✔	Environment - DHT11/12/22 SONOFF2301/7021	DHT22		❶	GPIO-13 (D7)	Temperature: 19.20 Humidity: 46.10
Edit	3	✔	Energy (AC) - Eastron SDM120C/220T/230/630	SDM220C	SoftwareSerial	❶	RX: GPIO-12 (D6)TX: GPIO-14 (D5)	V: 238.31 A: 0.14 W: 34.34 cos_phi: 1.00
Edit	4	✔	Energy (AC) - Eastron SDM120C/220T/230/630	SDM220W	SoftwareSerial	❶	RX: GPIO-12 (D6)TX: GPIO-14 (D5)	Hz: 50.15 Wh_imp: 34095.60 Wh_exp: 211.04 Wh_tot: 34306.64
Edit	5	✔	Energy (AC) - Eastron SDM120C/220T/230/630	SDM220O	SoftwareSerial	❶	RX: GPIO-12 (D6)TX: GPIO-14 (D5)	VA: 34.38 VAr: 0.00 W: 34.34 Wh_tot: 34306.64

Old settings

Serial Settings:
Enable Serial port: [x]
Baud Rate: 115200

New settings

Serial Settings:
Enable Serial port: [x]
Baud Rate: 115200
Serial Port: HW Serial0: GPIO-3 (D9) <- TX / GPIO-1 (D10) -> RX

Thou I only use the Serial Console for provisioning and debugging.

Would the timing stats be helpful in any way?

[TD-er]

I got a hypothesis about this, still need to verify....
In the most recent changes to the serial stuff (read: complete rewrite of all related to serial) I did add support for Software Serial to ESP32.
However for this I needed to update the SWserial library.
For ESP8266 this meant an increase in build size, so for ESP8266 I made it a bit more 'dynamic'.
On builds made with LIMIT_BUILD_SIZE defined, I still use the older version of this library and on the other builds I use the latest version.

I think the 'energy' build on ESP8266 is not having LIMIT_BUILD_SIZE defined and thus using the latest version of SW serial.

Just wondering, why aren't you using GPIO 13 and 15 for this Eastron device? You need an extra PNP transistor for it (because GPIO-15 is pulled down and must be pulled down at boot), but it takes a lot less resources to use HWserial0 swapped compared to SW serial.
SW serial triggers so many interrupts it will also affect WiFi performance/stability when doing a lot of I/O, especially at lower baud rates.

So you could also try to set the unit to at least 9600 baud.

Anyway, I will look into the SW serial issue.

[TD-er]

Hmm apparently I already reverted that so all ESP8266 builds should now use the older SWserial library:

#ifndef USES_SW_SERIAL
# ifndef DISABLE_SOFTWARE_SERIAL
#  define USES_SW_SERIAL 1
#  ifndef USES_LATEST_SOFTWARE_SERIAL_LIBRARY
#   ifdef ESP32
#    define USES_LATEST_SOFTWARE_SERIAL_LIBRARY 1
#   elif defined(ESP8266)
#    define USES_LATEST_SOFTWARE_SERIAL_LIBRARY 0
#   else // ifdef ESP32
#    define USES_LATEST_SOFTWARE_SERIAL_LIBRARY 1
#   endif // ifdef ESP32
#  endif // ifndef USES_LATEST_SOFTWARE_SERIAL_LIBRARY
# else // ifndef DISABLE_SOFTWARE_SERIAL
#  define USES_SW_SERIAL 0
# endif // ifndef DISABLE_SOFTWARE_SERIAL
#endif // ifndef USES_SW_SERIAL

Is it possible for you to set the Eastron unit to 9600 baud?
2400 baud really is keeping interrupts active too long for SWserial

[LeeNX]

Can't really change the baud rate, without using the setup until, if I remember correctly? But, my main unit is set at 9600 and had the same problem, so I don't think that would help much.

As for the reason for not using the hardware serial, was that I was debugging in the past and did not think I could use both serial console for debugging and ModBus Eastron meter. Can say that it's not the most stable setup, but these meters were only used for reporting and the meter maintains the metrics, not perfect, but gave me what I needed at the time.

Not sure if this is worth debugging? Would anybody else want to use SW Serial on ESP8266 devices with these meters? Would it work better on ESP32?

I can test and debug firmware, but I think I am going to work toward removing these devices as the solar inverter gives me the similar data that I can run my automation with.

[TD-er]

Yep ESP32 does have upto 3 hardware serial ports.

I'm personally quite pleased with those small ESP32-C3 units with external antenna. (Seeed Studio XIAO - ESP32-C3)
Really small and quite responsive.
See the different versions of ESP32 here: https://espeasy.readthedocs.io/en/latest/ESPEasy/ESPchips.html

Those ESP32-variants with native USB allow to have the ESPeasy console via USB, so that doesn't cost you a hardware serial port.

Better not use the ESP32-S2 as that one is a bit strange regarding native USB support.
It does work, but I had to jump through a lot of hoops to get it to work.
But if you have one laying around and it has enough HW serial ports with an external USB to UART chip, go for it.

[TD-er]

Here my extremely professional test bench setup....

I have set it up on GPIO 12/13/14

It is running a build I made yesterday and it should be the same as this GH Actions build: https://github.com/letscontrolit/ESPEasy/actions/runs/5728205521

But just to be sure, here is the exact custom build I am using

I did not get it to work without a separate linke between a GPIO pin and both DE/RE pins.

So I will take a look at the signalling as it should also work with DE/RE connected to the TX pin.

N.B. I have the MAX485 chip running at 5V here in my test setup. When using these GPIO pins, it should make no difference as these pins should be able to handle 5V.
Not sure if GPIO 1/3 will be able to handle 5V signals as those are connected to the USB to serial chip.

[TD-er]

Just took some measurements on the RO pin (connected to the ESP RX input) and its signal level is the same as the Vcc voltage of the MAX485.

N.B. using a battery powered scope for this, just to be sure.

N.B.2 I also received my isolation transformer for testing stuff like these kWh meters and working on mains power connected stuff like Sonoff units.

[TD-er]

When this test build is ready, you can try testing as I'm quite confident about how it is now working :)

Tested on ESP32 and ESP8266 board.

• SDM120
• SDM220
• SDM630

They are all 3 connected to the same board and working just fine.

Tested using

• SW serial (on ESP8266)
• HW Serial (on ESP8266) with software DE/RE pin
• HW Serial (on ESP32) with hardware controlled DE/RE pin :)

Meaning that on ESP32 we don't even need to generate the low signal during TX transfers anymore as I now use the RTS signal to generate this exactly when we're sending data.
Extremely stable timings :)

As a bonus, I now also changed the SDM library to allow for async communications.
ESPEasy now creates a list of registers (and Modbus addresses) to read.
This is done on every PLUGIN_TEN_PER_SECOND, so when all modules are set to 9600 baud, you can read upto 10 registers per second.
This is internally used to keep track of the peak values (when Stats are enabled).
Also the interactions with the Eastron modules only take 13 - 14 msec per register read, instead of 2.5 seconds it was before during a call to PLUGIN_READ
When a module is not connected or not replying, it doesn't hold back the ESPEasy node anymore. It used to block ESPEasy close to 8 seconds, effectively rendering access to the node impossible.

I also reduced the build size, while maintaining all registers for all Eastron modules out there.

There are now commands in ESPEasy to set/change baud rate and Modbus address for those modules with only a single button (e.g. SDM120)

See the documentation

[LeeNX]

Downloading and will test in the next few days, as I have not yet had a chance to change the hardware.

Thanks @TD-er , you are a rock star!

[LeeNX]

Custom build ESP_Easy_mega_20230801_energy_ESP8266_4M1M.bin. I did have to ungzip the bin and it seems like the setting option/command is not working, but will look further tomorrow.

Using 2400 baud on SDM120 is working at the moment, tomorrow I will try and bump the baud to 9600.

[tonhuisman]

I did have to ungzip the bin

When using the OTA feature you can select ether the .bin.gz or the .bin file, the OTA feature knows how to handle both.

[TD-er]

it seems like the setting option/command is not working, but will look further tomorrow.

Please have a look at the documentation of these new commands: https://espeasy.readthedocs.io/en/latest/Plugin/P078.html#set-baudrate-id
When changing the baud rate, you need to press the button first for 3 seconds and after setting the baud rate you need to press the button again (I think, not 100% sure as I have tried several times myself, so not entirely sure which steps are absolutely needed)

Also the changed baud rate is only active after a power cycle of the Eastron module.
And don't forget to call Eastron,pause before sending the command to change baud rate and/or ID.

I know it is working as I have done it myself on a SDM120 which only has a single button and thus only allows changing settings via Modbus.

[LeeNX]

Thank @TD-er , I did find the new docs, which are awesome.

Using Eastron,pause, from the web interface, I am getting Command unknown: Eastron,pause.

Node has been running since last night, which now has an uptime of 10h+ with only Checksum (pass/fail): 15924/1, using the default 2400 baud. Seems way more stable than it's been in the past, but this is just one day.

[LeeNX]

I did have to ungzip the bin

When using the OTA feature you can select ether the .bin.gz or the .bin file, the OTA feature knows how to handle both.

Thanks @tonhuisman , might be I am using OSX? But I can't select .bin.gz firmware files.

Thou it's FireFox, don't know if that should be a problem?

[TD-er]

The .bin.gz files (only for >1M ESP8266 builds) are exactly the same as the .bin files.
Only they are compressed using GZip.
When performing OTA updates, you can also use those .bin.gz files, which will then be decompressed on the fly while flashing.
This did help me a few times on nodes which were either very busy, or had really bad WiFi connection as it could mean a higher success rate when flashing.
But it is by no means required to use the .bin.gz files, nor does it have any advantage other than just being a smaller file during flashing.

[TD-er]

Thank @TD-er , I did find the new docs, which are awesome.

Using Eastron,pause, from the web interface, I am getting Command unknown: Eastron,pause.

Node has been running since last night, which now has an uptime of 10h+ with only Checksum (pass/fail): 15924/1, using the default 2400 baud. Seems way more stable than it's been in the past, but this is just one day.

Hmm have to look at the code, perhaps I forgot to add a success = true; on handling those commands.
You can simply see whether the reading had paused, as the read counter will not increment.

You can still keep using the 2400 baud rate, but in my tests here, I noticed the SW serial gave way more often a watchdog reset when using SW serial at such a lower baud rate.
N.B. when using the core SDK 3.1.x library, it was more stable running SW serial, but had significant less free memory and resulted in larger build files. Thus decided not to set those 'energy' builds to use the latest Arduino core libs.

[tonhuisman]

Hmm have to look at the code, perhaps I forgot to add a success = true; on handling those commands.

The code is correct.
Not sure @LeeNX what exact build you are running? These commands have been added quite shortly before the code was merged. Best build to use would be from this mega merge Actions run

[tonhuisman]

might be I am using OSX?

Well, I don't have OSX available, but the file-browser does get *.bin;*.bin.gz as the file selection spec, (both Chrome and Firefox) so it might indeed be an OSX thing 🤔. But as TD-er explained, no issue.

[LeeNX]

Thanks for the info @TD-er
Thanks @tonhuisman for the update. I think I was using the suggested custom test build from the 1st of August - ESP_Easy_mega_20230801_energy_ESP8266_4M1M Aug 1 2023, let me try the workflow build from 8th August.

Just checked on another 8266 node, gz firmware not selectable in the firmware upload dialog from my Mac. Also noted that there is not bin for the energy collection, need to ungzip, unlike some of the other collections. Not sure if this is an oversight or a bug?

[LeeNX]

Tried on none energy 8266 node, using Chrome and still can select gz firmware. Just in case I am been a newbie

Not the end of the world, just noting.

[tonhuisman]

Also noted that there is not bin for the energy collection, need to ungzip, unlike some of the other collections. Not sure if this is an oversight or a bug?

That it's not supported on a Mac is most likely a Mac-thing... 😸

I just downloaded the .zip file from the link I gave you and opened it:

In the 20230801 zip you show it was probably too large to be accepted, but the .bin.gz was accepted, TD-er did put in a lot of work to reduce the size of that and other .bin files, but most of that work went in another PR, that was merged somewhat later, and that also has some more improvements to P078 😮
NB: The commands to pause, resume, etc. where added on August 6...

[LeeNX]

Thanks @tonhuisman . I don't know it's not supported on Mac, almost sure I have used the dialog box to upload zips.

Some local testing, seems like <input type="file" accept=".bin,.bin.gz" name="firmware"> is not a defined behaviour, <input type="file" accept=".bin, .gz" name="firmware"> works, but does not limit to only .bin.gz, as two levels of file extension is not defined or at least not listed on the few sites I did look at - https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/file#accept
I try and few ideas of escaping, but I did not get anything to work other than the .gz

I did see that the bin file for energy was included in 08th August build, did only check after I had posted. Oh boy, my test unit did not want to flash the new firmware. Had to wait 15min after cold boot before the firmware flashing worked.

I see the eastron command in the plugin (tried coping and pasting from there), but it still does not work on 8th August build.

Can I use the commands in the tools page command input field?

Not to make firmware bigger, but adding eastron status/settings/info command to make sure it works an idea? I see there is some commented out logging.

Ran into another odd issue, in Devices -> Task Settings updating the Values -> Name for the P078 does not seem to save. Changing the Decimals or enabling the Stats does seem to save.

[tonhuisman]

Ran into another odd issue, in Devices -> Task Settings updating the Values -> Name for the P078 does not seem to save. Changing the Decimals or enabling the Stats does seem to save.

That is caused by the selectors for the output Values, they overwrite with their default names on save, AFAICS.

Can I use the commands in the tools page command input field?

Yes, you can type any command there, including the Eastron command. Only requirement is that the plugin (that handles the command if it's not an internal/core command) is Enabled.

[tonhuisman]

I try and few ideas of escaping, but I did not get anything to work other than the .gz

It seems that OSX is somewhat picky here. This SO answer lists the requirements quite clearly, but the description for 'starts with a period then a file-extension' is open for interpretation, and OSX obviously does that different from Windows (haven't tried on Linux, as I don't use Linux machines that have a GUI...).
No need to do any further investigation, I think, as the Updater page for both ESP8266 and ESP32 are hidden in the Arduino libraries, and are quite hard to override. (That's also why they don't follow the dark theme setting...)

[LeeNX]

Thanks, Values -> Name that can't be set or are not unique, seems like a bug. Looking at my mqtt and seeing that I am only getting one zero value for three reading. I could work around this by using different tasks.

[TD-er]

And what if you change the value names yourself and press submit?
These 'kWh' names are just being generated.
I wasn't sure whether I should make more descriptive default names or not...
At least on 3-phase modules, the phase is added to the default name.

[LeeNX]

I did try change the names and hit submit, but the return page and devices keeps returning to the value between ( ).

Other plugins work as expected with updated names.

[TD-er]

Check!
Will have a look to see what I might have done wrong here...

[LeeNX]

I see PR #4763, I will wait for a new release and test again. Thanks!

[TD-er]

Ton already noticed some issue with that PR, probably related to caching on ESP32, so it is not 100% fixed yet...

[TD-er]

It is all working well in the latest build, so this one can be closed

From left to right...

• 2x Sonoff THR316D
• 5V/2A DIN power supply
• 3D printed D1 mini DIN rail case Thingiverse link
• Eastron SDM120
• Eastron SDM220
• TAC1100 (Eastron SDM120 clone)