Enhancement displaying jpeg possible?

[KjellVerb]

Enhancement Summary

display image

Detailed Description

Would it in any way be possible to download and display an image from an http server or does the NSpanel architecture with the nextion display prevent this?
It would be awesome to have the panel display a photo from the doorbell camera whenever somebody rings it.

Additional Context

No additional context...

[edwardtfn]

Please take a look at this:

• Image display possible #1270

Technically, transfer an image is feasible, however the settings used by Nextion isn't the best for this.
Currentlyl we are communicating to the display at 115200bps, and if we transfer pixel by pixel for a 480x320 pixels, 16bit color, it would take at best more than 20sec (just the theoretical communication without any error control). But the normal Nextion protocol requires several bytes per pixel.
We can increase this rate (up to 921600bps is supported) but will still a slow process, most likely driving to buffer overflows (already an issue today with the normal pages).
Another option would be converting the image to more advanced geometry, as Nextion accepts lines, circles and rectangles so we could reduce the communication when areas of the picture have the same color, however this will require way more advanced logic in the Blueprint (or ESPHome) side, and I'm not sure I can do it without additional hands. 😩

[KjellVerb]

Apologies that I did not find the case you referenced. The necessary transfer time would make it useless anyway for the case that I proposed. Thanks for your answer and I will close this case now.

[edwardtfn]

No need for apologies, please. :)

I still wanna find some time to look at this anyways. I do consider changing the baud rate to 921600bps, but that would require tons of tests.

[KjellVerb]

Were there any signal integrity measurements executed on the UART? I could take a look at it to see whether reliable 921600bps would be achievable.

There is limited info on dynamic image display but it seems like this should be possible: https://community.home-assistant.io/t/upload-an-image-to-nextion-intelligent-series/403764
I'm happy to collaborate but not very skilled at SW design as a HW engineer.

[KjellVerb]

Collecting some more relevant links here in case I or anyone else feels like picking this up.
https://www.youtube.com/watch?v=hTtUgQ8Vkek
https://www.youtube.com/watch?v=JLrLsmf9c4M
https://www.youtube.com/watch?v=nKL0M23c_HU

[edwardtfn]

Collecting some more relevant links here in case I or anyone else feels like picking this up.

Thanks for sharing!! I wanna go deep on this some time in the future, but would be really nice if someone else could take a look sooner.

About the links you shared, these are based on twfile command, which is not available on the Nextion model used on NSPanel. 😞

I believe for this model the best approach would be using advanced reparse mode to implement a protocol to transfer initial coordinates and a sequence of colors, and process it into the display itself to plot each pixel. With this we reduce a lot the amount of bytes needed on the transfer and therefore could make it possible to transfer in a reasonable amount of time.
I'm also working on changing the default baud rate to 921600 and, if this works fine, the transfer time would be 1/8 of what it is today.

[KjellVerb]

I'd love to support you in this investigation but I'm not sure I have the skills :-)

[edwardtfn]

Maybe you can help with specifications and testing at least. 😉

My idea was to have an action to display an image where the user provides a url for that image.
ESPHome should then download that image and convert it to a matrix of colors on it's PSRAM, then transfer all the pixels to Nextion, which will open a specific empty page, process that info and plot the pixels in the screen...

What should be the maximum time acceptable from sending the action from HA to the image being completely visible on the panels display?

[KjellVerb]

I did some further thinking about the specification:

• what type of image file can be supported --> resolution, extension, compression, ...?
• can we add a "tap screen to refresh" function? Or just in general an event when tapping this specific page that we can capture in HA?
• For me personally the maximum time would be in the order of 5 seconds. That feels like an acceptable time between the doorbell being pressed and me seeing who is there. We could also try some fancy stuff by scanning through the pixels in a certain order, for instance first plotting every odd row or column and then in a 2nd pass filling in the missing lines, that way a blurrier image is already visible in half the time.

[edwardtfn]

My idea is to start with the minimum functional solution, then improve it over time...

• what type of image file can be supported --> resolution, extension, compression, ...?

I would try to use as much as possible the Online Image component currently available on ESPHome to transfer the image to PSRAM and then get the color of each pixel to be sent to Nextion (to be developed).
I'm not sure which formats are supported, but all their docs mentions PNG only. Anyhow, if adding support to additional formats, I woukd prefer to add on that component, and not an implementation specifically for NSPanel.

• can we add a "tap screen to refresh" function? Or just in general an event when tapping this specific page that we can capture in HA?

This wouldn't be hard... Even to pass a refresh interval as a parameter when calling the action to plot the picture.

• For me personally the maximum time would be in the order of 5 seconds. That feels like an acceptable time between the doorbell being pressed and me seeing who is there. We could also try some fancy stuff by scanning through the pixels in a certain order, for instance first plotting every odd row or column and then in a 2nd pass filling in the missing lines, that way a blurrier image is already visible in half the time.

Ok, I think I will start trying to make this working within 5s. It should be feasible, but must be tested as many things could interfere (including the processing limitations on both sides).
After having it functional, improve it to interlacing lines and/or columns shouldn't be hard.

[KjellVerb]

I would try to use as much as possible the Online Image component currently available on ESPHome to transfer the image to PSRAM and then get the color of each pixel to be sent to Nextion (to be developed).
I'm not sure which formats are supported, but all their docs mentions PNG only. Anyhow, if adding support to additional formats, I would prefer to add on that component, and not an implementation specifically for NSPanel.

For now I would indeed start with PNG as that seems to be the only one currently supported by that ESPHome component. Handling conversion and even rescaling in HA is probably the best starting point. Then your code can assume it gets an image with the correct extension and size.

One possible option to handle the HA "preprocessing" of the image is to create a shell script that calls imagekick:
sudo apt-get install imagemagick
convert -resize 480X320 source.jpg dest.png