existing Sming 4.2 based app - OTA to 5.1

[pljakobs]

so, I'm getting to the point of releasing my updated firmware to the first testing community. There are a few 1000 devices out there with the old, Sming 4.2 based firmware that this shall replace. Many of those users do not have the means to flash the device, in many places, the devices are hard to access, so upgrading through OTA is a must.

I've read up on the PRs when adding partition table support to rBoot and where to put them but I'm still unclear on a few things.

What I have now:
the original OTA code only knows of the ROM and SPIFFS memory blocks, it will get those and write those, so it will neither change rBoot nor will it write a partition table.
As for now, the OTA fails, failing back to the old code, so not switching ROMs, I suppose (not a whole lot of debug output on the old code).
this is the boot code output:

rBoot v1.4.2 - [email protected]
Flash Size:   32 Mbit
Flash Mode:   QIO
Flash Speed:  40 MHz
rBoot Option: Big flash
rBoot Option: RTC data

Booting rom 0.
ota1 not set
ota2 not set
rf cal sector: 1019
freq trace enable 0

flashing the new layout yields

rBoot v1.4.2 - [email protected]
Flash Size:   32 Mbit
Flash Mode:   DIO
Flash Speed:  40 MHz
rBoot Option: RTC data

Found 'rom0' @ 0x00002000, size 0x000f8000, subtype 0x10
Found 'rom1' @ 0x00102000, size 0x000f8000, subtype 0x11
Booting rom 1 at 102000, load addr 17b2a0.
ota1 not set
ota2 not set
V2
Mo

rf cal sector: 1019
freq trace enable 0

My key question here is: how do I get from the original state to the new rBoot + partition table layout? I suppose the best way is a two step upgrade?
Have a small ROM to OTA into from the old firmware that does nothing but OTA pull the four images for rBoot, ROM, spiffs and partitionTable and write those?

I'm still wondering why my full build, when OTA'd, won't boot when the boot loader has not changed.

[mikee47]

I'm still wondering why my full build, when OTA'd, won't boot when the boot loader has not changed.

If there's no valid partition table then boot will fail. See
https://github.com/SmingHub/Sming/blob/develop/Sming/Arch/Esp8266/Components/esp8266/startup.cpp#L47.

My key question here is: how do I get from the original state to the new rBoot + partition table layout? I suppose the best way is a two step upgrade? Have a small ROM to OTA into from the old firmware that does nothing but OTA pull the four images for rBoot, ROM, spiffs and partitionTable and write those?

Yes, that was the original thinking. Bit clunky, though isn't it?! Perhaps we can do this in one step, like this:

Add to application's component.mk:

EXTRA_LDFLAGS := $(call Wrap,user_pre_init)
USER_CFLAGS += -DPARTITION_TABLE_OFFSET=$(PARTITION_TABLE_OFFSET)

Add this to your application:

namespace
{
// Note: This file won't exist on initial build!
IMPORT_FSTR(partitionTableData, PROJECT_DIR "/out/Esp8266/debug/firmware/partitions.bin")
}

extern "C" void __wrap_user_pre_init(void)
{
	static_assert(PARTITION_TABLE_OFFSET == 0x3fa000, "Bad PTO");
	Storage::initialize();
        auto& flash = *Storage::spiFlash;
	if(!flash.partitions()) {
		LOAD_FSTR(data, partitionTableData)
		flash.erase_range(PARTITION_TABLE_OFFSET, flash.getBlockSize());
		flash.write(PARTITION_TABLE_OFFSET, data, partitionTableData.size());
		flash.loadPartitions(PARTITION_TABLE_OFFSET);
	}

	extern void __real_user_pre_init(void);
	__real_user_pre_init();
}

Note: You will get a 'file not found' error because the partition table gets built after compiling the application. You can run make partmap-build manually first to get around this.

Alternatively we could build the table directly ( and we don't need to set USER_CFLAGS):

extern "C" void __wrap_user_pre_init(void)
{
	Storage::initialize();

	auto& table = Storage::spiFlash->editablePartitions();
	if(!table) {
		using SubType = Storage::Partition::SubType;
		table.add(F("rom0"), SubType::App::ota0, 0x2000, 0xf8000);
		table.add(F("rom1"), SubType::App::ota1, 0x102000, 0xf8000);
		table.add(F("spiffs0"), SubType::Data::spiffs, 0x200000, 0xc0000);
		table.add(F("spiffs1"), SubType::Data::spiffs, 0x2c0000, 0xc0000);
		table.add(F("rf_cal"), SubType::Data::rfCal, 0x3fb000, 0x1000);
		table.add(F("phy_init"), SubType::Data::phy, 0x3fc000, 0x1000);
		table.add(F("sys_param"), SubType::Data::sysParam, 0x3fd000, 0x3000);
	}

	extern void __real_user_pre_init(void);
	__real_user_pre_init();
}

IMPORTANT! I spotted a bug which needs fixing first:

diff --git a/Sming/Components/Storage/src/include/Storage/PartitionTable.h b/Sming/Components/Storage/src/include/Storage/PartitionTable.h
index b54fcee3..2497178f 100644
--- a/Sming/Components/Storage/src/include/Storage/PartitionTable.h
+++ b/Sming/Components/Storage/src/include/Storage/PartitionTable.h
@@ -24,7 +24,7 @@ public:
 
 	explicit operator bool() const
 	{
-		return mEntries.isEmpty();
+		return !mEntries.isEmpty();
 	}
 
 	/**

[pljakobs]

iteresting. When is this called? How does the old bootloader know that entrypoint?
I think the first option with the built-in partition file is pretty elegant.

Thinking about it: the max size for a rom partition on the esp8266 is still just under 1MB, right? I was hoping to fit my webapp into FSTRINGS in the firmware and stay <1MB, but that will only work if I sparsify the webfonts I use (to get a few material icons). That may still be worth it, but there seems to be no way to do that as part of the build process.

I was hoping to have two 1.5MB rom partitions and one 1MB filesystem - but that's not an option on the ESP, I guess.

I complained to a web-developer friend of mine that my quasar frontend is more than 512kB in size when gzipped and he fell over backwards laughing. Sometimes I feel us embedded people are ... rather unusual 🗡️

[mikee47]

When is this called? How does the old bootloader know that entrypoint?

This function is documented in the NON-OS-SDK and was introduced in version 3. It is called by the SDK before user_init() so nothing else in the framework has yet been initialised, including any C++ static initialisers. (The bootloader doesn't care, it calls the SDK entrypoint.)

The EXTRA_LDFLAGS := $(call Wrap,user_pre_init) we add to component.mk adds -Wl,-wrap,user_pre_init to the linker. (We use this often enough that a helper macro is included in build.mk.) See https://ftp.gnu.org/old-gnu/Manuals/ld-2.9.1/html_mono/ld.html for details on the -wrap directive. The result here is that the original user_pre_init() function (in startup.cpp) is renamed to __real_user_pre_init(), and our replacement __wrap_user_pre_init() is called anywhere the original function was called.

... sparsify the webfonts I use (to get a few material icons). That may still be worth it, but there seems to be no way to do that as part of the build process.

I'm sure there is :-)

[pljakobs]

very cool stuff, thanks for he insight!

[mikee47]

NB. Forgot to call erase_range - have amended above.

Here's a third way building the partition table layout in code. No external file dependencies.

// Support updating legacy devices without partition tables (Sming 4.2 and earlier)
#ifdef ARCH_ESP8266

#include <Storage/partition_info.h>

extern "C" void __wrap_user_pre_init(void)
{
	static_assert(PARTITION_TABLE_OFFSET == 0x3fa000, "Bad PTO");

	Storage::initialize();

	auto& flash = *Storage::spiFlash;
	if(!flash.partitions()) {
		using FullType = Storage::Partition::FullType;
		using SubType = Storage::Partition::SubType;
#define PT_ENTRY(name, fulltype, offset, size) \
	{ ESP_PARTITION_MAGIC, FullType(fulltype).type, FullType(fulltype).subtype, offset, size, name, 0 }

		static constexpr Storage::esp_partition_info_t partitionTableData[] PROGMEM{
			PT_ENTRY("spiFlash", Storage::Device::Type::flash, 0, 0x400000),
			PT_ENTRY("rom0", SubType::App::ota0, 0x2000, 0xf8000),
			PT_ENTRY("rom1", SubType::App::ota1, 0x102000, 0xf8000),
			PT_ENTRY("spiffs0", SubType::Data::spiffs, 0x200000, 0xc0000),
			PT_ENTRY("spiffs1", SubType::Data::spiffs, 0x2c0000, 0xc0000),
			PT_ENTRY("rf_cal", SubType::Data::rfCal, 0x3fb000, 0x1000),
			PT_ENTRY("phy_init", SubType::Data::phy, 0x3fc000, 0x1000),
			PT_ENTRY("sys_param", SubType::Data::sysParam, 0x3fd000, 0x3000),
		};

		uint8_t buffer[sizeof(partitionTableData)];
		memcpy(buffer, partitionTableData, sizeof(partitionTableData));
		flash.erase_range(PARTITION_TABLE_OFFSET, flash.getBlockSize());
		flash.write(PARTITION_TABLE_OFFSET, buffer, sizeof(buffer));
		flash.loadPartitions(PARTITION_TABLE_OFFSET);
	}

	extern void __real_user_pre_init(void);
	__real_user_pre_init();
}

#endif // ARCH_ESP8266

Just another small patch to keep the code simple as possible:

diff --git a/Sming/Components/Storage/src/include/Storage/Device.h b/Sming/Components/Storage/src/include/Storage/Device.h
index 7ebaa32d..b05017e0 100644
--- a/Sming/Components/Storage/src/include/Storage/Device.h
+++ b/Sming/Components/Storage/src/include/Storage/Device.h
@@ -40,6 +40,7 @@ public:
 	 * @brief Storage type
 	 */
 	enum class Type : uint8_t {
+		partitionType = uint8_t(Partition::Type::storage),
 #define XX(type, value, desc) type = value,
 		STORAGE_TYPE_MAP(XX)
 #undef XX

[pljakobs]

I like using the external partition table rather than building it in code. I would think that keeps it more in line with the idea of hwconfig. Otherwise, for a future maintainer (and future me), there will be two code changes to make when really only one would suffice.
Also, the way this is written today, it's esp8266 only (due to the assert) and at least, it's relevant for the 8266 only - while we may have similar changes in the future, I see no reason to include this on other SOCs today, so it should be an #ifdef ARCH_ESP8266, I guess. Actually, I guess the entrypoint doesn't even exist in anything but 8266 nonos, right?

[mikee47]

I like using the external partition table rather than building it in code. I would think that keeps it more in line with the idea of hwconfig. Otherwise, for a future maintainer (and future me), there will be two code changes to make when really only one would suffice.

Once the partition table has been created this additional code wouldn't run again. I guess once hardware has all migrated to the new version it can be dropped entirely from the codebase. Or just leave it there - no reason to ever change it as newly flashed devices will already have a partition table so the code will have no effect.

I'm going to assume here that partition layout for existing devices wouldn't ever be changed, and (subject to testing) the bootloader shouldn't require updating either. If future devices use a different partition layout then the worst case should be that updates fail gracefully if the target partitions are too small; It won't matter if the position changes.

Therefore this third option provides a 'cut & paste' solution to this 'one-time' issue without external depdendencies which is why I give it here as an option. (It's also slightly more compact - the partitions.bin file is 0xc00 bytes and is mostly padding.)

Also, the way this is written today, it's esp8266 only (due to the assert) and at least, it's relevant for the 8266 only - while we may have similar changes in the future, I see no reason to include this on other SOCs today, so it should be an #ifdef ARCH_ESP8266, I guess. Actually, I guess the entrypoint doesn't even exist in anything but 8266 nonos, right?

Yes, esp8266 only, doesn't apply to other architectures since they had partition tables from inception.

[pljakobs]

My thinking is that I want to move the webapp into flash strings, provided I have the space. For that reason, I am currently distinguishing between partition layout v1 (two Roms, two spiffs) and v2 (two Roms one spiffs, or maybe Littlefs).
So the plan is to maintain both going forward, but since I can write the partition table, I might end up transitioning everything to v2 if that ever works out.

Anyway: I have today rebuilt with your first suggestion and am happy to report that I could successfully boot into the new code.
I had a few issues trying to downgrade afterwards (bootloader complaining about rf_cal[0] !=0x05,is 0x7B) but that is not a required use case.

[pljakobs]

hmmm... maybe I was a bit too quick here. When OTA'ing the same controller a 2nd time, I'm getting a garbled spiffs partition.

so

old Firmware -> new firmware, running rom is rom1 and spiffs is spiffs1 -> all is well
using the OTA for that to now OTA into rom0 / spiffs0, I get

** Enumerate registered SPIFFS partitions
[/home/pjakobs/devel/esp_rgbww_firmware/app/application.cpp:336] checking filesystem  spiffs0 @0x00200000
2165382 mount() returned 0 (Success)
2165511 File system initialised
OK, listing files:
  VERSION
  index.html.gz
  assets
2169319 rtc time init...
  J
  �#@�
  ��A��87%A؆
  �'@,
  .cfg
  .ota
  .color
10 files found

[/home/pjakobs/devel/esp_rgbww_firmware/app/application.cpp:336] checking filesystem  spiffs1 @0x00300000
2226184 mount() returned 0 (Success)
2226296 File system initialised
OK, listing files:
  VERSION
  index.html.gz
  assets
  icons
  .ota
  .color
  .cfg
7 files found

so, it looks like the spiffs0 has partially been written but has some garbled directory content. I believe that the "rtc time init..." line is part of the boot process, but at least at this time, it always shows up and after that there is mojibake.

I looked at the Sming 4.2 component.mk

# SPI EEPROM SIZE
SPI_SIZE = 4M

#### SPIFFS options ####
# folder with files to include
SPIFF_FILES = webapp

# size of filesystem
SPIFF_SIZE ?= 786432 #~768KB spiffs size
#SPIFF_SIZE      = 284288 #~512KB spiffs size


#### rBoot options ####
# use rboot build mode
RBOOT_ENABLED = 1

#  enable tmp rom switching
RBOOT_RTC_ENABLED = 1

# enable big flash support (for multiple roms, each in separate 1mb block of flash)
RBOOT_BIG_FLASH = 1

# two rom mode (where two roms sit in the same 1mb block of flash)
RBOOT_TWO_ROMS  = 0

# where does the filesystem reside
RBOOT_SPIFFS_0  = 0x100000
RBOOT_SPIFFS_1  = 0x300000 

and the RBOOT_SPIFFS_0 = 0x100000 bugs me - I thought with RBOOT_BIG_FLASH = 1, the first 1MB flash blocks would be the two ROM as is witht he modern two-roms-two-spiffs.hw? how can spiffs0 start at 0x100000?

my partition as written to flash is this:

Partition map read from device:
Device            Start       End         Size        Type      SubType   Name              Filename
----------------  ----------  ----------  ----------  --------  --------  ----------------  ------------
spiFlash          0x00002000  0x000f9fff        992K  app       ota_0     rom0              
spiFlash          0x000fa000  0x00101fff         32K                      (unused)          
spiFlash          0x00102000  0x001f9fff        992K  app       ota_1     rom1              
spiFlash          0x001fa000  0x001fffff         24K                      (unused)          
spiFlash          0x00200000  0x002bffff        768K  data      spiffs    spiffs0           
spiFlash          0x002c0000  0x002fffff        256K                      (unused)          
spiFlash          0x00300000  0x003bffff        768K  data      spiffs    spiffs1           
spiFlash          0x003c0000  0x003fafff        236K                      (unused)          
spiFlash          0x003fb000  0x003fbfff          4K  data      rfcal     rf_cal            
spiFlash          0x003fc000  0x003fcfff          4K  data      phy       phy_init          
spiFlash          0x003fd000  0x003fffff         12K  data      sysparam  sys_param         

that makes total sense to me, but would indicate that the 2nd MByte block is used for rom1 - as it should.

technically, at this point, nothing from the old layout should be around anymore, whatever it was, the new images would have overwritten the locations for rom1 and spiffs1 first and rom0 and spiffs0 then - but I have not seen this issue with any of the controllers that I fully flashed on the command line.

Any hint would be appreciatd.

[mikee47]

I can only conclude that your original firmware defines RBOOT_SPIFFS_0 at a different location, or it gets overridden? I might be inspecting disassembly at this point...

[pljakobs]

but would that still have any bearing? As said, at this time, both ROM partitions should be (and I believe are) the new firmware, so anything that remains of the original content is rBoot itself

the scan above is just as a test, after that, this

    auto romPartition=app.ota.getRomPartition();

    debug_i("Application::init - got rom partition %s @0x%#08x", romPartition.name(),romPartition.address());
    auto spiffsPartition=app.ota.findSpiffsPartition(romPartition);
    debug_i("Application::init - mounting filesystem  %s @0x%#08x",spiffsPartition.name(),spiffsPartition.address());
    _fs_mounted=spiffs_mount(spiffsPartition);

with getRomPartition being

    Storage::Partition getRomPartition() { return ota.getRunningPartition();}

and getSpiffsPartition

    Storage::Partition getSpiffsPartition() { return findSpiffsPartition(ota.getBootPartition());}

both defined in the otaupdate component of the app

[/home/pjakobs/devel/esp_rgbww_firmware/app/application.cpp:181] Application::init - got rom partition rom0 @0x00002000
[/home/pjakobs/devel/esp_rgbww_firmware/app/application.cpp:183] Application::init - mounting filesystem  spiffs0 @0x00200000
2267645 mount() returned 0 (Success)
2267744 File system initialised

so, to me, this looks all to be in the right place and dong the right thing ... I'm starting to wonder if the image might be corrupted.

oh wait, I just noticed something,

 ets Jan  8 2013,rst cause:1, boot mode:(3,6)

load 0x40100000, len 2292, room 16 
tail 4
chksum 0xb7
load 0x3ffe8000, len 772, room 4 
tail 0
chksum 0xe9
csum 0xe9

rBoot v1.4.2 - [email protected]
Flash Size:   32 Mbit
Flash Mode:   QIO
Flash Speed:  40 MHz
rBoot Option: Big flash
rBoot Option: RTC data

Booting rom 0.
ota1 not set
ota2 not set
V2

whereas in a fully flashed chip, I get this:

 ets Jan  8 2013,rst cause:1, boot mode:(3,6)

load 0x40100000, len 1692, room 16 
tail 12
chksum 0xee
ho 0 tail 12 room 4
load 0x3ffe8000, len 744, room 12 
tail 12
chksum 0xe3
csum 0xe3


rBoot v1.4.2 - [email protected]
Flash Size:   32 Mbit
Flash Mode:   DIO
Flash Speed:  40 MHz
rBoot Option: RTC data

Found 'rom0' @ 0x00002000, size 0x000f8000, subtype 0x10
Found 'rom1' @ 0x00102000, size 0x000f8000, subtype 0x11
Booting rom 1 at 102000, load addr 17a600.
ota1 not set
ota2 not set
V2
Mo

so,
a) rBoot in the old setup uses QIO, the new one ses DIO and
b) in the new system, rBoot finds the rom partitions correctly, the other one doesn't.

but both are rBoot v1.4.2, so I guess it's really about configuration not re-flashing rBoot, is it?

[mikee47]

Look at your otaupdate.cpp file, line 51:

otaUpdater->addItem(spiffsurl,spiffsPart, new Storage::PartitionStream(spiffsPart));

The blockErase parameter defaults to false, hence the corruption. This should read:

otaUpdater->addItem(spiffsurl,spiffsPart, new Storage::PartitionStream(spiffsPart, true));

[mikee47]

The Ota library provides a UpgradeOutputStream which kind of duplicates PartitionStream. We should probably sort this. Perhaps also defaulting the blockErase parameter is a bad idea... See #2729

[pljakobs]

ok, I believe that while clearing the flash bock is a good thing, I think I found the real issue here:

[/home/pjakobs/devel/esp_rgbww_firmware/app/otaupdate.cpp:42] ApplicationOTA::start nextBootPartition: rom1 102000
[/home/pjakobs/devel/esp_rgbww_firmware/app/otaupdate.cpp:50] ApplicationOTA::start spiffspart: spiffs1
[/home/pjakobs/devel/esp_rgbww_firmware/app/otaupdate.cpp:53] ApplicationOTA::start added spiffsurl: http://home.mobileme.de:8080/Esp8266/v1/spiff_rom.bin with blockerase=true
[/home/pjakobs/devel/esp_rgbww_firmware/app/otaupdate.cpp:105] ApplicationOTA::beforeOTA
[/home/pjakobs/devel/esp_rgbww_firmware/app/otaupdate.cpp:62] Free heap before OTA: 16936
[/home/pjakobs/devel/esp_rgbww_firmware/app/otaupdate.cpp:64] Current running partition: rom1
[/home/pjakobs/devel/esp_rgbww_firmware/app/otaupdate.cpp:65] OTA target partition: rom1
[/home/pjakobs/devel/esp_rgbww_firmware/app/otaupdate.cpp:66] configured OTA item list
[/home/pjakobs/devel/esp_rgbww_firmware/app/otaupdate.cpp:67] ========================
[/home/pjakobs/devel/esp_rgbww_firmware/app/otaupdate.cpp:70]   URL: http://home.mobileme.de:8080/Esp8266/v1/rom0.bin
[/home/pjakobs/devel/esp_rgbww_firmware/app/otaupdate.cpp:71]   Partition: rom1
[/home/pjakobs/devel/esp_rgbww_firmware/app/otaupdate.cpp:72]   Size: 0
[/home/pjakobs/devel/esp_rgbww_firmware/app/otaupdate.cpp:73]   ---------
[/home/pjakobs/devel/esp_rgbww_firmware/app/otaupdate.cpp:70]   URL: http://home.mobileme.de:8080/Esp8266/v1/spiff_rom.bin
[/home/pjakobs/devel/esp_rgbww_firmware/app/otaupdate.cpp:71]   Partition: spiffs1
[/home/pjakobs/devel/esp_rgbww_firmware/app/otaupdate.cpp:72]   Size: 0
[/home/pjakobs/devel/esp_rgbww_firmware/app/otaupdate.cpp:73]   ---------
[/home/pjakobs/devel/esp_rgbww_firmware/app/otaupdate.cpp:77] Starting OTA ...

ota.getNextBootPartition() returns rom1 and ota.getRunningPartition() yields the same!
I have tried calling rboot_get_current_rom(); and that shows 0 even though I'm clearly running rom1/spiffs1, so somehow, that first boot after ota gets rBoot confused, I'd think.

since getNextBootPartition calls rboot_get_current_rom()

	uint8_t currentSlot = rboot_get_current_rom();
		if(startFrom) {
			currentSlot = getSlotForPartition(startFrom);
		}
		return getPartitionForSlot(currentSlot ? 0 : 1);

it will return rom1/spiffs1 even though that's what is currently running and therefore spiffs0 is never flashed.

[edit]
looking a bit deeper - could it be that I'm in MODE_TEMP_ROM after the first reboot?

Partition RbootUpgrader::getRunningPartition()
{
	uint8_t slot = rboot_get_current_rom();
#ifdef BOOT_RTC_ENABLED
	rboot_rtc_data rtc;
	if(rboot_get_rtc_data(&rtc) && rtc.last_mode == MODE_TEMP_ROM) {
		slot = rtc.last_rom;
	}
#endif
	return getPartitionForSlot(slot);
}

ok, so immediately after OTA reboot, I print

due to

#ifdef ARCH_ESP8266
    // load boot information
    uint8 bootmode, bootslot;
    debug_i("Application::init - loading boot info");
    if (rboot_get_last_boot_mode(&bootmode)) {
        if (bootmode == MODE_TEMP_ROM) {
            debug_i("Application::init - booting after OTA");
        } else {
            debug_i("Application::init - normal boot");
        }
        _bootmode = bootmode;
    }
#endif

That would at least begin to explain it. If I initiate the next OTA in TEMP mode, the same partitions would be written to but after that, a switch would happen. The garbled content of spiffs0 is just due to the spiffs metadata format change.

Question is: should this be this way? do I need to handle something in TEMP mode or is this a bug in handling the rboot conf ?

[mikee47]

I agree a check is required to reject any firmware update when running in temporary mode.

auto current = ota.getRunningPartition();
auto next = ota.getNextBootPartition();
if(next == current) {
   // FAIL UPDATE! - reboot first
}

However, I don't see any message in your boot logs indicating a temporary boot...

[pljakobs]

I had to stop yesterday. Back at it today.

So first, I've gotten debug from HttpUprader:

[Libraries/OtaNetwork/src/HttpUpgrader.cpp:23] Download file:
    (0) http://home.mobileme.de:8080/Esp8266/v1/rom0.bin -> rom0 @ 0x2000
[Libraries/OtaNetwork/src/HttpUpgrader.cpp:23] Download file:
    (0) http://home.mobileme.de:8080/Esp8266/v1/spiff_rom.bin -> spiffs0 @ 0x200000
[Libraries/OtaNetwork/src/HttpUpgrader.cpp:61] Finished: URL: http://home.mobileme.de:8080/Esp8266/v1/rom0.bin, Offset: 0x2000, Length: 541712


***** Fatal exception 0

this looks ok to me, the size corresponds to the file size and 0x2000 is the right flash location for rom0.
But then clearly something happens, and this only started when pushing the newer rboot. I will test without it again.

---

so after some testing... I'm getting those fatal exceptions even if I flash the original rboot.bin

I have addedd a string to the rboot version tag and it seems that, after the first OTA, it still is not executed, at least I still get the same output, so it seems that the new rboot is not used.

manually flashing the new rboot seems to fix the whole thing.

I am uncertain what to think, but it feels like there's a mismatch in what the original rboot "thinks" of what rom slot is being run and what the ota code "thinks".

Anyway, it would seem that flashing rboot with the ota is the right course of action, but it currently doesn't seem to work, though I don't know why.

[edit]
oh, one more thing: building the rboot.bin is difficult once it's deleted.
make rboot-build only creates the object file, only the full app build will create the linked bin.
I've circumvented this by including a dummy file in the FSTRING but that's cumbersome. If we want to be able to OTA rboot, there ought to be a better way.

[mikee47]

As indicated above are you absolutely sure that the flash layout for your existing devices is correct? My earlier suggestion about disassembly was to inspect the deployed original code to confirm where all the partitions should be. This very much feels like a mismatch issue.

[pljakobs]

Well, no.
I had posted this further up and was confused about it

# SPI EEPROM SIZE
SPI_SIZE = 4M

#### SPIFFS options ####
# folder with files to include
SPIFF_FILES = webapp

# size of filesystem
SPIFF_SIZE ?= 786432 #~768KB spiffs size
#SPIFF_SIZE      = 284288 #~512KB spiffs size


#### rBoot options ####
# use rboot build mode
RBOOT_ENABLED = 1

#  enable tmp rom switching
RBOOT_RTC_ENABLED = 1

# enable big flash support (for multiple roms, each in separate 1mb block of flash)
RBOOT_BIG_FLASH = 1

# two rom mode (where two roms sit in the same 1mb block of flash)
RBOOT_TWO_ROMS  = 0

# where does the filesystem reside
RBOOT_SPIFFS_0  = 0x100000
RBOOT_SPIFFS_1  = 0x300000 

That would indicate to me that spiffs0 is in flash block 0x100000 - which contradicts the RBOOT_BIG_FLASH = 1, which puts rom1 right there.
maybe that has been wrong all the time and nobody ever noticed?

It would still leave me with the problem.

I have a flash image that I can look at when I get back home.

---

to complete the confusion (and add credence to the thought that it may be a mismatch)

• after flashing the new rboot with esptool.py -> everything works as expected
• later on, flashing the original rboot again -> everything still works

However, it seems that the code writing rboot as part of the OTA file does not seem to work. While the updated signature is in the rom0.bin, it is never started. I still have to look at the flash content if it has actually been written - maybe there's a chance that during an OTA reboot, rboot is executed from IRAM?

[mikee47]

maybe there's a chance that during an OTA reboot, rboot is executed from IRAM?

LOL. Nope. Boot ROM sees to that. And flash caching is disabled initially at boot...

[pljakobs]

hmm.. but this is reboot after OTA:

ets Jan  8 2013,rst cause:1, boot mode:(3,6)

load 0x40100000, len 2292, room 16 
tail 4
chksum 0xb7
load 0x3ffe8000, len 772, room 4 
tail 0
chksum 0xe9
csum 0xe9

rBoot v1.4.2 - [email protected]
Flash Size:   32 Mbit
Flash Mode:   QIO
Flash Speed:  40 MHz
rBoot Option: Big flash
rBoot Option: RTC data

Booting temp rom.
Booting rom 1.

I have just added "testing" to the version tag, and that only shows on the next boot.

[mikee47]

I've also spotted some debug statements which need fixing, such as otaupdate.cpp line 42:

debug_i("ApplicationOTA::start nextBootPartition: %s %#06x", part.name(), part.address());

Must use part.name().c_str(). (Only works here since strings are <= 10 bytes and stored directly in String object.)

[pljakobs]

thanks, fixed those.

[mikee47]

actually, one thing that I'm thinking here: it would be good practice to revert a rom switch if the corresponding spiffs (or other IFS) partition is not usable. Question is: how do I check that reliably. If, as I'm thinking, the issue here is metadata size, it would technically be a consistent spiffs, right? I think I saw there's something like a file system structure check somewhere, but can't find it right now.

You'll want the fileSystemCheck() call. Also, if application and SPIFFS partitions are updated together there shouldn't be an issue with SPIFFS metadata size.

I'm not sure exactly what you store in SPIFFS, but if there's a lot of static (immutable) files have you considered using FWFS? See https://sming.readthedocs.io/en/latest/_inc/Sming/Components/IFS/index.html

[pljakobs]

[moved]

[pljakobs]

On the file system: this is as of yet a part I have not modified. Yes, the majority of things is static (the webapp) and just a few files are being changed at runtime.
As said, if possible (size wise) I want the webapp to be built onto the rom partitions using FSTRING, but it remains to be seen if I can get it below what is it, 998kB.
Once I get there, I would ideally only have a single data partition, so I don't need to touch it in OTA at all. I would also be looking for a Filesystem that is a bit more friendly to flash (with the old layout, a value in flash is updated sometimes a couple of dozen times a day, wearing out the flash rather quickly as I understand the original spiffs only does static wear levelling which I believe means it won't move a single file around?)

[mikee47]

Found a bug #2730 which explains the absence of the rBoot Option: Big flash message after OTA. This could be part of the problem. If it's visible in the logs then bootloader pre-dates #1817, i.e. release 4.0.0.

[mikee47]

I think you're heading down a rabbit hole here; there is absolutely no reason to be messing around at this level!

[pljakobs]

ups, you must have send your response further up before I sent mine.

my thinking is: the rom header (the first four bytes) are part of rboot.bin, ok
but it seems that esptool overwrites them when flashing rboot, at least for the fourth byte, where the high nibble signify the size of the flash chip. I was trying to see if hardcoding that in would do the trick.

[pljakobs]

but then - how to correct the rom header for the 32MBit layout?

[mikee47]

I was a bit dubious about but have just tested by building Basic_Blink for a 1M device then flashing it using esptool with 4M size. Lo and behold, yes, it modifies the boot sector. FFS.

[mikee47]

Your absolutely right here. Values passed to esptool for flash_freq, flash_mode and flash_size are used to update the bootloader image (as in esptool/cmds.py _update_image_flash_params() function). So the rboot image was indeed incorrectly generated somehow.

For future reference, python3 $SMING_HOME/Components/esptool/esptool/esptool.py image_info -v2 out/Esp8266/debug/firmware/rboot.bin shows this information. I'll add that as a make target.

[mikee47]

flash the Sming4 based firmware

What commit from your esp_rbgww_firmware repo. are you using?

[pljakobs]

for the old one, I just use the pre-built binaries here https://github.com/verybadsoldier/esp_rgbww_firmware/wiki/Flashing-&-Compiling
I have pulled a full image of the result available here: http:/home.mobileme.de:8080/old/full_image_v4.bin (the browser will possibly report an ssl error, wget conveniently ignores that)
The new software is currently up to date in https://github.com/pljakobs/esp_rgbww_firmware/tree/feature/newOTA
for the rboot, I've just used a hex editor to set byte 4 to 0x40.

[mikee47]

OK, so the partition layout is still incorrect (AFAIK) so let's try and get that sorted first. Forget about the boot sector for now - the existing boot sector should work. Try this:

    // Support updating legacy devices without partition tables (Sming 4.2 and earlier)
    #ifdef ARCH_ESP8266

    #include <Storage/partition_info.h>

    extern "C" void __wrap_user_pre_init(void)
    {
        static_assert(PARTITION_TABLE_OFFSET == 0x3fa000, "Bad PTO");

        Storage::initialize();

        auto& flash = *Storage::spiFlash;
        if(!flash.partitions()) {
            using FullType = Storage::Partition::FullType;
            using SubType = Storage::Partition::SubType;
    #define PT_ENTRY(name, fulltype, offset, size) \
        { Storage::ESP_PARTITION_MAGIC, FullType(fulltype).type, FullType(fulltype).subtype, offset, size, name, 0 }

            static constexpr Storage::esp_partition_info_t partitionTableData[] PROGMEM{
                PT_ENTRY("spiFlash", Storage::Device::Type::flash, 0, 0x400000),
                PT_ENTRY("rom0", SubType::App::ota0, 0x2000, 0xf8000),
                PT_ENTRY("spiffs0", SubType::Data::spiffs, 0x100000, 0xc0000),
                PT_ENTRY("rom1", SubType::App::ota1, 0x202000, 0xf8000),
                PT_ENTRY("spiffs1", SubType::Data::spiffs, 0x300000, 0xc0000),
                PT_ENTRY("rf_cal", SubType::Data::rfCal, 0x3fb000, 0x1000),
                PT_ENTRY("phy_init", SubType::Data::phy, 0x3fc000, 0x1000),
                PT_ENTRY("sys_param", SubType::Data::sysParam, 0x3fd000, 0x3000),
            };

            uint8_t buffer[sizeof(partitionTableData)];
            memcpy(buffer, partitionTableData, sizeof(partitionTableData));
            flash.erase_range(PARTITION_TABLE_OFFSET, flash.getBlockSize());
            flash.write(PARTITION_TABLE_OFFSET, buffer, sizeof(buffer));
            flash.loadPartitions(PARTITION_TABLE_OFFSET);
        }

        extern void __real_user_pre_init(void);
        __real_user_pre_init();
    }

    #endif // ARCH_ESP8266

[pljakobs]

ok, wait, I see the difference here, you've laid out the partitions as rom0, spiffs0, rom1, spiffs1.
I had it rom0, rom1, spiffs0, spiffs1 (as per the two-rom-one-spiff plus one more spiff).
I pointed at this yesterday but more as a point of confusion (#2727 (reply in thread))
That would of course explain why spiffs0 would be corrupted when flashing what OTA thinks should be rom0.

testing that now.

[pljakobs]

does that code compile for you?
I get

/home/pjakobs/devel/esp_rgbww_firmware/Sming/Sming/Components/Storage/src/include/Storage/Partition.h:156:74: error: 'partitionType' is not a member of 'Storage::Device::Type'
  156 |   template <typename T> constexpr FullType(T subType) : FullType(Type(T::partitionType), uint8_t(subType))
      |                                                                          ^~~~~~~~~~~~~

and a bunch of follow-on errors.

[mikee47]

Patch from above:

diff --git a/Sming/Components/Storage/src/include/Storage/Device.h b/Sming/Components/Storage/src/include/Storage/Device.h
index 7ebaa32d..b05017e0 100644
--- a/Sming/Components/Storage/src/include/Storage/Device.h
+++ b/Sming/Components/Storage/src/include/Storage/Device.h
@@ -40,6 +40,7 @@ public:
 	 * @brief Storage type
 	 */
 	enum class Type : uint8_t {
+		partitionType = uint8_t(Partition::Type::storage),
 #define XX(type, value, desc) type = value,
 		STORAGE_TYPE_MAP(XX)
 #undef XX

[pljakobs]

dang, that did it.
So you were right all along, and I just didn't understand that I could put spiffs0 in 0x0100000 and rom1 in 0x0200000

[/home/pjakobs/devel/esp_rgbww_firmware/app/application.cpp:352] checking filesystem  spiffs0 @0x00100000
[Core/FileSystem.cpp:45] mount() returned 0 (Success)
[Core/FileSystem.cpp:54] File system initialised
OK, listing files:
  VERSION
  index.html.gz
  assets
  icons
4 files found

[/home/pjakobs/devel/esp_rgbww_firmware/app/application.cpp:352] checking filesystem  spiffs1 @0x00300000
[Core/FileSystem.cpp:45] mount() returned 0 (Success)
[Core/FileSystem.cpp:54] File system initialised
OK, listing files:
  VERSION
  index.html.gz
  assets
  icons
  .color
  .cfg
  .ota
7 files found

[/home/pjakobs/devel/esp_rgbww_firmware/app/application.cpp:197] Application::init - got rom partition rom0 @0x00002000
[/home/pjakobs/devel/esp_rgbww_firmware/app/application.cpp:199] Application::init - mounting filesystem  spiffs0 @0x00100000
[Core/FileSystem.cpp:45] mount() returned 0 (Success)
[Core/FileSystem.cpp:54] File system initialised

that's after the 2nd OTA

so - I don't know what to say, thank you for your help and patience.

This was the deep end for me, please apologize if I made this more difficult and confusing than it should have been.

I have btw reverted to the partition file method after I could not compile the code above. Works just as fine once the partition file is built.

The only thing for now is that I'm still on the old rboot, but that doesn't seem to be an issue.

[mikee47]

The two patches above are included in latest Sming develop branch.