Documentation/CI fixes (#2783)

This PR contains a few minor fixes relating to CI testing and documentation. There are no functional changes to the framework.

**Documentation fixes**

- Noticed that `:issue:` links don't work! Just a typo in the URL...
- The `Storage` component API documentation contained everything from the FatIFS and DiskStorage libraries
too since they live in the same namespace. I've broken things out a bit to avoid that.
- Doxygen API index page isn't appearing at https://sming.readthedocs.io/en/latest/api/, just the placeholder page. The pages are all there but Sphinx hasn't overwritten the placeholder index.html as it should do. Fixed by adding a manual copy at end of sphinx build. Tested on readthedocs and locally.
- Document `STRICT` setting and add upgrade note about Y2038 warning.
- Add tips to vscode documentation.
- Upgrade sphinx to latest 7.3.7

**Test fixes**

- Simplify HostTests partition layout
- Fix uf2conf python regex warning

.gitignore

@@ -1,11 +1,8 @@
 # generated binaries
 out
 
-# generated documentation
-docs/api
-docs/build
-docs/source/_inc
-*.pyc
+# Python cache
+__pycache__
 
 # Eclipse
 .project

Sming/Arch/Rp2040/Components/uf2/uf2conv.py

@@ -176,7 +176,7 @@ def get_drives():
                                      "get", "DeviceID,", "VolumeName,",
                                      "FileSystem,", "DriveType"])
         for line in r.decode().split('\n'):
-            words = re.split('\s+', line)
+            words = re.split(r'\s+', line)
             if len(words) >= 3 and words[1] == "2" and words[2] == "FAT":
                 drives.append(words[0])
     else:

Sming/Components/Storage/README.rst

@@ -343,7 +343,9 @@ Entries are fixed 32-byte structures, :cpp:class:`Storage::esp_partition_info_t`
 Partition API
 -------------
 
-This is a C++ interface. Some examples::
+This is a C++ interface. Some examples:
+
+.. code-block:: c++
 
    Storage::Partition part = Storage::findPartition("spiffs0"); // Find by name
    if(part) {
@@ -371,7 +373,9 @@ This is usually :cpp:var:`Storage::spiFlash` for the main flash device.
 
 Other devices must be registered via :cpp:func:`Storage::PartitionTable::registerStorageDevice`.
 
-You can query partition entries from a Storage object directly, for example::
+You can query partition entries from a Storage object directly, for example:
+
+.. code-block:: c++
 
    #include <Storage/SpiFlash.h>
 
@@ -400,5 +404,52 @@ See :library:`DiskStorage` for how devices such as SD flash cards are managed.
 API
 ---
 
-.. doxygennamespace:: Storage
+Core Functions
+~~~~~~~~~~~~~~
+
+.. doxygenfunction:: Storage::initialize
+.. doxygenfunction:: Storage::getDevices
+.. doxygenfunction:: Storage::registerDevice
+.. doxygenfunction:: Storage::unRegisterDevice
+.. doxygenfunction:: Storage::findDevice
+.. doxygenfunction:: Storage::findPartition(const String&)
+.. doxygenfunction:: Storage::findPartition(Partition::Type, uint8_t)
+.. doxygenvariable:: Storage::spiFlash
+
+
+Main classes
+~~~~~~~~~~~~
+
+.. doxygenclass:: Storage::Device
    :members:
+.. doxygenclass:: Storage::SpiFlash
+   :members:
+.. doxygenclass:: Storage::Partition
+   :members:
+.. doxygenclass:: Storage::PartitionTable
+   :members:
+.. doxygenclass:: Storage::FileDevice
+   :members:
+
+
+Streaming
+~~~~~~~~~
+
+.. doxygenenum:: Storage::Mode
+.. doxygenclass:: Storage::PartitionStream
+   :members:
+.. doxygenclass:: Storage::StreamDevice
+   :members:
+
+
+Debugging
+~~~~~~~~~
+
+.. doxygennamespace:: Storage::Debug
+   :members:
+
+.. doxygenvariable:: Storage::progMem
+.. doxygenclass:: Storage::ProgMem
+
+.. doxygenvariable:: Storage::sysMem
+.. doxygenclass:: Storage::SysMem

Sming/Components/Storage/src/include/Storage/StreamDevice.h

@@ -14,10 +14,19 @@ namespace Storage
 class StreamDevice : public Device
 {
 public:
+	/**
+	 * @brief Create a Device object using data from a stream with restricted size
+	 * @param stream Backing data source for this device
+	 * @param size Size of device in bytes
+	 */
 	StreamDevice(IDataSourceStream* stream, size_t size) : Device(nameOf(stream), size), mStream(stream)
 	{
 	}
 
+	/**
+	 * @brief Create a Device object using data from a stream with all available data
+	 * @param stream Backing data source for this device
+	 */
 	StreamDevice(IDataSourceStream* stream) : StreamDevice(stream, size_t(stream->available()))
 	{
 	}

Sming/README.rst

@@ -49,14 +49,38 @@ Change it like this::
 
 .. note::
    If you change these settings and want them applied to Sming, not just your project, then you'll
-   need to recompile all components like this:
-
-   ::
+   need to recompile all components like this::
 
       make components-clean
       make DEBUG_VERBOSE_LEVEL=3
 
 
+.. envvar:: STRICT
+
+   * default: undefined (standard warnings, treat as errors)
+   * 1: Enable all warnings but do not treat as errors
+
+   By default, Sming builds code with a few warnings disabled:
+
+   - sign-compare: Comparison between signed and unsigned values could produce an incorrect result when the signed value is converted to unsigned.
+   - parentheses: Parentheses are omitted in certain contexts.
+   - unused-variable: A local or static variable is unused aside from its declaration.
+   - unused-but-set-variable: A local variable is assigned to, but otherwise unused.
+   - strict-aliasing: Casts which can violate alignment rules.
+
+   These can be indicative of problematic code rather than errors.
+   Because new compiler releases can increase the level of checking, this set may also need to change
+   but it is kept minimal. See ``Sming/build.mk`` for the actual settings.
+   See https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html for further details
+   about all compiler settings.
+   See also :doc:`/information/develop/clang-tools`.
+
+   Any remaining warnings will be treated as errors and compilation will be halted.
+
+   It is a good idea to check your codebase with ``STRICT=1`` which enables **all** warnings.
+   However, it does not treat them as errors because of the above which could be false-positives.
+
+
 Release builds
 ~~~~~~~~~~~~~~
 

docs/.gitignore

@@ -0,0 +1,5 @@
+# generated documentation
+api
+build
+source/_inc
+*.pyc

docs/requirements.txt

@@ -1,6 +1,6 @@
 # Requirements file for pip
 # list of Python packages used in documentation build
-sphinx==7.2.6
+sphinx==7.3.7
 sphinx-rtd-theme==2.0.0
 m2r2==0.3.3.post2
 breathe==4.35.0

docs/source/conf.py

@@ -14,13 +14,12 @@
 import os
 import sys
 import subprocess
-from docutils import nodes, utils
-from sphinx import roles, addnodes
-from sphinx.util.nodes import set_role_source_info, split_explicit_title
+from sphinx.util import logging
 # So our custom extensions can be found
 sys.path.insert(0, os.path.abspath('.'))
 from filemap import buildFileMap
 
+logger = logging.getLogger(__name__)
 
 # -- Project information -----------------------------------------------------
 
@@ -126,7 +125,7 @@
 #
 # on_rtd is whether we are on readthedocs.org
 env_readthedocs = os.environ.get('READTHEDOCS', None)
-print(env_readthedocs)
+logger.info(env_readthedocs)
 
 if env_readthedocs:
     # Start the build from a clean slate
@@ -141,3 +140,20 @@
 subprocess.call('make -C .. setup api API_VERSION="' + version + '"', shell=True)
 
 buildFileMap(html_context)
+
+
+def setup(app):
+    '''
+    Sphinx requires a placeholder 'api/index.rst' which is overwritten by the doxygen-generated version
+    via the above `html_extra_path` setting above.
+    As of Sphinx 7.3.7 (and 7.2.6) this doesn't happen, though it does if run a second time!
+    Simple fix is to just copy the one `index.html` file after sphinx has done it's build.
+    '''
+    def fix_api_index(app, exception):
+        import shutil
+        src_file = os.path.join(app.srcdir, '../api/html/api/index.html')
+        dst_path = os.path.join(app.outdir, 'api')
+        logger.info(f'>> Copy from "{src_file}" -> "{dst_path}"')
+        shutil.copy(src_file, dst_path)
+
+    app.connect('build-finished', fix_api_index)

docs/source/link-roles.py

@@ -33,7 +33,7 @@ def get_github_rev():
 
 def setup(app):
     app.add_role('source', SourceRole())
-    app.add_role('issue', autolink('Issue #{0} <' + github_url + '/issue/{0}>'))
+    app.add_role('issue', autolink('Issue #{0} <' + github_url + '/issues/{0}>'))
     app.add_role('pull-request', autolink('Pull Request #{0} <' + github_url + '/pull/{0}>'))
 
     app.add_role('sample', SampleRole)

docs/source/tools/vscode.rst

@@ -13,6 +13,17 @@ Software involved
 -  `Visual Studio Code <https://code.visualstudio.com/>`__
 -  `C/C++ extension <https://marketplace.visualstudio.com/items?itemName=ms-vscode.cpptools>`__
 
+.. note::
+
+   Linux users may prefer `VSCodium <https://vscodium.com/>`__ which does not contain telemetry or tracking.
+
+   Standard C/C++ language support is available via the `cpptools <https://github.com/microsoft/vscode-cpptools>`__
+   extension which is not available in the vscodium repositories.
+
+   Visit https://marketplace.visualstudio.com/items?itemName=ms-vscode.cpptools and go to ``Version History``
+   to download the .vsix file. Open the ``Extensions`` pane in vscodium and drag the file there to install,
+   or run ``codium --install-extension NAME-OF-FILE.vsix``.
+
 
 Installation
 ------------
@@ -70,6 +81,22 @@ To debug your application, follow these steps:
    VS Code debug selection
 
 
+Editor window titles
+--------------------
+
+As Sming is a multi-architecture framework there are lots of files with the same name.
+By default editor window titles contain only the filename, but in vscode this can be changed
+to something more useful, like including the parent directory name.
+
+Open user settings JSON (via F1 hotkey) and add this to the config:
+
+.. code-block:: json
+
+    "workbench.editor.customLabels.patterns": {
+        "**/*.*": "${dirname}/${filename}.${extname}"
+    }
+
+
 Manual configuration changes
 ----------------------------
 

docs/source/upgrading/5.1-5.2.rst

@@ -10,12 +10,14 @@ The intended default behaviour is read-only, however previously this also allowe
 This can result in corrupted flash contents where the flash has not been explicitly erased beforehand.
 
 The new constructors instead use a :cpp:enum:`Storage::Mode` so behaviour is more explicit.
-The default is read-only and writes will now be failed.
+The default is read-only and writes will now be failed. For example::
+
+    Storage::PartitionStream stream(myPartition, Storage::Mode::BlockErase);
 
 
 **64-bit time_t**
 
-There is some variability in whether `time_t` is 32 or 64 bits. See issue #2758.
+There is some variability in whether `time_t` is 32 or 64 bits. See :issue:`2758`.
 
 This is dependent on the toolchain and accompanying C library.
 
@@ -25,11 +27,18 @@ This is dependent on the toolchain and accompanying C library.
 - Windows Host (using mingw)
 - Linux host builds prior to Sming v5.2
 
+Building for these will generate a warning ``**Y2038** time_t is only 32-bits in this build configuration``.
+If you cannot upgrade then build with :envvar:`STRICT` set to 1.
+
 Range of time_t:
 
-    -0x80000000: 1901-12-13 20:45:52
-    0x00000000: 1970-01-01 00:00:00
-    0x7fffffff: 2038-01-19 03:14:07
+===========     ===================
+value           Timestamp
+===========     ===================
+-0x80000000     1901-12-13 20:45:52
+0x00000000      1970-01-01 00:00:00
+0x7fffffff      2038-01-19 03:14:07
+===========     ===================
 
 All others use 64-bit values.
 
@@ -39,8 +48,8 @@ Rp2040 builds with standard ARM toolkit so probably accommodated by the standard
 
 Espressif toolchains use forks:
 
-esp8266: https://github.com/earlephilhower/newlib-xtensa/blob/xtensa-4_0_0-lock-arduino/newlib/libc/
-esp32: https://github.com/espressif/newlib-esp32/blob/esp-4.3.0/newlib/libc/
+- esp8266: https://github.com/earlephilhower/newlib-xtensa/blob/xtensa-4_0_0-lock-arduino/newlib/libc/
+- esp32: https://github.com/espressif/newlib-esp32/blob/esp-4.3.0/newlib/libc/
 
 
 **Toolchain versions updated**

tests/HostTests/host-tests-1m.hw

@@ -5,12 +5,10 @@
         "1m"
     ],
     "partitions": {
-        "fwfs0": {
-            "address": "0x000c0000",
-            "size": "128K"
+        "fwfs_httprequest": {
+            "address": "0x000c0000"
         },
         "spiffs0": {
-            "size": "0x10000",
             "address": "0x000e0000"
         }
     }

tests/HostTests/host-tests.hw

@@ -1,53 +1,21 @@
 {
 	"name": "Host Tests profile",
 	"base_config": "spiffs",
-	"devices": {
-		"testDevice": {
-			"type": "spiram",
-			"size": "0x40000000"
-		}
-	},
 	"partitions": {
 		"spiffs0": {
 			"size": "0x10000",
 			"filename": ""
 		},
-		"fwfs0": {
+		"fwfs_httprequest": {
 			"address": "0x220000",
-			"size": "0x40000",
+			"size": "0x20000",
 			"type": "data",
 			"subtype": "fwfs",
 			"filename": "$(FW_BASE)/fwfs0.bin",
 			"build": {
 				"target": "fwfs-build",
 				"config": "fwfs0.json"
 			}
-		},
-		"external1": {
-			"device": "testDevice",
-			"address": 0,
-			"size": "4M",
-			"type": "data",
-			"subtype": "spiffs",
-			"filename": "$(FW_BASE)/test-spiffs.bin",
-			"build": {
-				"target": "spiffsgen",
-				"files": "resource"
-			}
-		},
-		"external2": {
-			"device": "testDevice",
-			"address": "0x600000",
-			"size": "240K",
-			"type": "data",
-			"subtype": 37
-		},
-		"external3": {
-			"device": "testDevice",
-			"address": "0x800000",
-			"size": "240M",
-			"type": "data",
-			"subtype": "nvs"
 		}
 	}
 }