Update windows installer (#2859)

The Windows installer makes use of custom chocolatey packages, which are out of date. See issue #2514. These packages are separate from the main Sming repository and represent an un-necessary maintenance overhead. This PR updates the installation scripts so these custom chocolatey packages are no longer required. The scripts were previously only used for CI testing so have been reworked appropriately. The installation procedure is now similar to that for linux/MacOS. The initial bootstrap installation remains unchanged: The ``choco-install.cmd`` script is fetched from the Sming ``develop`` branch and executed. Note: Contrary to other installs and the documentation, the current sming choco script installs the `master` branch. The user runs a second install script `tools\install` to install required toolchains. This is separate because: - Administrative priviledges are not required - The user gets more control over which toolchains get installed - The script doesn't change anything outside of the `C:\tools` directory (other than python packages) The directory can also be customised by setting `SMING_TOOLS_DIR`. The documentation at https://sming.readthedocs.io/en/latest/getting-started/windows/index.html has been updated to reflect these changes. **CI Compiler Cache** I've moved caching to just before the build starts, so that configuration and zeroing ccache is handled by the build script. This is more logical than doing it as part of the install phase. **Esp32 install/build** A standard IDF tools installation includes the following packages: - cmake - ninja - ccache As Sming installs these system-wide, these are not required. Also, as we now have Ninja in the system path for all platforms some logic in build.mk can be removed. These additional items are not used: - idf-exe - dfu-util The Sming Esp32 installer omits all the above items, for linux/MacOS also. Turns out we can just pass the filter `"*elf*"` to the installer to get all the stuff we **do** actually need. **Python** This defaulted to `python3`, which failed in testing as current installers don't provide it, so change to `python`. The `PYTHON` variable now defaults to the full path determined by `which python`. In Windows, the makefile resolves this to something like `/c/Python312/python.exe`. **MinGW** I've also added `MinGW-2021-04-12.7z` to the [SmingTools](https://github.com/SmingHub/SmingTools/releases). This differs from `MinGW-2020-10-19.7z` which was previously used (in both choco package and CI install) in the following ways: - MinGW version changed from 5.4.1 to 5.4.2 - Files unpack into `./` instead of `MinGw/` so makes it easier to relocate during installation if required - Contents of `var/cache` removed to reduce size of archive (48MB vs 133MB) No changes to GCC, still 9.2.0. **Docs build** Documentation build fails in Windows, problems with libcairo-2.dll. Initially, the error was about not finding the DLL. Some reserach suggests installing gtk+ runtime should fix this. Still fails but with error "can't find .so". Leaving this as a known issue, just build in Linux or WSL.
SmingHub · Jul 14, 2024 · fd165a5 · fd165a5
1 parent a9864da
commit fd165a5
Show file tree

Hide file tree

Showing 25 changed files with 499 additions and 334 deletions.
diff --git a/.github/workflows/ci-esp32.yml b/.github/workflows/ci-esp32.yml
@@ -75,12 +75,6 @@ jobs:
           /opt/esp32
         key: ${{ matrix.os }}-idf-${{ matrix.idf_version }}
 
-    - name: Compiler Cache
-      uses: actions/cache@v4
-      with:
-        path: .ccache
-        key: ${{ matrix.os }}-ccache-${{ matrix.variant }}-${{ matrix.idf_version }}
-
     - name: Install build tools for Ubuntu / MacOS
       if: matrix.os != 'windows-latest'
       run: |
@@ -89,13 +83,19 @@ jobs:
     - name: Install build tools for Windows
       if: matrix.os == 'windows-latest'
       run: |
-         . Tools/ci/setenv.ps1
-         Tools/ci/install.cmd
+        . Tools/ci/setenv.ps1
+        Tools/ci/install.cmd
+
+    - name: Compiler Cache
+      uses: actions/cache@v4
+      with:
+        path: .ccache
+        key: ${{ matrix.os }}-ccache-${{ matrix.variant }}-${{ matrix.idf_version }}
 
     - name: Build and test for ${{matrix.variant}} with IDF v${{matrix.idf_version}} on Ubuntu / MacOS
       if: matrix.os != 'windows-latest'
       run: |
-        source $SMING_HOME/../Tools/export.sh
+        . Tools/export.sh
         Tools/ci/build.sh
 
     - name: Build and test for ${{matrix.variant}} with IDF v${{matrix.idf_version}} on Windows

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -64,12 +64,6 @@ jobs:
         "CI_BUILD_DIR=" + (Resolve-Path ".").path >> $env:GITHUB_ENV
         "SMING_HOME=" + (Resolve-Path "Sming").path >> $env:GITHUB_ENV
 
-    - name: Compiler Cache
-      uses: actions/cache@v4
-      with:
-        path: .ccache
-        key: ${{ matrix.os }}-ccache-${{ matrix.toolchain }}-${{ matrix.variant }}
-
     - name: Install build tools for Ubuntu / MacOS
       if: matrix.os != 'windows-latest'
       run: |
@@ -78,15 +72,21 @@ jobs:
     - name: Install build tools for Windows
       if: matrix.os == 'windows-latest'
       run: |
-         . Tools/ci/setenv.ps1
-         Tools/ci/install.cmd
+        . Tools/ci/setenv.ps1
+        Tools/ci/install.cmd
+
+    - name: Compiler Cache
+      uses: actions/cache@v4
+      with:
+        path: .ccache
+        key: ${{ matrix.os }}-ccache-${{ matrix.toolchain }}-${{ matrix.variant }}
 
     - name: Build and test for ${{matrix.variant}} on Ubuntu / MacOS
       env:
         CLANG_FORMAT: clang-format-8
       if: matrix.os != 'windows-latest'
       run: |
-        source $SMING_HOME/../Tools/export.sh
+        . Tools/export.sh
         Tools/ci/build.sh
 
     - name: Build and test for ${{matrix.variant}} on Windows

diff --git a/.github/workflows/library.yml b/.github/workflows/library.yml
@@ -121,12 +121,6 @@ jobs:
           /opt/esp32
         key: ${{ matrix.os }}-idf-${{ env.INSTALL_IDF_VER }}
 
-    - name: Compiler Cache
-      uses: actions/cache@v4
-      with:
-        path: .ccache
-        key: ${{ matrix.os }}-ccache-${{ matrix.variant }}-${{ matrix.arch == 'Esp32' && env.INSTALL_IDF_VER || '' }}
-
     - name: Install build tools for Ubuntu / MacOS
       if: matrix.os != 'windows-latest'
       run: |
@@ -139,6 +133,12 @@ jobs:
         . Tools/ci/setenv.ps1
         Tools/ci/install.cmd
 
+    - name: Compiler Cache
+      uses: actions/cache@v4
+      with:
+        path: .ccache
+        key: ${{ matrix.os }}-ccache-${{ matrix.variant }}-${{ matrix.arch == 'Esp32' && env.INSTALL_IDF_VER || '' }}
+
     - name: Build and Test for ${{matrix.arch}} on Ubuntu / MacOS
       env:
         CLANG_FORMAT: clang-format-8

diff --git a/Sming/Arch/Esp32/Tools/install.cmd b/Sming/Arch/Esp32/Tools/install.cmd
@@ -3,16 +3,24 @@ REM Esp32 install.cmd
 if "%IDF_PATH%"=="" goto :undefined
 if "%IDF_TOOLS_PATH%"=="" goto :undefined
 
-if exist "%IDF_PATH%" goto :cloned
+if "%INSTALL_IDF_VER%"=="" set INSTALL_IDF_VER=5.2
+
+echo.
+echo ** Installing Esp32 IDF version %INSTALL_IDF_VER%
+echo.
+
+for %%I in (%IDF_PATH%) do set IDF_CLONE_PATH=%%~dpIesp-idf-%INSTALL_IDF_VER%
+
+if exist "%IDF_CLONE_PATH%" goto :cloned
 
 if "%IDF_REPO%"=="" set IDF_REPO="https://github.com/mikee47/esp-idf.git"
-if "%INSTALL_IDF_VER%"=="" set INSTALL_IDF_VER=5.2
 set IDF_BRANCH="sming/release/v%INSTALL_IDF_VER%"
 
 if "%CI_BUILD_DIR%" NEQ "" (
     set IDF_INSTALL_OPTIONS=--depth 1 --recurse-submodules --shallow-submodules
 )
-git clone -b %IDF_BRANCH% %IDF_REPO% %IDF_PATH% %IDF_INSTALL_OPTIONS%
+git clone -b %IDF_BRANCH% %IDF_REPO% %IDF_CLONE_PATH% %IDF_INSTALL_OPTIONS%
+if errorlevel 1 goto :EOF
 
 goto :setup
 
@@ -26,24 +34,29 @@ goto :EOF
 
 :cloned
 echo.
-echo ** Skipping ESP-IDF clone: '%IDF_PATH%' exists
+echo ** Skipping ESP-IDF clone: '%IDF_CLONE_PATH%' exists
 echo.
 goto :setup
 
 
 :setup
 
-REM Skip installation for CI if already present
-if "%CI_BUILD_DIR%" NEQ "" (
+@echo Create link from "%IDF_PATH%" to "%IDF_CLONE_PATH%"
+if exist "%IDF_PATH%" rmdir /q %IDF_PATH%
+mklink /j %IDF_PATH% %IDF_CLONE_PATH%
+
+if "%CI_BUILD_DIR%"=="" (
+set IDF_TOOL_PACKAGES=*openocd*
+) else (
+    REM Skip installation for CI if already present
     if exist "%IDF_TOOLS_PATH%\tools" (
         goto :EOF
     )
 )
 
 REM Install IDF tools and packages
-python "%IDF_PATH%\tools\idf_tools.py" --non-interactive install
-python -m pip install %SMINGTOOLS%/gevent-1.5.0-cp39-cp39-win_amd64.whl
-python3 "%IDF_PATH%\tools\idf_tools.py" --non-interactive install-python-env
+python "%IDF_PATH%\tools\idf_tools.py" --non-interactive install install "*elf*" %IDF_TOOL_PACKAGES%
+python "%IDF_PATH%\tools\idf_tools.py" --non-interactive install-python-env
 
 if "%CI_BUILD_DIR%" NEQ "" (
     del /q "%IDF_TOOLS_PATH%\dist\*"

diff --git a/Sming/Arch/Esp32/Tools/install.sh b/Sming/Arch/Esp32/Tools/install.sh
@@ -65,7 +65,7 @@ ln -s "$IDF_CLONE_PATH" "$IDF_PATH"
 if [ -z "$CI_BUILD_DIR" ] || [ ! -d "$IDF_TOOLS_PATH/tools" ]; then
 
 # Install IDF tools and packages
-python3 "$IDF_PATH/tools/idf_tools.py" --non-interactive install
+python3 "$IDF_PATH/tools/idf_tools.py" --non-interactive install "*elf*"
 if [ -n "$VIRTUAL_ENV" ]; then
     unset VIRTUAL_ENV
     unset VIRTUAL_ENV_PROMPT

diff --git a/Sming/Arch/Esp32/build.mk b/Sming/Arch/Esp32/build.mk
@@ -86,30 +86,10 @@ IDF_PATH_LIST := \
 	$(ESP32_ULP_PATH)/$(ESP_VARIANT)ulp-elf-binutils/bin \
 	$(ESP32_OPENOCD_PATH)/openocd-esp32/bin
 
-ifeq ($(UNAME),Windows)
-DEBUG_VARS += ESP32_NINJA_PATH
-ifndef ESP32_NINJA_PATH
-ESP32_NINJA_PATH	:= $(call FindTool,tools/ninja/)
-endif
-ifeq (,$(wildcard $(ESP32_NINJA_PATH)/ninja.exe))
-$(error Failed to find NINJA)
-endif
-IDF_PATH_LIST += $(ESP32_NINJA_PATH)
-
-DEBUG_VARS += ESP32_IDFEXE_PATH
-ifndef ESP32_IDFEXE_PATH
-ESP32_IDFEXE_PATH := $(call FindTool,tools/idf-exe/)
-endif
-IDF_PATH_LIST += $(ESP32_IDFEXE_PATH)
-endif
-
 ifeq ($(ENABLE_CCACHE),1)
 export IDF_CCACHE_ENABLE := 1
 endif
 
-DEBUG_VARS += NINJA
-NINJA := $(if $(ESP32_NINJA_PATH),$(ESP32_NINJA_PATH)/,)ninja
-
 empty:=
 space:= $(empty) $(empty)
 

diff --git a/Sming/Arch/Esp8266/Tools/install.cmd b/Sming/Arch/Esp8266/Tools/install.cmd
@@ -3,12 +3,16 @@ REM Esp8266 install.cmd
 if "%ESP_HOME%" == "" goto :undefined
 if exist "%ESP_HOME%" goto :installed
 
+echo.
+echo ** Installing ESP8266 toolchain
+echo.
+
 set EQT_REPO=https://github.com/earlephilhower/esp-quick-toolchain/releases/download/3.2.0-gcc10.3
 set EQT_TOOLCHAIN=x86_64-w64-mingw32.xtensa-lx106-elf-c791b74.230224.zip
 
 mkdir %ESP_HOME%
-curl -Lo %DOWNLOADS%/%EQT_TOOLCHAIN% %EQT_REPO%/%EQT_TOOLCHAIN%
-7z -o%ESP_HOME% x %DOWNLOADS%/%EQT_TOOLCHAIN%
+curl -Lo %DOWNLOADS%/%EQT_TOOLCHAIN% %EQT_REPO%/%EQT_TOOLCHAIN% || goto :EOF
+7z -o%ESP_HOME% x %DOWNLOADS%/%EQT_TOOLCHAIN% || goto :EOF
 
 goto :EOF
 

diff --git a/Sming/Arch/Rp2040/Tools/install.cmd b/Sming/Arch/Rp2040/Tools/install.cmd
@@ -3,6 +3,10 @@ REM Rp2040 install.cmd
 if "%PICO_TOOLCHAIN_PATH%"=="" goto :undefined
 if exist "%PICO_TOOLCHAIN_PATH%/arm-none-eabi" goto :installed
 
+echo.
+echo ** Installing Rp2040 toolchain
+echo.
+
 set TOOLCHAIN_VERSION=13.2.rel1
 set TOOLCHAIN_BASE_URL=https://developer.arm.com/-/media/Files/downloads/gnu
 set TOOLCHAIN_NAME=arm-gnu-toolchain-%TOOLCHAIN_VERSION%-mingw-w64-i686-arm-none-eabi

diff --git a/Sming/build.mk b/Sming/build.mk
@@ -160,7 +160,7 @@ DEBUG_VARS += PYTHON
 ifdef PYTHON
 export PYTHON := $(call FixPath,$(PYTHON))
 else
-PYTHON := python3
+PYTHON := $(shell which python)
 endif
 
 PYTHON_VERSION := $(shell $(PYTHON) --version 2>&1)

diff --git a/Sming/project.mk b/Sming/project.mk
@@ -609,7 +609,7 @@ TCP_SERIAL_REDIRECT = $(SMING_HOME)/../Tools/tcp_serial_redirect.py $(COM_PORT)
 tcp-serial-redirect: ##Redirect COM port to TCP port
 	$(info Starting serial redirector)
 ifeq ($(WSL_COMPORT_POWERSHELL),1)
-	$(Q) cmd.exe /c start /MIN python3 $(WSL_ROOT)/$(TCP_SERIAL_REDIRECT)
+	$(Q) cmd.exe /c start /MIN python $(WSL_ROOT)/$(TCP_SERIAL_REDIRECT)
 else
 	$(Q) $(call DetachCommand,$(PYTHON) $(TCP_SERIAL_REDIRECT))
 endif

diff --git a/Tools/check-packages.cmd b/Tools/check-packages.cmd
@@ -0,0 +1,18 @@
+@echo off
+
+REM Check pre-requisites are installed and working, from whatever source
+
+set MISSING_PACKAGES=
+call :check_package python Python
+call :check_package mingw-get MinGW
+call :check_package cmake CMake
+call :check_package ninja Ninja
+where /q 7z
+if errorlevel 1 set MISSING_PACKAGES=%MISSING_PACKAGES% 7zip
+goto :EOF
+
+REM Could use 'where' but Windows has a stub for python which isn't helpful
+:check_package
+%1 --version >nul 2>&1
+if errorlevel 1 set MISSING_PACKAGES=%MISSING_PACKAGES% %2
+goto :EOF
diff --git a/Tools/choco-install.cmd b/Tools/choco-install.cmd
@@ -4,16 +4,50 @@ REM
 REM Sming installer bootstrap script for Windows
 REM
 
+echo.
+echo Welcome to Sming bootstrap installer
+echo.
+
+REM Check we have administrative priviledge
+net session >nul 2>&1
+if errorlevel 1 (
+    echo Opening an administrative command prompt to continue installation...
+    powershell -Command "Start-Process cmd.exe -ArgumentList '/K %~f0' -Verb RunAs"
+    goto :EOF
+)
+
+REM Sync working directory
+cd /d %~dp0
+
+REM See if choco is already installed
+choco --version >nul 2>&1
+if not errorlevel 1 goto :install_packages
+
 REM Fetch the Chocolatey powershell installer script.
 REM Note: Local scripts are blocked by default for 'security reasons'
 REM Microsoft have kindly set up some hoops for us to jump through
 REM (presumably making this 'secure'). OK, lets jump now.
 REM Warning: Get the fire extinguisher, hoops may be on fire.
 @powershell -NoProfile -ExecutionPolicy unrestricted -Command "iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))"
-if "%errorlevel%" == "1" then goto :EOF
+if errorlevel 1 goto :EOF
+
+setlocal
+SET PATH=%ALLUSERSPROFILE%\chocolatey\bin;%PATH%
+
+:install_packages
+
+REM Install standard packages
+choco install -y git Python CMake Ninja 7zip ccache || goto :EOF
 
-SET PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin
+REM Fetch Sming
+echo Fetch sming...
+git clone https://github.com/SmingHub/Sming sming || goto :EOF
+REM Fix ownership - we'd prefer to just clone as a regular user but that requires impersonation. Don't go there.
+icacls sming /setowner %USERNAME% /T /C /L /Q
 
-choco sources add -name smingrepo -source 'https://www.myget.org/F/sming/'
+REM Install MinGW: Fine to leave this with admin priviledge as it should be considered read-only
+call sming\tools\mingw-install.cmd
+if errorlevel 1 goto :EOF
 
-choco install -y sming
+echo.
+echo OK. Now open a regular (non-administrative) command prompt and run 'sming\tools\install.cmd' to install required toolchains
diff --git a/Tools/ci/build.cmd b/Tools/ci/build.cmd
@@ -9,6 +9,13 @@ if "%BUILD_DOCS%"=="true" (
     goto :EOF
 )
 
+REM Configure ccache
+if "%ENABLE_CCACHE%"=="1" (
+    ccache --set-config cache_dir="%CI_BUILD_DIR%\.ccache"
+    ccache --set-config max_size=500M
+    ccache -z
+)
+
 cd /d %SMING_HOME%
 call Arch\%SMING_ARCH%\Tools\ci\build.setup.cmd || goto :error
 

diff --git a/Tools/ci/build.sh b/Tools/ci/build.sh
@@ -9,6 +9,13 @@ if [ "$BUILD_DOCS" = "true" ]; then
     exit 0
 fi
 
+# Configure ccache
+if [ "$ENABLE_CCACHE" == "1" ]; then
+    ccache --set-config cache_dir="$CI_BUILD_DIR/.ccache"
+    ccache --set-config max_size=500M
+    ccache -z
+fi
+
 # Build times benefit from parallel building
 export MAKE_PARALLEL="make -j$(nproc)"
 

diff --git a/Tools/ci/install.cmd b/Tools/ci/install.cmd
@@ -13,14 +13,19 @@ if "%BUILD_DOCS%" == "true" (
     set INSTALL_OPTS=doc
 )
 
-call %SMING_HOME%\..\Tools\install.cmd %SMING_ARCH% %INSTALL_OPTS%
+REM Python and CMake are preconfigured
+choco install ninja ccache -y --no-progress || goto :error
 
-REM Configure ccache
-if "%ENABLE_CCACHE%"=="1" (
-    ccache --set-config cache_dir="%CI_BUILD_DIR%\.ccache"
-    ccache --set-config max_size=500M
-    ccache -z
-)
+call %~dp0..\mingw-install.cmd || goto :error
+
+call %~dp0..\install.cmd %SMING_ARCH% %INSTALL_OPTS% || goto :error
 
 REM Clean up tools installation
-python "%SMING_HOME%\..\Tools\ci\clean-tools.py" clean --delete
+python "%~dp0clean-tools.py" clean --delete
+
+goto :EOF
+
+
+:error
+echo Failed with error #%errorlevel%.
+exit /b %errorlevel%
diff --git a/Tools/ci/install.sh b/Tools/ci/install.sh
@@ -23,13 +23,6 @@ sudo chown "$USER" /opt
 
 fi
 
-# Configure ccache
-if [ "$ENABLE_CCACHE" == "1" ]; then
-    ccache --set-config cache_dir="$CI_BUILD_DIR/.ccache"
-    ccache --set-config max_size=500M
-    ccache -z
-fi
-
 # Clean up tools installation
 source "$SMING_HOME/../Tools/export.sh"
 python "$SMING_HOME/../Tools/ci/clean-tools.py" clean --delete