[Report] Detecting new versions of Visual C++ Runtime (and other dependencies)

[Arcitec]

#259 introduced versioned URLs which means that we can no longer detect when the upstream Visual C++ runtime has new releases with bugfixes.

We used to get hash errors since we retrieved the "latest version" URL at all times. Now we silently install older versions instead. This leads to a risk of not discovering updates.

But I wrote a small script to demonstrate how we can detect when new runtimes are available. Does Bottles have a dependency version checker/updater script somewhere where this can be integrated?

import urllib.request

urls = [
    # vcredist2019
    "https://aka.ms/vs/16/release/VC_redist.x86.exe",
    "https://aka.ms/vs/16/release/VC_redist.x64.exe",
    # vcredist2022
    "https://aka.ms/vs/17/release/VC_redist.x86.exe",
    "https://aka.ms/vs/17/release/VC_redist.x64.exe",
]

for url in urls:
    request = urllib.request.Request(url, method="HEAD")
    try:
        response = urllib.request.urlopen(request)
        final_url = response.geturl()
        print(f'"{url}": "{final_url}"')
    except urllib.error.URLError as e:
        print(f"Error fetching URL {url}: {e}")

Result:

"https://aka.ms/vs/16/release/VC_redist.x86.exe": "https://download.visualstudio.microsoft.com/download/pr/5efdbcb7-d3bd-4432-a2fb-b267c386e2f3/49545CB0F6499C4A65E1E8D5033441EEEB4EDFAE465A68489A70832C6A4F6399/VC_redist.x86.exe"
"https://aka.ms/vs/16/release/VC_redist.x64.exe": "https://download.visualstudio.microsoft.com/download/pr/453680ea-b88a-411f-80fd-5db37fdc9dbb/5D9999036F2B3A930F83B7FE3E2186B12E79AE7C007D538F52E3582E986A37C3/VC_redist.x64.exe"
"https://aka.ms/vs/17/release/VC_redist.x86.exe": "https://download.visualstudio.microsoft.com/download/pr/5319f718-2a84-4aff-86be-8dbdefd92ca1/DD1A8BE03398367745A87A5E35BEBDAB00FDAD080CF42AF0C3F20802D08C25D4/VC_redist.x86.exe"
"https://aka.ms/vs/17/release/VC_redist.x64.exe": "https://download.visualstudio.microsoft.com/download/pr/c7dac50a-e3e8-40f6-bbb2-9cc4e3dfcabe/1821577409C35B2B9505AC833E246376CC68A8262972100444010B57226F0940/VC_redist.x64.exe"

If there's no infrastructure for detecting dependency updates yet, then it could be an excellent idea to extend the current dependency .yml format so that any field which takes a url: can also take a x-version-checker: array with details such as how and what to check (similar to Flathub's flatpak-external-data-checker.

There, we'd be able to say "check the redirect-target of this URL, and if the target URL changes, hash the new file and its size and update the dependency manifest with the new URL". It may even be possible to use the Flatpak checker library to do it, since it's written in Python and has all functions we'd need.

Their "rotating-url" checker, without any "pattern" specified, is able to follow redirects and detect new redirect URLs, for example.

Ping: @mirkobrombin @commiterate

[Arcitec]

Speaking of which... https://github.com/bottlesdevs/dependencies/blob/main/Essentials/vcredist2022.yml is outdated. ;)

[commiterate]

I don't think dependency updates is something that can be easily automated since there may be metadata in the dependency manifest (e.g. full version numbers) which needs manual intervention to update.

For Microsoft Visual C++ Redistributable (vcredist*) versions, I used this repo to get both the version-specific download links and the human-readable version (e.g. 14.42.34433.0 which is now the newest version).

https://github.com/abbodi1406/vcredist/tree/master/source_links

The tool would only really tell us that something is out of date, but the download links in certain cases aren't very enlightening. For example, the vcredist resolved links just contain a bunch of hashes in the URL and no actual version number indicator. You'd need to dissect the binary itself to get that information.

[Arcitec]

@commiterate After getting the URL, if we see a changed URL, we need to download it and hash it. At the same time we'll use pefile to extract the exe version.

[Arcitec]

Here's a quick example of how we'll do that:

from pathlib import Path
import pefile


def get_exe_version(file_path: Path) -> str:
    pe = pefile.PE(file_path)
    
    # Access the VS_VERSIONINFO resource in the PE file.
    for file_info_section in pe.FileInfo:
        for file_info in file_info_section:
            if file_info.Key == b"StringFileInfo":
                for string_table in file_info.StringTable:
                    if b"FileVersion" in string_table.entries:
                        return string_table.entries[b"FileVersion"].decode("utf-8")

    return "Version information not found"

file_path = Path.home() / ".var/app/com.usebottles.bottles/data/bottles/temp/vcredist2022_x64.exe"
version = get_exe_version(file_path)
print(f"EXE Version: {version}")

Result:

EXE Version: 14.42.34430.0

[commiterate]

Does that library handle .msi files as well? The Wikipedia page for Portable Executable linked in the README lists a bunch of extensions but .msi (for Microsoft/Windows Installer) doesn't seem to be among those.

[Arcitec]

MSI is not a PE format. They are some kind of zip-like format with embedded CAB archives.

A RedHat employee wrote a tool for inspecting those files:

https://manpages.ubuntu.com/manpages/focal/man1/msiinfo.1.html

It's part of msitools:

https://wiki.gnome.org/msitools

Those can be called via subprocess(). So yeah that's solvable. There's also a package which bundles those binaries as a Python package here:

https://pypi.org/project/packagedcode-msitools/

Edit: MSI files seem to store their version in a Property named ProductVersion.

Edit: There's also a reference here from Microsoft. Seems like PID_REVNUMBER and PID_TITLE are the interesting fields.

https://learn.microsoft.com/en-us/windows/win32/msi/summary-information-stream-property-set

The Unix msiinfo tool may even display it directly when querying msiinfo suminfo thefile.msi since it's listed as a "package summary" command and probably includes version (I have not tried this tool).

[commiterate]

I don't see an existing auto-update tool, but it'll probably be straightforward to write something that parses all the dependency YAML files, looks at the Steps list, and updates the links + hashes, then write out the changes.

Version information in the Description block is a bit all over the place so not sure how that should be handled. We definitely need it though for multi-binary dependencies like vcredist where the x86 and x64 installer versions should be identical.

It's not clear how the tool would handle that case if there's version mismatches between the variants, so it probably just needs to give up and ask for human intervention.

Also not clear on how to communicate that multiple install binaries in a given dependency manifest are just architectural variants rather than a dependency that just wraps multiple programs into a single manifest.

That might need a schema change for the install directives where all variants are encoded in a single install step. For example:

Steps:
- action: install_exe
  variants:
    - for:
        - win32
      url: "{x86 installer URL}"
      checksum: "{md5 checksum}"
      file_size: "{file size}"
      arguments: "{arguments}"
    - for:
        - win64
      url: "{x64 installer URL}"
      checksum: "{md5 checksum}"
      file_size: "{file size}"
      arguments: "{arguments}"

[Arcitec]

Yeah, and the dependency updater would need to know the upstream URL that points at the latest version, which is where something like this becomes necessary:

Steps:
  - action: install_exe
    file_name: VC_redist.x86.exe
    url: https://download.visualstudio.microsoft.com/download/pr/e9ff90f1-424e-4489-9302-28cbaed0fec1/E57FF114114F08F97977887A56975AF754374888E534D87622CEFEB7448653AE/VC_redist.x86.exe
    rename: vcredist2022_x86.exe
    file_checksum: 822551b098e93c504de2c7865216928f
    file_size: 13949168
    arguments: /quiet /norestart
    x-version-checker:
        type: rotating-url
        url: https://aka.ms/vs/17/release/VC_redist.x86.exe

Type is inspired by Flathub's automatic dependency updater. They have the rotating-url implementation. In case we add their repo as a dependency and directly import their functions for our use, since they have some interesting checkers such as "latest github release" etc. Although we don't know yet what checkers we'll need besides the "URL redirect" one, so might as well start with just implementing URL redirect check as in my first code example. Can always swap out and expand later.

So... an initial plan would be:

• Load with PyYAML.
• Look at all Steps.
• Look for x-version-checker.
• Look for a supported type: (such as rotating-url).
• Do the URL check. If the redirect URL differs, then:
  • Update the step's url property.
  • Download the file.
  • Hash it and check the filesize, then update the file_checksum and file_size.
  • Get version information:
    • For exe files, use pelib as shown here.
    • For MSI files, use msitools as shown here

Edit: And you are right, there's no clear solution for how to denote the version when a dependency installer has multiple EXE/MSI files in its Steps. So we'd need to think about that... I can imagine a YAML layout where one of the Steps is marked as "this is the file to get the version information from". Something like a get_version_info: true field. Which is only legally allowed to be on ONE Step. And then instead of having version randomly somewhere in the Description, we could have a dedicated Version field at the root of the YAML.

Something like this (* shows new fields):

Name: vcredist2022
Description: Microsoft Visual C++ Redistributable (2015-2022)
*Version: 14.42.34430.0
Steps:
  - action: install_exe
    file_name: VC_redist.x86.exe
    url: https://download.visualstudio.microsoft.com/download/pr/e9ff90f1-424e-4489-9302-28cbaed0fec1/E57FF114114F08F97977887A56975AF754374888E534D87622CEFEB7448653AE/VC_redist.x86.exe
    rename: vcredist2022_x86.exe
    file_checksum: 822551b098e93c504de2c7865216928f
    file_size: 13949168
    arguments: /quiet /norestart
    *x-version-checker:
        type: rotating-url
        url: https://aka.ms/vs/17/release/VC_redist.x86.exe
        *get_version_info: true

The Bottles GUI could show the dependency description as "{Description} v{Version}" then.

Edit: And if it's difficult to extract version for some dependency (nested zip files perhaps), the manifest would omit get_version_info and the updater would then emit a warning to remind the person to manually update the Version field when a dependency update was triggered.

To do that, it's a matter of counting how many get_version_info: true exist inside all x-version-checker in the entire manifest, then:

• If 0, emit Warning: Dependency "vcredist2022" has been updated. Please manually set its Version field.
• If 2+, emit Error: Dependency "vcredist2022" has multiple "get_version_info" sources. Please fix the manifest.
• If 1, perfect. Use that file to extract the Version tag.