Windows Bug: Shorten your project path down to no more than 10 characters

[majorzpley]

I was compiling the /inro/hardware-check project,but i met this issue:

error: failed to run custom build command for esp-idf-sys v0.33.7

Caused by:
process didn't exit successfully: C:\Users\wyx12\Desktop\std-training\intro\hardware-check\target\debug\build\esp-idf-sys-647092daf06bb09c\build-script-build (exit code: 1)
--- stderr
Error: Too long output directory: \\?\C:\Users\wyx12\Desktop\std-training\intro\hardware-check\target\riscv32imc-esp-espidf\debug\build\esp-idf-sys-fef687b8d1970f2c\out. Shorten your project path down to no more than 10 characters (or use WSL2 and its native Linux filesystem). Note that tricks like Windows subst do NOT work!
warning: build failed, waiting for other jobs to finish...

[Vollbrecht]

Its not a bug, but a problem on the windows platform where you cant have long path names as a default. For more info you can read about it here As you see they are ways to allow for bigger file_path length.

After reading the windows article and understand + tested that your system can work with that long path names you can check out this env var and set it to ignore the error. Keep in mind that if you don't take the steps and just ignore the error we cant guarantee that the build system runs into unknown wired errors on windows.

Simplar solutions might be to just do what the error message tell's you to do -> Move the project directly into C:/ or some othere base folder or use WSL

[majorzpley]

fine，thanke for ur answer.

[David-OConnor]

Hi! Is there a reason this is coming up here in particular? Running the template program for ESP-IDF (Not non-IDF) along triggers this. Of note, I've not encountered this before, and have used Rust quite a bit on PCs and embedded. Thanks!

Does this fix involve forking esp-idf-sys to modify build/common.rs? edit: No: Instructions here

[Vollbrecht]

1. It has nothing to do with rust
2. The esp-idf build system produces long path names, they sometimes become quite long
3. You don't need to fork or modify anything
4. Either use ultra short path's as a base - use the method described in the microsoft article to allow for longer path names and then set the env variable to ignore it - or use WSL as your dev env.

[David-OConnor]

I appreciate the explanation! I got it working by following the instructions in the linked thread, eg the addition to config.TOML.

[ivmarkov]

I appreciate the explanation! I got it working by following the instructions in the linked thread, e.g. the addition to config.TOML.

This will work until it stops. Setting the parameter to warn or ignore is NOT a solution.
Also, the linked Windows article does NOT solve the problem. People do not read it fully - it is not enough to do a regedit. You need to have ALL executables which participate in the build modified with a special manifest, and I'm pretty sure most of them don't have that manifest.

Really, only two options:

• Use very short project path names
• Switch to WSL

[hydra]

I'm coming here as someone that just added esp-idf-sys as a dependency to a template project and I now can't build and get the same error message and came here.

I'd just like to say that:
a) it's 2024 not 1970.
b) using WSL is not an option, don't want it, don't need it, don't want to maintain it, manage it, update it, no no no. if i wanted Linux I'd be running a linux system. I'm running Windows, my users use windows, I have to use Windows.
c) polluting the root of a drive with projects is bad filesytem organization, we have directories and paths that allow a decent folder structure for a reason.
d) the message doesn't say what a 'project path' is, it only shows me the 'output directory' so I've no idea what to change anyway.
e) forcing a user to relocate all their files, which often includes updating many paths, launch configurations, etc, just because of a dependency addition, is madness.

IMHO, more effort should be placed in fixing the underlying issue.

[ivmarkov]

Folks, if you really feel strongly about having long path-names support on native Windows, then you can actually help!

The issue at hand is not easy, but it isn't rocket science either!
Here's a rough action plan:

• Read this
• Enable the reg key mentioned above using regedit
• Disable the check in esp-idf-sys that checks long path names (check the BUILD-OPTINOS.md readme in the repo how to disable or turn into a warning)
• Generate a really long binary project name using esp-idf-template. By the way, last time I checked ESP IDF was choking on path names with spaces in it, not sure if they addressed it in the meantime, but maybe one step at a time, and let's just not use spaces in the really long project name initially, shall we?
• (Here's the hard part)
  • Audit every single executable which is called during the esp-idf-sys build process, whether it has the Application Manifest key mentioned in the above article
  • If it does, great!
  • If it doesn't:
    • report here
    • ... and then we need to figure out who to poke so that a newer executable with long path names support is used during the build (this is primarily ESP IDF build toolchain, including CMake, ninja and whatnot but also GIT) - you can find the ESP-IDF executables which are auto-downloaded (except git and python that you need to have pre-installed) in the ./embuild directory that will be created inside your project

Well, that's it! We need a hero with a fast machine running Windows who is willing to help!

[ivmarkov]

BTW: Ideally, the above exercise should be done separately for:

• ESP IDF 4.4X
• ESP IDF 5.0 or greater

That's because every single ESP IDF release might be using its own toolchain (for example, the GCC toolchain for ESP IDF 4.4.X is different than the one for ESP IDF 5.0; not sure if they upgraded it for 5.1 or 5.2)

... but: let's initially just concentrate on a recent ESP IDF 5.X. Say, ESP IDF 5.1.X

• ESP IDF 4.4 is getting out of support soon
• Even if older versions don't work, that would become less and less of a problem over time

[ivmarkov]

BTW: Ideally, and if every executable used has the manifest key enabled, then you should build - the project-with-the-really-long-path-name - without issues. Unfortunately, this definitely wasn't the case before. But maybe things had changed with recent ESP IDF 5.X versions?

[Vollbrecht]

IMHO, more effort should be placed in fixing the underlying issue.

This project is free & open source software. If you want to prioritize it feel free to work on it if you have questions, we can try to help answer them. Demanding time from other people is a pretty selfish thing to do.

using WSL is not an option, don't want it, don't need it, don't want to maintain it, manage it, update it, no no no. if i wanted Linux I'd be running a linux system. I'm running Windows, my users use windows, I have to use Windows.

If you reflect on what you are saying here for a short amount of time. Its the year 2024 and Microsoft still struggles with basic shell input output. Yes they fixed the problem that you now can have longer absolute path. But wait have a look, oh no relative path's are still a huge pile of shit cant have more than 260 charts-12 chars for a potential filename. Oh you can now "opt-in" for a new behavior since a certain windows version builds. Benevolent. It works only on a subset of the windows api - nice. No problem we runtime detect a) that the user opt-in to that new max path length b) make sure that in every binary used on windows only the allowed functions are called. That needs to be checked for "git cloning submodules" (creates pretty long relative paths), rustc buildartifacts, cmake, ninja, xtensa & riscv gcc for there linker and python and the esp-idf-component manager for the esp-idf components.

So yeah rust build artifacts sometimes have longer path and it gets worse when that is piped in mixed into cmake. We have multiple places in the buildsystem already where we try to mitigate this kinds of problems. For example we are writing path output into a file and retrieve them later "only on windows" because it cant cope. Still its not enough, since its only the part of the rustbuild itself that we controll there. And why all that, because you are using a system that is shit, blaming us for it and most of us don't even use that system.

Would i love to have it working? Obvious.
Would it be easier for us to just say Windows is not supported? Yes.
Would i personally spend 50h to fix the underlying issue and tripple check everything making sure its working. Since we have a "workaround" and we have also alot of other stuff that can see love i would say most likely NO. The workaround is not good enough for you? Fine go ahead provide the fix.
Would i help someone fixing the issue if he has questions. That was never a debate.

I'm coming here as someone that just added esp-idf-sys as a dependency to a template project and I now can't build and get the same error message and came here.

Its unfortunate, but that is just reality. No one is claiming that the current situation is nice.

[hydra]

Hi Fredrick,

IMHO, more effort should be placed in fixing the underlying issue.

This project is free & open source software. If you want to prioritize it feel free to work on it if you have questions, we can try to help answer them. Demanding time from other people is a pretty selfish thing to do.

I wasn't demanding, IMHO means In My Honest Opinion. I'm just hoping, that someone, alas not me as I'm rather time poor at the moment, can work or assist on this issue. I contribute to open-source projects where I can, sometimes with a PR, sometimes by giving other devs what the need to fix issues, logs, testing, etc, and sometimes like today by simply giving an opinion so that devs and maintainers better understand users of the things they create.

using WSL is not an option, don't want it, don't need it, don't want to maintain it, manage it, update it, no no no. if i wanted Linux I'd be running a linux system. I'm running Windows, my users use windows, I have to use Windows.

If you reflect on what you are saying here for a short amount of time. Its the year 2024 and Microsoft still struggles with basic shell input output.

I'm aware that there's a lot of legacy baggage with Windows and MS software, but the same is also true of Linux, Posix, BSD, and OSX, just in different ways...

I'm coming here as someone that just added esp-idf-sys as a dependency to a template project and I now can't build and get the same error message and came here.

Its unfortunate, but that is just reality. No one is claiming that the current situation is nice.

we agree on that at least 😄

[David-OConnor]

Hydra - at least there's a feasible (although awkward!) workaround here - BF, PX4, and AP all are linux-only toolchains... for an ARM target! That one is beyond me.

[ivmarkov]

Folks, perhaps what is not so clear is that Espressif is not really investing developer time in the esp-idf-* crates (unlike baremetal).

I can (and probably should) have lobbied for this to change, but alas, that's the status quo. Which reminds me that @Vollbrecht perhaps we should be a tad more explicit about that in the readme's of all esp-idf-* crates.

Because otherwise what happens is us, the two maintainers (and all other contributors of course) who dedicate free time to maintain and improve - - we get patronizing comments like this (and I'm 26 years in this industry, so when i read these I'm not sure whether I should be angry or rolling out laughing really) which are not exactly motivating. :)

Anyway and back on topic: perhaps the rabbit hole is not as deep as I thought. First, checking the executables shouldn't be that hard as they have to manifest themselves that they are long-path compatible, with the app XML manifest described by Microsoft which is easy to check. Second - recollecting what @N3xed mentioned in the past - I think the main offender at the time was collect2 and probably git. So if these are fixed in the meantime (rust and cargo surely are - at least the msvcrt versions) maybe we have a chance!

[hydra]

Folks, perhaps what is not so clear is that Espressif is not really investing developer time in the esp-idf-* crates (unlike baremetal).

as a n00bie to the esp32 rust ecosystem that wasn't clear to me. I'm just reporting issues and commenting as I progress though things, in the hopes that some action is taken at some point.

Because otherwise what happens is us, the two maintainers (and all other contributors of course) who dedicate free time to maintain and improve - - we get patronizing comments like this (and I'm 26 years in this industry, so when i read these I'm not sure whether I should be angry or rolling out laughing really) which are not exactly motivating. :)

yeah, I've been a software developer for 40 years now (C64/BBC Micro B) and fully understand dedicating free time as I myself have been doing for years. Agreed with the sentiment of laugh vs cry on so many things, on this issue I was crying though due to the pain of error message and seeing others with the same problem coming here and reporting issues like this. 😢

maybe we have a chance!

It that would be AWESOME to see some progress on this, when someone has time of course. It would really help out all the users coming here trying out these esp-idf-* crates.

[georgik]

@ivmarkov Regarding the manifest. I did a validation with Rust calling another binary stored in deep directory.

The result is that manifest does not help much, because it's just opt-in which tell Windows not to truncate some paths. It does not influence EXEC.

So the only option is that all tools must be using UNC \\?\ prefix and Unicode Windows API.

When I change the execution to UNC path in Rust, then it's able to enter even deep directories like 5000 characters and execute application even without manifest.

Our Toolchain team will take a look about possibilities to build tools with enabled long paths.

[ivmarkov]

@ivmarkov Regarding the manifest. I did a validation with Rust calling another binary stored in deep directory.

The result is that manifest does not help much, because it's just opt-in which tell Windows not to truncate some paths. It does not influence EXEC.

So the only option is that all tools must be using UNC \\?\ prefix and Unicode Windows API.

When I change the execution to UNC path in Rust, then it's able to enter even deep directories like 5000 characters and execute application even without manifest.

Our Toolchain team will take a look about possibilities to build tools with enabled long paths.

@georgik Let's first maybe get to the bottom of this before changing everything to handle UNC (which might be next to impossible for a legacy code base)

• Besides creating the manifest, did you also change the reg-key as per the article I linked and rebooted your machine?
• Is your Windows version later than the one described in the article?
• Do you have admin rights on the machine?

A useful resource for understanding the details might be this issue I just found out and I'm reading on.
But at least the initial comments do seem to suggest that manifest + reg key is a valid alternative to using UNC everywhere. Maybe not for all Windows APIs but only for a subset (i.e. EXEC might need UNC unconditionally), but still, this subset could make all the difference w.r.t. required effort?

[georgik]

All tests were done with enabled LongPath registry.
BTW: Windows idf-installer has check and sets this registry, this behavior was added long time ago.

Tests were conducted both on latest Windows 10 and the latest Windows 11.

Registry and Manifest helps with accessing files. It does not solve exec problem. The execution is only possible with UNC path.

As @igrr mentioned: As is stated in the docs that the manifest removes the path length limitations for file and directory APIs only. It doesn't include CreateProcess* APIs which I guess Rust's Command::new implementation is using.

[ivmarkov]

@georgik I think you are (unfortunately) probably right in the following: the absence of the app manifest does not mean, that the app is incompatible with long paths! It might be using only UNC paths all the way!

So... we can't really tell easily if an app is long-paths compatible or not just by looking if it has the manifest or not. We need to test the app on very long path names, as my original comment actually also suggests.

[ivmarkov]

All tests were done with enabled LongPath registry. BTW: Windows idf-installer has check and sets this registry, this behavior was added long time ago.

Tests were conducted both on latest Windows 10 and the latest Windows 11.

Registry and Manifest helps with accessing files. It does not solve exec problem. The execution is only possible with UNC path.

As @igrr mentioned: As is stated in the docs that the manifest removes the path length limitations for file and directory APIs only. It doesn't include CreateProcess* APIs which I guess Rust's Command::new implementation is using.

Right. And as I said:

Maybe not for all Windows APIs but only for a subset (i.e. EXEC might need UNC unconditionally), but still, this subset could make all the difference w.r.t. required effort?

I mean, if you think converting all the Espressif tooling to only handle UNC paths is feasible, great then.
And by the way, I don't fear our own (Rust) based tooling, like espup and friends. It is about the ESP-IDF tooling, like GCC & friends.

[ivmarkov]

By the way what you (and I) call a "UNC" path, and what you've used in your example is called "verbatim" path in this table. Not that this is an official terminology, but UNC path is something else.

This link seems to imply that GCC STILL has long-path problems, nobody has taken the effort to fix that yet inside their codebase, and seems to suggest that slapping an app manifest to the GCC executable would alleviate at least some problems.

[ivmarkov]

BTW: Windows idf-installer has check and sets this registry, this behavior was added long time ago.

I don't think folks using the esp-idf-* crates are using the Windows idf-installer thing. Why should they?
Just mentioning in case you think we should not tell them to touch the registry setting by themselves.