Environment resolution uses UNC paths with Python 3.10, 3.7 was using mounted network drives. #1438

KelSolaar · 2023-02-02T20:09:34Z

Hello,

As we are trying to update our Rez version from 2.91, I noted that the resolution now uses UNC paths:

Rez v2.112.0

C:\Users\tmansencal>rez-env dcraw

You are now in a rez-configured environment.

resolved by tmansencal@***, on Fri Feb 03 08:48:25 2023, using Rez v2.112.0

requested packages:
dcraw
~platform==windows           (implicit)
~arch==AMD64                 (implicit)
~os==windows-10.0.***.SP0  (implicit)

resolved packages:
arch-AMD64        c:\***\packages\***\arch\AMD64
dcraw-9.27        \\***\***\packages\thirdparty\dcraw\9.27\platform-windows\arch-AMD64
platform-windows  c:\***\packages\***\platform\windows

Rez v2.2.91.0

C:\Users\tmansencal>rez-env dcraw

You are now in a rez-configured environment.

resolved by tmansencal@***, on Fri Feb 03 08:53:09 2023, using Rez v2.91.0

requested packages:
dcraw
~platform==windows           (implicit)
~arch==AMD64                 (implicit)
~os==windows-10.0.***.SP0  (implicit)

resolved packages:
arch-AMD64        c:\***\packages\***\arch\AMD64
dcraw-9.27        s:\packages\thirdparty\dcraw\9.27\platform-windows\arch-AMD64
platform-windows  c:\***\packages\***\platform\windows

Environment

OS: Window 10
Rez version (eg "2.100.0"): v2.112.0
Rez python version (output of "rez-python --version"): Python 3.10.9

I do think that this is strictly caused by the Python difference but would be keen to know if anyone has seen that already before I start digging.

To Reproduce

Build Rez with Python 3.7, resolve environment from mounted drive.
Build Rez with Python 3.10, resolve environment from mounted drive.
Compare the resolved package paths.

Expected behavior
The mounted drive package path should be used as UNC paths wreck havoc in many CMD scripts.

Actual behavior
The UNC package path is used instead.

The text was updated successfully, but these errors were encountered:

flord · 2023-02-06T19:24:38Z

I would argue this should be a config. Mapped drive letters can wreck havoc in other setups, like the one we have here. We only use powershell where UNC works fine.

KelSolaar · 2023-02-06T19:30:22Z

It is already wrecking havoc! :) The default behaviour on something as critical as the way paths are handled should not change without huge warnings.

instinct-vfx · 2023-02-06T20:02:58Z

I would expect that the config is used as-is and not transformed. Can you share your config (or any environment overrides) respectively?

KelSolaar · 2023-02-06T20:07:11Z

@instinct-vfx: There is nothing extra-ordinary here, only the packages_path variable that sets various Windows paths, all of them being drive letter-based.

instinct-vfx · 2023-02-06T21:01:10Z

I just did parallel installs of 2.91.0 and 2.112.0 and i can not reproduce this. Are you sure there are not other differences (one installation being python2.7 and one being python3.x maybe?)

KelSolaar · 2023-02-06T21:03:27Z

Hey @instinct-vfx,

As I was saying in the description, I think it is related to the Python version and not Rez itself, or indirectly at least. Did you try with Python 3.7 vs 3.10?

Cheers,

Thomas

instinct-vfx · 2023-02-06T21:22:38Z

Sorry, i must have skipped that on first read :( I did try with 3.9 in my case. We will need to find out which specific change at what version broke this and see what's the best way to make it behave backwards compatible. These are really tricky to detect. Next time we can detect it by adding tests, but it's unlikely to change again anytime soon i guess :/

yanshil · 2023-02-17T18:22:10Z

Hey guys, let me give you an insight.

This is an issue caused by different behavior of how UNC path resolved by os.path.realpath. It occurs at py-3.7 <-> py-3.8

Note: The rez version on the screenshot is not the actual one. We applied some local changes but didn't version if with another tag.

yanshil · 2023-02-17T18:26:07Z

And, this is the path resolve logic : https://github.com/AcademySoftwareFoundation/rez/blob/master/src/rez/utils/filesystem.py#L526

KelSolaar · 2023-02-17T19:56:38Z

Great finding @yanshil!

KelSolaar · 2023-02-19T21:44:12Z

Confirming the behaviour on our end:

C:\Users\tmansencal>rez-env python-2.7 -- python -c "import os;print(os.path.realpath('S:\\****'))"
S:\****

C:\Users\tmansencal>rez-env python-3.6 -- python -c "import os;print(os.path.realpath('S:\\****'))"
S:\****

C:\Users\tmansencal>rez-env python-3.7 -- python -c "import os;print(os.path.realpath('S:\\****'))"
S:\****

C:\Users\tmansencal>rez-env python-3.9 -- python -c "import os;print(os.path.realpath('S:\\****'))"
\\****\****\****

C:\Users\tmansencal>rez-env python-3.10 -- python -c "import os;print(os.path.realpath('S:\\****'))"
\\****\****\****

KelSolaar · 2023-02-20T00:43:03Z

For completeness:

JeanChristopheMorinPerso · 2023-02-22T01:32:24Z

Thanks @yanshil for jumping in! I'm not a Windows expert and I also don't have the bandwidth right now to look into this specific issue. What do you think would be the next step on this? WOUld there be a relatively easy fix to this? @instinct-vfx any opinion?

KelSolaar · 2023-02-22T01:49:20Z

I think we should first figure out what was the intent for using os.path.realpath. Does it make sense to resolve symlinks in this context or would os.path.abspath suffice here? Maybe @nerdvegas can comment?

JeanChristopheMorinPerso · 2023-02-22T02:32:18Z

Here is where the canonical_path function is called. Each call site has a comment explaining why it's called.

rez/src/rez/resolved_context.py

Lines 1807 to 1811 in 8968662

    
           # Must make canonical otherwise a symlinked path will cause it not 
        
           # to match the repo location, which is always canonical. 
        
           # 
        
           location = os.path.join(bundle_path, repo_path) 
        
           location = canonical_path(location, platform_)

rez/src/rezplugins/package_repository/filesystem.py

Lines 498 to 500 in 8968662

    
           # ensure that differing case doesn't get interpreted as different repos 
        
           # on case-insensitive platforms (eg windows) 
        
           location = canonical_path(location, platform_)

rez/src/rezplugins/package_repository/filesystem.py

Lines 818 to 825 in 8968662

    
           # It appears that sometimes, the handle location can differ to the 
        
           # repo location even though they are the same path (different 
        
           # mounts). We account for that here. 
        
           # 
        
           # https://github.com/AcademySoftwareFoundation/rez/pull/957 
        
           # 
        
           if location != self.location: 
        
               location = canonical_path(location, platform_)

KelSolaar · 2023-02-26T22:09:20Z

It seems like the intent was to resolve symlinks for Linux. The behavioural change of os.path.realpath affects Windows only.

I would be inclined to have a specific case for Windows that resolves the path in a backward compatible way with a Config option to be able to have the new symlink and UNC path resolution for people that wants it.

Using os.path.abspath should hopefully provides the expected behaviour:

C:\Users\tmansencal>rez-env python -- python
Python 3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.path.realpath(".")
'C:\\Users\\tmansencal'
>>> os.path.abspath(".")
'C:\\Users\\tmansencal'

KelSolaar · 2023-03-06T01:19:34Z

Spent a bit of time on this one and we are going to go with a sitecustomize.py file:

This addresses our needs and fixes the regression. I'm still keen get an upstream fix/solution for the problem that said!

JeanChristopheMorinPerso · 2023-03-16T20:47:04Z

From the TSC meeting: Let's try to replace all os.path.realpath with rez.utils.filesystem. canonical_path.

Seems introduced by #775.

Will need some thinking and testing.

yanshil · 2023-03-29T06:12:33Z

Ahhh, actually I did came across with an issue related iwth UNC path, and I started a PR at that time but I didn't keep working on it.

I would suggest testing should also include this path mismatch issue.
#1329

KelSolaar added the bug label Feb 2, 2023

JeanChristopheMorinPerso added the os:windows Windows-specific label Feb 2, 2023

KelSolaar mentioned this issue Mar 6, 2023

TSC Meeting 2023-03-16 #1456

Closed

6 tasks

JeanChristopheMorinPerso added the shell Shell related issues label Mar 11, 2023

maxnbk mentioned this issue Sep 30, 2023

Add new use_canonical_path option in rezconfig.py #1543

Open

cfxegbert linked a pull request Oct 15, 2024 that will close this issue

fix: realpath on windows changed behavior in python 3.8 to expand mapped drive letters to their UNC path names #1856

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Environment resolution uses UNC paths with Python 3.10, 3.7 was using mounted network drives. #1438

Environment resolution uses UNC paths with Python 3.10, 3.7 was using mounted network drives. #1438

KelSolaar commented Feb 2, 2023

flord commented Feb 6, 2023

KelSolaar commented Feb 6, 2023

instinct-vfx commented Feb 6, 2023

KelSolaar commented Feb 6, 2023 •

edited

Loading

instinct-vfx commented Feb 6, 2023

KelSolaar commented Feb 6, 2023

instinct-vfx commented Feb 6, 2023

yanshil commented Feb 17, 2023 •

edited

Loading

yanshil commented Feb 17, 2023

KelSolaar commented Feb 17, 2023

KelSolaar commented Feb 19, 2023

KelSolaar commented Feb 20, 2023

JeanChristopheMorinPerso commented Feb 22, 2023 •

edited

Loading

KelSolaar commented Feb 22, 2023 •

edited

Loading

JeanChristopheMorinPerso commented Feb 22, 2023

KelSolaar commented Feb 26, 2023 •

edited

Loading

KelSolaar commented Mar 6, 2023

JeanChristopheMorinPerso commented Mar 16, 2023 •

edited

Loading

yanshil commented Mar 29, 2023

Environment resolution uses UNC paths with Python 3.10, 3.7 was using mounted network drives. #1438

Environment resolution uses UNC paths with Python 3.10, 3.7 was using mounted network drives. #1438

Comments

KelSolaar commented Feb 2, 2023

Rez v2.112.0

Rez v2.2.91.0

flord commented Feb 6, 2023

KelSolaar commented Feb 6, 2023

instinct-vfx commented Feb 6, 2023

KelSolaar commented Feb 6, 2023 • edited Loading

instinct-vfx commented Feb 6, 2023

KelSolaar commented Feb 6, 2023

instinct-vfx commented Feb 6, 2023

yanshil commented Feb 17, 2023 • edited Loading

yanshil commented Feb 17, 2023

KelSolaar commented Feb 17, 2023

KelSolaar commented Feb 19, 2023

KelSolaar commented Feb 20, 2023

JeanChristopheMorinPerso commented Feb 22, 2023 • edited Loading

KelSolaar commented Feb 22, 2023 • edited Loading

JeanChristopheMorinPerso commented Feb 22, 2023

KelSolaar commented Feb 26, 2023 • edited Loading

KelSolaar commented Mar 6, 2023

JeanChristopheMorinPerso commented Mar 16, 2023 • edited Loading

yanshil commented Mar 29, 2023

KelSolaar commented Feb 6, 2023 •

edited

Loading

yanshil commented Feb 17, 2023 •

edited

Loading

JeanChristopheMorinPerso commented Feb 22, 2023 •

edited

Loading

KelSolaar commented Feb 22, 2023 •

edited

Loading

KelSolaar commented Feb 26, 2023 •

edited

Loading

JeanChristopheMorinPerso commented Mar 16, 2023 •

edited

Loading