llvm@11, chapel: migrate to Python 3.10 #90708
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BrewTestBot
added
the
python
6 tasks
|
Needs |
carlocab
added
CI-long-timeout
XuehaiPan
force-pushed
the
[email protected]
branch
from
ed432f1
to
2e4fb4e
Compare
iMichka
added
CI-long-timeout
branchvincent
force-pushed
the
[email protected]
branch
from
2e4fb4e
to
03043fa
Compare
|
iMichka
removed
the
CI-long-timeout
mppf
added a commit
to chapel-lang/chapel
that referenced
this pull request
Dec 15, 2021
fix(makefile): support python 3.10 [PR by @branchvincent - thanks! Reviewed, tested, and merged by @mppf] This fixes a bug detecting the current Python version, noticed when building with Python 3.10 in Homebrew/homebrew-core#90708 Before: ```console $ make -C third-party/chpl-venv test-venv PYTHON=python3.10 python3 version 3.5 or later is required to install chpldoc and start_test dependencies. See https://www.python.org/ ``` After: ```console $ make -C third-party/chpl-venv test-venv PYTHON=python3.10 ... Successfully installed ... ```
branchvincent
force-pushed
the
[email protected]
branch
from
03043fa
to
de9d40b
Compare
iMichka
added
the
CI-long-timeout
iMichka
force-pushed
the
[email protected]
branch
from
de9d40b
to
78b817d
Compare
|
|
|
|
There was a commit to sphinx to fix this: sphinx-doc/sphinx@36e06d6 Need to check how this is can be fixed in chapel first. @Maxrimus, @bradcray, @mppf or @ronawho might know how to help. |
|
Hi - sorry, most of the folks you've pinged are on holiday break right now. I don't know much about homebrew formulas and such, but it does look like updating the Sphinx version we rely on to 4.3.2 doesn't cause problems for us and should include the commit you mentioned. So I think modifying line 195 and 217 to use 4.3.2 would be okay? |
|
I've opened chapel-lang/chapel#18894 on our repo, if that helps with adjusting the patches at the end of the formula |
SMillerDev
removed
the
CI-long-timeout
|
In looking at the failure on that PR, this is not just a simple change, I believe we need to update a separate repository we distribute so that it's compatible with the new version of Sphinx. I'll be taking a deeper look, but since the majority of our team are on holiday break, it might not be able to be reviewed until the new year. What's your timeframe on making this change? |
|
I don't think this is urgent. It can wait till people have come back from holidays. |
|
Hello, just curious to know if this is on the plate to be resolved soon? I'd very much appreciate your input. Thank you all for all you do. |
|
Yup, I've got the update to our alternate repo made and have a PR incorporating it into our main repo. I expect it to go in soon, thank you for your patience :) |
|
chapel-lang/chapel#18894 has been merged and appears to be passing our nightly testing so far. Only one of the file changes is relevant to homebrew, the adjustments to @mppf also merged a PR this morning to integrate the patches we'd included as part of the Chapel formula into the main repo itself, which performs the split of that file into three files and includes the changes from my PR: chapel-lang/chapel#18926 I'd advise waiting until that second PR has had a night of testing on it but I think we're probably in good shape. Is there anything more we can do to help? |
|
Second PR was clean! |
|
I tried to apply the patches to But we might wait for March/April. I would like to move the Python 3.10 migration forward, but I also know this takes time so we have plenty of other formulae to fix until this one is ready. |
|
Can you go more into what you tried and how it didn't work? |
These patches are already hosted on GitHub. They can download by |
Install diff --git a/Formula/chapel.rb b/Formula/chapel.rb
index 0eebb5cba99..fddf02c3b27 100644
--- a/Formula/chapel.rb
+++ b/Formula/chapel.rb
@@ -4,6 +4,7 @@ class Chapel < Formula
url "https://github.com/chapel-lang/chapel/releases/download/1.25.1/chapel-1.25.1.tar.gz"
sha256 "0c13d7da5892d0b6642267af605d808eb7dd5d4970766f262f38b94fa2405113"
license "Apache-2.0"
+ revision 1
bottle do
sha256 arm64_monterey: "e11d484b8dbeb19649b46ebadb1ff063e79a3fa7c3e1befc258c6fd28a4712a2"
@@ -16,11 +17,21 @@ class Chapel < Formula
depends_on "gmp"
depends_on "llvm@11"
- depends_on "[email protected]"
+ depends_on "[email protected]"
# can be removed after https://github.com/chapel-lang/chapel/pull/18880 merged
patch :DATA
+ patch do
+ url "https://github.com/chapel-lang/chapel/pull/18894.patch"
+ sha256 "ff27ffcd76aa89b3b7722b02918bfe3d315469b398bdc9b47e7ee2285246ebe6"
+ end
+
+ patch do
+ url "https://github.com/chapel-lang/chapel/pull/18926.patch"
+ sha256 "c888619ca8e7dc29cb054b137af82b0d9ace42dc3e040182fdc98eaede39f48a"
+ end
+
def install
libexec.install Dir["*"]
# Chapel uses this ENV to work out where to install.
diff --git a/Formula/[email protected] b/Formula/[email protected]
index 1e76fbe2e94..b7ce2827671 100644
--- a/Formula/[email protected]
+++ b/Formula/[email protected]
@@ -5,7 +5,7 @@ class LlvmAT11 < Formula
sha256 "74d2529159fd118c3eac6f90107b5611bccc6f647fdea104024183e8d5e25831"
# The LLVM Project is under the Apache License v2.0 with LLVM Exceptions
license "Apache-2.0" => { with: "LLVM-exception" }
- revision 3
+ revision 4
# This should be removed when LLVM 13 is released, so we only check the
# current version (the `llvm` formula) and one major version before it
@@ -35,7 +35,7 @@ class LlvmAT11 < Formula
# See: Homebrew/homebrew-core/issues/35513
depends_on "cmake" => :build
depends_on "swig" => :build
- depends_on "[email protected]"
+ depends_on "[email protected]"
uses_from_macos "libedit"
uses_from_macos "libffi", since: :catalinaGot: |
|
Right I would expect that But also note that the 1.25.1 chapel.rb has patches at the bottom. To get everything to apply cleanly, we need to regenerate the patches for chapel.rb to include the changes needed in the formula from chapel-lang/chapel#18894 and chapel-lang/chapel#18926 . The PR chapel-lang/chapel#18880 has a summary of patches that were included in the formula. Going to the individual PRs and grabbing commit shas led me to these commands: From there I grabbed the chapel.rb from this PR and made these changes:
I have uploaded the resulting chapel.rb to https://gist.github.com/mppf/32869664c6bb251dcd1101d783763780 but I have not tested it at all. Hope that is helpful to you & you can either re-run the commands above or use that file. |
|
Oh, I think I forgot to include one commit from PR chapel-lang/chapel#18880 in the above -- chapel-lang/chapel@60df192 . I will update it. |
|
OK I've edited my previous comment to fix it to include chapel-lang/chapel@60df192 . Next step is to try some testing. (I am not planning to do that, myself, right now). |
XuehaiPan
force-pushed
the
[email protected]
branch
from
78b817d
to
f464f5d
Compare
|
I removed some unrelated diffs in patch https://gist.github.com/mppf/32869664c6bb251dcd1101d783763780. Then, I can build For testing, I got: $ brew install -s llvm@11
$ brew install -s chapel
$ brew test chapel
==> Testing chapel
==> util/test/checkChplInstall
Last 15 lines from /home/linuxbrew/.cache/Homebrew/Logs/chapel/test.01.checkChplInstall:
[Info] Compiling $CHPL_HOME/examples/hello6-taskpar-dist.chpl
[Info] Compiling with CHPL_TARGET_COMPILER=
[Fail] Test job failed to compile - Chapel is not installed correctly
[Fail] Compilation output:
Traceback (most recent call last):
File "/home/linuxbrew/.linuxbrew/Cellar/chapel/1.25.1_1/libexec/util/chplenv/printchplenv.py", line 48, in <module>
from chplenv import *
File "/home/linuxbrew/.linuxbrew/Cellar/chapel/1.25.1_1/libexec/util/chplenv/chplenv.py", line 1, in <module>
import chpl_cpu
File "/home/linuxbrew/.linuxbrew/Cellar/chapel/1.25.1_1/libexec/util/chplenv/chpl_cpu.py", line 9, in <module>
import chpl_comm, chpl_compiler, chpl_platform, overrides
File "/home/linuxbrew/.linuxbrew/Cellar/chapel/1.25.1_1/libexec/util/chplenv/chpl_compiler.py", line 6, in <module>
from distutils.spawn import find_executable
ModuleNotFoundError: No module named 'distutils.spawn'
error: 'CHPL_HOME=/home/linuxbrew/.linuxbrew/Cellar/chapel/1.25.1_1/libexec CHPL_RUNTIME_INCL=/home/linuxbrew/.linuxbrew/Cellar/chapel/1.25.1_1/libexec/runtime/include CHPL_RUNTIME_LIB=/home/linuxbrew/.linuxbrew/Cellar/chapel/1.25.1_1/libexec/lib CHPL_THIRD_PARTY=/home/linuxbrew/.linuxbrew/Cellar/chapel/1.25.1_1/libexec/third-party CHPLENV_SUPPRESS_WARNINGS=true /home/linuxbrew/.linuxbrew/Cellar/chapel/1.25.1_1/libexec/util/printchplenv --all --internal --no-tidy --simple' did not run successfully
Error: chapel: failed
An exception occurred within a child process:
BuildError: Failed executing: util/test/checkChplInstall
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/formula.rb:2309:in `block in system'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/formula.rb:2245:in `open'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/formula.rb:2245:in `system'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/chapel.rb:81:in `block (3 levels) in <class:Chapel>'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/utils.rb:588:in `with_env'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/chapel.rb:80:in `block (2 levels) in <class:Chapel>'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.8/lib/ruby/2.6.0/fileutils.rb:128:in `chdir'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.8/lib/ruby/2.6.0/fileutils.rb:128:in `cd'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/chapel.rb:79:in `block in <class:Chapel>'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/formula.rb:2109:in `block (3 levels) in run_test'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/utils.rb:588:in `with_env'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/formula.rb:2108:in `block (2 levels) in run_test'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/formula.rb:948:in `with_logging'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/formula.rb:2107:in `block in run_test'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/mktemp.rb:63:in `block in run'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/mktemp.rb:63:in `chdir'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/mktemp.rb:63:in `run'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/formula.rb:2360:in `mktemp'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/formula.rb:2101:in `run_test'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/test.rb:43:in `block in <main>'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.8/lib/ruby/2.6.0/timeout.rb:93:in `block in timeout'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.8/lib/ruby/2.6.0/timeout.rb:33:in `block in catch'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.8/lib/ruby/2.6.0/timeout.rb:33:in `catch'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.8/lib/ruby/2.6.0/timeout.rb:33:in `catch'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.8/lib/ruby/2.6.0/timeout.rb:108:in `timeout'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/test.rb:48:in `<main>'
from distutils.spawn import find_executableI make a new patch for this: --- a/third-party/llvm/llvm-src/tools/clang/utils/check_cfc/setup.py
+++ b/third-party/llvm/llvm-src/tools/clang/utils/check_cfc/setup.py
@@ -2,7 +2,12 @@
python.exe setup.py py2exe
"""
from __future__ import absolute_import, division, print_function
-from distutils.core import setup
+
+try:
+ from distutils.core import setup
+except ImportError:
+ from setuptools import setup
+
try:
import py2exe
except ImportError:
--- a/third-party/llvm/llvm-src/tools/clang/utils/creduce-clang-crash.py
+++ b/third-party/llvm/llvm-src/tools/clang/utils/creduce-clang-crash.py
@@ -18,7 +18,6 @@ import pipes
import shlex
import tempfile
import shutil
-from distutils.spawn import find_executable
verbose = False
creduce_cmd = None
@@ -43,12 +42,12 @@ def check_cmd(cmd_name, cmd_dir, cmd_path=None):
if cmd_path:
# Make the path absolute so the creduce test can be run from any directory.
cmd_path = os.path.abspath(cmd_path)
- cmd = find_executable(cmd_path)
+ cmd = shutil.which(cmd_path)
if cmd:
return cmd
sys.exit("ERROR: executable `%s` not found" % (cmd_path))
- cmd = find_executable(cmd_name, path=cmd_dir)
+ cmd = shutil.which(cmd_name, path=cmd_dir)
if cmd:
return cmd
--- a/third-party/llvm/llvm-src/utils/update_cc_test_checks.py
+++ b/third-party/llvm/llvm-src/utils/update_cc_test_checks.py
@@ -16,11 +16,11 @@ from __future__ import print_function
import argparse
import collections
-import distutils.spawn
import json
import os
import re
import shlex
+import shutil
import subprocess
import sys
import tempfile
@@ -139,7 +139,7 @@ def config():
args.clang = 'clang'
else:
args.clang = os.path.join(args.llvm_bin, 'clang')
- if not distutils.spawn.find_executable(args.clang):
+ if not shutil.which(args.clang):
print('Please specify --llvm-bin or --clang', file=sys.stderr)
sys.exit(1)
@@ -160,7 +160,7 @@ def config():
args.opt = 'opt'
else:
args.opt = os.path.join(args.llvm_bin, 'opt')
- if not distutils.spawn.find_executable(args.opt):
+ if not shutil.which(args.opt):
# Many uses of this tool will not need an opt binary, because it's only
# needed for updating a test that runs clang | opt | FileCheck. So we
# defer this error message until we find that opt is actually needed.
--- a/util/chplenv/chpl_compiler.py
+++ b/util/chplenv/chpl_compiler.py
@@ -1,10 +1,9 @@
#!/usr/bin/env python3
import optparse
import os
+import shutil
import sys
-from distutils.spawn import find_executable
-
import chpl_platform, overrides
from utils import error, memoize, warning
@@ -193,7 +192,7 @@ def get(flag='host'):
elif platform_val.startswith('pwr'):
compiler_val = 'ibm'
elif platform_val == 'darwin' or platform_val == 'freebsd':
- if find_executable('clang'):
+ if shutil.which('clang'):
compiler_val = 'clang'
else:
compiler_val = 'gnu'
--- a/util/chplenv/chpl_launcher.py
+++ b/util/chplenv/chpl_launcher.py
@@ -1,5 +1,5 @@
#!/usr/bin/env python3
-from distutils.spawn import find_executable
+import shutil
import sys
import chpl_comm, chpl_comm_substrate, chpl_platform, overrides
@@ -7,7 +7,7 @@ from utils import error, memoize, warning
def slurm_prefix(base_launcher, platform_val):
""" If salloc is available and we're on a cray-cs/hpe-apollo, prefix with slurm-"""
- if platform_val in ('cray-cs', 'hpe-apollo') and find_executable('salloc'):
+ if platform_val in ('cray-cs', 'hpe-apollo') and shutil.which('salloc'):
return 'slurm-{}'.format(base_launcher)
return base_launcher
@@ -29,8 +29,8 @@ def get():
platform_val = chpl_platform.get('target')
if platform_val.startswith('cray-x') or platform_val.startswith('hpe-cray-'):
- has_aprun = find_executable('aprun')
- has_slurm = find_executable('srun')
+ has_aprun = shutil.which('aprun')
+ has_slurm = shutil.which('srun')
if has_aprun and has_slurm:
launcher_val = 'none'
elif has_aprun:
@@ -60,7 +60,7 @@ def get():
elif substrate_val == 'ofi':
launcher_val = slurm_prefix('gasnetrun_ofi', platform_val)
else:
- if platform_val in ('cray-cs', 'hpe-apollo') and find_executable('srun'):
+ if platform_val in ('cray-cs', 'hpe-apollo') and shutil.which('srun'):
launcher_val = 'slurm-srun'
else:
launcher_val = 'none'
--- a/util/chplenv/chpl_llvm.py
+++ b/util/chplenv/chpl_llvm.py
@@ -1,8 +1,8 @@
#!/usr/bin/env python3
import optparse
import os
+import shutil
import sys
-from distutils.spawn import find_executable
import re
import chpl_bin_subdir, chpl_arch, chpl_compiler, chpl_platform, overrides
@@ -302,7 +302,7 @@ def get_gcc_prefix():
if not gcc_prefix:
# When 'gcc' is a command other than '/usr/bin/gcc',
# compute the 'gcc' prefix that LLVM should use.
- gcc_path = find_executable('gcc')
+ gcc_path = shutil.which('gcc')
if gcc_path == '/usr/bin/gcc' :
# In this common case, nothing else needs to be done,
# because we can assume that clang can find this gcc.
--- a/util/chplenv/chpl_make.py
+++ b/util/chplenv/chpl_make.py
@@ -1,5 +1,5 @@
#!/usr/bin/env python3
-from distutils.spawn import find_executable
+import shutil
import sys
import chpl_platform, overrides
@@ -14,7 +14,7 @@ def get():
if platform_val.startswith('cygwin') or platform_val == 'darwin':
make_val = 'make'
elif platform_val.startswith('linux'):
- if find_executable('gmake'):
+ if shutil.which('gmake'):
make_val = 'gmake'
else:
make_val = 'make' |
XuehaiPan
force-pushed
the
[email protected]
branch
from
f464f5d
to
68a1a9e
Compare
XuehaiPan
force-pushed
the
[email protected]
branch
from
68a1a9e
to
e4f2585
Compare
|
May need label |
|
Regarding
I wouldn't expect that you need to patch anything in third-party/llvm/llvm-src since this directory will not be used since Chapel is using the Homebrew LLVM 11. Also we would appreciate it upstream if you can make a PR with the changes to util/chplenv. Regarding this
Can you say more about which diffs you removed from there? Thanks |
Get it. I will remove them then.
Files where their paths start with |
|
Sounds great, thanks for clarifying |
XuehaiPan
force-pushed
the
[email protected]
branch
from
e4f2585
to
7c49b21
Compare
carlocab
added
CI-long-timeout
|
All checks have passed. |
iMichka
approved these changes
Feb 5, 2022
|
Nice. Thanks all. I naively tried to apply the patches above, but I see there was more work to get things done correctly. Thanks all! |
|
Can we move the embedded Chapel patch to formula-patches? It can wait until this is merged. |
|
🤖 A scheduled task has triggered a merge. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Migrate
llvm@11family to Python 3.10. See #87075 (comment)