Add recipe for pytorch (C++ interface only) #8388

iarspider · 2023-03-15T21:07:14Z

No description provided.

cmsbuild · 2023-03-15T21:07:40Z

A new Pull Request was created by @iarspider for branch IB/CMSSW_13_1_X/master.

@cmsbuild, @smuzaffar, @aandvalenzuela, @iarspider can you please review it and eventually sign? Thanks.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.
cms-bot commands are listed here

iarspider · 2023-03-15T21:10:01Z

please test

cmsbuild · 2023-03-15T21:10:10Z

Pull request #8388 was updated.

cmsbuild · 2023-03-16T07:10:34Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d4402c/31301/summary.html
COMMIT: 0181b0c
CMSSW: CMSSW_13_1_X_2023-03-15-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/8388/31301/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@cms-sw Update tag for RecoEgamma-EgammaPhotonProducers to V00-03-00 #8387

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d4402c/31301/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d4402c/31301/git-merge-result

Comparison Summary

Summary:

You potentially added 11 lines to the logs
Reco comparison results: 17 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3550756
DQMHistoTests: Total failures: 170
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3550564
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 309.217 KiB( 48 files compared)
DQMHistoSizes: changed ( 1001.0 ): 309.217 KiB AlCaReco/SiStripHitEfficiency
Checked 213 log files, 164 edm output root files, 49 DQM output files
TriggerResults: no differences found

cmsbuild · 2023-03-16T08:30:06Z

Pull request #8388 was updated.

iarspider · 2023-03-16T08:30:39Z

please test for el8_ppc64le_gcc11

cmsbuild · 2023-03-16T11:40:29Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d4402c/31307/summary.html
COMMIT: 317acbd
CMSSW: CMSSW_13_1_X_2023-03-15-2300/el8_ppc64le_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/8388/31307/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

+ tar xfz '/scratch/cmsbuild/jenkins_a/workspace/ib-run-pr-tests/testBuildDir/SOURCES/external/py3-torch/1.13.1-c52873272d6ed54dadebd1dc97e7c1eb/%{pkgsource}'
tar (child): /scratch/cmsbuild/jenkins_a/workspace/ib-run-pr-tests/testBuildDir/SOURCES/external/py3-torch/1.13.1-c52873272d6ed54dadebd1dc97e7c1eb/%{pkgsource}: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
error: Bad exit status from /scratch/cmsbuild/jenkins_a/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.9nIz8V (%build)


RPM build errors:
line 37: It's not recommended to have unversioned Obsoletes: Obsoletes: external+py3-torch+1.13.1-c52873272d6ed54dadebd1dc97e7c1eb
Bad exit status from /scratch/cmsbuild/jenkins_a/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.9nIz8V (%build)

iarspider · 2023-03-16T12:24:41Z

please test

smuzaffar · 2023-12-04T12:36:44Z

please test for el9_amd64_gcc13

cmsbuild · 2023-12-04T13:51:41Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d4402c/36292/summary.html
COMMIT: cbd56e4
CMSSW: CMSSW_14_0_X_2023-12-03-2300/el8_aarch64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/8388/36292/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@mkirsano upgrade lhapdf to 6.5.4, sets to 6.5.1c #8852

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d4402c/36292/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d4402c/36292/git-merge-result

cmsbuild · 2023-12-04T14:45:49Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d4402c/36290/summary.html
COMMIT: cbd56e4
CMSSW: CMSSW_14_0_X_2023-12-03-2300/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/8388/36290/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@mkirsano upgrade lhapdf to 6.5.4, sets to 6.5.1c #8852

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d4402c/36290/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d4402c/36290/git-merge-result

Comparison Summary

Summary:

You potentially added 75 lines to the logs
Reco comparison results: 16 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3370032
DQMHistoTests: Total failures: 81
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3369929
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 214 log files, 167 edm output root files, 50 DQM output files
TriggerResults: no differences found

GPU Comparison Summary

Summary:

You potentially added 12 lines to the logs
Reco comparison results: 24 differences found in the comparisons
DQMHistoTests: Total files compared: 3
DQMHistoTests: Total histograms compared: 39740
DQMHistoTests: Total failures: 932
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 38808
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 2 files compared)
Checked 8 log files, 10 edm output root files, 3 DQM output files
TriggerResults: no differences found

cmsbuild · 2023-12-04T18:12:06Z

-1

Failed Tests: Build HeaderConsistency
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d4402c/36306/summary.html
COMMIT: cbd56e4
CMSSW: CMSSW_14_0_X_2023-12-01-2300/el9_amd64_gcc13
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/8388/36306/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d4402c/36306/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d4402c/36306/git-merge-result

Build

I found compilation error when building:

      |                                           ~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/pytorch/2.1.1-815fe07924977ebb6c7bffd29d80d4f3/include/torch/csrc/api/include/torch/nn/modules/container/sequential.h:122:52: note: remove 'std::move' call
>> Building binary testTorchTimeSeries
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/gcc/13.2.0-1b0a3367d04f48f01ad3ccf40e55475c/bin/../lib/gcc/x86_64-redhat-linux-gnu/13.2.0/../../../../x86_64-redhat-linux-gnu/bin/ld.bfd: cannot find -ltorch_cuda: No such file or directory
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/gcc/13.2.0-1b0a3367d04f48f01ad3ccf40e55475c/bin/../lib/gcc/x86_64-redhat-linux-gnu/13.2.0/../../../../x86_64-redhat-linux-gnu/bin/ld.bfd: cannot find -lc10_cuda: No such file or directory
collect2: error: ld returned 1 exit status
>> Deleted: tmp/el9_amd64_gcc13/src/PhysicsTools/PythonAnalysis/test/testTorchTimeSeries/testTorchTimeSeries
gmake: *** [tmp/el9_amd64_gcc13/src/PhysicsTools/PythonAnalysis/test/testTorchTimeSeries/testTorchTimeSeries] Error 1
>> Compiling  /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_0_X_2023-12-01-2300/src/PhysicsTools/PythonAnalysis/test/test_PyMVA.cpp
>> Building binary test_PyMVA
Copying tmp/el9_amd64_gcc13/src/PhysicsTools/PythonAnalysis/test/test_PyMVA/test_PyMVA to productstore area:

iarspider · 2023-12-07T13:10:26Z

please test

cmsbuild · 2023-12-07T16:18:17Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d4402c/36368/summary.html
COMMIT: cbd56e4
CMSSW: CMSSW_14_0_X_2023-12-07-1100/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/8388/36368/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 431 lines to the logs
Reco comparison results: 1 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3430794
DQMHistoTests: Total failures: 73
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3430699
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 214 log files, 167 edm output root files, 50 DQM output files
TriggerResults: no differences found

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 32 differences found in the comparisons
DQMHistoTests: Total files compared: 3
DQMHistoTests: Total histograms compared: 39740
DQMHistoTests: Total failures: 746
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 38994
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 2 files compared)
Checked 8 log files, 10 edm output root files, 3 DQM output files
TriggerResults: no differences found

tvami · 2023-12-13T19:44:49Z

hi @iarspider do you think it would be helpful to squash to commits? Is there anything more missing or should we merge after the squashing is done?

smuzaffar · 2023-12-14T10:05:15Z

please test

lets refresh the tests
@tvami , sure I will squash merge this PR

cmsbuild · 2023-12-14T14:34:36Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d4402c/36490/summary.html
COMMIT: cbd56e4
CMSSW: CMSSW_14_0_X_2023-12-13-2300/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/8388/36490/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@cms-sw Update tag for Configuration-Generator to V01-08-00 #8882

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d4402c/36490/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d4402c/36490/git-merge-result

Comparison Summary

There are some workflows for which there are errors in the baseline:
10024.3 step 3
10024.4 step 3
2017.13 step 1
2018.13 step 1
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

You potentially added 83 lines to the logs
Reco comparison results: 7 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3429858
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3429830
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 214 log files, 167 edm output root files, 50 DQM output files
TriggerResults: no differences found

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 35 differences found in the comparisons
DQMHistoTests: Total files compared: 3
DQMHistoTests: Total histograms compared: 39740
DQMHistoTests: Total failures: 1156
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 38584
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 2 files compared)
Checked 8 log files, 10 edm output root files, 3 DQM output files
TriggerResults: no differences found

tvami · 2023-12-14T19:07:32Z

+analysis

smuzaffar · 2023-12-14T21:35:44Z

+externals

cmsbuild · 2023-12-14T21:36:07Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_14_0_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @antoniovilela, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

iarspider added 3 commits March 14, 2023 14:44

Initial commit of py3-torch recipe

dd2f95c

Merge branch 'IB/CMSSW_13_1_X/master' into add-py3-torch

15e8919

Fix the recipe

b1f0dca

cmsbuild added tests-pending externals-pending pending-signatures orp-pending labels Mar 15, 2023

Add py3-torch to python_tools

0181b0c

cmsbuild added tests-started and removed tests-pending labels Mar 15, 2023

cmsbuild added tests-approved and removed tests-started labels Mar 16, 2023

Add relocation

317acbd

cmsbuild added tests-pending and removed tests-approved labels Mar 16, 2023

cmsbuild added tests-started and removed tests-pending labels Mar 16, 2023

Remove pthreadpoool - only needed for nnpack or mobile

b05c5d3

cmsbuild added tests-rejected and removed tests-started labels Mar 16, 2023

cmsbuild added tests-started and removed tests-rejected labels Mar 16, 2023

cmsbuild added tests-approved and removed tests-started labels Dec 4, 2023

cmsbuild added tests-started and removed tests-approved labels Dec 7, 2023

cmsbuild added tests-approved and removed tests-started labels Dec 7, 2023

cmsbuild added tests-started and removed tests-approved labels Dec 14, 2023

cmsbuild added tests-approved and removed tests-started labels Dec 14, 2023

cmsbuild added analysis-approved and removed analysis-pending labels Dec 14, 2023

cmsbuild added externals-approved fully-signed and removed externals-pending pending-signatures labels Dec 14, 2023

smuzaffar merged commit 5354a4c into IB/CMSSW_14_0_X/master Dec 14, 2023
27 of 29 checks passed

smuzaffar deleted the add-py3-torch branch December 18, 2023 18:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add recipe for pytorch (C++ interface only) #8388

Add recipe for pytorch (C++ interface only) #8388

iarspider commented Mar 15, 2023 •

edited

Loading

cmsbuild commented Mar 15, 2023 •

edited

Loading

iarspider commented Mar 15, 2023

cmsbuild commented Mar 15, 2023

cmsbuild commented Mar 16, 2023

cmsbuild commented Mar 16, 2023

iarspider commented Mar 16, 2023

cmsbuild commented Mar 16, 2023

iarspider commented Mar 16, 2023

smuzaffar commented Dec 4, 2023

cmsbuild commented Dec 4, 2023

cmsbuild commented Dec 4, 2023

cmsbuild commented Dec 4, 2023

iarspider commented Dec 7, 2023

cmsbuild commented Dec 7, 2023

tvami commented Dec 13, 2023

smuzaffar commented Dec 14, 2023

cmsbuild commented Dec 14, 2023

tvami commented Dec 14, 2023

smuzaffar commented Dec 14, 2023

cmsbuild commented Dec 14, 2023

Add recipe for pytorch (C++ interface only) #8388

Add recipe for pytorch (C++ interface only) #8388

Conversation

iarspider commented Mar 15, 2023 • edited Loading

cmsbuild commented Mar 15, 2023 • edited Loading

iarspider commented Mar 15, 2023

cmsbuild commented Mar 15, 2023

cmsbuild commented Mar 16, 2023

Comparison Summary

cmsbuild commented Mar 16, 2023

iarspider commented Mar 16, 2023

cmsbuild commented Mar 16, 2023

External Build

iarspider commented Mar 16, 2023

smuzaffar commented Dec 4, 2023

cmsbuild commented Dec 4, 2023

cmsbuild commented Dec 4, 2023

Comparison Summary

GPU Comparison Summary

cmsbuild commented Dec 4, 2023

Build

iarspider commented Dec 7, 2023

cmsbuild commented Dec 7, 2023

Comparison Summary

GPU Comparison Summary

tvami commented Dec 13, 2023

smuzaffar commented Dec 14, 2023

cmsbuild commented Dec 14, 2023

Comparison Summary

GPU Comparison Summary

tvami commented Dec 14, 2023

smuzaffar commented Dec 14, 2023

cmsbuild commented Dec 14, 2023

iarspider commented Mar 15, 2023 •

edited

Loading

cmsbuild commented Mar 15, 2023 •

edited

Loading