Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] use sklearnex azure pipelines template for sklearnex CI step #2950

Draft
wants to merge 40 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
f090e79
Update ci.yml
icfaust Oct 22, 2024
3accb1b
Update ci.yml
icfaust Oct 23, 2024
35f06d0
Update ci.yml
icfaust Oct 24, 2024
c55ed6d
Update ci.yml
icfaust Oct 24, 2024
3712fd8
Update ci.yml
icfaust Oct 24, 2024
a3ec692
Update ci.yml
icfaust Oct 28, 2024
b80336f
Update ci.yml
icfaust Oct 28, 2024
ffd1477
Update ci.yml
icfaust Oct 28, 2024
7882048
Update ci.yml
icfaust Oct 28, 2024
997277d
Update ci.yml
icfaust Oct 28, 2024
bc679e3
Update ci.yml
icfaust Oct 28, 2024
a9f84ed
Update ci.yml
icfaust Oct 28, 2024
a0ec883
Update ci.yml
icfaust Oct 28, 2024
8c11015
Update ci.yml
icfaust Oct 28, 2024
ea94d36
Update ci.yml
icfaust Oct 28, 2024
5db2a48
Merge branch 'main' into dev/use_sklearnex_testing
icfaust Oct 28, 2024
6d3153d
Update ci.yml
icfaust Oct 28, 2024
7a70904
Update ci.yml
icfaust Oct 28, 2024
4357439
Update ci.yml
icfaust Oct 29, 2024
7946276
Update ci.yml
icfaust Oct 29, 2024
aaad2d3
Update ci.yml
icfaust Oct 29, 2024
4fb04bb
Update ci.yml
icfaust Oct 30, 2024
b07c807
Update ci.yml
icfaust Oct 30, 2024
3bab93b
Update ci.yml
icfaust Oct 30, 2024
540e9dd
Merge branch 'oneapi-src:main' into dev/use_sklearnex_testing
icfaust Oct 30, 2024
e3d9e37
Update ci.yml
icfaust Oct 30, 2024
955a7bc
Update ci.yml
icfaust Oct 31, 2024
2f5f327
Update ci.yml
icfaust Oct 31, 2024
99fbf92
Update ci.yml
icfaust Nov 1, 2024
dc8be7b
Update apt.sh
icfaust Nov 1, 2024
9988328
Update ci.yml
icfaust Nov 1, 2024
d0551ea
Merge branch 'oneapi-src:main' into dev/use_sklearnex_testing
icfaust Nov 1, 2024
a31b23d
Update ci.yml
icfaust Nov 1, 2024
325d86f
Update ci.yml
icfaust Nov 1, 2024
186aa77
Update ci.yml
icfaust Nov 1, 2024
de29e51
Update ci.yml
icfaust Nov 1, 2024
6f1522d
Update ci.yml
icfaust Nov 1, 2024
fe1877f
Update ci.yml
icfaust Nov 2, 2024
54c4214
remove sklearn conformance scripts
icfaust Nov 2, 2024
d3f8fd4
Merge branch 'oneapi-src:main' into dev/use_sklearnex_testing
icfaust Nov 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion .ci/env/apt.sh
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,10 @@ function install_mkl {
sudo apt-get install -y intel-oneapi-mkl-devel-2025.0 intel-oneapi-tbb-devel-2022.0
}

function install_tbb {
sudo apt-get install -y intel-oneapi-tbb-devel-2022.0
}

function install_clang-format {
sudo apt-get install -y clang-format-14
}
Expand Down Expand Up @@ -100,6 +104,9 @@ if [ "${component}" == "dpcpp" ]; then
elif [ "${component}" == "mkl" ]; then
add_repo
install_mkl
elif [ "${component}" == "tbb" ]; then
add_repo
install_tbb
elif [ "${component}" == "gnu-cross-compilers" ]; then
update
install_gnu-cross-compilers "$2"
Expand All @@ -126,6 +133,6 @@ elif [ "${component}" == "build-sysroot" ] ; then
build_sysroot "$2" "$3" "$4" "$5"
else
echo "Usage:"
echo " $0 [dpcpp|mkl|gnu-cross-compilers|clang-format|dev-base|qemu-apt|qemu-deb|llvm-version|build-sysroot]"
echo " $0 [dpcpp|mkl|tbb|gnu-cross-compilers|clang-format|dev-base|qemu-apt|qemu-deb|llvm-version|build-sysroot]"
exit 1
fi
97 changes: 35 additions & 62 deletions .ci/pipeline/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,14 @@ variables:
WINDOWS_BASEKIT_URL: 'https://registrationcenter-download.intel.com/akdlm/IRC_NAS/b380d914-366b-4b77-a74a-05e3c38b3514/intel-oneapi-base-toolkit-2025.0.0.882_offline.exe'
WINDOWS_DPCPP_COMPONENTS: 'intel.oneapi.win.mkl.devel:intel.oneapi.win.tbb.devel'

resources:
repositories:
- repository: sklearnex
type: github
name: icfaust/scikit-learn-intelex
ref: dev/azure_pipelines_without_dal-devel
endpoint: intel-daal-connection

jobs:
- job: 'FormatterChecks'
pool:
Expand Down Expand Up @@ -81,6 +89,12 @@ jobs:
targetPath: '$(Build.Repository.LocalPath)/$(release.dir)'
displayName: 'Upload build artifacts'
continueOnError: true
- task: PublishPipelineArtifact@1
inputs:
artifactName: 'oneDAL environment'
targetPath: '$(Build.Repository.LocalPath)/.ci/env'
displayName: 'Upload environment artifacts'
continueOnError: true
- script: |
source /opt/intel/oneapi/setvars.sh
.ci/scripts/test.sh --test-kind examples --build-dir $(release.dir) --compiler gnu --interface daal/cpp --build-system cmake
Expand Down Expand Up @@ -582,83 +596,42 @@ jobs:
timeoutInMinutes: 0
variables:
release.dir: '__release_lnx_gnu'
platform.type : 'lnx32e'
platform.type: 'lnx32e'
DALROOT: '$(Pipeline.Workspace)/daal/latest'
TBBROOT: '/opt/intel/oneapi/tbb/latest'
NO_DPC: 1
NO_DIST: 1
pool:
vmImage: '$(VM_IMAGE)'
maxParallel: 2
strategy:
matrix:
Python311:
python.version: '3.11'
PYTHON_VERSION: '3.11'
SKLEARN_VERSION: '1.5'
steps:
- checkout: none
- script: |
git clone https://github.com/icfaust/scikit-learn-intelex.git .
git switch dev/azure_pipelines_without_dal-devel
displayName: 'Clone sklearnex'
- task: DownloadPipelineArtifact@2
inputs:
artifact: 'oneDAL environment'
path: '$(Build.Repository.LocalPath)/.ci/env'
- task: DownloadPipelineArtifact@2
inputs:
artifact: '$(platform.type) build'
path: $(Pipeline.Workspace)
- script: .ci/env/apt.sh dev-base
displayName: 'apt-get'
- script: |
.ci/scripts/describe_system.sh
displayName: 'System info'
- script: |
conda update -y -q conda
conda create -q -y -n CB -c conda-forge python=$(python.version) mpich
displayName: 'Conda create'
- script: |
git clone https://github.com/intel/scikit-learn-intelex.git sklearnex
displayName: Clone sklearnex
- script: |
source /usr/share/miniconda/etc/profile.d/conda.sh
conda activate CB
pip install -r sklearnex/dependencies-dev
pip install -r sklearnex/requirements-test.txt
pip install tbb==2022.*
displayName: Create python environment
- script: |
source /usr/share/miniconda/etc/profile.d/conda.sh
conda activate CB
export DALROOT=$(Pipeline.Workspace)/daal/latest
source ${DALROOT}/env/vars.sh
export PREFIX=${CONDA_PREFIX}
cd sklearnex
export PYTHON=python
./conda-recipe/build.sh
displayName: sklearnex build
chmod -R 755 .ci/env
.ci/env/apt.sh tbb
displayName: 'tbb installation'
- template: .ci/pipeline/build-and-test-lnx.yml@sklearnex
- task: PublishPipelineArtifact@1
inputs:
artifactName: '$(platform.type) sklearnex build'
targetPath: '$(Build.Repository.LocalPath)/sklearnex'
displayName: 'Upload sklearnex build artifacts'
continueOnError: true
- script: |
source /usr/share/miniconda/etc/profile.d/conda.sh
conda activate CB
source $(Pipeline.Workspace)/daal/latest/env/vars.sh
./sklearnex/conda-recipe/run_test.sh
timeoutInMinutes: 20
displayName: sklearnex test
- script: |
source /usr/share/miniconda/etc/profile.d/conda.sh
conda activate CB
source $(Pipeline.Workspace)/daal/latest/env/vars.sh
ret_code=0
python -m sklearnex sklearnex/tests/run_examples.py
ret_code=$(($ret_code + $?))
exit $ret_code
displayName: sklearnex examples
- script: |
source /usr/share/miniconda/etc/profile.d/conda.sh
conda activate CB
source $(Pipeline.Workspace)/daal/latest/env/vars.sh
$(Pipeline.Workspace)/s/.ci/scripts/conformance_test.sh --build-dir $(Pipeline.Workspace) --conda-dir $HOME/miniconda --oneapi-dir /opt/intel/oneapi
timeoutInMinutes: 15
displayName: Conformance tests
- task: PublishPipelineArtifact@1
inputs:
artifactName: '$(platform.type) conformance tests report'
targetPath: '$(Pipeline.Workspace)/s/.ci/scripts/conformance-scripts/'
displayName: 'Upload conformance tests artifacts'
continueOnError: true
targetPath: '$(Build.Repository.LocalPath)/build'

- job: 'WindowsMakeVC'
timeoutInMinutes: 0
Expand Down
11 changes: 0 additions & 11 deletions .ci/scripts/conformance-scripts/algorithms.txt

This file was deleted.

73 changes: 0 additions & 73 deletions .ci/scripts/conformance-scripts/download_tests.sh

This file was deleted.

49 changes: 0 additions & 49 deletions .ci/scripts/conformance-scripts/run_tests.py

This file was deleted.

15 changes: 0 additions & 15 deletions .ci/scripts/conformance-scripts/run_tests_with_context.py

This file was deleted.

65 changes: 0 additions & 65 deletions .ci/scripts/conformance-scripts/run_xpu_conformance.py

This file was deleted.

Loading