Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added FIL model REST protocol test on triton for Kserve (#2018) Backport #2123

Open
wants to merge 3 commits into
base: releases/2.16.0
Choose a base branch
from
Open

Added FIL model REST protocol test on triton for Kserve (#2018) Backport #2123

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
762e06a
Added FIL model REST protocol test on triton for Kserve (#2018)
rpancham Dec 4, 2024
5d02a8a
Merge branch 'releases/2.16.0' into fil_rest_backport
rpancham Dec 9, 2024
74c9b33
Merge branch 'releases/2.16.0' into fil_rest_backport
rpancham Dec 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ods_ci/tests/Resources/Files/triton/kserve-triton-fil-rest-input.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"inputs":[{"name":"input__0","shape":[1,393],"datatype":"FP32","data":[0.3284832835197449,0.8268185257911682,0.78487628698349,0.9519302845001221,0.12701496481895447,0.5136651992797852,0.6298874020576477,0.45734643936157227,0.9838705658912659,0.20956359803676605,0.8379985690116882,0.5718668103218079,0.019412444904446602,0.38271811604499817,0.6325977444648743,0.03927919268608093,0.48838457465171814,0.4702889323234558,0.4560658633708954,0.6781136989593506,0.5536667704582214,0.4988061487674713,0.12478730827569962,0.1059209406375885,0.20966315269470215,0.4273585081100464,0.653721034526825,0.45579633116722107,0.14691787958145142,0.33020544052124023,0.05173210799694061,0.11969181895256042,0.16853050887584686,0.4049186110496521,0.755652666091919,0.7312496900558472,0.10803086310625076,0.457566499710083,0.3736112415790558,0.1936161369085312,0.8652274012565613,0.4038611948490143,0.5566637516021729,0.06435327231884003,0.566157877445221,0.6998145580291748,0.15337392687797546,0.6230511665344238,0.7362670302391052,0.5723153352737427,0.6682037115097046,0.3011724650859833,0.042016711086034775,0.6602086424827576,0.5835669040679932,0.0284038707613945,0.1020161584019661,0.1951567679643631,0.4169561266899109,0.6155021786689758,0.2703559100627899,0.07744283229112625,0.051553454250097275,0.46595120429992676,0.38126346468925476,0.8140078186988831,0.6293876767158508,0.9310184717178345,0.7771000266075134,0.2593490481376648,0.2669752538204193,0.5493547916412354,0.017035549506545067,0.6309475898742676,0.7316464185714722,0.6175021529197693,0.7201271057128906,0.08383309096097946,0.9262048006057739,0.43735557794570923,0.3239584267139435,0.047336988151073456,0.36462053656578064,0.7607018947601318,0.8412635326385498,0.9601210951805115,0.6327416896820068,0.8613588213920593,0.24567070603370667,0.5343116521835327,0.828964114189148,0.8069473505020142,0.40954720973968506,0.4407307207584381,0.8217458128929138,0.8649827241897583,0.555292010307312,0.8169907927513123,0.9104942083358765,0.3510514795780182,0.313764363527298,0.706859827041626,0.2965836524963379,0.40725183486938477,0.5099840760231018,0.132075235247612,0.43063321709632874,0.3924606144428253,0.28096944093704224,0.32640188932418823,0.7225360870361328,0.5635764598846436,0.5840679407119751,0.7885806560516357,0.21381767094135284,0.8454336524009705,0.17979982495307922,0.5565366148948669,0.23402656614780426,0.7756942510604858,0.03492642194032669,0.5121867060661316,0.557293713092804,0.6212375164031982,0.6052588224411011,0.9269223809242249,0.305753618478775,0.45190030336380005,0.6436784863471985,0.7278525233268738,0.5478901863098145,0.9916322231292725,0.9576631188392639,0.07307152450084686,0.19526472687721252,0.949763834476471,0.6788548827171326,0.33683764934539795,0.6835283041000366,0.6119109988212585,0.25176212191581726,0.220492884516716,0.7287231087684631,0.9773213267326355,0.6861571669578552,0.7782228589057922,0.06823815405368805,0.9881873726844788,0.6924068331718445,0.1236552745103836,0.40915706753730774,0.6072324514389038,0.1805514097213745,0.4088282883167267,0.7606160044670105,0.27846649289131165,0.4318022131919861,0.10575707256793976,0.4982239305973053,0.14627672731876373,0.14424724876880646,0.7654518485069275,0.4624658226966858,0.027059316635131836,0.48425596952438354,0.5849552750587463,0.531979501247406,0.33964803814888,0.6425493359565735,0.10979773104190826,0.5976448655128479,0.9113814234733582,0.6277787089347839,0.43358638882637024,0.023739399388432503,0.6573179960250854,0.8587101101875305,0.2997402548789978,0.6855675578117371,0.2859164774417877,0.38119083642959595,0.8460462689399719,0.7710570096969604,0.3703351318836212,0.5643764138221741,0.5831498503684998,0.8303821086883545,0.8836601376533508,0.8476974964141846,0.9472933411598206,0.25329846143722534,0.11601176112890244,0.9747433066368103,0.34176814556121826,0.8251526355743408,0.17527931928634644,0.5483436584472656,0.8419793248176575,0.9838578701019287,0.07968159765005112,0.09145355969667435,0.826462984085083,0.23544777929782867,0.3038272559642792,0.8828607201576233,0.344532310962677,0.2766166627407074,0.057731930166482925,0.7391823530197144,0.47562375664711,0.7556608319282532,0.2841605246067047,0.6486907601356506,0.2636808753013611,0.023107757791876793,0.6709638833999634,0.6070297956466675,0.2201913446187973,0.8700276613235474,0.12464558333158493,0.8486520648002625,0.235423743724823,0.9058780670166016,0.03418419882655144,0.7535243630409241,0.5563464760780334,0.8332407474517822,0.5922387838363647,0.40195754170417786,0.25363990664482117,0.9972585439682007,0.9446473717689514,0.44357454776763916,0.5888155102729797,0.3688054382801056,0.2110026776790619,0.16790752112865448,0.010274024680256844,0.5046085715293884,0.7927681803703308,0.07285496592521667,0.8019525408744812,0.5883767604827881,0.0405036099255085,0.4820171296596527,0.188790425658226,0.22243718802928925,0.6020722389221191,0.7702606320381165,0.07360490411520004,0.7017942070960999,0.018562905490398407,0.6460157632827759,0.29041850566864014,0.5751222968101501,0.40912067890167236,0.7265976071357727,0.6026481986045837,0.26270273327827454,0.2948898375034332,0.665690541267395,0.47435808181762695,0.36917752027511597,0.06358526647090912,0.565690815448761,0.04643912985920906,0.6920033693313599,0.6904599666595459,0.7231625914573669,0.5998018383979797,0.5368053913116455,0.9239375591278076,0.3611655533313751,0.8831526041030884,0.8676901459693909,0.7948328852653503,0.4128318130970001,0.4721153676509857,0.39066994190216064,0.2203458845615387,0.7242655158042908,0.8013365268707275,0.552757978439331,0.6674852967262268,0.9250481128692627,0.7821402549743652,0.13559220731258392,0.7416204810142517,0.9529541730880737,0.5500898361206055,0.3610351085662842,0.6199477314949036,0.7445921897888184,0.5828084945678711,0.22443918883800507,0.10699205845594406,0.6624839901924133,0.2649187445640564,0.9852810502052307,0.6515886187553406,0.5583882331848145,0.3834531009197235,0.9359939098358154,0.8240406513214111,0.631750226020813,0.03376126289367676,0.8696344494819641,0.17219533026218414,0.0512053407728672,0.1792028546333313,0.1741051971912384,0.1618472784757614,0.4589572846889496,0.5846876502037048,0.7165510654449463,0.6666379570960999,0.8034464716911316,0.8408963084220886,0.33244580030441284,0.9043223261833191,0.4367749094963074,0.5260583162307739,0.17052626609802246,0.7733922004699707,0.34092459082603455,0.7339818477630615,0.9219359159469604,0.236061692237854,0.05947384610772133,0.7501789927482605,0.9122900366783142,0.22693008184432983,0.4718751907348633,0.4210229218006134,0.11501990258693695,0.4955475628376007,0.7880254983901978,0.1566125601530075,0.379436194896698,0.28470107913017273,0.8385181427001953,0.669925332069397,0.44045940041542053,0.21030208468437195,0.43408194184303284,0.6881758570671082,0.999151349067688,0.8823217153549194,0.6883155107498169,0.41020721197128296,0.8052054643630981,0.5057960152626038,0.340267539024353,0.43431583046913147,0.5931105017662048,0.017704544588923454,0.7937710285186768,0.7223883271217346,0.41894444823265076,0.5870479941368103,0.5931740999221802,0.377535879611969,0.568333625793457,0.7279545664787292,0.9516356587409973,0.7008804082870483,0.5602816343307495,0.9451645016670227,0.04009874165058136,0.5847292542457581,0.04684118181467056,0.8882943391799927,0.12886029481887817,0.26304569840431213,0.6296922564506531,0.4929397702217102,0.26196181774139404,0.6638117432594299,0.33967679738998413,0.8484995365142822,0.634166955947876,0.38546454906463623,0.05189669504761696,0.2459883987903595,0.25851762294769287,0.789595901966095,0.3542202413082123,0.09381032735109329,0.06850702315568924,0.5028350949287415,0.6331533193588257,0.8120883107185364,0.027338240295648575]}]}
1 change: 1 addition & 0 deletions ods_ci/tests/Resources/Files/triton/kserve-triton-fil-rest-output.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"model_name":"fil","model_version":"1","outputs":[{"name":"output__0","datatype":"FP32","shape":[1,2],"data":[0.9988755583763123,0.0011244702618569136]}]}
1 change: 0 additions & 1 deletion ods_ci/tests/Resources/Files/triton/kserve-triton-keras-rest-input.json
View file
Open in desktop

This file was deleted.

3 changes: 3 additions & 0 deletions ods_ci/tests/Resources/Files/triton/triton_onnx_rest_servingruntime.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,9 @@ spec:
name: triton
priority: 1
version: "2"
- autoSelect: true
name: xgboost
version: "1"
- autoSelect: true
name: python
version: "1"
90 changes: 85 additions & 5 deletions ...l_serving/1009__model_serving_triton_on_kserve/1009__model_serving_triton_on_kserve.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,16 +50,23 @@
${EXPECTED_INFERENCE_PYTHON_REST_OUTPUT_FILE}= tests/Resources/Files/triton/kserve-triton-python-rest-output.json
${INFERENCE_REST_INPUT_PYTHON}= @tests/Resources/Files/triton/kserve-triton-python-rest-input.json
${KERAS_RUNTIME_NAME}= triton-keras-rest
${KERAS_MODEL_NAME}= resnet50
${KERAS_RUNTIME_FILEPATH}= ${RESOURCES_DIRPATH}/triton_keras_rest_servingruntime.yaml
${INFERENCE_REST_INPUT_KERAS}= @tests/Resources/Files/triton/kserve-triton-keras-rest-input.json
${EXPECTED_INFERENCE_REST_OUTPUT_FILE_KERAS}= tests/Resources/Files/triton/kserve-triton-keras-rest-output.json
${KERAS_RUNTIME_NAME_GRPC}= triton-keras-grpc
${KERAS_GRPC_RUNTIME_FILEPATH}= ${RESOURCES_DIRPATH}/triton_keras_gRPC_servingruntime.yaml
${INFERENCE_GRPC_INPUT_KERAS}= tests/Resources/Files/triton/kserve-triton-keras-gRPC-input.json
${EXPECTED_INFERENCE_GRPC_OUTPUT_FILE_KERAS}= tests/Resources/Files/triton/kserve-triton-keras-gRPC-output.json
${FIL_MODEL_NAME}= fil
${FIL_RUNTIME_NAME_REST}= triton-fil-rest
${INFERENCE_REST_INPUT_FIL}= @tests/Resources/Files/triton/kserve-triton-fil-rest-input.json
${EXPECTED_INFERENCE_REST_OUTPUT_FILE_FIL}= tests/Resources/Files/triton/kserve-triton-fil-rest-output.json
${PYTHON_MODEL_NAME}= python
${EXPECTED_INFERENCE_GRPC_OUTPUT_FILE_PYTHON}= tests/Resources/Files/triton/kserve-triton-python-gRPC-output.json
${INFERENCE_GRPC_INPUT_PYTHON}= tests/Resources/Files/triton/kserve-triton-python-gRPC-input.json

*** Test Cases ***
Test Onnx Model Rest Inference Via UI (Triton on Kserve) # robocop: off=too-long-test-case

Check notice

Code scanning / Robocop

Variable '{{ name }}' is assigned but not used Note test

Variable '${FIL_RUNTIME_NAME_REST}' is assigned but not used
[Documentation] Test the deployment of an onnx model in Kserve using Triton
[Tags] Sanity RHOAIENG-11565
Open Data Science Projects Home Page
Expand Down Expand Up @@ -213,6 +220,50 @@
... AND
... Delete Serving Runtime Template From CLI displayed_name=triton-tensorflow-grpc

Test KERAS Model Grpc Inference Via UI (Triton on Kserve) # robocop: off=too-long-test-case
[Documentation] Test the deployment of an keras model in Kserve using Triton
[Tags] Sanity RHOAIENG-10327
Open Data Science Projects Home Page
Create Data Science Project title=${PRJ_TITLE} description=${PRJ_DESCRIPTION}
... existing_project=${FALSE}
Open Dashboard Settings settings_page=Serving runtimes
Upload Serving Runtime Template runtime_filepath=${KERAS_GRPC_RUNTIME_FILEPATH}
... serving_platform=single runtime_protocol=gRPC
Serving Runtime Template Should Be Listed displayed_name=${KERAS_RUNTIME_NAME_GRPC}
... serving_platform=single
Recreate S3 Data Connection project_title=${PRJ_TITLE} dc_name=model-serving-connection
... aws_access_key=${S3.AWS_ACCESS_KEY_ID} aws_secret_access=${S3.AWS_SECRET_ACCESS_KEY}
... aws_bucket_name=ods-ci-s3
Deploy Kserve Model Via UI model_name=${PYTORCH_MODEL_NAME} serving_runtime=triton-keras-grpc
... data_connection=model-serving-connection path=tritonkeras/model_repository/ model_framework=tensorflow - 2
... token=${TRUE}
Wait For Pods To Be Ready label_selector=serving.kserve.io/inferenceservice=${PYTORCH_MODEL_NAME}
... namespace=${PRJ_TITLE}
${EXPECTED_INFERENCE_GRPC_OUTPUT_KERAS}= Load Json File file_path=${EXPECTED_INFERENCE_GRPC_OUTPUT_FILE_KERAS}
... as_string=${TRUE}
${EXPECTED_INFERENCE_GRPC_OUTPUT_KERAS}= Load Json String ${EXPECTED_INFERENCE_GRPC_OUTPUT_KERAS}
${EXPECTED_INFERENCE_GRPC_OUTPUT_KERAS}= Evaluate json.dumps(${EXPECTED_INFERENCE_GRPC_OUTPUT_KERAS})

Check warning

Code scanning / Robocop

Line is too long ({{ line_length }}/{{ allowed_length }}) Warning test

Line is too long (123/120)
Log ${EXPECTED_INFERENCE_GRPC_OUTPUT_KERAS}
Open Model Serving Home Page
${host_url}= Get Model Route Via UI model_name=${PYTORCH_MODEL_NAME}
${host}= Evaluate re.search(r"${PATTERN}", r"${host_url}").group(1) re
Log ${host}
${token}= Get Access Token Via UI single_model=${TRUE} model_name=resnet50 project_name=${PRJ_TITLE}
${inference_output}= Query Model With GRPCURL host=${host} port=443
... endpoint=inference.GRPCInferenceService/ModelInfer
... json_body=@ input_filepath=${INFERENCE_GRPC_INPUT_KERAS}
... insecure=${True} protobuf_file=${PROTOBUFF_FILE} json_header="Authorization: Bearer ${token}"
Log ${inference_output}
${inference_output}= Evaluate json.dumps(${inference_output})
Log ${inference_output}
${result} ${list}= Inference Comparison ${EXPECTED_INFERENCE_GRPC_OUTPUT_KERAS} ${inference_output}
[Teardown] Run Keywords Get Kserve Events And Logs model_name=${PYTORCH_MODEL_NAME}
... project_title=${PRJ_TITLE}

Check warning

Code scanning / Robocop

Test case '{{ test_name }}' has too many keywords inside ({{ keyword_count }}/{{ max_allowed_count }}) Warning test

Test case 'Test KERAS Model Grpc Inference Via UI (Triton on Kserve)' has too many keywords inside (22/10)
... AND
... Clean All Models Of Current User
... AND
... Delete Serving Runtime Template From CLI displayed_name=triton-keras-grpc

Test KERAS Model Inference Via UI(Triton on Kserve)
[Documentation] Test the deployment of an keras model in Kserve using Triton
[Tags] Sanity RHOAIENG-10328
Expand All @@ -226,24 +277,24 @@
Serving Runtime Template Should Be Listed displayed_name=${KERAS_RUNTIME_NAME}
... serving_platform=single
Recreate S3 Data Connection project_title=${PRJ_TITLE} dc_name=model-serving-connection
... aws_access_key=${S3.AWS_ACCESS_KEY_ID} aws_secret_access=${S3.AWS_SECRET_ACCESS_KEY}

Check warning

Code scanning / Robocop

Line is too long ({{ line_length }}/{{ allowed_length }}) Warning test

Line is too long (122/120)
... aws_bucket_name=ods-ci-s3
Deploy Kserve Model Via UI model_name=${KERAS_MODEL_NAME} serving_runtime=triton-keras-rest
Deploy Kserve Model Via UI model_name=${PYTORCH_MODEL_NAME} serving_runtime=triton-keras-rest
... data_connection=model-serving-connection path=tritonkeras/model_repository/ model_framework=tensorflow - 2
Wait For Pods To Be Ready label_selector=serving.kserve.io/inferenceservice=${KERAS_MODEL_NAME}
Wait For Pods To Be Ready label_selector=serving.kserve.io/inferenceservice=${PYTORCH_MODEL_NAME}
... namespace=${PRJ_TITLE} timeout=180s
${EXPECTED_INFERENCE_REST_OUTPUT_KERAS}= Load Json File
... file_path=${EXPECTED_INFERENCE_REST_OUTPUT_FILE_KERAS} as_string=${TRUE}
Log ${EXPECTED_INFERENCE_REST_OUTPUT_KERAS}
Run Keyword And Continue On Failure Verify Model Inference With Retries
... ${KERAS_MODEL_NAME} ${INFERENCE_REST_INPUT_KERAS} ${EXPECTED_INFERENCE_REST_OUTPUT_KERAS}
... ${PYTORCH_MODEL_NAME} ${INFERENCE_REST_INPUT_KERAS} ${EXPECTED_INFERENCE_REST_OUTPUT_KERAS}
... token_auth=${FALSE} project_title=${PRJ_TITLE}
[Teardown] Run Keywords Get Kserve Events And Logs model_name=${KERAS_MODEL_NAME}
[Teardown] Run Keywords Get Kserve Events And Logs model_name=${PYTORCH_MODEL_NAME}
... project_title=${PRJ_TITLE}
... AND
... Clean All Models Of Current User
... AND
... Delete Serving Runtime Template From CLI displayed_name=triton-keras-rest

Check notice

Code scanning / Robocop

Variable '{{ name }}' is assigned but not used Note test

Variable '${result}' is assigned but not used

Check notice

Code scanning / Robocop

Variable '{{ name }}' is assigned but not used Note test

Variable '${list}' is assigned but not used

Test Python Model Grpc Inference Via UI (Triton on Kserve) # robocop: off=too-long-test-case
[Documentation] Test the deployment of an python model in Kserve using Triton
Expand Down Expand Up @@ -321,6 +372,35 @@
... AND
... Delete Serving Runtime Template From CLI displayed_name=triton-kserve-rest

Test FIL Model Rest Inference Via UI (Triton on Kserve) # robocop: off=too-long-test-case
[Documentation] Test the deployment of an fil model in Kserve using Triton
[Tags] Sanity RHOAIENG-15649
Open Data Science Projects Home Page
Create Data Science Project title=${PRJ_TITLE} description=${PRJ_DESCRIPTION}
... existing_project=${FALSE}
Open Dashboard Settings settings_page=Serving runtimes
Upload Serving Runtime Template runtime_filepath=${ONNX_RUNTIME_FILEPATH}
... serving_platform=single runtime_protocol=REST
Serving Runtime Template Should Be Listed displayed_name=${ONNX_RUNTIME_NAME}
... serving_platform=single
Recreate S3 Data Connection project_title=${PRJ_TITLE} dc_name=model-serving-connection
... aws_access_key=${S3.AWS_ACCESS_KEY_ID} aws_secret_access=${S3.AWS_SECRET_ACCESS_KEY}
... aws_bucket_name=ods-ci-s3
Deploy Kserve Model Via UI model_name=${FIL_MODEL_NAME} serving_runtime=triton-kserve-rest
... data_connection=model-serving-connection path=triton/model_repository/ model_framework=xgboost - 1
Wait For Pods To Be Ready label_selector=serving.kserve.io/inferenceservice=${FIL_MODEL_NAME}
... namespace=${PRJ_TITLE}
${EXPECTED_INFERENCE_REST_OUTPUT_FIL}= Load Json File file_path=${EXPECTED_INFERENCE_REST_OUTPUT_FILE_FIL}
... as_string=${TRUE}
Run Keyword And Continue On Failure Verify Model Inference With Retries
... ${FIL_MODEL_NAME} ${INFERENCE_REST_INPUT_FIL} ${EXPECTED_INFERENCE_REST_OUTPUT_FIL}
... token_auth=${FALSE} project_title=${PRJ_TITLE}
[Teardown] Run Keywords Get Kserve Events And Logs model_name=${FIL_MODEL_NAME}
... project_title=${PRJ_TITLE}
... AND
... Clean All Models Of Current User
... AND
... Delete Serving Runtime Template From CLI displayed_name=triton-kserve-rest

*** Keywords ***
Triton On Kserve Suite Setup
Expand Down
Loading