Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: Increase initial time to stabilize for FVTs #429

Merged
merged 4 commits into from
Sep 13, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions fvt/fvtclient.go
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,8 @@ import (
torchserveapi "github.com/kserve/modelmesh-serving/fvt/generated/torchserve/apis"
)

const PredictorTimeout = time.Second * 120 // absolute time to wait for predictor to become ready
const TimeForStatusToStabilize = time.Second * 5 // time to wait between watcher events before assuming a stable state
const PredictorTimeout = time.Second * 120 // absolute time to wait for predictor to become ready
const TimeForStatusToStabilize = time.Second * 10 // time to wait between watcher events before assuming a stable state

type ModelServingConnectionType int

Expand Down
2 changes: 1 addition & 1 deletion fvt/predictor/predictor_suite_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ var _ = SynchronizedBeforeSuite(func() []byte {
FVTClientInstance.CreateTLSSecrets()

// ensure a stable deploy state
WaitForStableActiveDeployState(time.Second * 30)
WaitForStableActiveDeployState(time.Second * 45)

return nil
}, func(_ []byte) {
Expand Down
24 changes: 22 additions & 2 deletions fvt/storage/storage_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -130,11 +130,21 @@ var _ = Describe("ISVCs", func() {
// from the old to the new pod

// make a shallow copy of default configmap (don't modify the DefaultConfig reference)
// keeping 1 pod per runtime and don't scale to 0
config := make(map[string]interface{})
for k, v := range DefaultConfig {
config[k] = v
}

// scale to 0 for resource-constrained environments (only 2 CPUs on GH actions)
// to stop and remove runtimes which are not used for this test
// Warning FailedScheduling pod/modelmesh-serving-mlserver-1.x-...
// 0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available:
// 1 No preemption victims found for incoming pod.
config["scaleToZero"] = map[string]interface{}{
"enabled": true,
"gracePeriodSeconds": 5,
}

// update the model-serving-config to allow any PVC
config["allowAnyPVC"] = true

Expand Down Expand Up @@ -194,11 +204,21 @@ var _ = Describe("ISVCs", func() {

It("should fail with non-existent PVC", func() {
// make a shallow copy of default configmap (don't modify the DefaultConfig reference)
// keeping 1 pod per runtime and don't scale to 0
config := make(map[string]interface{})
for k, v := range DefaultConfig {
config[k] = v
}

// scale to 0 for resource-constrained environments (only 2 CPUs on GH actions)
// to stop and remove runtimes which are not used for this test
// Warning FailedScheduling pod/modelmesh-serving-mlserver-1.x-...
// 0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available:
// 1 No preemption victims found for incoming pod.
config["scaleToZero"] = map[string]interface{}{
"enabled": true,
"gracePeriodSeconds": 5,
}

// update the model-serving-config to allow any PVC
config["allowAnyPVC"] = true
FVTClientInstance.ApplyUserConfigMap(config)
Expand Down