Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Scale from 0" and "MultipleFuncInvoke" tests often fail in vHive CRI CI #655

Open
ustiugov opened this issue Dec 29, 2022 · 2 comments
Open
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@ustiugov
Copy link
Member

ustiugov commented Dec 29, 2022

Describe the bug
vHive CRI tests exhibit a sporadic (but quite frequent) failure when scaling microVMs from 0:

=== RUN   TestAutoscaler/Scale_from_0
=== CONT  TestAutoscaler
    cri_test.go:166: 
        	Error Trace:	/root/_work/vHive/vHive/cri/cri_test.go:166
        	            				/root/_work/vHive/vHive/cri/cri_test.go:99
        	            				/root/_work/vHive/vHive/cri/cri_test.go:109
        	Error:      	Received unexpected error:
        	            	rpc error: code = DeadlineExceeded desc = context deadline exceeded
        	Test:       	TestAutoscaler
        	Messages:   	Failed to get response from function
=== CONT  TestAutoscaler/Scale_from_0
    testing.go:1336: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test
--- FAIL: TestAutoscaler (268.62s)
    --- PASS: TestAutoscaler/Scale_fn_with_concurrency_1 (8.58s)
    --- FAIL: TestAutoscaler/Scale_from_0 (260.04s)
=== RUN   TestMultipleFuncInvoke
    cri_test.go:167: 
        	Error Trace:	/root/_work/vHive/vHive/cri/cri_test.go:167
        	            				/root/_work/vHive/vHive/cri/cri_test.go:132
        	            				/root/_work/vHive/vHive/cri/asm_amd64.s:1571
        	Error:      	Received unexpected error:
        	            	rpc error: code = DeadlineExceeded desc = context deadline exceeded
        	Test:       	TestMultipleFuncInvoke
        	Messages:   	Failed to get response from function
--- FAIL: TestMultipleFuncInvoke (60.04s)

Hence, we exclude these tests from the CI until a fix is found.

@ustiugov ustiugov added bug Something isn't working help wanted Extra attention is needed labels Dec 29, 2022
@ustiugov ustiugov changed the title "Scale from 0" test fails in vHive CRI CI "Scale from 0" and "MultipleFuncInvoke" tests often fail in vHive CRI CI Dec 29, 2022
@leokondrashov
Copy link
Contributor

@lrq619 Was this fixed in your last CI update?

@lrq619
Copy link
Collaborator

lrq619 commented Aug 22, 2023

@lrq619 Was this fixed in your last CI update?

No, it is not fixed, there will still be sporadic failures

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

3 participants