Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

REQ: Add post mortem logs in case cluster-up failed #1329

Open
oshoval opened this issue Nov 25, 2024 · 4 comments
Open

REQ: Add post mortem logs in case cluster-up failed #1329

oshoval opened this issue Nov 25, 2024 · 4 comments
Labels
kind/enhancement sig/ci Denotes an issue or PR as being related to sig-ci, marks changes to the CI system.

Comments

@oshoval
Copy link
Contributor

oshoval commented Nov 25, 2024

Is your feature request related to a problem? Please describe:
Upon Kubevirtci cluster-up failures we don't have logs, such as kubectl pods, logs, describe etc

example https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/pr-logs/pull/kubevirt_kubevirt/13282/pull-kubevirt-e2e-k8s-1.30-ipv6-sig-network/1859255594477686784

16:41:01: Waiting on istio operator to become ready failed after maximum retries: Istio operator failed to move to Healthy status after max retries
16:41:02: /home/prow/go/src/github.com/kubevirt/kubevirt/kubevirtci/cluster-up/up.sh: line 34: pop_var_context: head of shell_variables not a function context
make: *** [Makefile:163: cluster-up] Error 1

Describe the solution you'd like:
Would be good to have collected artifacts on CI in case the cluster-up itself failed.

We need for example the following commands:
kubectl get pods -A
for each pod, we need please the kubectl describe, and the logs (for each of its containers)
Or whatever crucial info that we find beside that on the fly.

Describe alternatives you've considered:

Additional context:
Note that we do have some logs collecting as far as i remember, need to adapt / see if something miss.
(they are mostly about the docker BM containers, not the pods)

@oshoval
Copy link
Contributor Author

oshoval commented Nov 25, 2024

/cc @brianmcarey

@dosubot dosubot bot added the sig/ci Denotes an issue or PR as being related to sig-ci, marks changes to the CI system. label Nov 25, 2024
@aerosouund
Copy link
Member

@oshoval
Yes, i think this is a great suggestion and was a pain point for me in kubevirtci while developing
I will investigate how it can be approached

@oshoval
Copy link
Contributor Author

oshoval commented Nov 28, 2024

kubevirt/ipam-extensions#57

some code here might help (especially if you use go style)

@oshoval
Copy link
Contributor Author

oshoval commented Nov 28, 2024

Btw two more things that we can open issues for if desired:

  1. Once a cluster fails
    it will try to do some operations and we will see this error that we can stop the flow before that, to reduce noise
Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
  1. This error happens once a cluster fails, and also can be fixed
16:41:02: /home/prow/go/src/github.com/kubevirt/kubevirt/kubevirtci/cluster-up/up.sh: line 34: pop_var_context: head of shell_variables not a function context

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/enhancement sig/ci Denotes an issue or PR as being related to sig-ci, marks changes to the CI system.
Projects
None yet
Development

No branches or pull requests

2 participants