Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Function deployment fails in multi node kind clusters #126

Open
shyamjesal opened this issue Feb 18, 2021 · 5 comments
Open

Function deployment fails in multi node kind clusters #126

shyamjesal opened this issue Feb 18, 2021 · 5 comments
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@shyamjesal
Copy link
Collaborator

Setup:
1 master node
2 worker nodes
Image used: vhiveease/vhive_dev_env
Error:
Knative function containers fail to boot. Status loops between [Error, CrashLoopBackOff, Terminating].
Error log:

root@kind-control-plane:/vhive# kubectl logs helloworld-0-00001-deployment-7b8fcdb447-knnhf queue-proxy
time="2021-02-18T13:27:16Z" level=info msg="This is the vHive QP code"
required key GUEST_ADDR missing value
@shyamjesal shyamjesal added enhancement New feature or request help wanted Extra attention is needed labels Feb 18, 2021
@vfoehn
Copy link

vfoehn commented Apr 5, 2021

I experience the same error when trying to run vHive on a single-node minikube cluster. However, I don't think the cluster setup has anything to do with the error. It seems like the image docker.io/vhiveease/queue-39be6f1d08a095bd076a71d288d295b6 doesn't run correctly unless GUEST_ADDR is set (perhaps as an environment variable).

The following lines (introduced on 27.11.2020 and 30.11.2020) inside the file cri/container_create.go seem to require a value for GUEST_ADDR inside the function createQueueProxy(...):

guestIPEnv = "GUEST_ADDR"
...
guestIPKeyVal := &criapi.KeyValue{Key: guestIPEnv, Value: vmConfig.guestIP}

Note: I have tried both versions (i.e., latest and c3d2e0ad00457c972b642a1b6cf7c67ac1a58a5dcb83207e244c3cf6e19befbe) of the image docker.io/vhiveease/queue-39be6f1d08a095bd076a71d288d295b6, but neither works for me. Since both versions are younger than the code changes described above, this isn't a surprise to me.

If I decide to work with vHive in the future, I will take a closer look at this issue.

@ustiugov
Copy link
Member

ustiugov commented Apr 5, 2021

@vfoehn thank you for your comment.

vHive starts each VM with a tap and its corresponding IP address (this is assigned by the vHive code not the CNI plugin), using the environment variable that you mentioned. This is the intended behavior.

This issue is about a multi-node setup in a kind container (i.e., a kubernetes cluster running in Docker containers instead of VMs or bare metal). The multi-node setup should work on bare metal or VMs without any problem.

If you use vHive in a different scenario. Please raise a separate issue and we could try to investigate it together.

@vfoehn
Copy link

vfoehn commented Apr 6, 2021

@ustiugov thanks for the quick reply and the information.

I didn't follow the instructions from docs/quickstart_guide.md to a T since I wanted to run vHive on a minikube cluster (as opposed to creating a new cluster using kubeadm) I had already installed on my machine. Consequently, it is very possible that the mistake was caused by me or my setup. This is why I'm hesitant to raise a separate issue just yet. However, if I start using vHive for my project, I will investigate it more closely.

@prateeksahu
Copy link

Hi @ustiugov ,
I am running into a similar issue using a local single node cluster setup using the single node steps described in the quick-start guide.

Since I used the create_one_node_cluster and did not see any errors, I am assuming all the deployment have been done correctly. Any insight into it is appreciated.

@ustiugov
Copy link
Member

Hi @prateeksahu,

This issue was open for kind containers, are you running in kind or on a bare metal / VM? If your issue is not related to the kind setup, please open a new issue.

Then, could you please specify the exact steps (which you execute on a clean Ubuntu 18 node or a vm) and supply relevant snippets of the logs, attaching full logs?

HermioneKT pushed a commit that referenced this issue Jan 31, 2024
@KTCyber @HermioneKT
# This is a combination of 154 commits.
6674807 
# This is the 1st commit message:

refactor

# This is the commit message #2:

modify powerNodeSelector

# This is the commit message #3:

modify powerNodeSelector

# This is the commit message #4:

modify powerNodeSelector

# This is the commit message #5:

modify namespace

# This is the commit message #6:

modify namespace

# This is the commit message #7:

add namespace

# This is the commit message #8:

init power manager script

# This is the commit message #9:

add go code

# This is the commit message #10:

test

# This is the commit message #11:

test

# This is the commit message #12:

test

# This is the commit message #13:

test

# This is the commit message #14:

go.sum

# This is the commit message #15:

go sum

# This is the commit message #16:

go mod tidy

# This is the commit message #17:

test

# This is the commit message #18:

test

# This is the commit message #19:

test

# This is the commit message #20:

test

# This is the commit message #21:

test

# This is the commit message #22:

test

# This is the commit message #23:

test

# This is the commit message #24:

test

# This is the commit message #25:

test

# This is the commit message #26:

test

# This is the commit message #27:

test

# This is the commit message #28:

test

# This is the commit message #29:

print output

# This is the commit message #30:

test

# This is the commit message #31:

test

# This is the commit message #32:

test

# This is the commit message #33:

test

# This is the commit message #34:

test

# This is the commit message #35:

test

# This is the commit message #36:

test

# This is the commit message #37:

test

# This is the commit message #38:

test

# This is the commit message #39:

test

# This is the commit message #40:

refactor

# This is the commit message #41:

change val

# This is the commit message #42:

test

# This is the commit message #43:

test

# This is the commit message #44:

test

# This is the commit message #45:

exp1

# This is the commit message #46:

add url

# This is the commit message #47:

test

# This is the commit message #48:

test

# This is the commit message #49:

test auth

# This is the commit message #50:

test

# This is the commit message #51:

test

# This is the commit message #52:

test

# This is the commit message #53:

test

# This is the commit message #54:

test

# This is the commit message #55:

fix bug

# This is the commit message #56:

remove log

# This is the commit message #57:

test

# This is the commit message #58:

test

# This is the commit message #59:

test

# This is the commit message #60:

fix loop

# This is the commit message #61:

increase invocation

# This is the commit message #62:

test

# This is the commit message #63:

test

# This is the commit message #64:

test

# This is the commit message #65:

test

# This is the commit message #66:

test

# This is the commit message #67:

test

# This is the commit message #68:

test

# This is the commit message #69:

Test

# This is the commit message #70:

test

# This is the commit message #71:

test

# This is the commit message #72:

test

# This is the commit message #73:

test

# This is the commit message #74:

test

# This is the commit message #75:

test

# This is the commit message #76:

test

# This is the commit message #77:

test

# This is the commit message #78:

test

# This is the commit message #79:

test

# This is the commit message #80:

test

# This is the commit message #81:

test

# This is the commit message #82:

test

# This is the commit message #83:

test

# This is the commit message #84:

test

# This is the commit message #85:

test

# This is the commit message #86:

test

# This is the commit message #87:

test

# This is the commit message #88:

test

# This is the commit message #89:

test

# This is the commit message #90:

test1

# This is the commit message #91:

test

# This is the commit message #92:

test

# This is the commit message #93:

test

# This is the commit message #94:

test

# This is the commit message #95:

test

# This is the commit message #96:

test

# This is the commit message #97:

test

# This is the commit message #98:

test

# This is the commit message #99:

test

# This is the commit message #100:

test

# This is the commit message #101:

test

# This is the commit message #102:

test

# This is the commit message #103:

test

# This is the commit message #104:

test

# This is the commit message #105:

test

# This is the commit message #106:

test

# This is the commit message #107:

test

# This is the commit message #108:

test

# This is the commit message #109:

test1

# This is the commit message #110:

test

# This is the commit message #111:

test

# This is the commit message #112:

Test

# This is the commit message #113:

test

# This is the commit message #114:

test

# This is the commit message #115:

test

# This is the commit message #116:

test

# This is the commit message #117:

test

# This is the commit message #118:

test

# This is the commit message #119:

Test

# This is the commit message #120:

Test

# This is the commit message #121:

test

# This is the commit message #122:

test

# This is the commit message #123:

Test

# This is the commit message #124:

test

# This is the commit message #125:

test

# This is the commit message #126:

test

# This is the commit message #127:

test

# This is the commit message #128:

test

# This is the commit message #129:

test

# This is the commit message #130:

test

# This is the commit message #131:

test

# This is the commit message #132:

test

# This is the commit message #133:

test

# This is the commit message #134:

test

# This is the commit message #135:

test

# This is the commit message #136:

test

# This is the commit message #137:

test

# This is the commit message #138:

test

# This is the commit message #139:

test

# This is the commit message #140:

test

# This is the commit message #141:

test

# This is the commit message #142:

test

# This is the commit message #143:

Test

# This is the commit message #144:

Test

# This is the commit message #145:

Test

# This is the commit message #146:

Test

# This is the commit message #147:

test

# This is the commit message #148:

test

# This is the commit message #149:

test

# This is the commit message #150:

test

# This is the commit message #151:

test

# This is the commit message #152:

test

# This is the commit message #153:

test

# This is the commit message #154:

Test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@prateeksahu @ustiugov @shyamjesal @vfoehn