Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fail to spin up stargate service but Cassandra is running #1084

Open
chenweisomebody126 opened this issue Oct 6, 2023 · 1 comment
Open

Fail to spin up stargate service but Cassandra is running #1084

chenweisomebody126 opened this issue Oct 6, 2023 · 1 comment
Labels
question Further information is requested

Comments

@chenweisomebody126
Copy link

chenweisomebody126 commented Oct 6, 2023 •
edited by sync-by-unito bot
Loading

What did you do?
I followed the steps in https://docs.k8ssandra.io/install/local/single-cluster-helm/ to Cassandra.

Did you expect to see some different?
I did not see stargate service

 % kubectl get service  -n k8ssandra-operator

NAME                                               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                                                 AGE
demo-dc1-additional-seed-service                   ClusterIP   None           <none>        <none>                                                  2d
demo-dc1-all-pods-service                          ClusterIP   None           <none>        9042/TCP,8080/TCP,9103/TCP,9000/TCP                     2d
demo-dc1-service                                   ClusterIP   None           <none>        9042/TCP,9142/TCP,8080/TCP,9103/TCP,9000/TCP,9160/TCP   2d
demo-seed-service                                  ClusterIP   None           <none>        <none>                                                  2d
k8ssandra-operator-cass-operator-webhook-service   ClusterIP   10.96.14.228   <none>        443/TCP                                                 2d
k8ssandra-operator-webhook-service                 ClusterIP   10.96.49.100   <none>        443/TCP

Environment
Cassandra version: 3.11.9

  cassandra:
    serverVersion: "3.11.9"

helm inspection:

% helm ls -A
NAME              	NAMESPACE         	REVISION	UPDATED                             	STATUS  	CHART                   	APP VERSION
cert-manager      	cert-manager      	1       	2023-10-04 16:44:36.043497 -0400 EDT	deployed	cert-manager-v1.13.0    	v1.13.0    
k8ssandra-operator	k8ssandra-operator	1       	2023-10-04 17:12:18.210535 -0400 EDT	deployed	k8ssandra-operator-1.8.1	1.8.1

  • K8ssandra Operator version:

    Insert image tag or Git SHA here

    * Kubernetes version information: `kubectl version````Client Version: v1.28.2

Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.25.3

* Kubernetes cluster kind:

I ran `scripts/setup-kind-multicluster.sh --clusters 1 --kind-worker-nodes 4`

* Manifests:

I ran `kubectl apply -n k8ssandra-operator -f k8c1.yml`
apiVersion: k8ssandra.io/v1alpha1

kind: K8ssandraCluster
metadata:
name: demo
spec:
cassandra:
serverVersion: "3.11.9"
datacenters:
- metadata:
name: dc1
size: 3
storageConfig:
cassandraDataVolumeClaimSpec:
storageClassName: standard
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
stopped: false
config:
jvmOptions:
heapSize: 512M
stargate:
size: 1
heapSize: 256M


* K8ssandra Operator Logs:

2023-10-06T21:19:39.560Z INFO client::callNodeMgmtEndpoint {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"demo","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "demo", "reconcileID": "895ab605-7c14-429d-baac-90d5e151b5e1", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""}
2023-10-06T21:19:39.562Z ERROR Failed to CALL list keyspaces system_traces on pod demo-dc1-default-sts-0 {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"demo","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "demo", "reconcileID": "895ab605-7c14-429d-baac-90d5e151b5e1", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": "", "error": "incorrect status code of 404 when calling endpoint"}
github.com/k8ssandra/k8ssandra-operator/pkg/cassandra.(*defaultManagementApiFacade).ListKeyspaces
/workspace/pkg/cassandra/management.go:195
github.com/k8ssandra/k8ssandra-operator/pkg/cassandra.(*defaultManagementApiFacade).EnsureKeyspaceReplication
/workspace/pkg/cassandra/management.go:289
github.com/k8ssandra/k8ssandra-operator/controllers/k8ssandra.(*K8ssandraClusterReconciler).updateReplicationOfSystemKeyspaces
/workspace/controllers/k8ssandra/schemas.go:157
github.com/k8ssandra/k8ssandra-operator/controllers/k8ssandra.(*K8ssandraClusterReconciler).checkSchemas
/workspace/controllers/k8ssandra/schemas.go:43
github.com/k8ssandra/k8ssandra-operator/controllers/k8ssandra.(*K8ssandraClusterReconciler).reconcileDatacenters
/workspace/controllers/k8ssandra/datacenters.go:199
github.com/k8ssandra/k8ssandra-operator/controllers/k8ssandra.(*K8ssandraClusterReconciler).reconcile
/workspace/controllers/k8ssandra/k8ssandracluster_controller.go:143
github.com/k8ssandra/k8ssandra-operator/controllers/k8ssandra.(*K8ssandraClusterReconciler).Reconcile
/workspace/controllers/k8ssandra/k8ssandracluster_controller.go:91
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235
2023-10-06T21:19:39.562Z ERROR Failed to update replication {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"demo","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "demo", "reconcileID": "895ab605-7c14-429d-baac-90d5e151b5e1", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": "", "keyspace": "system_traces", "error": "CALL list keyspaces system_traces failed on all datacenter dc1 pods"}
github.com/k8ssandra/k8ssandra-operator/controllers/k8ssandra.(*K8ssandraClusterReconciler).updateReplicationOfSystemKeyspaces
/workspace/controllers/k8ssandra/schemas.go:161
github.com/k8ssandra/k8ssandra-operator/controllers/k8ssandra.(*K8ssandraClusterReconciler).checkSchemas
/workspace/controllers/k8ssandra/schemas.go:43
github.com/k8ssandra/k8ssandra-operator/controllers/k8ssandra.(*K8ssandraClusterReconciler).reconcileDatacenters
/workspace/controllers/k8ssandra/datacenters.go:199
github.com/k8ssandra/k8ssandra-operator/controllers/k8ssandra.(*K8ssandraClusterReconciler).reconcile
/workspace/controllers/k8ssandra/k8ssandracluster_controller.go:143
github.com/k8ssandra/k8ssandra-operator/controllers/k8ssandra.(*K8ssandraClusterReconciler).Reconcile
/workspace/controllers/k8ssandra/k8ssandracluster_controller.go:91
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235
2023-10-06T21:19:39.562Z DEBUG events CALL list keyspaces system_traces failed on all datacenter dc1 pods {"type": "Warning", "object": {"kind":"K8ssandraCluster","namespace":"k8ssandra-operator","name":"demo","uid":"093f5222-14f7-49a4-845e-1fd74972505f","apiVersion":"k8ssandra.io/v1alpha1","resourceVersion":"62623"}, "reason": "Reconcile Error"}
2023-10-06T21:19:39.572Z INFO updated k8ssandracluster status {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"demo","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "demo", "reconcileID": "895ab605-7c14-429d-baac-90d5e151b5e1", "K8ssandraCluster": "k8ssandra-operator/demo"}
2023-10-06T21:19:39.572Z ERROR Reconciler error {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"demo","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "demo", "reconcileID": "895ab605-7c14-429d-baac-90d5e151b5e1", "error": "CALL list keyspaces system_traces failed on all datacenter dc1 pods"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235

**Anything else we need to know?**:
I use Mac M1 chip, so I actually manually use `kubectl edit ` to update the value of `config-builder` from `1.0-ubi7` to `1.0` as pointed in https://github.com/k8ssandra/cass-operator/issues/541#issuecomment-1683403321
image_config.yaml: |
apiVersion: [config.k8ssandra.io/v1beta1](http://config.k8ssandra.io/v1beta1)
kind: ImageConfig
images:
  system-logger: "k8ssandra/system-logger:v1.16.0"
  config-builder: "datastax/cass-config-builder:1.0




┆Issue is synchronized with this [Jira Story](https://datastax.jira.com/browse/K8OP-67) by [Unito](https://www.unito.io)
┆Issue Number: K8OP-67
@chenweisomebody126 chenweisomebody126 added the question Further information is requested label Oct 6, 2023
@chenweisomebody126 chenweisomebody126 mentioned this issue Oct 6, 2023
Closed
@burmanm
Copy link
Contributor

burmanm commented Oct 16, 2023

The problem is Cassandra 3.11.9, which is a broken version and for that reason we don't build new images for it and are not supporting it in k8ssandra-operator.

Update to 3.11.16 if you really wish to use 3.11.x series.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question Further information is requested
Projects
No open projects
K8ssandra
Status: No status
Milestone
No milestone
Development

No branches or pull requests
2 participants
@burmanm @chenweisomebody126