Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Indexer crashes after a restart command due to plugin load failure #505

Closed
aghosh0605 opened this issue Oct 30, 2024 · 4 comments
Closed
Labels
reporter/community Issue reported by the community

Comments

@aghosh0605
Copy link

aghosh0605 commented Oct 30, 2024

Describe the bug
Errors are showing up while loading the plugins after the indexer is restarted

To Reproduce
Steps to reproduce the behaviour:

  1. systemctl restart wazuh-indexer

Expected behaviour
The indexer crashes after using a simple restart command due to a plugin load failure. This issue occurred unexpectedly after a long period without changes or updates, and specific reproducible steps are currently not clear to me.

Plugins

opensearch-alerting
opensearch-anomaly-detection
opensearch-asynchronous-search
opensearch-cross-cluster-replication
opensearch-geospatial
opensearch-index-management
opensearch-job-scheduler
opensearch-knn
opensearch-ml
opensearch-neural-search
opensearch-notifications
opensearch-notifications-core
opensearch-observability
opensearch-performance-analyzer
opensearch-reports-scheduler
opensearch-security
opensearch-security-analytics
opensearch-sql

Screenshots
Logs of the error are attached below.

[2024-10-29T20:25:39,037][INFO ][o.o.p.PluginsService     ] [node-1] loaded plugin [opensearch-reports-scheduler]
[2024-10-29T20:25:39,038][INFO ][o.o.p.PluginsService     ] [node-1] loaded plugin [opensearch-security]
[2024-10-29T20:25:39,039][INFO ][o.o.p.PluginsService     ] [node-1] loaded plugin [opensearch-security-analytics]
[2024-10-29T20:25:39,039][INFO ][o.o.p.PluginsService     ] [node-1] loaded plugin [opensearch-sql]
[2024-10-29T20:25:39,105][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[2024-10-29T20:25:39,110][INFO ][o.o.e.ExtensionsManager  ] [node-1] ExtensionsManager initialized
[2024-10-29T20:25:39,134][INFO ][o.o.e.NodeEnvironment    ] [node-1] using [1] data paths, mounts [[/var (/dev/sda7)]], net usable_space [2.5tb], net total_space [2.7tb], types [xfs]
[2024-10-29T20:25:39,135][INFO ][o.o.e.NodeEnvironment    ] [node-1] heap size [20gb], compressed ordinary object pointers [true]
[2024-10-29T20:25:39,940][INFO ][o.o.n.Node               ] [node-1] node name [node-1], node ID [9DD3fUT4S26KIxJSf8wiOQ], cluster name [wazuh-indexer-cluster], roles [ingest, remote_cluster_client, data, cluster_manager]
[2024-10-29T20:25:42,096][ERROR][o.o.b.Bootstrap          ] [node-1] Exception
java.lang.IllegalArgumentException: unknown setting [org.opensearch.bootstrap.OpenSearchnode.master] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:608) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:549) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:519) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.common.settings.SettingsModule.<init>(SettingsModule.java:179) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.node.Node.<init>(Node.java:548) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.node.Node.<init>(Node.java:381) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) [opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) [opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) [opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.8.0.jar:2.8.0]
        at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) [opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) [opensearch-2.8.0.jar:2.8.0]
[2024-10-29T20:25:42,098][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [org.opensearch.bootstrap.OpenSearchnode.master] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.8.0.jar:2.8.0]
        at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.8.0.jar:2.8.0]
Caused by: java.lang.IllegalArgumentException: unknown setting [org.opensearch.bootstrap.OpenSearchnode.master] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:608) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:549) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:519) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.common.settings.SettingsModule.<init>(SettingsModule.java:179) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.node.Node.<init>(Node.java:548) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.node.Node.<init>(Node.java:381) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.8.0.jar:2.8.0]
        ... 6 more

Host/Environment (please complete the following information):

  • OS: Ubuntu 20.04.03
  • Version 4.7.1
@aghosh0605 aghosh0605 added level/task Task issue type/bug Bug issue labels Oct 30, 2024
@AlexRuiz7 AlexRuiz7 added reporter/community Issue reported by the community and removed level/task Task issue type/bug Bug issue labels Oct 30, 2024
@AlexRuiz7
Copy link
Member

Hello @aghosh0605

Looking at the logs, this looks more like a configuration issue. It seems there is a typo in any of the settings preventing the service from starting.

java.lang.IllegalArgumentException: unknown setting [org.opensearch.bootstrap.OpenSearchnode.master] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

Have you performed any configuration changes lately? Have you upgraded wazuh-indexer lately or installed, removed or updated any of the plugins?

Review your opensearch.yml or share with us so we can check it. In that case, remove any sensitive information such as public IPs.

Regards,
Alex

@aghosh0605
Copy link
Author

Hello @aghosh0605

Looking at the logs, this looks more like a configuration issue. It seems there is a typo in any of the settings preventing the service from starting.

java.lang.IllegalArgumentException: unknown setting [org.opensearch.bootstrap.OpenSearchnode.master] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

Have you performed any configuration changes lately? Have you upgraded wazuh-indexer lately or installed, removed or updated any of the plugins?

Review your opensearch.yml or share with us so we can check it. In that case, remove any sensitive information such as public IPs.

Regards, Alex

Hey, thanks for your support!

Mostly till we remember we hadn't done any changes after setup as it was running fine!
I attached the below configuration for opensearch.yml

org.opensearch.bootstrap.OpenSearchnode.master: true
node.data: true
node.ingest: true

cluster.name: wazuh-indexer-cluster
cluster.routing.allocation.disk.threshold_enabled: false

node.max_local_storage_nodes: "3"
path.data: /var/lib/wazuh-indexer
path.logs: /var/log/wazuh-indexer


plugins.security.ssl.http.pemcert_filepath: /etc/wazuh-indexer/certs/node-1.pem
plugins.security.ssl.http.pemkey_filepath: /etc/wazuh-indexer/certs/node-1-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
plugins.security.ssl.transport.pemcert_filepath: /etc/wazuh-indexer/certs/node-1.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/wazuh-indexer/certs/node-1-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.resolve_hostname: false
plugins.security.ssl.http.enabled_ciphers:
  - "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
  - "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"
  - "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"
  - "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"
plugins.security.ssl.http.enabled_protocols:
  - "TLSv1.2"
plugins.security.authcz.admin_dn:
- "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.restapi.roles_enabled:
- "all_access"
- "security_rest_api_access"

plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]

### Option to allow Filebeat-oss 7.10.2 to work ###
compatibility.override_main_response_version: true
node.name: node-1
network.host: 0.0.0.0
cluster.initial_master_nodes: node-1
plugins.security.nodes_dn:
        - CN=node-1,OU=Wazuh,O=Wazuh,L=California,C=US

@AlexRuiz7
Copy link
Member

AlexRuiz7 commented Oct 30, 2024

org.opensearch.bootstrap.OpenSearchnode.master: true

That's the unrecognized setting. I don't recognize node.data and node.ingest either.

Wazuh 4.7.1 is based on OpenSearch 2.8.0. I've been reviewing the configuration and some fields are deprecated. These configurations are not included on our builds.

[deprecated] node.master: true
[deprecated] node.data: true
[deprecated] node.ingest: true
[deprecated] node.max_local_storage_nodes: "3" 

Although they are deprecated, they should work. The problem is on the first line, as org.opensearch.bootstrap.OpenSearchnode.master is obviously not a valid setting. It should be node.master: true.

Update: node.max_local_storage_nodesis in fact included in our opensearch.yml. The other 3 settings are not.

@aghosh0605
Copy link
Author

org.opensearch.bootstrap.OpenSearchnode.master: true

That's the unrecognized setting. I don't recognize node.data and node.ingest either.

Wazuh 4.7.1 is based on OpenSearch 2.8.0. I've been reviewing the configuration and some fields are deprecated. These configurations are not included on our builds.

[deprecated] node.master: true
[deprecated] node.data: true
[deprecated] node.ingest: true
[deprecated] node.max_local_storage_nodes: "3" 

Although they are deprecated, they should work. The problem is on the first line, as org.opensearch.bootstrap.OpenSearchnode.master is obviously not a valid setting. It should be node.master: true.

Update: node.max_local_storage_nodesis in fact included in our opensearch.yml. The other 3 settings are not.

Hey! Thanks for the help. I don't know how it came but you are correct.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
reporter/community Issue reported by the community
Projects
None yet
Development

No branches or pull requests

2 participants