Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix startup errors on STIG compliant systems due to noexec filesystems #533

Merged
merged 2 commits into from
Nov 25, 2024

Conversation

QU3B1M
Copy link
Member

@QU3B1M QU3B1M commented Nov 9, 2024

Description

To avoid errors when starting wazuh-indexer on STIG compliant systems, where the /var/log directory is noexec, we have moved the temporary directory to /var/lib, which (almost every case) is not set to noexec. Additionally, the .restart file, which indicates when the system should be restarted after an upgrade, has been relocated to the new ../tmp directory (previously located in /tmp).

The creation of the ../tmp directory has been moved from the postinst step to the preinst step for consistency.

Related Issues

Resolves #501

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Moved tmp dir creation from postinst to preinst

Update .restart tmp file to be stored in the new tmp dir
@QU3B1M
Copy link
Member Author

QU3B1M commented Nov 11, 2024

Reproduced the error installing wazuh-indexer package without the fix on a STIG compliant system (/var/log, /var/tmp, /tmp being noexec)

Nov 11 12:58:44 ubuntu2204.localdomain systemd-entrypoint[4549]: WARNING: A terminally deprecated method in java.lang.System has been called
Nov 11 12:58:44 ubuntu2204.localdomain systemd-entrypoint[4549]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/s>
Nov 11 12:58:44 ubuntu2204.localdomain systemd-entrypoint[4549]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Nov 11 12:58:44 ubuntu2204.localdomain systemd-entrypoint[4549]: WARNING: System::setSecurityManager will be removed in a future release
Nov 11 12:58:44 ubuntu2204.localdomain systemd-entrypoint[4549]: Nov 11, 2024 12:58:44 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
Nov 11 12:58:44 ubuntu2204.localdomain systemd-entrypoint[4549]: WARNING: COMPAT locale provider will be removed in a future release
Nov 11 12:58:45 ubuntu2204.localdomain systemd-entrypoint[4549]: WARNING: A terminally deprecated method in java.lang.System has been called
Nov 11 12:58:45 ubuntu2204.localdomain systemd-entrypoint[4549]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/sha>
Nov 11 12:58:45 ubuntu2204.localdomain systemd-entrypoint[4549]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Nov 11 12:58:45 ubuntu2204.localdomain systemd-entrypoint[4549]: WARNING: System::setSecurityManager will be removed in a future release
Nov 11 12:58:50 ubuntu2204.localdomain systemd-entrypoint[4549]: ERROR: [1] bootstrap checks failed
Nov 11 12:58:50 ubuntu2204.localdomain systemd-entrypoint[4549]: [1]: system call filters failed to install; check the logs and fix your configuration or disable system>
Nov 11 12:58:50 ubuntu2204.localdomain systemd-entrypoint[4549]: ERROR: OpenSearch did not exit normally - check the logs at /var/log/wazuh-indexer/wazuh-cluster.log
Nov 11 12:58:50 ubuntu2204.localdomain systemd-entrypoint[4549]: fatal error in thread [Thread-3], exiting
Nov 11 12:58:50 ubuntu2204.localdomain systemd-entrypoint[4549]: java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
Nov 11 12:58:50 ubuntu2204.localdomain systemd-entrypoint[4549]:         at org.opensearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:48)
Nov 11 12:58:50 ubuntu2204.localdomain systemd-entrypoint[4549]:         at java.base/java.security.AccessController.doPrivileged(AccessController.java:319)
Nov 11 12:58:50 ubuntu2204.localdomain systemd-entrypoint[4549]:         at org.opensearch.systemd.Libsystemd.<clinit>(Libsystemd.java:47)
Nov 11 12:58:50 ubuntu2204.localdomain systemd-entrypoint[4549]:         at org.opensearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:126)
Nov 11 12:58:50 ubuntu2204.localdomain systemd-entrypoint[4549]:         at org.opensearch.systemd.SystemdPlugin.close(SystemdPlugin.java:152)
Nov 11 12:58:50 ubuntu2204.localdomain systemd-entrypoint[4549]:         at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89)
Nov 11 12:58:50 ubuntu2204.localdomain systemd-entrypoint[4549]:         at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131)
Nov 11 12:58:50 ubuntu2204.localdomain systemd-entrypoint[4549]:         at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:114)
Nov 11 12:58:50 ubuntu2204.localdomain systemd-entrypoint[4549]:         at org.opensearch.node.Node.close(Node.java:1791)
Nov 11 12:58:50 ubuntu2204.localdomain systemd-entrypoint[4549]:         at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89)
Nov 11 12:58:50 ubuntu2204.localdomain systemd-entrypoint[4549]:         at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131)
Nov 11 12:58:50 ubuntu2204.localdomain systemd-entrypoint[4549]:         at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:81)
Nov 11 12:58:50 ubuntu2204.localdomain systemd-entrypoint[4549]:         at org.opensearch.bootstrap.Bootstrap$4.run(Bootstrap.java:206)
Nov 11 12:58:50 ubuntu2204.localdomain systemd-entrypoint[4549]: Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.UnsatisfiedLinkError: /var/log/wa>

Validate the package with the fix can be correctly installed and started on the same system

  1. Install wazuh-indexer package with the fix
    bash 01_download_and_install_package.sh -id 11777760990 -n wazuh-indexer_4.10.2-0_amd64_c996476.deb
    Fetching artifacts list...
    Checking wazuh-indexer_4.10.2-0_amd64_c996476.deb package is generated for workflow run 11777760990
    Wazuh indexer artifact detected. Artifact ID: 2170967024
    Downloading wazuh-indexer package from GitHub artifactory...
    (It could take a couple of minutes)
    Package downloaded successfully
    Decompressing wazuh-indexer package...
    Archive:  ./package.zip
      inflating: wazuh-indexer_4.10.2-0_amd64_c996476.deb  
    Package decompressed
    Installing wazuh-indexer package...
    Package installed successfully.
  2. Start wazuh-indexer service
    systemctl start wazuh-indexer
    systemctl status wazuh-indexer
    
    ● wazuh-indexer.service - wazuh-indexer
         Loaded: loaded (/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: enabled)
         Active: active (running) since Mon 2024-11-11 13:12:23 UTC; 10s ago
           Docs: https://documentation.wazuh.com
       Main PID: 4857 (java)
          Tasks: 62 (limit: 4558)
         Memory: 1.3G
            CPU: 19.094s
         CGroup: /system.slice/wazuh-indexer.service
                 └─4857 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+A>
    
    Nov 11 13:12:16 ubuntu2204.localdomain systemd-entrypoint[4857]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/s>
    Nov 11 13:12:16 ubuntu2204.localdomain systemd-entrypoint[4857]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
    Nov 11 13:12:16 ubuntu2204.localdomain systemd-entrypoint[4857]: WARNING: System::setSecurityManager will be removed in a future release
    Nov 11 13:12:16 ubuntu2204.localdomain systemd-entrypoint[4857]: Nov 11, 2024 1:12:16 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
    Nov 11 13:12:16 ubuntu2204.localdomain systemd-entrypoint[4857]: WARNING: COMPAT locale provider will be removed in a future release
    Nov 11 13:12:17 ubuntu2204.localdomain systemd-entrypoint[4857]: WARNING: A terminally deprecated method in java.lang.System has been called
    Nov 11 13:12:17 ubuntu2204.localdomain systemd-entrypoint[4857]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/sha>
    Nov 11 13:12:17 ubuntu2204.localdomain systemd-entrypoint[4857]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
    Nov 11 13:12:17 ubuntu2204.localdomain systemd-entrypoint[4857]: WARNING: System::setSecurityManager will be removed in a future release
    Nov 11 13:12:23 ubuntu2204.localdomain systemd[1]: Started wazuh-indexer.
    

@QU3B1M
Copy link
Member Author

QU3B1M commented Nov 11, 2024

Check service maintains its previous status on a upgrade using this PR's package

  1. Check actual status
    systemctl status wazuh-indexer
    ● wazuh-indexer.service - wazuh-indexer
         Loaded: loaded (/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: enabled)
         Active: active (running) since Mon 2024-11-11 13:12:23 UTC; 10s ago
           Docs: https://documentation.wazuh.com
       Main PID: 4857 (java)
          Tasks: 62 (limit: 4558)
         Memory: 1.3G
            CPU: 19.094s
         CGroup: /system.slice/wazuh-indexer.service
                 └─4857 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+A>
  2. Upgrade wazuh-indexer using the package with the fix
    dpkg -i wazuh-indexer_4.10.2-0_amd64_c996476.deb 
    (Reading database ... 77523 files and directories currently installed.)
    Preparing to unpack wazuh-indexer_4.10.2-0_amd64_c996476.deb ...
    Running Wazuh Indexer Pre-Installation Script
    Stop existing wazuh-indexer.service
    Unpacking wazuh-indexer (4.10.2-0) over (4.10.2-0) ...
    Setting up wazuh-indexer (4.10.2-0) ...
    Running Wazuh Indexer Post-Installation Script
    Restarting wazuh-indexer service...
  3. Check current status
    systemctl status wazuh-indexer
    ● wazuh-indexer.service - wazuh-indexer
         Loaded: loaded (/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: enabled)
         Active: active (running) since Mon 2024-11-11 13:18:26 UTC; 1min 49s ago
           Docs: https://documentation.wazuh.com
       Main PID: 5168 (java)
          Tasks: 69 (limit: 4558)
         Memory: 1.3G
            CPU: 23.221s
         CGroup: /system.slice/wazuh-indexer.service
                 └─5168 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+A>

@f-galland f-galland self-requested a review November 25, 2024 14:18
@f-galland
Copy link
Member

Packages from this branch get installed and run properly on an almalinux 9 installation with /tmp, /var/log and /var/tmp mounted with the noexec flag:

[root@node-1 scripts]# cat /etc/os-release 
NAME="AlmaLinux"
VERSION="9.3 (Shamrock Pampas Cat)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.3"
PLATFORM_ID="platform:el9"
PRETTY_NAME="AlmaLinux 9.3 (Shamrock Pampas Cat)"
ANSI_COLOR="0;34"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:almalinux:almalinux:9::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"

ALMALINUX_MANTISBT_PROJECT="AlmaLinux-9"
ALMALINUX_MANTISBT_PROJECT_VERSION="9.3"
REDHAT_SUPPORT_PRODUCT="AlmaLinux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.3"


[root@node-1 scripts]# mount | grep noexec
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime,seclabel)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,nosuid,noexec,relatime)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime,seclabel)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime,seclabel)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime,seclabel)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
none on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,seclabel,mode=700)
none on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,seclabel,mode=700)
/dev/mapper/almalinux_alma9-root on /tmp type xfs (rw,noexec,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/mapper/almalinux_alma9-root on /var/log type xfs (rw,noexec,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/mapper/almalinux_alma9-root on /var/tmp type xfs (rw,noexec,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
none on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,seclabel,mode=700)

[root@node-1 scripts]# systemctl status wazuh-indexer --no-pager -l
● wazuh-indexer.service - wazuh-indexer
     Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; disabled; preset: disabled)
     Active: active (running) since Mon 2024-11-25 15:34:30 UTC; 2min 39s ago
       Docs: https://documentation.wazuh.com
   Main PID: 5641 (java)
      Tasks: 81 (limit: 24731)
     Memory: 1.3G
        CPU: 28.488s
     CGroup: /system.slice/wazuh-indexer.service
             └─5641 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.security.manager=allow -Djava.locale.providers=SPI,COMPAT -Xms1g -Xmx1g -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Djava.io.tmpdir=/var/lib/wazuh-indexer/tmp -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/wazuh-indexer -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log "-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m" -Djava.security.manager=allow -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory -Dclk.tck=100 -Djdk.attach.allowAttachSelf=true -Djava.security.policy=file:///etc/wazuh-indexer/opensearch-performance-analyzer/opensearch_security.policy --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED -XX:MaxDirectMemorySize=536870912 -Dopensearch.path.home=/usr/share/wazuh-indexer -Dopensearch.path.conf=/etc/wazuh-indexer -Dopensearch.distribution.type=rpm -Dopensearch.bundled_jdk=true -cp "/usr/share/wazuh-indexer/lib/*" org.opensearch.bootstrap.OpenSearch -p /run/wazuh-indexer/wazuh-indexer.pid --quiet

Nov 25 15:34:23 node-1 systemd-entrypoint[5641]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.16.0.jar)
Nov 25 15:34:23 node-1 systemd-entrypoint[5641]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Nov 25 15:34:23 node-1 systemd-entrypoint[5641]: WARNING: System::setSecurityManager will be removed in a future release
Nov 25 15:34:23 node-1 systemd-entrypoint[5641]: Nov 25, 2024 3:34:23 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
Nov 25 15:34:23 node-1 systemd-entrypoint[5641]: WARNING: COMPAT locale provider will be removed in a future release
Nov 25 15:34:24 node-1 systemd-entrypoint[5641]: WARNING: A terminally deprecated method in java.lang.System has been called
Nov 25 15:34:24 node-1 systemd-entrypoint[5641]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.16.0.jar)
Nov 25 15:34:24 node-1 systemd-entrypoint[5641]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Nov 25 15:34:24 node-1 systemd-entrypoint[5641]: WARNING: System::setSecurityManager will be removed in a future release
Nov 25 15:34:30 node-1 systemd[1]: Started wazuh-indexer.


[root@node-1 scripts]# curl -sku admin:admin https://localhost:9200
{
  "name" : "node-1",
  "cluster_name" : "wazuh-cluster",
  "cluster_uuid" : "LPWidYpFS4uM5fNvKZ1X0w",
  "version" : {
    "number" : "7.10.2",
    "build_type" : "rpm",
    "build_hash" : "c99647645cd8e8871653c65d6c451a32711ded50",
    "build_date" : "2024-11-11T11:37:51.521499Z",
    "build_snapshot" : false,
    "lucene_version" : "9.11.1",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

Copy link
Member

@f-galland f-galland left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@AlexRuiz7 AlexRuiz7 merged commit 999f5ab into 4.10.2 Nov 25, 2024
11 checks passed
@AlexRuiz7 AlexRuiz7 deleted the ci/501-fix-error-with-varlog-noexec branch November 25, 2024 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] 4.9.1 initial startup of indexer fails due to noexec filesystems from STIG compliance + workaround
3 participants