Added logserver and fixed Glidein log token and URL propagation

Added php and python example for a server receiving Glidein logs

Fixed JWT generation and GLIDEIN_LOG_RECIPIENTS_FACTORY param for Factory logserver, improved also code and docstrings, and added documentation for logging and log server

Added the glideinwms-logserver RPM

.reuse/dep5

@@ -13,6 +13,10 @@ Files: .codecov.yml .coveragerc .editorconfig .gitattributes .gitignore .gitmodu
 Copyright: 2009 Fermi Research Alliance, LLC
 License: Apache-2.0
 
+Files: logserver/logging_config.json
+Copyright: 2009 Fermi Research Alliance, LLC
+License: Apache-2.0
+
 Files: .github/ISSUE_TEMPLATE/* bigfiles/* etc/* config/* creation/templates/*.service creation/templates/*.cron creation/templates/*.timer doc/tags*txt lib/logging.conf test/bats/fixtures/* unittests/*.fixture unittests/fixtures/*
 Copyright: 2009 Fermi Research Alliance, LLC
 License: Apache-2.0

CHANGELOG.md

@@ -11,6 +11,7 @@ Changes since the last release
 
 -   item one of the list
 -   item N
+-   Added custom log server example (glideinwms-logging) (Issue #398, PR #467)
 
 ### Changed defaults / behaviours
 
@@ -21,6 +22,7 @@ Changes since the last release
 ### Bug Fixes
 
 -   Fixed early truncation in log files configuration and inconsistent documentation (Issue #464, PR #462, PR #463)
+-   Fixed JWT logging credentials not transferred to the Glidein. This includes removal of DictFile.append() and use of add_environment() for JWT tokens (Issue #398, PR #467)
 
 ### Testing / Development
 

build/packaging/rpm/glideinwms.spec

@@ -33,6 +33,8 @@
 %define factory_web_base %{_localstatedir}/lib/gwms-factory/web-base
 %define factory_dir %{_localstatedir}/lib/gwms-factory/work-dir
 %define condor_dir %{_localstatedir}/lib/gwms-factory/condor
+%define logserver_dir %{_localstatedir}/lib/gwms-logserver
+%define logserver_web_dir %{_localstatedir}/lib/gwms-logserver/web-area
 %define systemddir %{_prefix}/lib/systemd/system
 
 Name:           glideinwms
@@ -58,6 +60,7 @@ Source8:        gwms-frontend.sysconfig
 Source9:        gwms-factory.sysconfig
 Source11:       creation/templates/frontend_startup_sl7
 Source12:       creation/templates/factory_startup_sl7
+Source13:       gwms-logserver.conf.httpd
 
 BuildRequires:  python3
 BuildRequires:  python3-devel
@@ -153,6 +156,7 @@ This subpackage includes the Glidein components for the Frontend.
 %package vofrontend-httpd
 Summary:        The Apache http configuration for GWMS Frontend.
 Requires: httpd
+Requires: mod_ssl
 %description vofrontend-httpd
 This subpackage includes the minimal configuration to start Apache to
 serve the Frontend files to the pilot and the monitoring pages.
@@ -279,6 +283,7 @@ Factory. Created to separate out the httpd server.
 %package factory-httpd
 Summary:        The Apache httpd configuration for the GWMS Factory
 Requires: httpd
+Requires: mod_ssl
 %description factory-httpd
 This subpackage includes the minimal configuration to start Apache to
 serve the Factory files to the pilot and the monitoring pages.
@@ -293,6 +298,17 @@ This is a package including condor_config for a full one-node
 install of wmscollector + wms factory
 
 
+%package logserver
+Summary:        The Glidein Log server and its Apache http configuration.
+Requires: httpd
+Requires: mod_ssl
+Requires: php
+Requires: php-fpm
+%description logserver
+This subpackage includes an example of the files and Apache configuration
+to implement a simple server to receive Glidein logs.
+
+
 %prep
 %setup -q -n glideinwms
 # Apply the patches here if any
@@ -469,7 +485,6 @@ install -d $RPM_BUILD_ROOT%{factory_web_dir}/monitor/group_main/total
 install -m 644 creation/web_base/nodes.blacklist $RPM_BUILD_ROOT%{web_dir}/stage/nodes.blacklist
 install -m 644 creation/web_base/nodes.blacklist $RPM_BUILD_ROOT%{web_dir}/stage/group_main/nodes.blacklist
 
-
 # Install the logs
 install -d $RPM_BUILD_ROOT%{_localstatedir}/log/gwms-frontend/frontend
 install -d $RPM_BUILD_ROOT%{_localstatedir}/log/gwms-frontend/group_main
@@ -589,10 +604,11 @@ install -m 0755 install/glidecondor_createSecCol $RPM_BUILD_ROOT%{_sbindir}/glid
 install -m 0644 etc/checksum.frontend $RPM_BUILD_ROOT%{frontend_dir}/checksum.frontend
 install -m 0644 etc/checksum.factory $RPM_BUILD_ROOT%{factory_dir}/checksum.factory
 
-#Install web area conf
+# Install web area conf
 install -d $RPM_BUILD_ROOT/%{_sysconfdir}/httpd/conf.d
 install -m 0644 %{SOURCE3} $RPM_BUILD_ROOT/%{_sysconfdir}/httpd/conf.d/gwms-frontend.conf
 install -m 0644 %{SOURCE5} $RPM_BUILD_ROOT/%{_sysconfdir}/httpd/conf.d/gwms-factory.conf
+install -m 0644 %{SOURCE13} $RPM_BUILD_ROOT/%{_sysconfdir}/httpd/conf.d/gwms-logserver.conf
 
 install -d $RPM_BUILD_ROOT%{web_base}/../creation
 install -d $RPM_BUILD_ROOT%{web_base}/../creation/templates
@@ -603,6 +619,15 @@ install -d $RPM_BUILD_ROOT%{factory_web_base}/../creation/templates
 install -m 0644 creation/templates/factory_initd_startup_template $RPM_BUILD_ROOT%{factory_web_base}/../creation/templates/
 install -m 0644 creation/templates/frontend_initd_startup_template $RPM_BUILD_ROOT%{web_base}/../creation/templates/
 
+# Install the logserver
+install -d $RPM_BUILD_ROOT%{logserver_dir}
+install -d $RPM_BUILD_ROOT%{logserver_web_dir}
+install -d $RPM_BUILD_ROOT%{logserver_web_dir}/uploads
+install -d $RPM_BUILD_ROOT%{logserver_web_dir}/uploads_unauthorized
+cp logserver/put.php $RPM_BUILD_ROOT%{logserver_web_dir}/put.php
+cp logserver/jwt.php $RPM_BUILD_ROOT%{logserver_web_dir}/jwt.php
+cp logserver/logging_config.json $RPM_BUILD_ROOT%{logserver_web_dir}/logging_config.json
+
 
 %post usercollector
 /sbin/service condor condrestart > /dev/null 2>&1 || true
@@ -680,6 +705,10 @@ systemctl daemon-reload
 # Protecting from failure in case it is not running/installed
 /sbin/service httpd reload > /dev/null 2>&1 || true
 
+%post logserver
+# Protecting from failure in case it is not running/installed
+/sbin/service httpd reload > /dev/null 2>&1 || true
+
 %pre vofrontend-core
 # Add the "frontend" user and group if they do not exist
 getent group frontend >/dev/null || groupadd -r frontend
@@ -754,6 +783,10 @@ fi
 # Protecting from failure in case it is not running/installed
 /sbin/service httpd reload > /dev/null 2>&1 || true
 
+%postun logserver
+# Protecting from failure in case it is not running/installed
+/sbin/service httpd reload > /dev/null 2>&1 || true
+
 %postun factory-core
 # Protecting from failure in case it is not running/installed
 /sbin/service condor condrestart > /dev/null 2>&1 || true
@@ -1052,6 +1085,18 @@ rm -rf $RPM_BUILD_ROOT
 %config(noreplace) %{_sysconfdir}/condor/certs/condor_mapfile
 #%config(noreplace) %{_sysconfdir}/condor/scripts/frontend_condortoken
 
+%files logserver
+%defattr(-,root,root,-)
+%config(noreplace) %{_sysconfdir}/httpd/conf.d/gwms-logserver.conf
+%attr(-, root, apache) %{logserver_web_dir}
+%attr(-, root, apache) %{logserver_web_dir}/logging_config.json
+%attr(-, apache, apache) %{logserver_web_dir}/uploads
+%attr(-, apache, apache) %{logserver_web_dir}/uploads_unauthorized
+%attr(-, gfactory, gfactory) %{factory_web_base}
+%attr(-, gfactory, gfactory) %{factory_web_base}/../creation
+%attr(-, root, root) %{logserver_dir}
+MMDB
+
 %changelog
 * Mon Nov 25 2024 Marco Mambelli <[email protected]> - 3.10.8
 - Glideinwms v3.10.8

build/packaging/rpm/gwms-logging.conf.httpd

@@ -0,0 +1,63 @@
+# SPDX-FileCopyrightText: 2009 Fermi Research Alliance, LLC
+# SPDX-License-Identifier: Apache-2.0
+
+# This is the httpd conf file
+# GlideinWMS Logging Server web configuration
+
+Alias /logserver /var/lib/gwms-logserver/web-area
+<Directory /var/lib/gwms-logserver/web-area/>
+    <IfModule mod_authz_core.c>
+        # Apache 2.4
+        Require all granted
+    </IfModule>
+    <IfModule !mod_authz_core.c>
+        # Apache 2.2
+        Order allow,deny
+        Allow from all
+    </IfModule>
+    <IfModule mod_authnz_jwt>
+        AuthType jwt
+        AuthName "private area"
+        # Require valid-user
+    </IfModule>
+    # Options +Indexes
+    AllowMethods PUT GET POST OPTIONS
+    # Require method PUT GET POST OPTIONS
+    #<Limit GET POST PUT OPTIONS DELETE PATCH HEAD>
+    #  Require all granted
+    #</Limit>
+</Directory>
+
+# Use the following version if you are using mod_jwt (authnz_mod_jwt)
+# This will allow a simpler PUT uploader because it will not need to validate the JWT
+
+#AuthJWTSignatureAlgorithm HS256
+#AuthJWTSignatureSharedSecret VG9rZVNlY3JldEtleQo=
+#AuthJWTIss factory-workspace.glideinwms.org
+#
+#Alias /logging /var/lib/gwms-logging/logging
+#<Directory /var/lib/gwms-logging/logging/>
+#    <IfModule mod_authz_core.c>
+#        # Apache 2.4
+#        Require all granted
+#    </IfModule>
+#    <IfModule !mod_authz_core.c>
+#        # Apache 2.2
+#        Order allow,deny
+#        Allow from all
+#    </IfModule>
+#    <IfModule mod_authnz_jwt>
+#        AuthType jwt
+#        AuthName "private area"
+#        # Require valid-user
+#    <IFModule>
+#    # Options +Indexes
+#    AllowMethods PUT GET POST OPTIONS
+#    # Require method PUT GET POST OPTIONS
+#</Directory>
+#
+# # This is needed to always forward the HTTP_AUTHORIZATION header
+# # See: https://www.codepunker.com/blog/php-a-primer-on-the-basic-authorization-header
+# RewriteEngine On
+# RewriteCond %{HTTP:Authorization} ^(.+)$
+# RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]

creation/lib/cgWCreate.py

@@ -113,11 +113,11 @@ def get_factory_log_recipients(entry):
         entry: dict-like object representing the entry configuration
 
     Returns:
-        list: list contaning the URLs of the log servers, empty if none present
+        list: list containing the URLs of the log servers, empty if none present
     """
     entr_attrs = entry.get_child_list("attrs")
     for attr in entr_attrs:
-        if attr["name"] == "LOG_RECIPIENTS_FACTORY":
+        if attr["name"] == "GLIDEIN_LOG_RECIPIENTS_FACTORY":
             return attr["value"].split()
     return []
 

creation/web_base/logging_utils.source

@@ -162,7 +162,7 @@ log_init() {
     fi
 
     # Setup tokens to authenticate with log servers
-    log_recipients=($(gconfig_get GLIDEIN_LOG_RECIPIENTS "${glidein_config}"))
+    log_recipients=($(gconfig_get GLIDEIN_LOG_RECIPIENTS_FACTORY "${glidein_config}") $(gconfig_get GLIDEIN_LOG_RECIPIENTS "${glidein_config}"))
     local no_send=0
     if [ "${#log_recipients[@]}" -eq 0 ]; then
         warn "log_init: no recipients configured. Logs will still be produced, but not forwarded to remote servers."
@@ -230,7 +230,7 @@ log_setup() {
     stdout_logfile=$(gconfig_get GLIDEIN_STDOUT_LOGFILE "${glidein_config}")
     stderr_logfile=$(gconfig_get GLIDEIN_STDERR_LOGFILE "${glidein_config}")
     log_logfile=$(gconfig_get GLIDEIN_LOG_LOGFILE "${glidein_config}")
-    log_recipients=($(gconfig_get GLIDEIN_LOG_RECIPIENTS "${glidein_config}"))
+    log_recipients=($(gconfig_get GLIDEIN_LOG_RECIPIENTS_FACTORY "${glidein_config}") $(gconfig_get GLIDEIN_LOG_RECIPIENTS "${glidein_config}"))
     log_no_send=$(gconfig_get GLIDEIN_LOG_NO_SEND "${glidein_config}")
     log_relative_basepath=$(gconfig_get GLIDEIN_LOG_RELATIVE_BASEPATH "${glidein_config}")
     curl_version=$(gconfig_get CURL_VERSION "${glidein_config}")

doc/factory/custom_scripts.html

@@ -107,6 +107,7 @@ <h3>Custom Scripts</h3>
           <li><a href="#glidein_config">Configuration file</a></li>
           <li><a href="#condor_vars">HTCondor vars file</a></li>
           <li><a href="#xml_output">Reporting script exit status</a></li>
+          <li><a href="#logging">Logging</a></li>
           <li><a href="#periodic">Periodic scripts</a></li>
           <li><a href="#order">Loading order</a></li>
           <li><a href="#examples">Examples</a></li>
@@ -273,7 +274,7 @@ <h2><a name="glidein_config"></a>The glidein configuration file</h2>
         </blockquote>
         <blockquote>
           # read an attributes (set by you or some other script)<br />
-          <i>myvar</i>=gconfig_get <i>myattribute</i>
+          <i>myvar</i>=$(gconfig_get <i>myattribute</i>)
         </blockquote>
       </div>
 
@@ -448,7 +449,30 @@ <h2><a name="xml_output"></a>Reporting script exit status</h2>
           code.
         </p>
       </div>
-
+      <div class="section">
+        <h2><a name="logging"></a>Logging</h2>
+        <p>
+          Standard output and standard error of all custom scripts (except the
+          periodic ones) is captured in the Glidein stdout and stderr and it is
+          transferred back to the Factory by HTCondor at the end of the Glidein.
+          Anyway this process may be insufficient: if the Glidein is killed the
+          transfer may not happen, if there are multi-Glideins all their
+          stdout/err are intermixed in the same files, and a user may desire to
+          have this output back earlier or in a different place. For all these
+          needs there is also a logging utility. It is defined in
+          <tt>logging_util.source</tt>, can be used in any custom script,
+          requires a web server to receive the logging messages, and needs to
+          set up <tt>GLIDEIN_LOG_RECIPIENTS_FACTORY</tt> as <tt>attr</tt> in the
+          Factory configuration. The Web servers at the URLs in
+          GLIDEIN_LOG_RECIPIENTS_FACTORY must be able to receive
+          JWT-authenticated PUT requests HS256-encoded with the secret set in
+          the Factory secret file
+          (<tt>/var/lib/gwms-factory/server-credentials/jwt_secret.key</tt>).
+          This secret file must be HMAC 256 compatible, e.g. a 32 bytes string.
+          The Factory will create the file at startup if it is not there or
+          empty.
+        </p>
+      </div>
       <div class="section">
         <h2><a name="periodic"></a>Periodic scripts</h2>
         <p>

doc/factory/custom_vars.html

@@ -1139,6 +1139,22 @@ <h3>
               </p>
             </td>
           </tr>
+          <tr>
+            <td><b>GLIDEIN_LOG_RECIPIENTS_FACTORY</b></td>
+            <td>String</td>
+            <td>""</td>
+
+            <td>
+              <p>
+                Space separated list of URLs to use to publish the
+                <a href="custom_scripts.html#logging">custom logs</a>.
+              </p>
+              <p>
+                A token-authenticated Web server must be running at the given
+                URL to receive the POST requests from the Glideins
+              </p>
+            </td>
+          </tr>
         </table>
       </div>
       <div class="section">