Perfsonar Pscheduler Service not running after updating External Certificate

[smallboy69]

hi Perfsonar Experts

hope this email finds you well

am a new user to perfsonar and did a toolkit installation

with all services running but once i updated the External Certificate Pscheduler is not running as above..

the database goes offline

looks like i need to update the API setting

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

with below error in pscheduler log

Failed to connect to the database.
Jun 13 15:05:23 fjperfsonarint pscheduler-api[1543] ERROR Exception: psycopg2.OperationalError: could not connect to server: No such file or directory
Jun 13 15:05:23 fjperfsonarint pscheduler-api[1543] ERROR Is the server running locally and accepting
Jun 13 15:05:23 fjperfsonarint pscheduler-api[1543] ERROR connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Jun 13 15:05:23 fjperfsonarint pscheduler-api[1543] ERROR Traceback (most recent call last):
Jun 13 15:05:23 fjperfsonarint pscheduler-api[1543] ERROR File "/usr/lib/python3/dist-packages/flask/app.py", line 1949, in full_dispatch_request
Jun 13 15:05:23 fjperfsonarint pscheduler-api[1543] ERROR rv = self.dispatch_request()
Jun 13 15:05:23 fjperfsonarint pscheduler-api[1543] ERROR File "/usr/lib/python3/dist-packages/flask/app.py", line 1935, in dispatch_request
Jun 13 15:05:23 fjperfsonarint pscheduler-api[1543] ERROR return self.view_functionsrule.endpoint
Jun 13 15:05:23 fjperfsonarint pscheduler-api[1543] ERROR File "/usr/share/pscheduler/api-server/pschedulerapiserver/tests.py", line 22, in tests
Jun 13 15:05:23 fjperfsonarint pscheduler-api[1543] ERROR return json_query("SELECT json FROM test"
Jun 13 15:05:23 fjperfsonarint pscheduler-api[1543] ERROR File "/usr/share/pscheduler/api-server/pschedulerapiserver/json.py", line 54, in json_query
Jun 13 15:05:23 fjperfsonarint pscheduler-api[1543] ERROR cursor = dbcursor_query(query, query_args)
Jun 13 15:05:23 fjperfsonarint pscheduler-api[1543] ERROR File "/usr/share/pscheduler/api-server/pschedulerapiserver/dbcursor.py", line 94, in dbcursor_query
Jun 13 15:05:23 fjperfsonarint pscheduler-api[1543] ERROR cursor = dbcursor()
Jun 13 15:05:23 fjperfsonarint pscheduler-api[1543] ERROR File "/usr/share/pscheduler/api-server/pschedulerapiserver/dbcursor.py", line 77, in dbcursor
Jun 13 15:05:23 fjperfsonarint pscheduler-api[1543] ERROR return getattr(module.threadlocal, "cursor", DBCursor()).cursor()
Jun 13 15:05:23 fjperfsonarint pscheduler-api[1543] ERROR File "/usr/share/pscheduler/api-server/pschedulerapiserver/dbcursor.py", line 64, in cursor
Jun 13 15:05:23 fjperfsonarint pscheduler-api[1543] ERROR raise reason;
Jun 13 15:05:23 fjperfsonarint pscheduler-api[1543] ERROR File "/usr/share/pscheduler/api-server/pschedulerapiserver/dbcursor.py", line 54, in cursor
Jun 13 15:05:23 fjperfsonarint pscheduler-api[1543] ERROR self.db = pscheduler.pg_connection(module.dsn, name="api")
Jun 13 15:05:23 fjperfsonarint pscheduler-api[1543] ERROR File "/usr/lib/python3/dist-packages/pscheduler/db.py", line 43, in pg_connection
Jun 13 15:05:23 fjperfsonarint pscheduler-api[1543] ERROR pg = psycopg2.connect(dsn)
Jun 13 15:05:23 fjperfsonarint pscheduler-api[1543] ERROR File "/usr/lib/python3/dist-packages/psycopg2/init.py", line 126, in connect
Jun 13 15:05:23 fjperfsonarint pscheduler-api[1543] ERROR conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
Jun 13 15:05:23 fjperfsonarint pscheduler-api[1543] ERROR psycopg2.OperationalError: could not connect to server: No such file or directory
Jun 13 15:05:23 fjperfsonarint pscheduler-api[1543] ERROR Is the server running locally and accepting
Jun 13 15:05:23 fjperfsonarint pscheduler-api[1543] ERROR connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?

[mfeit-internet2]

The database has no connection to certificates, so I'm a little unclear about what actions you took.

You'd need to look at the PostgreSQL logs to see what caused it to stop. The location varies by OS; on EL it's in /var/lib/pgsql/<version>/pg_log (or something close to that) and on Debian/Ubuntu, it's in /var/log/postgresql.

[smallboy69]

Thanks @mfeit-internet2 for the advise

below snippet from logs PostgreSQL and Pscheduler..

is there some parameter to tweak to fix this issue,
have chmod 640 to 600 for private key but its still offline

now pscheduler logs suggesting pschedule load..

Thanks

[mfeit-internet2]

now pscheduler logs suggesting pschedule load..

I need more-specific information than that. If there are error messages, please include them.

[smallboy69]

Thanks @mfeit-internet2

can please advise which other logs that would provide the most insight in identifying the Root Cause of the Pscheduler offline

please note below pscheduler log same repeated messages looping over and over again

pscheduler_1306.log

[smallboy69]

syslog.log

@mfeit-internet2 syslog as of this morning

[mfeit-internet2]

The root of the problem is PostgreSQL not starting up. Without that, none of pScheduler will work.

My recommendation at this point would be to roll back all changes made to install the certificate, reboot and see where things stand. If pScheduler starts to work, the problem is with the certificate you installed and, not being something we ship, is going to be up to you to diagnose.

[smallboy69]

Thank you @mfeit-internet2 for the advise, i tried reversing it but was not successfully so i reloaded OS with the latest Perfsonar 5.1

when i run the pscheduler troubleshoot at the command prompt it is successful but when i try to log into Grafana for administrator using the default credentials admin/admin according to documentation it does not work

https://docs.perfsonar.net/grafana_cookbook.html

appreciate if you can advise is there a way to reset the admin password via the backend for grafana webinterface

thanks

[smallboy69]

sudo grafana-cli admin reset-admin-password admin

Thanks @mfeit-internet2 found it online...

now to understand grafana...

learning never ends...

Regards

[mfeit-internet2]

Ah, yes, Grafana is a different can of worms.

[smallboy69]

Thank you @mfeit-internet2 for all your asssistance

God Bless