Merge branch 'master' into 5.1.1

.github/workflows/project_auto_add_issue.yml

@@ -11,7 +11,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       # NOTE: "uses" cannot be a variable so name and version hard-coded here
-      - uses: actions/add-to-project@v0.5.0
+      - uses: actions/add-to-project@v1.0.1
         with:
           project-url: ${{ vars.PROJECT_PS_URL }}
-          github-token: ${{ secrets.PAT_PROJECT_PS_AUTO_ADD }}
+          github-token: ${{ secrets.PAT_PROJECT_PS_AUTO_ADD }}

FAQ.rst

@@ -100,7 +100,6 @@ Q: Why can't my Debian/Ubuntu host find ping?
 
 **A:** Run ``apt reinstall iputils-ping`` to fix the issue. This was caused by a bug in the paris-traceroute package that installed a non-standard version of ping that required sudo. This was removed in perfSONAR 5.0.5 which left some systems without a ping command.
 
-
 Tool Questions
 ----------------
 

conf.py

@@ -55,7 +55,7 @@
 # The short X.Y version.
 version = '5.1.0'
 # The full version, including alpha/beta/rc tags.
-release = '5.1.0 Beta 1'
+release = '5.1.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

cookbook_central_archive.rst

@@ -2,8 +2,6 @@
 Central Archive with Grafana Cookbook
 *********************************************************************************************************************
 
-.. note:: This page uses the staging repo and the URL https://raw.githubusercontent.com/perfsonar/psconfig/5.1.0/psconfig/perfsonar-psconfig/doc/skeleton.json, which will change for final release
-
 This guide walks-through a standard perfSONAR setup where there are multiple perfSONAR Testpoint hosts writing measurements to central archive from which results can be displayed. 
 
 This guide will assume a setup that looks like the following:
@@ -34,7 +32,7 @@ In this step we'll setup the *archive host* store measurements. Specifically we'
 
 3. Run the personar install script to setup the package repositories and install the perfSONAR archive. The command will automatically detect the operating system and install the correct packages::
 
-    curl -s https://downloads.perfsonar.net/install | sh -s - --repo staging archive
+    curl -s https://downloads.perfsonar.net/install | sh -s - archive
 
 4. Let's quickly verify the archive is running with the *psarchive troubleshoot* utility. It will check that components such as OpenSearch and Logstash are running as well as verify authentication credentials. It can also check if the archive has data, but since we have not yet configured our measurement hosts we will skip that check with the `--skip-opensearch-data` option. Run the command as follows and if everything is marked as *OK* then proceed, otherwise follow the instructions in the command output to debug::
 
@@ -130,7 +128,7 @@ In this step we'll create a file that defines the measurements we want all the t
 
 2. Download the "skeleton" file which we will use as the starting point for our pSConfig template::
 
-    curl -o psconfig.json https://raw.githubusercontent.com/perfsonar/psconfig/5.1.0/psconfig/perfsonar-psconfig/doc/skeleton.json
+    curl -o psconfig.json https://raw.githubusercontent.com/perfsonar/psconfig/master/psconfig/perfsonar-psconfig/doc/skeleton.json
 
 3. The first thing we'll add to the pSConfig template is instructions that tell testpoints how to send results to the archive. There is a helper script that helps us generate this definition. You will pass it a `-n` that tells it the public address of your archive. In our example it is *archive.local* but change that to the address of your archive host. The command for this example looks like the following::
 
@@ -323,7 +321,7 @@ We will now logout of the archive host and login to a testpoint host. The steps
 
 3. Now we'll run the installation script again, but specify the tespoint package::
 
-    curl -s https://downloads.perfsonar.net/install | sh -s - --repo staging testpoint
+    curl -s https://downloads.perfsonar.net/install | sh -s - testpoint
  
 4. Let's verify our install worked by running the `pscheduler troubleshoot` command::
 

install_debian.rst

@@ -2,7 +2,7 @@
 Bundle Installation on Debian
 ***********************************
 
-perfSONAR combines various sets of measurement tools and services. We provide the whole perfSONAR Toolkit as Debian packages for six different architectures.  This should enable you to deploy a full perfSONAR node on one of the following distributions:
+perfSONAR combines various sets of measurement tools and services. We provide the whole perfSONAR Toolkit as Debian packages for four different architectures.  This should enable you to deploy a full perfSONAR node on one of the following distributions:
 
 * Debian 11 Bullseye
 * Debian 12 Bookworm
@@ -24,7 +24,7 @@ System Requirements
   * ARM64 (arm64)
   * PPC64 (ppc64el)
 
-* **Operating System:**  Any system running a Debian 10, Ubuntu 18 or 20 server OS is supported.  Other Debian flavours derived from Debian 10, Ubuntu 18 or 20 might work too but are not officially supported.
+* **Operating System:**  Any system running a Debian 11 or 12, Ubuntu 20 or 22 server OS is supported.  Other Debian flavours derived from Debian 11 or 12, Ubuntu 20 or 22 might work too but are not officially supported.
 
 * See :doc:`install_hardware` for hardware requirements and more.
 
@@ -65,8 +65,8 @@ Step 1: Configure APT
 ---------------------
 All you need to do is to configure the perfSONAR Debian repository source, along with our signing key, on your Debian/Ubuntu machine. **You will need to follow the steps below as privileged user**::
 
-    curl -o /etc/apt/sources.list.d/perfsonar-release.list http://downloads.perfsonar.net/debian/perfsonar-release.list
-    curl -s -o /etc/apt/trusted.gpg.d/perfsonar-release.gpg.asc http://downloads.perfsonar.net/debian/perfsonar-release.gpg.key
+    curl -o /etc/apt/sources.list.d/perfsonar-release.list https://downloads.perfsonar.net/debian/perfsonar-release.list
+    curl -s -o /etc/apt/trusted.gpg.d/perfsonar-release.gpg.asc https://downloads.perfsonar.net/debian/perfsonar-release.gpg.key
    
 * **Ubuntu only**. Additionnaly, if you're running a stripped down Ubuntu installation, you might need to enable the universe repository.  This is done with the following command::
 
@@ -252,35 +252,35 @@ Configuring perfSONAR through the web interface
 ------------------------------------------------
 After installing the perfsonar-toolkit bundle, you can refer to the general perfSONAR configuration from :doc:`install_config_first_time`.
 
-Upgrading from 4.4.x
+Upgrading from 5.0.x
 ====================
-If you had installed a perfSONAR 4.4.x bundle and you now want to upgrade to perfSONAR 5.0, you'll have to follow the instructions here below.  This will only work for Debian and Ubuntu versions supported on both releases, i.e. Debian 10 and Ubuntu 18.
+If you had installed a perfSONAR 5.0.x bundle and you now want to upgrade to perfSONAR 5.1, you'll have to follow the instructions here below.  This will only work for the OS version supported on both releases, i.e. Ubuntu 20.
 
 Upgrade the perfSONAR installation
 ----------------------------------
-If you have auto-update enabled and already using the ``perfsonar-release.list`` APT source file (as was instructed when installing 4.4), you should receive the 5.0 upgrade automatically. However, because of some dependency changes and repository name change, the full upgrade need to be done manually.
+If you have auto-update enabled and already using the ``perfsonar-release.list`` APT source file (as was instructed when installing 5.0), you should receive the 5.1 upgrade automatically. However, because of some dependency changes and repository name change, the full upgrade need to be done manually.
 
 If this is the case or you don't use the auto-update feature, to upgrade your perfsonar installation, you need to run::
 
    apt update
    apt dist-upgrade
 
-The measurements and the measurement archives that you already have defined in your 4.4.x installation will be migrated to the 5.0 toolkit automatically.
+The measurements and the measurement archives that you already have defined in your 5.0.x installation will be migrated to the 5.1 toolkit automatically.
 
-.. note:: You might see ``apt`` issuing a warning about conflicting distribution with a message like ``W: Conflicting distribution: http://downloads.perfsonar.net/debian perfsonar-release InRelease (expected perfsonar-4.4 but got perfsonar-5.0)``  This is expected and can be ignored because you indeed are upgrading from 4.4 to 5.0.
+.. note:: You might see ``apt`` issuing a warning about conflicting distribution with a message like ``W: Conflicting distribution: https://downloads.perfsonar.net/debian perfsonar-release InRelease (expected perfsonar-5.0 but got perfsonar-5.1)``  This is expected and can be ignored because you indeed are upgrading from 5.0 to 5.1.
 
 Upgrade to another bundle
 -------------------------
 If you want to move from the `perfsonar-testpoint` bundle to another bundle that we provide for Debian, you can do so by following the instructions above from :ref:`install_debian_step2`.
 
-Upgrade from Ubuntu 18 to Ubuntu 20 
+Upgrade from Ubuntu 20 to Ubuntu 22
 -----------------------------------
-If you have a perfSONAR host running Ubuntu 18 and you want to upgrade it to 20, we recommend you to follow the following steps:
+If you have a perfSONAR host running Ubuntu 20 and you want to upgrade it to 22, we recommend you to follow the following steps:
 
-* Upgrade Ubuntu 18 to Ubuntu 20 (following official instructions, here are `Focal Upgrades notes <https://help.ubuntu.com/community/FocalUpgrades>`_)
+* Upgrade Ubuntu 20 to Ubuntu 22 (following official instructions, here are `Jammy Upgrades notes <https://help.ubuntu.com/community/JammyUpgrades>`_)
 * Reboot your system unless already done in previous step.
 * Run ``apt-get update; apt-get dist-upgrade`` to get the latest version of perfSONAR.
 * Reboot your system one last time.
 
-Alternatively, do a fresh installation of perfSONAR on Ubuntu 20.
+Alternatively, do a fresh installation of perfSONAR on Ubuntu 22.
 

install_el.rst

@@ -33,7 +33,7 @@ perfSONAR provides a script that will automatically perform the first two steps
 
     curl -s https://downloads.perfsonar.net/install | sh -s - archive
 
-If you would prefer to perform these steps manually see :ref:`_install_el_step1` and :ref:`_install_el_step2`.
+If you would prefer to perform these steps manually see :ref:`install_el_step1` and :ref:`install_el_step2`.
 
 .. _install_el_installation:
 

install_rcs.rst

@@ -67,14 +67,14 @@ Testing Debian Installation (Manual)
 
 The beta packages for Debian can be found in the source list below:
 
-* http://downloads.perfsonar.net/debian/perfsonar-minor-staging.list
+* https://downloads.perfsonar.net/debian/perfsonar-minor-staging.list
 
 You may install this source list as follows::
     
-    curl -o /etc/apt/sources.list.d/perfsonar-minor-staging.list http://downloads.perfsonar.net/debian/perfsonar-minor-staging.list
-    curl -s -o /etc/apt/trusted.gpg.d/perfsonar-staging.gpg.asc http://downloads.perfsonar.net/debian/perfsonar-staging.gpg.key
+    curl -o /etc/apt/sources.list.d/perfsonar-minor-staging.list https://downloads.perfsonar.net/debian/perfsonar-minor-staging.list
+    curl -s -o /etc/apt/trusted.gpg.d/perfsonar-staging.gpg.asc https://downloads.perfsonar.net/debian/perfsonar-staging.gpg.key
 
-These Debian packages should work on Debian 10, Ubuntu 18 and 20.  The perfsonar-archive bundle should also work on Debian 11 and Ubuntu 22.
+These Debian packages should work on Debian 11 and 12, Ubuntu 20 and 22.
 
 On Ubuntu you need to make you have the **universe** repository enabled, this is done with the command ``add-apt-repository universe``
 

install_small_node_details.rst

@@ -24,15 +24,14 @@ Please note that the perfSONAR team is not formally endorsing any particular pro
 System Requirements
 ===================
 
-ARM-based devices seem to work best running Ubuntu, while the slightly larger, $200-class nodes will usually work with EL, Debian or Ubuntu. The :doc:`perfsonar-testpoint <install_options>` bundle works on CentOS 7 and Ubuntu 18/20, and Debian 10. We provide compatible Debian packages for 6 different hardware architectures:
+ARM-based devices seem to work best running Ubuntu, while the slightly larger, $200-class nodes will usually work with EL, Debian or Ubuntu. The :doc:`perfsonar-testpoint <install_options>` bundle works on CentOS 7 and Ubuntu 20/22, and Debian 11/12. We provide compatible Debian packages for 4 different hardware architectures:
 
   * 64-bit (amd64)
   * ARMv7 and up (armhf) - not recommended for the full testpoint bundle
   * ARM 64-bit (arm64)
   * PPC 64-bit (ppc64el)
 
 
-
 Installation Instructions
 =========================
 
@@ -42,7 +41,7 @@ The :doc:`perfsonar-testpoint <install_options>` bundle can be used to install e
 - :doc:`install_debian`
 
 Certain devices like the Liva use an EMMC drive that is only supported in the *deskop* version of Ubuntu. 
-If the standard server ISO installation doe not recognize the drive, it may be worth attempting installation using the desktop ISO such as `Ubuntu 20.04.6 Desktop <https://releases.ubuntu.com/20.04.6/ubuntu-20.04.6-desktop-amd64.iso>`_.
+If the standard server ISO installation doe not recognize the drive, it may be worth attempting installation using the desktop ISO such as `Ubuntu 20.04.6 Desktop <https://releases.ubuntu.com/20.04.6/ubuntu-20.04.6-desktop-amd64.iso>`_ or newer.
 
 Support
 =======

pscheduler_client_schedule.rst

@@ -11,6 +11,8 @@ It is often desirable to view the pScheduler schedule to plan new tests, debug a
     * ``pscheduler schedule`` - This command outputs the schedule in the requested timeframe as text. See :ref:`pscheduler_client_schedule-schedule`.
     * ``pscheduler plot-schedule`` - This command creates a PNG image file that provides a box plot of the schedule. See :ref:`pscheduler_client_schedule-plot_schedule`.
     * ``pscheduler monitor`` - This command provides top-like output showing near real-time state of runs in the database. See :ref:`pscheduler_client_schedule-monitor`.
+
+If you installed the ``perfsonar-toolkit`` bundle you can can visualize the pScheduler schedule directly in the Toolkit GUI. See :ref:`pscheduler_client_schedule-plot_schedule_toolkit`.
 
 .. _pscheduler_client_schedule-basics:
 
@@ -191,3 +193,18 @@ If you would like to monitor a pScheduler server on remote host you can add the
 
 If you are curious about any additional options or details, you can also run ``pscheduler monitor --help`` to get more information about this command. 
 
+.. _pscheduler_client_schedule-plot_schedule_toolkit:
+
+Visualizing the Schedule with Plot Schedule Graph in Toolkit
+------------------------------------------------------------
+With Toolkit installation, you can view the pScheduler schedule with **Plot Schedule Graph** menu option that provides easy access to visualize the hosts's schedule in the Toolkit GUI.
+
+#. Open *http://<hostname>* in a web browser where *<hostname>* is the name or address of your host. 
+#. Click on the **Configuration** button in the right-upper corner of the main page and login as the web administrator user.
+#. On the page that loads go to **Tests** tab and click on **Plot-Schedule Graph** under **Resources** menu. The page that loads can be used to visualize the schedule in your Toolkit host.
+
+    .. image:: images/pscheduler_client_schedule-plot-toolkit-menu.png
+#. You will be prompted to select from drop-down lists a time period in the past where you wish the graph to start, and a time period in the future where the graph is to stop. These correspond to ``delta`` or ``start`` and ``end`` options of the ``pscheduler plot-schedule`` command. You can select only one of the periods or both.
+#. Click the **Process Graph** button to generate and display the schedule as an image file. 
+
+    .. image:: images/pscheduler_client_schedule-plot-toolkit-graph.png

psconfig_grafana_agent.rst

@@ -287,6 +287,67 @@ Since technically `super_special_test` would match both because its both a throu
 
 Beyond changing thredholds, the default displays may be fine in most cases but you can add more entries to the displays section for different test types or have it grab different stats. Feel free to experiment with the different fields to learn more. 
 
+.. _psconfig_grafana_agent-org_dashboards:
+
+Organizing Grids and Dashboards
+-----------------------------------------------
+Grouping grids onto the same dashboard can be controlled through your pSConfig JSON template under the `tasks` section. There are two values that need to be set under the task `reference` section:
+
+    1. Set the `display-task-name` to get a human readable title for the task
+    2. Set the `display-task-group` to a list of dashboard names in which you want the grid included (NOTE: This must be an array even if it is just one element)
+
+See the snippet below that includes a grid displayed as *Example Throughput* in a dashboard named *Example Dashboard*::
+
+    "example_throughput" : {
+        "reference": {
+            "display-set-source": "{% jq .addresses[0]._meta.\"display-set\" %}",
+            "display-set-dest": "{% jq .addresses[1]._meta.\"display-set\" %}",
+            "display-task-name": "Example Throughput",
+            "display-task-group": ["Example Dashboard"]
+        },
+        "group" : "example_thr_group",
+        "schedule" : "schedule_0"
+        "test" : "example_thr_test"
+    }
+
+.. _psconfig_grafana_agent-multi_xfaces:
+
+Graphing Multiple Interfaces on Same Dashboard
+-----------------------------------------------
+A common use case is that a single measurement host may have multiple interfaces, such as one dedicated to throughput and another dedicated to latency. The address for each interface is defined as separate `address` objects in your pSConfig file. By default, when the Grafana dashboards are generated these addresses will not be on the same graphs since the agent does not know they belong together. If you want this data to be on the same graph page you need to set some parameters that indicate the addresses should be grouped together. This requires two changes to your pSConfig template: 
+
+    1. Update the `address` definitions with a `_meta` tag of `display-set`.
+    2. Update the `task` definitions to include a `reference` section with `display-set-source` and `display-set-dest` that includes the `dispay-set` value from the source and destination.
+
+Let's look at these steps in detail. First let's say we have a host two interfaces: *example-tp.perfsonar.net* and *example-lat.perfsonar.net* that we want shown on the same graphs. When we define the addresses, we create a `display-set` called `example.perfsonar.net` (this can be any string as long as it is consistent between the two addresses) in our pSConfig template JSON. For example::
+
+    "example-tp.perfsonar.net" : {
+      "_meta" : {
+         "display-set" : "example.perfsonar.net"
+      },
+      "address" : "example-tp.perfsonar.net"
+   },
+   "example-lat.perfsonar.net" : {
+        "_meta" : {
+           "display-set" : "example.perfsonar.net"
+        },
+        "address" : "example-lat.perfsonar.net"
+    }
+
+The next step is to make sure these values are passed to pScheduler when the tasks are created. This gives the agent enough information to build queries that can include results using both addresses on the graph. We do this by setting a `display-set-source` and `display-set-dest` in the `reference` section of the task. These use `jq` to dynamically set the values based on the source and destination. The task definition should look like the following and you can copy the definition of `display-set-source` and `display-set-dest` exactly to your template::
+
+     "example_throughput" : {
+        "reference": {
+            "display-set-source": "{% jq .addresses[0]._meta.\"display-set\" %}",
+            "display-set-dest": "{% jq .addresses[1]._meta.\"display-set\" %}",
+            "display-task-name": "Example Throughput",
+            "display-task-group": ["Example Dashboard"]
+        },
+        "group" : "example_thr_group",
+        "schedule" : "schedule_0"
+        "test" : "example_thr_test"
+    }
+
 .. _psconfig_grafana_agent-advanced:
 
 Advanced Configuration