diff --git a/.github/workflows/project_auto_add_issue.yml b/.github/workflows/project_auto_add_issue.yml index 92fc2eec..c1a47c48 100644 --- a/.github/workflows/project_auto_add_issue.yml +++ b/.github/workflows/project_auto_add_issue.yml @@ -11,7 +11,7 @@ jobs: runs-on: ubuntu-latest steps: # NOTE: "uses" cannot be a variable so name and version hard-coded here - - uses: actions/add-to-project@v0.5.0 + - uses: actions/add-to-project@v1.0.1 with: project-url: ${{ vars.PROJECT_PS_URL }} - github-token: ${{ secrets.PAT_PROJECT_PS_AUTO_ADD }} \ No newline at end of file + github-token: ${{ secrets.PAT_PROJECT_PS_AUTO_ADD }} diff --git a/FAQ.rst b/FAQ.rst index 3c6a8907..8944af91 100644 --- a/FAQ.rst +++ b/FAQ.rst @@ -100,7 +100,6 @@ Q: Why can't my Debian/Ubuntu host find ping? **A:** Run ``apt reinstall iputils-ping`` to fix the issue. This was caused by a bug in the paris-traceroute package that installed a non-standard version of ping that required sudo. This was removed in perfSONAR 5.0.5 which left some systems without a ping command. - Tool Questions ---------------- diff --git a/conf.py b/conf.py index 579ede6d..7a741d2f 100644 --- a/conf.py +++ b/conf.py @@ -55,7 +55,7 @@ # The short X.Y version. version = '5.1.0' # The full version, including alpha/beta/rc tags. -release = '5.1.0 Beta 1' +release = '5.1.0' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. diff --git a/cookbook_central_archive.rst b/cookbook_central_archive.rst index 77df65d7..53411f23 100644 --- a/cookbook_central_archive.rst +++ b/cookbook_central_archive.rst @@ -2,8 +2,6 @@ Central Archive with Grafana Cookbook ********************************************************************************************************************* -.. note:: This page uses the staging repo and the URL https://raw.githubusercontent.com/perfsonar/psconfig/5.1.0/psconfig/perfsonar-psconfig/doc/skeleton.json, which will change for final release - This guide walks-through a standard perfSONAR setup where there are multiple perfSONAR Testpoint hosts writing measurements to central archive from which results can be displayed. This guide will assume a setup that looks like the following: @@ -34,7 +32,7 @@ In this step we'll setup the *archive host* store measurements. Specifically we' 3. Run the personar install script to setup the package repositories and install the perfSONAR archive. The command will automatically detect the operating system and install the correct packages:: - curl -s https://downloads.perfsonar.net/install | sh -s - --repo staging archive + curl -s https://downloads.perfsonar.net/install | sh -s - archive 4. Let's quickly verify the archive is running with the *psarchive troubleshoot* utility. It will check that components such as OpenSearch and Logstash are running as well as verify authentication credentials. It can also check if the archive has data, but since we have not yet configured our measurement hosts we will skip that check with the `--skip-opensearch-data` option. Run the command as follows and if everything is marked as *OK* then proceed, otherwise follow the instructions in the command output to debug:: @@ -130,7 +128,7 @@ In this step we'll create a file that defines the measurements we want all the t 2. Download the "skeleton" file which we will use as the starting point for our pSConfig template:: - curl -o psconfig.json https://raw.githubusercontent.com/perfsonar/psconfig/5.1.0/psconfig/perfsonar-psconfig/doc/skeleton.json + curl -o psconfig.json https://raw.githubusercontent.com/perfsonar/psconfig/master/psconfig/perfsonar-psconfig/doc/skeleton.json 3. The first thing we'll add to the pSConfig template is instructions that tell testpoints how to send results to the archive. There is a helper script that helps us generate this definition. You will pass it a `-n` that tells it the public address of your archive. In our example it is *archive.local* but change that to the address of your archive host. The command for this example looks like the following:: @@ -323,7 +321,7 @@ We will now logout of the archive host and login to a testpoint host. The steps 3. Now we'll run the installation script again, but specify the tespoint package:: - curl -s https://downloads.perfsonar.net/install | sh -s - --repo staging testpoint + curl -s https://downloads.perfsonar.net/install | sh -s - testpoint 4. Let's verify our install worked by running the `pscheduler troubleshoot` command:: diff --git a/images/pscheduler_client_schedule-plot-toolkit-graph.png b/images/pscheduler_client_schedule-plot-toolkit-graph.png new file mode 100644 index 00000000..66278b2e Binary files /dev/null and b/images/pscheduler_client_schedule-plot-toolkit-graph.png differ diff --git a/images/pscheduler_client_schedule-plot-toolkit-menu.png b/images/pscheduler_client_schedule-plot-toolkit-menu.png new file mode 100644 index 00000000..1ec4c9d3 Binary files /dev/null and b/images/pscheduler_client_schedule-plot-toolkit-menu.png differ diff --git a/install_debian.rst b/install_debian.rst index f18c2795..862aaf61 100644 --- a/install_debian.rst +++ b/install_debian.rst @@ -2,7 +2,7 @@ Bundle Installation on Debian *********************************** -perfSONAR combines various sets of measurement tools and services. We provide the whole perfSONAR Toolkit as Debian packages for six different architectures. This should enable you to deploy a full perfSONAR node on one of the following distributions: +perfSONAR combines various sets of measurement tools and services. We provide the whole perfSONAR Toolkit as Debian packages for four different architectures. This should enable you to deploy a full perfSONAR node on one of the following distributions: * Debian 11 Bullseye * Debian 12 Bookworm @@ -24,7 +24,7 @@ System Requirements * ARM64 (arm64) * PPC64 (ppc64el) -* **Operating System:** Any system running a Debian 10, Ubuntu 18 or 20 server OS is supported. Other Debian flavours derived from Debian 10, Ubuntu 18 or 20 might work too but are not officially supported. +* **Operating System:** Any system running a Debian 11 or 12, Ubuntu 20 or 22 server OS is supported. Other Debian flavours derived from Debian 11 or 12, Ubuntu 20 or 22 might work too but are not officially supported. * See :doc:`install_hardware` for hardware requirements and more. @@ -65,8 +65,8 @@ Step 1: Configure APT --------------------- All you need to do is to configure the perfSONAR Debian repository source, along with our signing key, on your Debian/Ubuntu machine. **You will need to follow the steps below as privileged user**:: - curl -o /etc/apt/sources.list.d/perfsonar-release.list http://downloads.perfsonar.net/debian/perfsonar-release.list - curl -s -o /etc/apt/trusted.gpg.d/perfsonar-release.gpg.asc http://downloads.perfsonar.net/debian/perfsonar-release.gpg.key + curl -o /etc/apt/sources.list.d/perfsonar-release.list https://downloads.perfsonar.net/debian/perfsonar-release.list + curl -s -o /etc/apt/trusted.gpg.d/perfsonar-release.gpg.asc https://downloads.perfsonar.net/debian/perfsonar-release.gpg.key * **Ubuntu only**. Additionnaly, if you're running a stripped down Ubuntu installation, you might need to enable the universe repository. This is done with the following command:: @@ -252,35 +252,35 @@ Configuring perfSONAR through the web interface ------------------------------------------------ After installing the perfsonar-toolkit bundle, you can refer to the general perfSONAR configuration from :doc:`install_config_first_time`. -Upgrading from 4.4.x +Upgrading from 5.0.x ==================== -If you had installed a perfSONAR 4.4.x bundle and you now want to upgrade to perfSONAR 5.0, you'll have to follow the instructions here below. This will only work for Debian and Ubuntu versions supported on both releases, i.e. Debian 10 and Ubuntu 18. +If you had installed a perfSONAR 5.0.x bundle and you now want to upgrade to perfSONAR 5.1, you'll have to follow the instructions here below. This will only work for the OS version supported on both releases, i.e. Ubuntu 20. Upgrade the perfSONAR installation ---------------------------------- -If you have auto-update enabled and already using the ``perfsonar-release.list`` APT source file (as was instructed when installing 4.4), you should receive the 5.0 upgrade automatically. However, because of some dependency changes and repository name change, the full upgrade need to be done manually. +If you have auto-update enabled and already using the ``perfsonar-release.list`` APT source file (as was instructed when installing 5.0), you should receive the 5.1 upgrade automatically. However, because of some dependency changes and repository name change, the full upgrade need to be done manually. If this is the case or you don't use the auto-update feature, to upgrade your perfsonar installation, you need to run:: apt update apt dist-upgrade -The measurements and the measurement archives that you already have defined in your 4.4.x installation will be migrated to the 5.0 toolkit automatically. +The measurements and the measurement archives that you already have defined in your 5.0.x installation will be migrated to the 5.1 toolkit automatically. -.. note:: You might see ``apt`` issuing a warning about conflicting distribution with a message like ``W: Conflicting distribution: http://downloads.perfsonar.net/debian perfsonar-release InRelease (expected perfsonar-4.4 but got perfsonar-5.0)`` This is expected and can be ignored because you indeed are upgrading from 4.4 to 5.0. +.. note:: You might see ``apt`` issuing a warning about conflicting distribution with a message like ``W: Conflicting distribution: https://downloads.perfsonar.net/debian perfsonar-release InRelease (expected perfsonar-5.0 but got perfsonar-5.1)`` This is expected and can be ignored because you indeed are upgrading from 5.0 to 5.1. Upgrade to another bundle ------------------------- If you want to move from the `perfsonar-testpoint` bundle to another bundle that we provide for Debian, you can do so by following the instructions above from :ref:`install_debian_step2`. -Upgrade from Ubuntu 18 to Ubuntu 20 +Upgrade from Ubuntu 20 to Ubuntu 22 ----------------------------------- -If you have a perfSONAR host running Ubuntu 18 and you want to upgrade it to 20, we recommend you to follow the following steps: +If you have a perfSONAR host running Ubuntu 20 and you want to upgrade it to 22, we recommend you to follow the following steps: -* Upgrade Ubuntu 18 to Ubuntu 20 (following official instructions, here are `Focal Upgrades notes `_) +* Upgrade Ubuntu 20 to Ubuntu 22 (following official instructions, here are `Jammy Upgrades notes `_) * Reboot your system unless already done in previous step. * Run ``apt-get update; apt-get dist-upgrade`` to get the latest version of perfSONAR. * Reboot your system one last time. -Alternatively, do a fresh installation of perfSONAR on Ubuntu 20. +Alternatively, do a fresh installation of perfSONAR on Ubuntu 22. diff --git a/install_el.rst b/install_el.rst index d2dcf8f1..0bfe48d6 100644 --- a/install_el.rst +++ b/install_el.rst @@ -33,7 +33,7 @@ perfSONAR provides a script that will automatically perform the first two steps curl -s https://downloads.perfsonar.net/install | sh -s - archive -If you would prefer to perform these steps manually see :ref:`_install_el_step1` and :ref:`_install_el_step2`. +If you would prefer to perform these steps manually see :ref:`install_el_step1` and :ref:`install_el_step2`. .. _install_el_installation: diff --git a/install_rcs.rst b/install_rcs.rst index c077788d..000f35d4 100644 --- a/install_rcs.rst +++ b/install_rcs.rst @@ -67,14 +67,14 @@ Testing Debian Installation (Manual) The beta packages for Debian can be found in the source list below: -* http://downloads.perfsonar.net/debian/perfsonar-minor-staging.list +* https://downloads.perfsonar.net/debian/perfsonar-minor-staging.list You may install this source list as follows:: - curl -o /etc/apt/sources.list.d/perfsonar-minor-staging.list http://downloads.perfsonar.net/debian/perfsonar-minor-staging.list - curl -s -o /etc/apt/trusted.gpg.d/perfsonar-staging.gpg.asc http://downloads.perfsonar.net/debian/perfsonar-staging.gpg.key + curl -o /etc/apt/sources.list.d/perfsonar-minor-staging.list https://downloads.perfsonar.net/debian/perfsonar-minor-staging.list + curl -s -o /etc/apt/trusted.gpg.d/perfsonar-staging.gpg.asc https://downloads.perfsonar.net/debian/perfsonar-staging.gpg.key -These Debian packages should work on Debian 10, Ubuntu 18 and 20. The perfsonar-archive bundle should also work on Debian 11 and Ubuntu 22. +These Debian packages should work on Debian 11 and 12, Ubuntu 20 and 22. On Ubuntu you need to make you have the **universe** repository enabled, this is done with the command ``add-apt-repository universe`` diff --git a/install_small_node_details.rst b/install_small_node_details.rst index cd30ce81..b3f4c3d3 100644 --- a/install_small_node_details.rst +++ b/install_small_node_details.rst @@ -24,7 +24,7 @@ Please note that the perfSONAR team is not formally endorsing any particular pro System Requirements =================== -ARM-based devices seem to work best running Ubuntu, while the slightly larger, $200-class nodes will usually work with EL, Debian or Ubuntu. The :doc:`perfsonar-testpoint ` bundle works on CentOS 7 and Ubuntu 18/20, and Debian 10. We provide compatible Debian packages for 6 different hardware architectures: +ARM-based devices seem to work best running Ubuntu, while the slightly larger, $200-class nodes will usually work with EL, Debian or Ubuntu. The :doc:`perfsonar-testpoint ` bundle works on CentOS 7 and Ubuntu 20/22, and Debian 11/12. We provide compatible Debian packages for 4 different hardware architectures: * 64-bit (amd64) * ARMv7 and up (armhf) - not recommended for the full testpoint bundle @@ -32,7 +32,6 @@ ARM-based devices seem to work best running Ubuntu, while the slightly larger, $ * PPC 64-bit (ppc64el) - Installation Instructions ========================= @@ -42,7 +41,7 @@ The :doc:`perfsonar-testpoint ` bundle can be used to install e - :doc:`install_debian` Certain devices like the Liva use an EMMC drive that is only supported in the *deskop* version of Ubuntu. -If the standard server ISO installation doe not recognize the drive, it may be worth attempting installation using the desktop ISO such as `Ubuntu 20.04.6 Desktop `_. +If the standard server ISO installation doe not recognize the drive, it may be worth attempting installation using the desktop ISO such as `Ubuntu 20.04.6 Desktop `_ or newer. Support ======= diff --git a/pscheduler_client_schedule.rst b/pscheduler_client_schedule.rst index 8b09f0bd..b85f7da6 100644 --- a/pscheduler_client_schedule.rst +++ b/pscheduler_client_schedule.rst @@ -11,6 +11,8 @@ It is often desirable to view the pScheduler schedule to plan new tests, debug a * ``pscheduler schedule`` - This command outputs the schedule in the requested timeframe as text. See :ref:`pscheduler_client_schedule-schedule`. * ``pscheduler plot-schedule`` - This command creates a PNG image file that provides a box plot of the schedule. See :ref:`pscheduler_client_schedule-plot_schedule`. * ``pscheduler monitor`` - This command provides top-like output showing near real-time state of runs in the database. See :ref:`pscheduler_client_schedule-monitor`. + +If you installed the ``perfsonar-toolkit`` bundle you can can visualize the pScheduler schedule directly in the Toolkit GUI. See :ref:`pscheduler_client_schedule-plot_schedule_toolkit`. .. _pscheduler_client_schedule-basics: @@ -191,3 +193,18 @@ If you would like to monitor a pScheduler server on remote host you can add the If you are curious about any additional options or details, you can also run ``pscheduler monitor --help`` to get more information about this command. +.. _pscheduler_client_schedule-plot_schedule_toolkit: + +Visualizing the Schedule with Plot Schedule Graph in Toolkit +------------------------------------------------------------ +With Toolkit installation, you can view the pScheduler schedule with **Plot Schedule Graph** menu option that provides easy access to visualize the hosts's schedule in the Toolkit GUI. + +#. Open *http://* in a web browser where ** is the name or address of your host. +#. Click on the **Configuration** button in the right-upper corner of the main page and login as the web administrator user. +#. On the page that loads go to **Tests** tab and click on **Plot-Schedule Graph** under **Resources** menu. The page that loads can be used to visualize the schedule in your Toolkit host. + + .. image:: images/pscheduler_client_schedule-plot-toolkit-menu.png +#. You will be prompted to select from drop-down lists a time period in the past where you wish the graph to start, and a time period in the future where the graph is to stop. These correspond to ``delta`` or ``start`` and ``end`` options of the ``pscheduler plot-schedule`` command. You can select only one of the periods or both. +#. Click the **Process Graph** button to generate and display the schedule as an image file. + + .. image:: images/pscheduler_client_schedule-plot-toolkit-graph.png diff --git a/psconfig_grafana_agent.rst b/psconfig_grafana_agent.rst index 11e2626b..1fd261cd 100644 --- a/psconfig_grafana_agent.rst +++ b/psconfig_grafana_agent.rst @@ -287,6 +287,67 @@ Since technically `super_special_test` would match both because its both a throu Beyond changing thredholds, the default displays may be fine in most cases but you can add more entries to the displays section for different test types or have it grab different stats. Feel free to experiment with the different fields to learn more. +.. _psconfig_grafana_agent-org_dashboards: + +Organizing Grids and Dashboards +----------------------------------------------- +Grouping grids onto the same dashboard can be controlled through your pSConfig JSON template under the `tasks` section. There are two values that need to be set under the task `reference` section: + + 1. Set the `display-task-name` to get a human readable title for the task + 2. Set the `display-task-group` to a list of dashboard names in which you want the grid included (NOTE: This must be an array even if it is just one element) + +See the snippet below that includes a grid displayed as *Example Throughput* in a dashboard named *Example Dashboard*:: + + "example_throughput" : { + "reference": { + "display-set-source": "{% jq .addresses[0]._meta.\"display-set\" %}", + "display-set-dest": "{% jq .addresses[1]._meta.\"display-set\" %}", + "display-task-name": "Example Throughput", + "display-task-group": ["Example Dashboard"] + }, + "group" : "example_thr_group", + "schedule" : "schedule_0" + "test" : "example_thr_test" + } + +.. _psconfig_grafana_agent-multi_xfaces: + +Graphing Multiple Interfaces on Same Dashboard +----------------------------------------------- +A common use case is that a single measurement host may have multiple interfaces, such as one dedicated to throughput and another dedicated to latency. The address for each interface is defined as separate `address` objects in your pSConfig file. By default, when the Grafana dashboards are generated these addresses will not be on the same graphs since the agent does not know they belong together. If you want this data to be on the same graph page you need to set some parameters that indicate the addresses should be grouped together. This requires two changes to your pSConfig template: + + 1. Update the `address` definitions with a `_meta` tag of `display-set`. + 2. Update the `task` definitions to include a `reference` section with `display-set-source` and `display-set-dest` that includes the `dispay-set` value from the source and destination. + +Let's look at these steps in detail. First let's say we have a host two interfaces: *example-tp.perfsonar.net* and *example-lat.perfsonar.net* that we want shown on the same graphs. When we define the addresses, we create a `display-set` called `example.perfsonar.net` (this can be any string as long as it is consistent between the two addresses) in our pSConfig template JSON. For example:: + + "example-tp.perfsonar.net" : { + "_meta" : { + "display-set" : "example.perfsonar.net" + }, + "address" : "example-tp.perfsonar.net" + }, + "example-lat.perfsonar.net" : { + "_meta" : { + "display-set" : "example.perfsonar.net" + }, + "address" : "example-lat.perfsonar.net" + } + +The next step is to make sure these values are passed to pScheduler when the tasks are created. This gives the agent enough information to build queries that can include results using both addresses on the graph. We do this by setting a `display-set-source` and `display-set-dest` in the `reference` section of the task. These use `jq` to dynamically set the values based on the source and destination. The task definition should look like the following and you can copy the definition of `display-set-source` and `display-set-dest` exactly to your template:: + + "example_throughput" : { + "reference": { + "display-set-source": "{% jq .addresses[0]._meta.\"display-set\" %}", + "display-set-dest": "{% jq .addresses[1]._meta.\"display-set\" %}", + "display-task-name": "Example Throughput", + "display-task-group": ["Example Dashboard"] + }, + "group" : "example_thr_group", + "schedule" : "schedule_0" + "test" : "example_thr_test" + } + .. _psconfig_grafana_agent-advanced: Advanced Configuration diff --git a/pwa_docker_create.rst b/pwa_docker_create.rst index ec9f1f33..4966c691 100644 --- a/pwa_docker_create.rst +++ b/pwa_docker_create.rst @@ -19,7 +19,7 @@ Run this script to create your nginx certs, if you don't already have them: Download the ``docker-compose.yml`` file from here: -https://raw.githubusercontent.com/perfsonar/psconfig-web/master/deploy/docker/docker-compose.yml +https://raw.githubusercontent.com/perfsonar/psconfig-web/master/psconfig-web-admin/perfsonar-psconfig-web-admin/deploy/docker/docker-compose.yml Bring up the application: diff --git a/pwa_install_docker.rst b/pwa_install_docker.rst index 4bf350e9..2662efa1 100644 --- a/pwa_install_docker.rst +++ b/pwa_install_docker.rst @@ -5,7 +5,7 @@ Docker Installation .. role:: raw-html-m2r(raw) :format: html -**If you are installing with RPMs, SKIP THIS PAGE** +.. note:: If you are installing with RPMs, **SKIP THIS PAGE** VM Host @@ -78,11 +78,9 @@ This guide assumes you have already pulled down the sample PWA config and extrac .. code-block:: bash - wget https://github.com/perfsonar/psconfig-web/raw/master/deploy/docker/pwa.sample.tar.gz + wget https://github.com/perfsonar/psconfig-web/raw/master/psconfig-web-admin/perfsonar-psconfig-web-admin/deploy/docker/pwa.sample.tar.gz sudo tar -C /etc -xvf pwa.sample.tar.gz pwa && sudo tar -C /etc/pwa -xvf pwa.sample.tar.gz scripts - - Other Topics ============ diff --git a/pwa_install_rpm.rst b/pwa_install_rpm.rst index 77ac1526..2dbb6ab9 100644 --- a/pwa_install_rpm.rst +++ b/pwa_install_rpm.rst @@ -5,7 +5,7 @@ PWA RPM Installation .. role:: raw-html-m2r(raw) :format: html -**If you are installing with Docker, SKIP THIS PAGE** +.. note:: If you are installing with Docker, **SKIP THIS PAGE** Host Requirements ^^^^^^^^^^^^^^^^^ @@ -24,7 +24,7 @@ perfSONAR Repo You will need to have the main perfSONAR repo installed, if not already:: - rpm -hUv http://software.internet2.edu/rpms/el7/x86_64/latest/packages/perfSONAR-repo-0.10-1.noarch.rpm + rpm -hUv http://software.internet2.edu/rpms/el7/x86_64/latest/packages/perfSONAR-repo-0.11-1.noarch.rpm Installing PWA from RPMs ^^^^^^^^^^^^^^^^^^^^^^^^ @@ -67,8 +67,6 @@ Installing just the publisher: yum install -y perfsonar-psconfig-web-admin-publisher -Before you start the docker engine, you might want to add any VM specific configuration. For example, your VM might be using /usr/local as a primary partition for your VM. If so, you should have something like following.. - By default, your configuration files will be placed under: * ``/etc/perfsonar/psconfig-web``