docs: add docs for vulnerability related commands

dev-docs/explanations/how_caching_works_for_vulnerability_commands.md

@@ -0,0 +1,29 @@
+# How caching works for vulnerability commands
+
+For the vulnerability commands (both API and CLI commands), we use caching to speed up
+the commands on subsequent runs. We cache both our vulnerability JSON data and the
+parsed result of that vulnerability data. To support that, we have four distinct caches:
+
+* **data cache**: This is just raw JSON vulnerability data.
+* **published data cached**: This caches the publish date of the vulnerability data.
+  We use it to check if there is new version of the JSON data available.
+* **result cache**: This is the JSON data created by parsing the JSON vulnerability data in the
+  system.
+* **dpkg status cache**: This caches the `mtime` of **/var/lib/dpkg/status**. We use this cache to
+  verify if our result cache is stale. The reason is that any APT related changes should invalidate
+  the result cache.
+
+Given all that, the following data flow diagram provides a better visualization of the role of those
+caches when running a vulnerability API command:
+
+![](vulnerability-cache.png)
+
+
+## Are there situations when the caches are not produced ?
+
+There are other scenarios where we don't cache anything when running the commands.
+That will happen when the user:
+
+* Provides a manifest file for the vulnerability commands
+* Provides a static vulnerability JSON data for the vulnerability commands
+

docs/explanations.rst

@@ -34,18 +34,21 @@ selection of some of the commands -- what they do, and how they work.
     explanations/what_refresh_does.md
     explanations/purging_services.md
 
-Public Cloud Ubuntu Pro
-=======================
+Vulnerability commands
+======================
 
-Here we talk about Ubuntu Pro images for AWS, Azure and GCP, and the related
-tooling: the ``ubuntu-pro-auto-attach`` package.
+Here we'll show you the information related to ``pro vulnerability`` commands.
+
+On any ``pro vulnerability`` command, we consider a vulnerability (CVE or USN) as **fixable**,
+if and only if, at least **one** affected installed package associated with it can
+be fixed. That means that a vulnerability is unfixable if no affected installed packages
+related to the vulnerability can be fixed at the moment.
 
 ..  toctree::
     :maxdepth: 1
 
-    explanations/what_are_ubuntu_pro_cloud_instances.md
-    explanations/what_is_the_ubuntu_advantage_pro_package.md
-
+    explanations/pro_vulnerability_list.md
+    explanations/pro_vulnerability_show.md
 
 Handling CVEs and USNs
 ==============================
@@ -60,6 +63,20 @@ as well as some details related to it like CVEs and USNs.
     explanations/fix_scenarios.rst
     explanations/how_to_interpret_output_of_fix_plan_api.md
 
+Public Cloud Ubuntu Pro
+=======================
+
+Here we talk about Ubuntu Pro images for AWS, Azure and GCP, and the related
+tooling: the ``ubuntu-pro-auto-attach`` package.
+
+..  toctree::
+    :maxdepth: 1
+
+    explanations/what_are_ubuntu_pro_cloud_instances.md
+    explanations/what_is_the_ubuntu_advantage_pro_package.md
+
+
+
 Other Pro features explained
 ============================
 

docs/explanations/pro_vulnerability_list.rst

@@ -0,0 +1,99 @@
+.. _pro-vulnerability-list:
+
+The pro vulnerability list command
+********************************************************
+
+In the Pro Client version 35, we introduce the ``pro vulnerability list`` command.
+This command displays all the CVEs currently affecting the
+system. Let's take a look at an example when running this command:
+
+.. code-block:: text
+
+    Common vulnerabilities and exposures (CVE):
+    VULNERABILITY     PRIORITY    FIX AVAILABLE FROM  AFFECTED INSTALLED PACKAGES
+    CVE-2021-44730    critical    esm-infra           snapd
+    CVE-2021-44731    high        esm-infra           snapd
+    CVE-2022-3328     high        security            snapd
+    CVE-2019-11922    medium      esm-infra           libzstd1
+    CVE-2021-3155     medium      esm-infra           snapd
+    CVE-2021-4120     medium      esm-infra           snapd
+    CVE-2023-1523     medium      esm-infra           snapd
+    CVE-2021-46848    low         security            libtasn1-6
+    CVE-2023-24626    low         esm-infra           screen
+    CVE-2018-1000654  negligible  esm-infra           libtasn1-6
+
+    Vulnerabilities with applied fixes:
+        13 applied via Ubuntu Security (2 high, 6 medium, 5 low)
+
+    Vulnerabilities with fixes available:
+        8 fixable via Ubuntu Pro (1 critical, 1 high, 4 medium, 1 low, 1 negligible)
+        2 fixable via Ubuntu Security (1 high, 1 low)
+
+We can see that the command has three distinct sections. Let's analyse each of them:
+
+.. code-block:: text
+
+    Common vulnerabilities and exposures (CVE):
+    VULNERABILITY     PRIORITY    FIX AVAILABLE FROM  AFFECTED INSTALLED PACKAGES
+    CVE-2021-44730    critical    esm-infra           snapd
+    CVE-2021-44731    high        esm-infra           snapd
+    CVE-2022-3328     high        security            snapd
+    CVE-2019-11922    medium      esm-infra           libzstd1
+    CVE-2021-3155     medium      esm-infra           snapd
+    CVE-2021-4120     medium      esm-infra           snapd
+    CVE-2023-1523     medium      esm-infra           snapd
+    CVE-2021-46848    low         security            libtasn1-6
+    CVE-2023-24626    low         esm-infra           screen
+    CVE-2018-1000654  negligible  esm-infra           libtasn1-6
+
+The first part of the output is a table showing all **fixable** CVEs in the system.
+The table contains these four distinct columns:
+
+* **VULNERABILITY**: The name of the CVE.
+* **PRIORITY**: The Ubuntu priority for the CVE.
+* **FIX AVAILABLE FROM**: The Ubuntu pocket where the fix can be found:
+	* ``esm-infra``: Fix is availabe from the ``esm-infra`` pocket. The user must have the ``esm-infra`` service enabled through Pro on the machine to access it.
+	* ``esm-apps``: Fix is availabe from the ``esm-apps`` pocket. The user must have the ``esm-apps`` service enabled through Pro on the machine to access it.
+	* ``fips``:     Fix is availabe from the ``fips`` pocket. The user must have the ``fips`` service enabled through Pro on the machine to access it.
+	* ``fips-updates``: Fix is availabe from the ``fips-updates`` pocket. The user must have the ``fips-updates`` service enabled through Pro on the machine to access it.
+	* ``security``: The Ubuntu ``security`` pocket.
+	* ``updates``: The Ubuntu ``updates`` pocket.
+	* ``no-fix``: No fix is available yet for this CVE.
+* **AFFECTED INSTALLED PACKAGES**: The installed packages affected by the CVE.
+
+Finally, note that the table is ordered by **PRIORITY**. This means that CVEs with higher
+priority will appear first in the table.
+
+Now, let's look at the next section:
+
+.. code-block:: text
+
+    Vulnerabilities with applied fixes:
+        13 applied via Ubuntu Security (2 high, 6 medium, 5 low)
+
+This section details the CVEs that already have fixes applied for them.
+For example, suppose we have a CVE named **CVE-2021-12345** and this
+CVE affects two packages, ``pkg1`` and ``pkg2``. If the we detect that ``pkg1`` is already fixed
+on the machine, then even if ``pkg2`` is still affected, we include this CVE here.
+
+Therefore, this section presents a summary of all CVEs that have at least **one** fix already
+applied for it. Note that we also order the count by Ubuntu priority as well.
+
+Finally, let's look at the last section:
+
+.. code-block:: text
+
+    Vulnerabilities with fixes available:
+        8 fixable via Ubuntu Pro (1 critical, 1 high, 4 medium, 1 low, 1 negligible)
+        2 fixable via Ubuntu Security (1 high, 1 low)
+
+While the previous section showed CVEs with fixes already applied, this section shows CVEs with fixes
+available, but not yet applied. The user can actively fix those CVEs by upgrading
+the affected packages. However, note that we also split the vulnerability according to the source of
+the fix. If the fix is
+available from a service that requires Pro, it will fall under **Ubuntu Pro**. Otherwise,
+we will classify the CVE as falling under **Ubuntu Security** instead. This means that some fixes will only be available
+if the user is attached to a Pro subscription and has the necessary service enabled (i.e. ``esm-infra``).
+
+Note that we group the count by Ubuntu priority here as well. If the table is long,
+breaking the count by priority provides better visualisation of the overall risk of the CVEs.

docs/explanations/pro_vulnerability_show.rst

@@ -0,0 +1,152 @@
+.. _pro-vulnerability-show:
+
+The pro vulnerability show command
+*****************************************************
+
+If you suspect that a CVE affects your system and want to verify
+that, you can use the ``pro vulnerability show`` command.
+This command shows you all the relevant information about a CVE **if** it affects
+your system.
+
+For example, let's assume that ``CVE-2022-2286`` affects your system and you run:
+
+.. code-block:: bash
+
+    pro vulnerability show CVE-2022-2286
+
+You will see an output similar to this one:
+
+.. code-block:: text
+
+   ● CVE-2022-2286
+   Public URL: https://ubuntu.com/security/CVE-2022-2286          
+   Published at: 2022-07-02 19:15:00
+   Ubuntu vulnerability data published at: 2024-10-22 23:37:41
+   APT package information updated at: 2024-10-23 19:37:11
+   Ubuntu priority: low
+   CVSS score: 7.8
+   CVSS severity: high
+
+   [DESCRIPTION]
+   Out-of-bounds Read in GitHub repository vim/vim prior to 9.0.
+
+   [AFFECTED INSTALLED PACKAGES]
+   NAME         STATUS  CURRENT VERSION        FIXED BY                     FIX AVAILABLE FROM
+   vim          fixed   2:7.4.1689-3ubuntu1.5  2:7.4.1689-3ubuntu1.5+esm19  esm-infra
+   vim-common   fixed   2:7.4.1689-3ubuntu1.5  2:7.4.1689-3ubuntu1.5+esm19  esm-infra
+   vim-runtime  fixed   2:7.4.1689-3ubuntu1.5  2:7.4.1689-3ubuntu1.5+esm19  esm-infra
+   vim-tiny     fixed   2:7.4.1689-3ubuntu1.5  2:7.4.1689-3ubuntu1.5+esm19  esm-infra
+
+   [RELATED USNs]
+   VULNERABILITY  TITLE                AFFECTED INSTALLED PACKAGES
+   USN-6270-1     Vim vulnerabilities  vim
+
+Let's break down the output of this command. It starts with some basic information for
+the CVE:
+
+* **Public URL**: The Ubuntu Security page for this CVE
+* **Published at**: The published date for the CVE
+* **Ubunut vulnerability data published at**: The published date for the vulnerability data we consume to generate this output
+* **APT package information updated at**: The last time the APT state was updated in the system
+  (i.e. running an ``apt install`` operation)
+* **Ubuntu priority**: The Ubuntu priority for this CVE
+* **CVSS score**: The CVSS score of the CVE
+* **CVESS severity**: The CVSS severity of the CVE
+
+After that, we display the CVE description and related notes (if the CVE has notes).
+
+The next block shows which packages installed on the machine are affected by the CVE.
+We use a table to display the affected packages with the following columns:
+
+* **NAME**: The package name
+* **STATUS**: The CVE status for this package
+* **CURRENT VERSION**: The current installed version of the package
+* **FIXED BY**: The version of the package that fixes the CVE issue
+* **FIX AVAILABLE FROM**: The pocket where the fix version can be found
+
+Finally, we also display any USNs related to the CVE.
+
+What if the CVE doesn't affect my system ?
+==========================================
+
+If the CVE doesn't affect your system, running ``pro vulnerability show`` will show
+you an output like this:
+
+.. code-block:: text
+
+    ● CVE-2024-7542
+    Public URL: https://ubuntu.com/security/CVE-2024-7542
+    Ubuntu vulnerability data published at: 2024-10-22 23:37:41
+    APT package information updated at: 2024-10-23 19:37:11
+
+    CVE-2024-7542 does not affect your system.
+
+Note that we directly inform you that the CVE doesn't affect your system.
+
+What if some packages cannot be fixed for a CVE?
+================================================
+
+When that happens, our affected installed packages table will inform you about that.
+For example, here is the table output for the situation where some packages cannot be
+fixed:
+
+.. code-block:: text
+
+    NAME                  STATUS      CURRENT VERSION          FIXED BY  FIX AVAILABLE FROM
+    libpython3.5          vulnerable  3.5.2-2ubuntu0~16.04.13  -         no-fix
+    libpython3.5-minimal  vulnerable  3.5.2-2ubuntu0~16.04.13  -         no-fix
+    libpython3.5-stdlib   vulnerable  3.5.2-2ubuntu0~16.04.13  -         no-fix
+    python3.5             vulnerable  3.5.2-2ubuntu0~16.04.13  -         no-fix
+    python3.5-minimal     vulnerable  3.5.2-2ubuntu0~16.04.13  -         no-fix
+
+The status column shows the package is **vulnerabable**, but the **FIXED BY** and
+**FIX AVAILABLE FROM** column tell you that no fix is available for the package.
+
+What if I want to see a USN instead ?
+=====================================
+
+To see information about a USN, you can provide a USN name instead. For example,
+to see information about USN-4976-2 you can run:
+
+.. code-block:: bash
+
+    pro vulnerability show USN-4976-2
+
+Assuming you are affected by the USN, you might see an output like this:
+
+.. code-block:: text
+
+    Public URL: https://ubuntu.com/security/notices/USN-4976-2
+    Published at: 2022-09-07 15:22:39
+    Ubuntu vulnerability data published at: 2024-10-22 23:37:41
+    APT package information updated at: 2024-10-23 19:37:11
+
+    [DESCRIPTION]
+    USN-4976-1 fixed a vulnerability in Dnsmasq. This update provides
+    the corresponding update for Ubuntu 16.04 ESM.
+
+    Dnsmasq has been updated to 2.79-1 for Ubuntu 16.04 ESM in order to fix
+    some security issues.
+
+    Original advisory details:
+
+     Petr Mensik discovered that Dnsmasq incorrectly randomized source ports in
+     certain configurations. A remote attacker could possibly use this issue to
+     facilitate DNS cache poisoning attacks.
+
+
+    [AFFECTED INSTALLED PACKAGES]
+    NAME          CURRENT VERSION         FIXED BY                    FIX AVAILABLE FROM
+    dnsmasq-base  2.75-1ubuntu0.16.04.10  2.79-1ubuntu0.16.04.1+esm1  esm-infra
+
+    [RELATED CVEs]
+    VULNERABILITY  PRIORITY  AFFECTED INSTALLED PACKAGES
+    CVE-2021-3448  low       dnsmasq
+
+
+The information presented here is similar to the output displayed for CVEs.
+However, we do not display information that is only relevant to a CVE, such as
+CVSS score and Ubuntu priority.
+
+Finally, note that the last table is called Related CVEs instead, since we are now
+showing the related CVEs for the given USN.

docs/howtoguides.rst

@@ -36,6 +36,17 @@ Enable services
 
    Enable... <howtoguides/enable_index>
 
+Vulnerability commands
+======================
+
+.. toctree::
+   :maxdepth: 1
+
+   Display USNs that affect the system... <howtoguides/how_to_display_usns_that_affect_the_system>
+   Provide manifest file to commands...  <howtoguides/how_to_use_manifest_file_for_pro_vulnerability>
+   Update vulnerability data... <howtoguides/how_to_update_vulnerability_data>
+
+
 Fix vulnerabilities
 ===================
 

docs/howtoguides/how_to_display_usns_that_affect_the_system.rst

@@ -0,0 +1,32 @@
+How to display USNs that affect the system
+******************************************
+
+To display which USNs affect the system, run the command:
+
+.. code-block:: bash
+
+    $ pro vulnerability list --usns
+
+
+When running the command you should see an output like this:
+
+.. code-block:: text
+
+    Ubuntu Security Notices (USN):
+    VULNERABILITY  FIX AVAILABLE FROM  AFFECTED INSTALLED PACKAGES
+    USN-5292-3     esm-infra           snapd
+    USN-5352-1     esm-infra           libtasn1-6
+    USN-5593-1     esm-infra           libzstd1
+    USN-5707-1     esm-infra           libtasn1-6
+    USN-5720-1     esm-infra           libzstd1
+
+    Vulnerabilities with applied fixes:
+        1 applied via Ubuntu Security
+
+    Vulnerabilities with fixes available:
+        5 fixable via Ubuntu Pro
+
+Note that this output is similar to what we describe in 
+:ref:`pro vulnerability list output <pro-vulnerability-list>`.
+The main differance is that we don't have a ubuntu priority for a USN and therefore
+this is not displayed in the output of the command.