Reduce pa11y runtimes

[caleywoods]

Pull request summary

Attempts to address #3752.

We're currently running pa11y scans using the sitemap which scans all of the URLs on the site which results in slowness. These changes use a two pronged approach to reducing the pa11y runtimes.

The Jekyll plugin

The first part is a Jekyll plugin that uses Jekyll hooks (:documents, :pages, and :site currently) to output a file named pa11y_targets to the filesystem during the build process. The plugin has the following behavior:

• When changes are made to a post or to any file in a user defined collection, those will be picked up by the :documents Jekyll hook and be included in the list of files to be scanned by pa11y. This takes care of the majority of file changes
• If a change is made to a file in the pages/ directory, the :pages Jekyll hook captures those to be included in the list of files to be scanned by pa11y (I'm not sure why :documents doesn't get these)
• If a layout is changed the plugin determines what files use that layout and includes all of them to be scanned by pa11y
• If a change is made to a "site-wide" file in assets, _includes, or _sass, the plugin does a sampling of pages across the site. These pages are: the homepage, three files that use each layout from _config.yml's defaults, as well as the first, middle, and last blog archive pages (/blog/page/:num)
• All files that are considered changed are written to the pa11y_targets file and each path is separated by a newline

There's RSpec tests to go along with this plugin.

The CircleCi config

The second part is a tweak to .circleci/config.yml when the Jekyll build step runs. This change does two things:

• It makes the contents of the pa11y_targets file be base64 encoded and exported into an environment variable called PA11Y_TARGETS in CircleCI's $BASH_ENV area which is stateful between job steps
• Two job steps later when pa11y is initiated, it reads the list of files to scan out of the previously stored PA11Y_TARGETS environment variable

Results

Without these changes, the full build takes around 25 minutes in CircleCI with the pa11y scan representing the bulk of that time taking around 20-21 minutes. Testing these changes showed CircleCI full builds in under 4 minutes with the pa11y scan portion being reduced to < 10 seconds in most cases. These are big gains for the common scenario of single file changes when someone is creating or editing a blog post.

• CircleCi current build example, 25m 35s runtime
• CircleCI _layouts/styled-container.html was changed, 3m 49s runtime (Note: this scan was prior to the :pages hook being added, _layouts/styled-container.html applies to three pages in the pages/ directory but adding three more files to the pa11y scan would not have caused a significant increase in the pa11y scan time)

This could still have some long running build times/pa11y scans depending on what files are touched. An example would be _layouts/post.html, when I checked locally, changing that particular layout resulted in pa11y scanning a little over 500 files.

[caleywoods]

Tagging @beechnut - I know you're not on TLC crew quite yet but you did a lot of great work on this and wanted to make sure you had a chance to look at it.

[caleywoods]

I also wonder if it's detrimental to never run a full pa11y scan. Should we consider adding a scheduled GitHub action to run a full scan maybe every two weeks?

[beechnut]

@caleywoods Yes, part of the scope of work is to add a nightly full scan. We shouldn't remove full scans from the pipeline without replacing them beforehand or in the same PR.

[cantsin]

LGTM!

[caleywoods]

@caleywoods Yes, part of the scope of work is to add a nightly full scan. We shouldn't remove full scans from the pipeline without replacing them beforehand or in the same PR.

I've opened #3832 to track this and I plan on working on it this week.