Sometime before March 2020, Russia's Foreign Intelligence Service (SVR) compromised SolarWinds, a software company based in Texas that makes the Orion network management tool. Note that Russia has denied involvement, but US officials have said the attack has the hallmarks of an SVR operation.
The SVR used its access to add malicious code (the payload) to Orion. Orion is used by thousands of organizations all over the world, and nearly eighteen thousand of them installed a compromised version of Orion. Among these eighteen thousand were US Pentagon, Treasury, Department of Homeland Security, and National Institutes of Health, not to mention other governments and companies.
While we won't know the full impact of this hack for years, this hack highlights how vulnerable the software supply chains we all rely on are.
To read more about the SolarWinds hack and see the articles from which I drew the information about the hack in this guide, see this blog post from Yubico, this essay from Bruce Schneier, and this NPR story.
While the SolarWinds hack is among the most recent and damaging supply-chain attack, there have been many others. Here are some from the Linux Foundation's "Open Source Software Supply Chain Security" publication from February 2020:
- In 2015, developers were tricked into using a malicious version of Xcode that inserted malware into their apps.
- In 2017, attackers introduced malicious libraries to the Python Package Index (PyPI). They chose library names that were similar to legitimate, popular packages so developers would install them by accident.
- In 2019, attackers compromised the Ruby account of the developer of
the popular Ruby package
strong_password. The attackers used this access to deploy a new version of the package that would download and execute arbitrary code of the attacker's choice on the developer's machine. - Later in 2019, 11 backdoored Ruby packages were found to have been infected with malicious code. Four were infected by compromising a developer's account.
When users run our software, they are trusting us to ensure it is free of malicious code. In this guide, we'll discuss how software projects get compromised and what you can do to keep your projects safe.
At a high level, you can think of your project as a mixture of code from you and your dependencies:
Dependency -----> +--------------+
Dependency -----> | Your Project | -----> Users
Dependency -----> +--------------+
^
|
|
Code You Write
From this view, we see that your project can be compromised in a few ways:
- Vulnerable dependencies
- Intentionally-introduced malicious code from you
- Accidentally-introduced security bugs from you
Let's take a look at each of these and how you can mitigate the risks they pose.
According to GitHub's Octoverse 2020 Security Report, an active repository in a supported package ecosystem like PyPI or NPM had a 59% chance of getting a Dependabot alert about a dependency with a security vulnerability in 2020. This means your repository will probaby have a dependency with a security vulnerability.
Luckily for us, GitHub will automatically send us Dependabot alerts whenever a dependency has patched a vulnerability. By staying on top of these alerts, you can keep your dependencies up-to-date.
Important
Act on Dependabot alerts quickly to upgrade vulnerable dependencies.
You might first think about one of your developers intentionally
introducing security bugs. In my opinion, this is pretty unlikely in the
small teams we work on. What small risk remains is pretty effectively
mitigated by the fact that git lets us track who introduced what
changes.
A bigger threat is an attacker compromising one of your developers' accounts. Recall that many of supply-chain attacks we discussed earlier involved compromising a developer's account. Here are some steps you as developers can take to secure your accounts:
- Use a strong, unique password for your GitHub account. I highly recommend using a password manager, for example KeePassXC or Bitwarden.
- Enable two-factor authentication on your GitHub account. Here are instructions. This is one of the easiest and most impactful steps you can take to improve your personal security. Do it right now!
As a bonus step, I strongly support cryptographically signing your git
commits. GitHub has instructions
you can follow. For example, if you look at our guides repository at Code the
Change, you'll see a Verified label next to our commits:
This label indicates that our commits have each been signed by the author with a secret key they alone control. This makes it easy to identify commits introduced by a hacker--they will be unsigned.
Important
Secure your GitHub account with strong, unique passwords and two-factor authentication. As a bonus steps, start signing your commits.
According to the 2020 Octoverse Security Report, only 17% of security vulnerabilities on GitHub were intentionally malicious, and these vulnerabilities caused just 0.2% of Dependabot alerts. Most software vulnerabilities are just mistakes.
Note
Most security vulnerabilites are mistakes. Unfortunately, they also go unnoticed for 4 years on average.
Unfortunately, there are no easy solutions to avoiding security bugs. GitHub has code scanning features that may be useful, though I haven't used them yet.
More generally, we stop security bugs the same ways we stop other bugs: defensive coding, thorough testing, and code review. You can ensure these steps aren't bypassed by setting up protected branches on GitHub.
Important
Enforce code review and passing automated tests for all code changes. Make sure your tests are comprehensive.
Just keeping vulnerabilities out of your code isn't enough. You also have to help users trust that the code they download is secure.
All software you ship should be digitally signed so your users can
verify that it hasn't been tampered with. I find it easiest to create a
signed git tag for each
release. If you then create a release on GitHub, you'll get a
Verified label just like with commits. Here's an example from one
of my projects:
Important
Securely sign all software releases to verify their authenticity.
If your repository is ever compromised, you want to be able to tell your users something like
"version 1.1.2 was compromised, so please make sure you have upgraded to version 1.1.3."
This is only possible if you release numbered versions of your software. I recommend following the Semantic Versioning numbering system, as it has become standard in open source software.
To get the most out of versions, you should also document the changes in each version using a changelog. In fact, the first compromise of a Ruby package in 2019 that we discussed above was caught when a user of the package noticed that a new version had been released without an update to the changelog.
Finally, you need to communicate to your users all the security practices you have in place to keep them safe. This could be in your documentation or README file. The important thing is to let your users know what practices you follow (e.g. signing releases) so they can spot anomalies (e.g. an unsigned release) that might indicate a compromise.
I also recommend letting people know how they can securely report security problems to you. Many of the hacks we discussed at the The earlier Ruby package compromise in 2019 was caught when a user reported the problem to the developer. This required the developer to have published secure ways to reach them. This should all go into your security policy, for instance a SECURITY.md file on GitHub.
Important
Be transparent about your security policies and how people can report vulnerabilities to you.
Copyright (c) 2020 U8N WXD
This work, including both this document and the source code in the associated GitHub repository, is licensed under a Creative Commons Attribution 4.0 International License.
This work was initially created for a workshop at Stanford Code the Change.