Scaling Public & Private Registries

[SteveLasker]

When designing key management solutions, there are a number of key scaling aspects to consider.

Before getting into the discussions of short-lived, long-lived, or rolling start keys, it helps to consider the number and type of registries, orgs and repos we will need to support.

Public Registry Expansion

When we talk about content promotion within and across registries, there's a growing landscape of registries to consider. Normally, we wouldn't think much about how many public and private registries that exist. However, as we talk about key management, and how we might prevent rollback attacks or other cross registry compromises, it becomes a factor to consider.

Before we add private registries, it's also important to recognize how many public registries exist, as Public Registries represent Software Registries and Distributors.

Public Registries

Public registries are essentially distributors of content. They represent content from multiple sources. For many content producers, this may be the primary form of distribution, or they may be hosted for redistribution for a specific target audience. This would be akin to buying a product from a distributor, such as buying XBox from BestBuy, Target, CostCo or other distributors. Some content creators may only sell through distributor channels as they're not big enough, or simply don't see the need to host their content directly.

Public registry distributors, or redistributors include:

• Docker Hub
• Quay.io
• GitHub Container Registry
• AWS ECR Public

Software Registries

For some content creators, they may choose to distribute directly, or the content may be redistributed. Using the XBox example, some users may choose to buy direct from Microsoft, or they may choose a distributor. In the case of software, the initial acquisition of content isn't necessarily tied to where the user may get subsequent updates, or other content from the same author. Some authors may contract with distributors to host the catalog, but they may serve the content directly. This is the case with MCR syndicating its catalog to Docker Hub.

Software registries include:

• Microsoft Container Registry (MCR)
• NVIDIA Registry
• Oracle Container Registry
• RedHat Container Registry

Private Registries

Following best practices for consuming public content users will promote the content they depend upon to a private registries. Private registries may be Cloud Hosted, or Product Instanced.

Cloud Hosted Private Registries

• AWS ecr
• Azure ACR
• Google Container Registry
• IBM Container Registry

Privately Instanced Product/Project Registries

• Quay.io
• Project Harbor
• JFrog

Scaling Key Management Across Public an Private Registries

With a minimum of 4 public registries and 4 software registries, we're already well beyond the single public registry. Just as retail business and distributors come and go, we'll continue to see an ever increasing expansion of software and distributors evolve.

To expand this further, it's a fair estimation that each cloud provider may have between 50,000 and 500,000 private registries. Within each of those private registries, customers sub-divide their permission boundaries. If we used a minimum average of 4 sub-divisions within a registry, we're talking a minimum of 200,000 to 2,000,000 privately scoped permission boundaries. While the software registries don't typically subdivide their permissions, the public distributors likely do.

Within the sub-division of a registry, we can now start counting the number of repositories. While many groups may share a set of keys across their repos within their sub-divided registry, others may decide to manage a dev, staging and prod root key.

Considering we're still at the early phase of container registries, a year from now, it's easy to predict 4 million privately scoped registries, with an average of 10 repos each suggests 40 million root keys is a reasonable estimate. Even if customers would allow us to create aggregated hashes of their data, which they don't, is it even reasonable to build a system that attempts to reconcile 40 million root keys, when many will be in air-gapped virtual networks?

[mnm678]

Scalability of the key management depends more on the keys needed by an individual user to verify images than it does on the total number of registries that exist. If a user can use a single trusted root key to find 1,000,000 artifact keys, the size of the registry doesn't matter as much.

This is exactly why it's important to allow aggregation on public registries when possible. Users will need to discover keys for every image they download from every registry. The more that these registries can consolidate roots, the less TOFU will sneak into the key management process as a shortcut. For example, with Notary v1-style repository roots, users need to discover root keys for every repository on a public registry. It would be much easier to instead have a smaller number of roots for large public registries (as proposed in the tuf-notary proposal, then use these roots to delegate keys to individual repositories. This minimizes the amount of external secure root key distribution that the user needs to do.

The story is different for private registries as aggregation may not be possible, and users will primarily be using a small number of these private registries. Because of this, users will only need to securely access root keys for a small number of private registries (or repositories on private registries).

[SteveLasker]

This is exactly why it's important to allow aggregation on public registries when possible

I totally get why a single source of truth would solve some problems while making others more difficult. Such as a single point of failure and a single point of hacking.

The more that these registries can consolidate roots, the less TOFU will sneak into the key management process as a shortcut.

I'm really pointing out the reality of the industry is we'll see more and more public and software registries. GitHub public and ecr public basically doubled the distributors just last year. And, I didn't even mention the additional registries in China and other sovereign boundaries. Our desire to want fewer registries isn't really related to the natural evolution of expansion.

The thing we're capturing here isn't the desire to consolidate, rather the reality of expansion of public & software registries, and the business need of customers to have isolation. It's a requirement for us to support this, and not really an option for us to consolidate as it's simply outside our control.

Because of this, users will only need to securely access root keys for a small number of private registries (or repositories on private registries).

What I was seeing in the previous proposal was some merkel tree aggregation of content across public and private registries. The purpose of this scaling discussion is to recognize we need to design a solution that:

1. allows for scoped permission boundaries that have no external correlation. Coke must be able to sign and validate their signed content without any correlation to, or awareness of content from Pepsi.
2. doesn't require all root keys to be continually updated just to prove they're still valid.

I'll write up some more on point 2, but I'd like to see us recognize this scaling problem makes short-lived keys problematic at best.

Can we design a solution that focuses on securely acknowledging the edge expriy/revocation cases, without requiring all the happy-path cases from having to be updated?

I do recognize the need to expire a window of content (based on a signature or key), without expiring all the content. Instead of short-lived keys, what if we had short-lived usage for signing new content for a key? This way, the key can live as long as makes sense (1-10 years), but users have the ability to say this key was only allowed to sign content from 12/1/2020-12/30/2020. It's valid till 12/31/2030 unless someone revokes it. But, the content that was signed from 11/1/2020-11/30/2020 is still valid, and the content that was signed from 1/1/2021-1/31/2021 is also still valid. Sure, it may overlap a broader range, but how often does content really need to be revoked, and when it does, is it ok to possibly have some collateral safety margins?

Rather than short-lived keys, they're short-life of signing keys. (we need a better term-naming is hard)

[dlorenc]

I can't really follow this discussion at all. I think you're talking about two separate things, or there's some kind of gross misunderstanding somewhere.

What I was seeing in the previous proposal was some merkel tree aggregation of content across public and private registries. The purpose of this scaling discussion is to recognize we need to design a solution that:

What previous proposal?

I'll write up some more on point 2, but I'd like to see us recognize this scaling problem makes short-lived keys problematic at best.

I really especially can't follow this. What is the scaling dimension that concerns you? Too many keys? Too much content to sign? Too many signing operations? It's hard to have scaling and performance discussions without first understanding the dimension.

Can we design a solution that focuses on securely acknowledging the edge expriy/revocation cases, without requiring all the happy-path cases from having to be updated?

I'm not sure how to parse this one. What happy-path cases are being updated?

[dlorenc]

@SteveLasker - could you point me to any of the docs in here around short-lived keys? I can't even find them mentioned in the notaryproject organization outside of these few issues you started to discuss the problems with them.

[mnm678]

This is exactly why it's important to allow aggregation on public registries when possible

I totally get why a single source of truth would solve some problems while making others more difficult. Such as a single point of failure and a single point of hacking.

I agree, thus the need for threshold signatures and offline keys for these roots.

The more that these registries can consolidate roots, the less TOFU will sneak into the key management process as a shortcut.

I'm really pointing out the reality of the industry is we'll see more and more public and software registries. GitHub public and ecr public basically doubled the distributors just last year. And, I didn't even mention the additional registries in China and other sovereign boundaries. Our desire to want fewer registries isn't really related to the natural evolution of expansion.

The thing we're capturing here isn't the desire to consolidate, rather the reality of expansion of public & software registries, and the business need of customers to have isolation. It's a requirement for us to support this, and not really an option for us to consolidate as it's simply outside our control.

Because of this, users will only need to securely access root keys for a small number of private registries (or repositories on private registries).

What I was seeing in the previous proposal was some merkel tree aggregation of content across public and private registries. The purpose of this scaling discussion is to recognize we need to design a solution that:

1. allows for scoped permission boundaries that have no external correlation. Coke must be able to sign and validate their signed content without any correlation to, or awareness of content from Pepsi.

2. doesn't require all root keys to be continually updated just to prove they're still valid.

My proposal only aggregates within the scope of the root of trust (see the Snapshot and timestamp process). This could be the registry or a subset of the registry. So in the Coke and Pepsi scenario they would each have their own root, and thus no correlation.

Also, the root keys themselves don't need to be continually updated. In fact, they should probably be longer lived in order to allow them to be stored offline and kept more secure. The only thing that needs to be updated regularly is the timestamp metadata. This one piece of regularly expiring metadata allows the user to ensure timeliness of all of the other TUF roles while allowing them to have long lived keys.

I do recognize the need to expire a window of content (based on a signature or key), without expiring all the content. Instead of short-lived keys, what if we had short-lived usage for signing new content for a key? This way, the key can live as long as makes sense (1-10 years), but users have the ability to say this key was only allowed to sign content from 12/1/2020-12/30/2020. It's valid till 12/31/2030 unless someone revokes it. But, the content that was signed from 11/1/2020-11/30/2020 is still valid, and the content that was signed from 1/1/2021-1/31/2021 is also still valid. Sure, it may overlap a broader range, but how often does content really need to be revoked, and when it does, is it ok to possibly have some collateral safety margins?

Rather than short-lived keys, they're short-life of signing keys. (we need a better term-naming is hard)

This is actually really similar to what the snapshot role in TUF allows. If a user wants to securely delete a piece of metadata, they can remove it from the snapshot role and all users will see that it should no longer be trusted. This is backed by a notion of timeliness from the timestamp role, so users can be assured that any signatures they are verifying are currently trusted.