Auto-detect applicable SCTs in multi-vhost setups

[BenBE]

When running a nginx setup with multiple vhosts for different domains that each have independent domains+certificates it would be nice, if SCTs for all certificates could be put in one directory with the module including only those applicable to the current connection's server certificate.

As I understood the code while skimming over it, the module currently puts all SCTs it finds into the TLS extension. It would be nice, if the module only added SCTs applicable to the current vhost's certificate (and if necessary trust chain).

Intention:

• Avoid configuration overhead by specifying SCT behaviour globally
• Centralize storage of SCTs for all certificates in one directory (e.g. all SCTs for all certs go to /etc/ssl/sct via cron or similar means).

[grahamedgecombe]

I agree this would be a nice feature, but it adds lots of complexity - nginx-ct would need to know about the keys of all the logs (presumably from reading the official log_list.json file, or similar) and it'd need code to verify the signatures (to tie each SCT to a certificate).

I also suspect it'd be much easier to do this outside of nginx in a higher level language than C - a separate program could take your bunch of certs/SCTs and split them into separate separate directories as required by nginx-ct. That said, I also understand the appeal of having it built into the module.

This isn't a feature my own setup requires, so it's currently low on my list of priorities. Patches are welcome, however!

[BenBE]

Totally understandable.

To simplify the "verification" you could even make the list of log server keys just that - similar to the list of trusted CAs for the certificate verification. Code for reading a list of keys should already be available in nginx - or is easily adopted from reading a list of certificates.

Armed with this list of public keys you could kinda pre-sort entries per log server, so that upon configuring the actual VHosts you just iterate each of the per-logserver lists of SCTs with the matching key and certificate and add all SCTs that "verify".

This approach originating from individual certificates eliminates the requirement for one ssl_ct_file per ssl_certificate directive. (Based on this the ssl_ct on directive might become advisory in the sense, that it is ignored for non-SSL hosts.)

This being said, I'll see what I can prepare for this.

[BenBE]

Minor heads-up on this:

• SCT validation and management of CT log servers is available in the OpenSSL v1.1.0 API onwards. Due to the hassle of maintaining two versions of the code, my patch will likely only address OpenSSL versions offering the CTLOG and SCT APIs.
• Huge portions of the existing code can be reused, especially the parts reading the SCTs from disk. But given I need the SCTs and not a fully encoded structure (encoding the final structure of SCTs to send can be done using OpenSSL API AFAIC), the functions for loading SCTs from disk will be somewhat rewritten (ABI-change) to return a list of SCTs read in a given directory.
• The way the SCT structure is attached to each certificate as extra data can be reused.
• OpenSSL comes with an API to manage a CTLOG_STORE. There will be a new directive to divert the system default one to a user-supplied one in the nginx configuration.

Backporting the changes to use OpenSSL 1.0.2 compatible API is mostly reinventing the wheel. My patch will thus target OpenSSL 1.1.0+. For use in old versions it will need someone else to copy OpenSSL source between the correct ifdef blocks.

[BenBE]

First part of the patch done; mostly fighting with keeping changes to a minimum and including OpenSSL API calls to CTLog/SCT API - which as expected is best explained by reading the OpenSSL source ...

So far:

• Specifying multiple SCT directories will work
• No need to configure SCT directories per VHost should work
• Detection of applicable SCTs per Certificate should work

Will likely make SCTs for unknown Log IDs a warning and add those SCT to all servers. User will then have to configure this additional log in system default or supplied CT log config. Other option would be dropping those SCT, which would be most unlike existing behaviour: Currently you can think of the module handling all SCTs in one directory as unknown logs.

Patch stats @ +112/-64 or +59/-11 (if ignoring indentation changes). Progress at around 60% ...

[BenBE]

Okay, time for a new status update:

Patch itself is written and is available at https://github.com/BenBE/nginx-ct/tree/check_sct for review.
Code neither tested if it compiles nor if it fully works. This both is still WIP.

I'd appreciate comments and suggestions. Especially the resource handling may still need quite some cleanup.

[BenBE]

Minor fixup during the week (missed to delete one LOC in the original patch).

Patch should do as expected; looking for reviews and tests.

[grahamedgecombe]

@BenBE thanks! I haven't forgotten about this, will take a look at the patch asap.

[BenBE]

Looking forward. If you need anything feel free to ask. Not yet tested (lack of suitable test system on my side). Confident it should work though.

[BenBE]

Further update: I included 5aa305a yesterday to fix some issues with compilation when using OpenSSL 1.1.1-dev from branch tls1.3-draft-19. While this compiles (and does not crash immediately) I still see some weird behaviour in my configuration (despite SCTs existing I receive a configuration failed without explicit error message). Will further investigate, but would be happy if somebody else could have a look too.

[BenBE]

And a second minor issue found with the ssl_ct_log directive I introduced: Forgot both to add it for mail servers AND to actually initialize it to unset ... Patch in 8d89b3f.

[BenBE]

The branch received some more refinements (somehow the ssl_ct_log directive got messed up and wasn't handled properly, among some other issues). Current set of changes can be found at https://github.com/grahamedgecombe/nginx-ct/compare/master...BenBE:check_sct?expand=1

Unfortunately I was not yet able to witness this code sending back SCTs, debugging.

[BenBE]

Latest version on my branch seems to work (it properly detects all the SCT to attach), but for some reason fails to include them in the ServerHello (debugging this part). Inclusion requires a ctlogs.cnf file to be present (either via OpenSSL config or set in the nginx config via ssl_ct_log). An almost complete file (the one I used for testing) is below:

enabled_logs = akamai, certly, cnnic, comodo-dodo, comodo-mammoth, comodo-sabre, ct-cn, digicert-ct1, digicert-ct2, gdca-ct, gdca-ctlog, google-aviator, google-deadalus, google-icarus, google-pilot, google-rocketeer, google-skydiver, google-submariner, google-testtube, izenpe-com, izenpe-eus, le-clicky, nordu-flimsy, nordu-plausible, sheca-1, sheca-2, sofa, startssl, symantec-ct, symantec-deneb, symantec-sirius, symantec-vega, venafi-gen1, venafi-gen2, wosign-ct, wosign-ctlog

[akamai]
description = Akamai CT Log
url = https://ct.akamai.com/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEQ3nrSVxQKkpqj1mTvMNCdsKZ+CeBPAZs0sgEj3R7tLUh8uOo3DO5/iXpPQT8P7SuQONFfoSSKthS6x8/cxPQyA==

[certly]
description = Certly.IO
url = https://log.certly.io/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAECyPLhWKYYUgEc+tUXfPQB4wtGS2MNvXrjwFCCnyYJifBtd2Sk7Cu+Js9DNhMTh35FftHaHu6ZrclnNBKwmbbSA==

[cnnic]
description = CNNIC CT Log
url = https://ctserver.cnnic.cn/
key = MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv7UIYZopMgTTJWPp2IXhhuAf1l6a9zM7gBvntj5fLaFm9pVKhKYhVnno94XuXeN8EsDgiSIJIj66FpUGvai5samyetZhLocRuXhAiXXbDNyQ4KR51tVebtEq2zT0mT9liTtGwiksFQccyUsaVPhsHq9gJ2IKZdWauVA2Fm5x9h8B9xKn/L/2IaMpkIYtd967TNTP/dLPgixN1PLCLaypvurDGSVDsuWabA3FHKWL9z8wr7kBkbdpEhLlg2H+NAC+9nGKx+tQkuhZ/hWR65aX+CNUPy2OB9/u2rNPyDydb988LENXoUcMkQT0dU3aiYGkFAY0uZjD2vH97TM20xYtNQIDAQAB

[comodo-dodo]
description = Comodo 'Dodo' Log
url = https://dodo.ct.comodo.com/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAELPXCMfVjQ2oWSgrewu4fIW4Sfh3lco90CwKZ061pvAI1eflh6c8ACE90pKM0muBDHCN+j0HV7scco4KKQPqq4A==

[comodo-mammoth]
description = Comodo 'Mammoth' Log
url = https://mammoth.ct.comodo.com/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE7+R9dC4VFbbpuyOL+yy14ceAmEf7QGlo/EmtYU6DRzwat43f/3swtLr/L8ugFOOt1YU/RFmMjGCL17ixv66MZw==

[comodo-sabre]
description = Comodo 'Sabre' Log
url = https://sabre.ct.comodo.com/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE8m/SiQ8/xfiHHqtls9m7FyOMBg4JVZY9CgiixXGz0akvKD6DEL8S0ERmFe9U4ZiA0M4kbT5nmuk3I85Sk4bagA==

[ct-cn]
description = PuChuangSiDa CT Log
url = https://www.certificatetransparency.cn/ct/
key = MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArM8vS3Cs8Q2Wv+gK/kSd1IwXncOaEBGEE+2M+Tdtg+QAb7FLwKaJx2GPmjS7VlLKA1ZQ7yR/S0npNYHd8OcX9XLSI8XjE3/Xjng1j0nemASKY6+tojlwlYRoS5Ez/kzhMhfC8mG4Oo05f9WVgj5WGVBFb8sIMw3VGUIIGkhCEPFow8NBE8sNHtsCtyR6UZZuvAjqaa9t75KYjlXzZeXonL4aR2AwfXqArVaDepPDrpMraiiKpl9jGQy+fHshY0E4t/fodnNrhcy8civBUtBbXTFOnSrzTZtkFJkmxnH4e/hE1eMjIPMK14tRPnKA0nh4NS1K50CZEZU01C9/+V81NwIDAQAB

[digicert-ct1]
description = DigiCert CT Server #1
url = https://ct1.digicert-ct.com/log/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEAkbFvhu7gkAW6MHSrBlpE1n4+HCFRkC5OLAjgqhkTH+/uzSfSl8ois8ZxAD2NgaTZe1M9akhYlrYkes4JECs6A==

[digicert-ct2]
description = DigiCert CT Server #2
url = https://ct2.digicert-ct.com/log/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEzF05L2a4TH/BLgOhNKPoioYCrkoRxvcmajeb8Dj4XQmNY+gxa4Zmz3mzJTwe33i0qMVp+rfwgnliQ/bM/oFmhA==

[gdca-ct]
description = GDCA CT Server #1
url = https://ct.gdca.com.cn/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAErQ8wrZ55pDiJJlSGq0FykG/7yhemrO7Gn30CBexBqMdBnTJJrbA5vTqHPnzuaGxg0Ucqk67hQPQLyDU8HQ9l0w==

[gdca-ctlog]
description = GDCA CT Server #2
url = https://ctlog.gdca.com.cn/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEW0rHAbd0VLpAnEN1lD+s77NxVrjT4nuuobE+U6qXM6GCu19dHAv6hQ289+Wg4CLwoInZCn9fJpTTJOOZLuQVjQ==

[google-aviator]
description = Google 'Aviator' Log
url = https://ct.googleapis.com/aviator/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE1/TMabLkDpCjiupacAlP7xNi0I1JYP8bQFAHDG1xhtolSY1l4QgNRzRrvSe8liE+NPWHdjGxfx3JhTsN9x8/6Q==

[google-deadalus]
description = Google 'Daedalus' Log
url = https://ct.googleapis.com.daedalus/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEbgwcuu4rakGFYB17fqsILPwMCqUIsz7VcCTRbR0ttrfzizbcI02VYxK75IaNzOnR7qFAot8LowYKMMqNrKQpVg==

[google-icarus]
description = Google 'Ikarus' Log
url = https://ct.googleapis.com/icarus/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAETtK8v7MICve56qTHHDhhBOuV4IlUaESxZryCfk9QbG9co/CqPvTsgPDbCpp6oFtyAHwlDhnvr7JijXRD9Cb2FA==

[google-pilot]
description = Google 'Pilot' Log
url = https://ct.googleapis.com/pilot/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEfahLEimAoz2t01p3uMziiLOl/fHTDM0YDOhBRuiBARsV4UvxG2LdNgoIGLrtCzWE0J5APC2em4JlvR8EEEFMoA==

[google-rocketeer]
description = Google 'Rocketeer' Log
url = https://ct.googleapis.com/rocketeer/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEIFsYyDzBi7MxCAC/oJBXK7dHjG+1aLCOkHjpoHPqTyghLpzA9BYbqvnV16mAw04vUjyYASVGJCUoI3ctBcJAeg==

[google-skydiver]
description = Google 'Skydiver' Log
url = https://ct.googleapis.com/skydiver/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEEmyGDvYXsRJsNyXSrYc9DjHsIa2xzb4UR7ZxVoV6mrc9iZB7xjI6+NrOiwH+P/xxkRmOFG6Jel20q37hTh58rA==

[google-submariner]
description = Google 'Submariner' Log
url = https://ct.googleapis.com/submariner/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEOfifIGLUV1Voou9JLfA5LZreRLSUMOCeeic8q3Dw0fpRkGMWV0Gtq20fgHQweQJeLVmEByQj9p81uIW4QkWkTw==

[google-testtube]
description = Google 'Testtbe' Log
url = https://ct.googleapis.com/testtube/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEw8i8S7qiGEs9NXv0ZJFh6uuOmR2Q7dPprzk9XNNGkUXjzqx2SDvRfiwKYwBljfWujozHESVPQyydGaHhkaSz/g==

[izenpe-com]
description = Izenpe CT Log
url = https://ct.izenpe.com/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEJ2Q5DC3cUBj4IQCiDu0s6j51up+TZAkAEcQRF6tczw90rLWXkJMAW7jr9yc92bIKgV8vDXU4lDeZHvYHduDuvg==

[izenpe-eus]
description = Izenpe 'Argi' Log
url = https://ct.izenpe.eus/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE18gOIz6eAjyauAdKKgX/SkuI1IpNOc73xfK2N+mj7eT1RQkOZxT9UyTVOpTy6rUT2R2LXKfD82vYPy07ZXJY1g==

[le-clicky]
description = Let's Encrypt 'Clicky' log
url = https://clicky.ct.letsencrypt.org/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEHxoVg3cAdWK5n/YGBe2ViYNBgZfn4NQz/na6O8lJws3xz/4ScNe+qCJfsqRnAntxrh2sqOnRCNXO7zN6w18A3A==

[nordu-flimsy]
description = Nordu 'flimsy' Log
url = https://flimsy.ct.nordu.net:8080/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE4qWq6afhBUi0OdcWUYhyJLNXTkGqQ9PMS5lqoCgkV2h1ZvpNjBH2u8UbgcOQwqDo66z6BWQJGolozZYmNHE2kQ==

[nordu-plausible]
description = Nordu 'plausible' Log
url = https://plausible.ct.nordu.net/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE9UV9+jO2MCTzkabodO2F7LM03MUBc8MrdAtkcW6v6GA9taTTw9QJqofm0BbdAsbtJL/unyEf0zIkRgXjjzaYqQ==

[sheca-1]
description = SHECA CT log 1
url = https://ctlog.sheca.com/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEEalgK7RxRWbgLt7VhzvV/vCSN/RoxpLdPxrivAwi1pljKW4yKBTAdiyAqCJRkdbrptjx7PAHfrD8dnB2cnyR6Q==

[sheca-2]
description = SHECA CT log 2
url = https://ct.sheca.com/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEsY4diqo6rM6Gy1N26KidWb4XiAMH8ifggr6x/Gc7Ru7T8Y3Wd+ijtNsJXKAJQ/xf0Gg0IyQIwk/Y0rad7dWM2w==

[sofa]
description = Behind the Sofa Log
url = https://ct.filippo.io/behindthesofa/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWTmyppTGMrn+Y2keMDujW9WwQ8lQHpWlLadMSkmOi4+3+MziW5dy1eo/sSFI6ERrf+rvIv/f9F87bXcEsa+Qjw==

[startssl]
description = StartSSL CT Log
url = https://ct.startssl.com/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAESPNZ8/YFGNPbsu1Gfs/IEbVXsajWTOaft0oaFIZDqUiwy1o/PErK38SCFFWa+PeOQFXc9NKv6nV0+05/YIYuUQ==

[symantec-ct]
description = Symantec CT Log
url = https://ct.ws.symantec.com/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEluqsHEYMG1XcDfy1lCdGV0JwOmkY4r87xNuroPS2bMBTP01CEDPwWJePa75y9CrsHEKqAy8afig1dpkIPSEUhg==

[symantec-deneb]
description = Symantec 'Deneb' Log
url = https://deneb.ws.symantec.com/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEloIeo806gIQel7i3BxmudhoO+FV2nRIzTpGI5NBIUFzBn2py1gH1FNbQOG7hMrxnDTfouiIQ0XKGeSiW+RcemA==

[symantec-sirius]
description = Symantec 'Sirius' Log
url = https://sirius.ws.symantec.com/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEowJkhCK7JewN47zCyYl93UXQ7uYVhY/Z5xcbE4Dq7bKFN61qxdglnfr0tPNuFiglN+qjN2Syxwv9UeXBBfQOtQ==

[symantec-vega]
description = Symantec 'Vega' Log
url = https://vega.ws.symantec.com/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE6pWeAv/u8TNtS4e8zf0ZF2L/lNPQWQc/Ai0ckP7IRzA78d0NuBEMXR2G3avTK0Zm+25ltzv9WWis36b4ztIYTQ==

[venafi-gen1]
description = Venafi Generation 1 CT Log
url = https://ctlog.api.venafi.com/
key = MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAolpIHxdSlTXLo1s6H1OCdpSj/4DyHDc8wLG9wVmLqy1lk9fz4ATVmm+/1iN2Nk8jmctUKK2MFUtlWXZBSpym97M7frGlSaQXUWyA3CqQUEuIJOmlEjKTBEiQAvpfDjCHjlV2Be4qTM6jamkJbiWtgnYPhJL6ONaGTiSPm7Byy57iaz/hbckldSOIoRhYBiMzeNoA0DiRZ9KmfSeXZ1rB8y8X5urSW+iBzf2SaOfzBvDpcoTuAaWx2DPazoOl28fP1hZ+kHUYvxbcMjttjauCFx+JII0dmuZNIwjfeG/GBb9frpSX219k1O4Wi6OEbHEr8at/XQ0y7gTikOxBn/s5wQIDAQAB

[venafi-gen2]
description = Venafi Generation 2 CT Log
url = https://ctlog-gen2.api.venafi.com/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEjicnerZVCXTrbEuUhGW85BXx6lrYfA43zro/bAna5ymW00VQb94etBzSg4j/KS/Oqf/fNN51D8DMGA2ULvw3AQ==

[wosign-ct]
description = WoSign CT Server #1
url = https://ct.wosign.com/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE1+wvK3VPN7yjQ7qLZWY8fWrlDCqmwuUm/gx9TnzwOrzi0yLcAdAfbkOcXG6DrZwV9sSNYLUdu6NiaX7rp6oBmw==

[wosign-ctlog]
description = WoSign CT Server #2
url = https://ctlog.wosign.com/
key = MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEzBGIey1my66PTTBmJxklIpMhRrQvAdPG+SvVyLpzmwai8IoCnNBrRhgwhbrpJIsO0VtwKAx+8TpFf1rzgkJgMQ==

Things to note when testing the patch:

• Be sure the CT log server you used to issue the SCT is listed (and enabled) in the config file.
• The SCT files reside all in one directory listed in the server config
• The SCT files contain only the raw SCT structure (beginning with the version id field).

If in doubt, ask nginx to log level debug and you should receive all the information you need (I chose to log the verification results inside the loop, despite this repeating quite a bit of log).

[BenBE]

Any updates on this issue/PR?