Infrastructure

I'm personally not fond of the official ACME client (somewhat bloated, too big
attack surface, no privilege separation, etc.) so I wrote my own earlier this
year:

https://tracker.debian.org/pkg/lacme

I mean

https://packages.debian.org/sid/lacme

You are the expert clearly :-) Some food for thought however:

• we should create two sets of private (4096 RSA) keys for each service
  • this helps key transition in case of regular change or compromised sytems, and e.g. HPKP requires two key pins to be mentioned (not sure about TLSA though, but it supports this at least)
  • e-mail should be hostmaster@tdf to get the notifications to the right list
  • SHA-512 where possible, not sure if Let's Encrypt gives any choice
  • when renewing a certificate, we should keep the private key, so TLSA/HPKP doesn't go havoc

Florian Effenberger wrote:

• we should create two sets of private (4096 RSA) keys for each service
  • this helps key transition in case of regular change or compromised sytems, and e.g. HPKP requires two key pins to be mentioned (not sure about TLSA though, but it supports this at least)

Agreed, but IMHO that's orthogonal to choosing Let's Encrypt as CA. While we do want to pre-generate spare private keys in the event of a compromise, so pinning clients (eg, through HPKP or TLSA) won't refuse to connect, generating the X.509 cert can always be done at a later time. In fact, I'd argue that the spare keys should not be kept on the same machine, so they are not compromised in the event of a privilege escalation attack; pre-generating the certs means we would have to take care of renewals as well, which is cumbersome when the private key isn't on the public service itself (this requires a CSR dance).

Also, I think we should two spare keys: another RSA 4096 (like the one in use), and an ECDSA key. This would ease transition in case of security concerns regarding a particular algorithm.

• e-mail should be hostmaster@tdf to get the notifications to the right list

If you mean the contact info on the account key, sure. If you are talking about a mailto: URI in the X509v3 SAN, last time I checked this was not supported by ACME (there is challenge type to verify this).

• SHA-512 where possible, not sure if Let's Encrypt gives any choice

This is up to the CA, and ACME doesn't let the user set a preference. The current signature algorithm is SHA-256.

• when renewing a certificate, we should keep the private key, so TLSA/HPKP doesn't go havoc

Of course :-)

Some setting proposals (please review and adapt as needed):

• dhparams
  

  openssl dhparam -out /etc/ssl/dhparams.pem 4096
  chmod 400 /etc/ssl/dhparams.pem
  chown root:root /etc/ssl/dhparams.pem
  

• nginx
  

   ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
   ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-   SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
   ssl_prefer_server_ciphers on;
   ssl_session_cache shared:ssl_session_cache:10m;
   ssl_dhparam /etc/ssl/dhparams.pem;
  

• OpenSSL
  

  KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
  Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
  MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-ripemd160-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160,umac-128@openssh.com
  RekeyLimit 64M
  

and

ssh-keygen -G /etc/ssh/moduli.all -b 4096
ssh-keygen -T /etc/ssh/moduli.safe -f /etc/ssh/moduli.all
mv /etc/ssh/moduli.safe /etc/ssh/moduli
rm /etc/ssh/moduli.all
sed -i /"ssh_host_dsa_key"/d /etc/ssh/sshd_config
sed -i /"ssh_host_ecdsa_key"/d /etc/ssh/sshd_config

Florian Effenberger wrote:

Some setting proposals (please review and adapt as needed):

• dhparams
  openssl dhparam -out /etc/ssl/dhparams.pem 4096

I agree that we should generate our own DH parameters of size >1024. However I'm sceptical about size 4096: it'll make non-elliptic ephemeral key exchange a lot slower (8x compared to 2048) with non obvious security benefits. Furthermore AFAIK not all clients support support such large primes; and modern clients will use ECDHE anyway so they are not the concern here.

chmod 400 /etc/ssl/dhparams.pem
chown root:root /etc/ssl/dhparams.pem

There is no harm in leaving the DH parameters public. They're shared with the client anyway. Furthermore, the above sequence (creation followed by chmod followed by chown) is racy. My favorite replacement is

install -oroot -groot -m0400 /dev/null /path/to/file
    openssl ... -out /path/to/file

• nginx
  [...]

I'd also disable RFC4507 ticket extension, add an HSTS header, and enable OCSP stapling.

ssl_session_tickets off;
    add_header Strict-Transport-Security 'max-age=15768000';
    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_trusted_certificate /path/to/lets-encrypt-x3-cross-signed.pem;

Mozilla has a great online tool to generate the SSL config based on the nginx and OpenSSL versions.

https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=nginx-1.6.2&openssl=1.0.1k&hsts=yes&profile=intermediate

• OpenSSL
  RekeyLimit 64M

Why? The maximum amount that can be processed depends on the cipher, but it's usually at least 4GB. Furthermore frequent rekeying will lead to a significant overhead (for dubious security gain IMHO), especially when using slow kex algorithms such as DHE.

Agreed, but IMHO that's orthogonal to choosing Let's Encrypt as CA.
While we do want to pre-generate spare private keys in the event of a
compromise, so pinning clients (eg, through HPKP or TLSA) won't refuse
to connect, generating the X.509 cert can always be done at a later
time. In fact, I'd argue that the spare keys should not be kept on the
same machine, so they are not compromised in the event of a privilege
escalation attack; pre-generating the certs means we would have to take
care of renewals as well, which is cumbersome when the private key isn't
on the public service itself (this requires a CSR dance).

Leaving that to you, my time is more than used before Christmas... ;-)

Also, I think we should two spare keys: another RSA 4096 (like the one
in use), and an ECDSA key. This would ease transition in case of
security concerns regarding a particular algorithm.

Dito, see above.

If you mean the contact info on the account key, sure. If you are

Yep, this one. :)

This is up to the CA, and ACME doesn't let the user set a preference.
The current signature algorithm is SHA-256.

Ok.

I agree that we should generate our own DH parameters of size >1024.
However I'm sceptical about size 4096: it'll make non-elliptic ephemeral
key exchange a lot slower (8x compared to 2048) with non obvious
security benefits. Furthermore AFAIK not all clients support support
such large primes; and modern clients will use ECDHE anyway so they are
not the concern here.

chmod 400 /etc/ssl/dhparams.pem
chown root:root /etc/ssl/dhparams.pem

There is no harm in leaving the DH parameters public. They're shared
with the client anyway. Furthermore, the above sequence (creation
followed by chmod followed by chown) is racy. My favorite replacement is

install -oroot -groot -m0400 /dev/null /path/to/file openssl ... -out /path/to/file

I'd also disable RFC4507 ticket extension, add an HSTS header, and
enable OCSP stapling.

Leaving that to you, too busy atm. ;-)

add_header Strict-Transport-Security 'max-age=15768000';

We should do this one on a per-service basis to not exclude legit
non-HTTPS uses. Not sure if there are any, but let's be very cautious
here to not lock anyone out.

Why? The maximum amount that can be processed depends on the cipher, but
it's usually at least 4GB. Furthermore frequent rekeying will lead to a
significant overhead (for dubious security gain IMHO), especially when
using slow kex algorithms such as DHE.

Took that from some documentation, but your call :-)

Alright, I'm done with the salt config so deployment should be a lot easier now :-)

For the record, Qualys SSL labs gives us an A for <https://survey.documentfoundation.org&gt;:

https://www.ssllabs.com/ssltest/analyze.html?d=survey.documentfoundation.org&s=89.238.68.136

AFAICT, the only room for improvement (since we want to keep compatibility with somewhat old browsers) is the addition of HSTS and HPKP, but as we said earlier we'll take care of those later.

Oops, forgot the question. That's a good opportunity for key rotation, no? Right now there are very few certs (hence key pairs) in place, but since ACME is free, easy to use, and doesn't support wildcard certificates, I think generating a dedicated key pair for each service is more appropriate with Let's Encrypt. If no one objects I'll start making issuance requests tomorrow evening.

Alright, I'm done with the salt config so deployment should be a lot
easier now :-)

For the record, Qualys SSL labs gives us an A for
<https://survey.documentfoundation.org&amp;gt;:

Fantastic! :-)

AFAICT, the only room for improvement (since we want to keep
compatibility with somewhat old browsers) is the addition of HSTS and
HPKP, but as we said earlier we'll take care of those later.

Agreed, let's defer that and decide on a per-service basis next year.

Oops, forgot the question. That's a good opportunity for key rotation,
no? Right now there are very few certs (hence key pairs) in place, but
since ACME is free, easy to use, and doesn't support wildcard
certificates, I think generating a dedicated key pair for each service
is more appropriate with Let's Encrypt. If no one objects I'll start
making issuance requests tomorrow evening.

Fine for me, I think keys are not stored client-side anywhere at the
moment. Only thing is, before touching productive services, we should
agree on a maintenance window and inform people beforehand, to avoid
outages before the Christmas period, when everyone is in a rush and
releases are coming up

Florian Effenberger wrote:

Only thing is, before touching productive services, we should agree
on a maintenance window and inform people beforehand, to avoid outages
before the Christmas period, when everyone is in a rush and releases
are coming up

Good point :-) However we have quite a few certs to roll out, and to avoid
hitting the 20-certs per domain limit I think we should start next week at
the latest if we want to migrate everything before the end of the year.

Where should I announce the planned maintenance? I can do this some night
while everyone is sleeping (on this side of the big pond at least), but I
guess there is also a fixed time for this?

Good point :-) However we have quite a few certs to roll out, and to avoid
hitting the 20-certs per domain limit I think we should start next week at
the latest if we want to migrate everything before the end of the year.

Sure, sounds sensible. Can you come up with a list of services to start
with? Ideally, we take those not so important as others - I'd leave out
gerrit, jenkins and bugzilla for the moment, and the pumbaa stuff maybe.

Where should I announce the planned maintenance? I can do this some night
while everyone is sleeping (on this side of the big pond at least), but I
guess there is also a fixed time for this?

Can you sync with Cloph on that? IMHO we sent to projects@, website@ and
affected lists, e.g. l10n@ for pootle, or the dev@ list for
jenkins&gerrit, the qa list for Bugzilla etc.

There's also a Github-hosted status page Cloph knows how to update.

So on Dec 25 night I requested and deployed certs for

https://devcentral.libreoffice.org
    https://perf.libreoffice.org
    https://crashreport.libreoffice.org
    https://translations.documentfoundation.org

My ACME client doesn't work out of the box on older Ubuntu, so I'll hurry up and migrate those to Debian 8 first.

Also, the website uses a wildcard in its virtual hostname, but Let's Encrypt won't issue certs for wildcard domains. We therefore need to have the full list of valid hostnames before performing the issuance request. Currently we have

server_name www.documentfoundation.org wwwtest.documentfoundation.org *.documentfoundation.org;

There is also a very long list of server_names I might have to split up due to the limit on the number of SAN in the CSR.

server_name ~?documentliberation.(org|net|info|biz|com) ~(www\.)?document-liberation.(org|net|info|biz|com) ~?documentsliberation.(org|net|info|biz|com) ~(www\.)?documents-liberation.(org|net|info|biz|com);

We therefore need to have the full list of valid hostnames before
performing the issuance request.

Send you the DNS zones yesterday ;-)

What's the current status of this?

• We do have Let's Encrypt for all hosts except the legacy ones that need migrating?
• These are automatically renewed ~30 days before they expire? (Except for pumbaa, which I handle manually for now)
• How is renewal handled? Via Cron? Salt?

Florian Effenberger wrote:

• We do have Let's Encrypt for all hosts except the legacy ones that need migrating?

Correct (for all production services in the documentfoundation.org or libreoffice.org domains, at least):

https://crt.sh/?Identity=%25documentfoundation.org&iCAID=16418&exclude=expired
  https://crt.sh/?Identity=%25libreoffice.org&iCAID=16418&exclude=expired

I also requested certs for other domains such as tdf.io or documentliberation.org. Generally, all http vhosts served by nginx or apache2 and hosted on salted machines should be covered.

• These are automatically renewed ~30 days before they expire? (Except for pumbaa, which I handle manually for now)
• How is renewal handled? Via Cron? Salt?

A cronjob is run each day at 3h07 machine time. Only certs expiring < 10 days (864000 secs) ahead are then automatically renewed, and the relevant service reloaded/restarted if need be. Private key material is always preserved.

Is this in Salt already? Can this ticket be closed, and we handle it via Salt when legacy hosts are migrated anyways?