Project

General

Profile

Task #2115

Request certs from Let's Encrypt for each HTTPS vhost

Added by Guilhem Moulin about 1 year ago. Updated 8 months ago.

Status:
Closed
Priority:
High
Category:
-
Target version:
Team - Q1/2017
Start date:
Due date:
% Done:

100%

Estimated time:
Tags:
URL:

Description

(Cf. floeff's mail on tdf-staff@ and the follow-ups.)

I'm personally not fond of the official ACME client (somewhat bloated, too big attack surface, no privilege separation, etc.)
so I wrote my own earlier this year:

https://tracker.debian.org/pkg/lacme

Let's Encrypt doesn't support wildcards, so the plan would be to have a separate cert for each service. One thing to keep in mind though, is the rate-limiting on Let's Encrypt's ACME server: currently 20 certificates per registered domain, excluding renewals.

https://letsencrypt.org/docs/rate-limits/

Then the next step is to set up 301 redirection from http:// to https:// everywhere :-)


Related issues

Related to Infrastructure - Task #2026: scan sites with observatory.mozilla.orgNew

History

#1 Updated by Guilhem Moulin about 1 year ago

I'm personally not fond of the official ACME client (somewhat bloated, too big
attack surface, no privilege separation, etc.) so I wrote my own earlier this
year:

https://tracker.debian.org/pkg/lacme

I mean

https://packages.debian.org/sid/lacme

#2 Updated by Florian Effenberger about 1 year ago

  • Related to Task #2026: scan sites with observatory.mozilla.org added

#3 Updated by Florian Effenberger about 1 year ago

You are the expert clearly :-) Some food for thought however:

  • we should create two sets of private (4096 RSA) keys for each service
    • this helps key transition in case of regular change or compromised sytems, and e.g. HPKP requires two key pins to be mentioned (not sure about TLSA though, but it supports this at least)
    • e-mail should be hostmaster@tdf to get the notifications to the right list
    • SHA-512 where possible, not sure if Let's Encrypt gives any choice
    • when renewing a certificate, we should keep the private key, so TLSA/HPKP doesn't go havoc

#4 Updated by Guilhem Moulin about 1 year ago

Florian Effenberger wrote:

  • we should create two sets of private (4096 RSA) keys for each service
    • this helps key transition in case of regular change or compromised sytems, and e.g. HPKP requires two key pins to be mentioned (not sure about TLSA though, but it supports this at least)

Agreed, but IMHO that's orthogonal to choosing Let's Encrypt as CA. While we do want to pre-generate spare private keys in the event of a compromise, so pinning clients (eg, through HPKP or TLSA) won't refuse to connect, generating the X.509 cert can always be done at a later time. In fact, I'd argue that the spare keys should not be kept on the same machine, so they are not compromised in the event of a privilege escalation attack; pre-generating the certs means we would have to take care of renewals as well, which is cumbersome when the private key isn't on the public service itself (this requires a CSR dance).

Also, I think we should two spare keys: another RSA 4096 (like the one in use), and an ECDSA key. This would ease transition in case of security concerns regarding a particular algorithm.

  • e-mail should be hostmaster@tdf to get the notifications to the right list

If you mean the contact info on the account key, sure. If you are talking about a mailto: URI in the X509v3 SAN, last time I checked this was not supported by ACME (there is challenge type to verify this).

  • SHA-512 where possible, not sure if Let's Encrypt gives any choice

This is up to the CA, and ACME doesn't let the user set a preference. The current signature algorithm is SHA-256.

  • when renewing a certificate, we should keep the private key, so TLSA/HPKP doesn't go havoc

Of course :-)

#5 Updated by Florian Effenberger about 1 year ago

Some setting proposals (please review and adapt as needed):

  • dhparams
    openssl dhparam -out /etc/ssl/dhparams.pem 4096
    chmod 400 /etc/ssl/dhparams.pem
    chown root:root /etc/ssl/dhparams.pem
    
  • nginx
     ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
     ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-   SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
     ssl_prefer_server_ciphers on;
     ssl_session_cache shared:ssl_session_cache:10m;
     ssl_dhparam /etc/ssl/dhparams.pem;
    
  • OpenSSL
    KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
    Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
    MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-ripemd160-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160,umac-128@openssh.com
    RekeyLimit 64M
    

and

ssh-keygen -G /etc/ssh/moduli.all -b 4096
ssh-keygen -T /etc/ssh/moduli.safe -f /etc/ssh/moduli.all
mv /etc/ssh/moduli.safe /etc/ssh/moduli
rm /etc/ssh/moduli.all
sed -i /"ssh_host_dsa_key"/d /etc/ssh/sshd_config
sed -i /"ssh_host_ecdsa_key"/d /etc/ssh/sshd_config

#6 Updated by Guilhem Moulin about 1 year ago

Florian Effenberger wrote:

Some setting proposals (please review and adapt as needed):

  • dhparams
    openssl dhparam -out /etc/ssl/dhparams.pem 4096

I agree that we should generate our own DH parameters of size >1024. However I'm sceptical about size 4096: it'll make non-elliptic ephemeral key exchange a lot slower (8x compared to 2048) with non obvious security benefits. Furthermore AFAIK not all clients support support such large primes; and modern clients will use ECDHE anyway so they are not the concern here.

chmod 400 /etc/ssl/dhparams.pem
chown root:root /etc/ssl/dhparams.pem

There is no harm in leaving the DH parameters public. They're shared with the client anyway. Furthermore, the above sequence (creation followed by chmod followed by chown) is racy. My favorite replacement is

install -oroot -groot -m0400 /dev/null /path/to/file
openssl ... -out /path/to/file
  • nginx
    [...]

I'd also disable RFC4507 ticket extension, add an HSTS header, and enable OCSP stapling.

ssl_session_tickets off;
add_header Strict-Transport-Security 'max-age=15768000';
ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /path/to/lets-encrypt-x3-cross-signed.pem;

Mozilla has a great online tool to generate the SSL config based on the nginx and OpenSSL versions.

https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=nginx-1.6.2&openssl=1.0.1k&hsts=yes&profile=intermediate
  • OpenSSL
    RekeyLimit 64M

Why? The maximum amount that can be processed depends on the cipher, but it's usually at least 4GB. Furthermore frequent rekeying will lead to a significant overhead (for dubious security gain IMHO), especially when using slow kex algorithms such as DHE.

#7 Updated by Florian Effenberger about 1 year ago

Agreed, but IMHO that's orthogonal to choosing Let's Encrypt as CA.
While we do want to pre-generate spare private keys in the event of a
compromise, so pinning clients (eg, through HPKP or TLSA) won't refuse
to connect, generating the X.509 cert can always be done at a later
time. In fact, I'd argue that the spare keys should not be kept on the
same machine, so they are not compromised in the event of a privilege
escalation attack; pre-generating the certs means we would have to take
care of renewals as well, which is cumbersome when the private key isn't
on the public service itself (this requires a CSR dance).

Leaving that to you, my time is more than used before Christmas... ;-)

Also, I think we should two spare keys: another RSA 4096 (like the one
in use), and an ECDSA key. This would ease transition in case of
security concerns regarding a particular algorithm.

Dito, see above.

If you mean the contact info on the account key, sure. If you are

Yep, this one. :)

This is up to the CA, and ACME doesn't let the user set a preference.
The current signature algorithm is SHA-256.

Ok.

#8 Updated by Florian Effenberger about 1 year ago

I agree that we should generate our own DH parameters of size >1024.
However I'm sceptical about size 4096: it'll make non-elliptic ephemeral
key exchange a lot slower (8x compared to 2048) with non obvious
security benefits. Furthermore AFAIK not all clients support support
such large primes; and modern clients will use ECDHE anyway so they are
not the concern here.

chmod 400 /etc/ssl/dhparams.pem
chown root:root /etc/ssl/dhparams.pem

There is no harm in leaving the DH parameters public. They're shared
with the client anyway. Furthermore, the above sequence (creation
followed by chmod followed by chown) is racy. My favorite replacement is

install -oroot -groot -m0400 /dev/null /path/to/file
openssl ... -out /path/to/file

I'd also disable RFC4507 ticket extension, add an HSTS header, and
enable OCSP stapling.

Leaving that to you, too busy atm. ;-)

add_header Strict-Transport-Security 'max-age=15768000';

We should do this one on a per-service basis to not exclude legit
non-HTTPS uses. Not sure if there are any, but let's be very cautious
here to not lock anyone out.

Why? The maximum amount that can be processed depends on the cipher, but
it's usually at least 4GB. Furthermore frequent rekeying will lead to a
significant overhead (for dubious security gain IMHO), especially when
using slow kex algorithms such as DHE.

Took that from some documentation, but your call :-)

#9 Updated by Guilhem Moulin about 1 year ago

Alright, I'm done with the salt config so deployment should be a lot easier now :-)

For the record, Qualys SSL labs gives us an A for <https://survey.documentfoundation.org&gt;:

https://www.ssllabs.com/ssltest/analyze.html?d=survey.documentfoundation.org&s=89.238.68.136

AFAICT, the only room for improvement (since we want to keep compatibility with somewhat old browsers) is the addition of HSTS and HPKP, but as we said earlier we'll take care of those later.

#10 Updated by Guilhem Moulin about 1 year ago

Oops, forgot the question. That's a good opportunity for key rotation, no? Right now there are very few certs (hence key pairs) in place, but since ACME is free, easy to use, and doesn't support wildcard certificates, I think generating a dedicated key pair for each service is more appropriate with Let's Encrypt. If no one objects I'll start making issuance requests tomorrow evening.

#11 Updated by Florian Effenberger about 1 year ago

Alright, I'm done with the salt config so deployment should be a lot
easier now :-)

For the record, Qualys SSL labs gives us an A for
<https://survey.documentfoundation.org&gt;:

Fantastic! :-)

AFAICT, the only room for improvement (since we want to keep
compatibility with somewhat old browsers) is the addition of HSTS and
HPKP, but as we said earlier we'll take care of those later.

Agreed, let's defer that and decide on a per-service basis next year.

#12 Updated by Florian Effenberger about 1 year ago

Oops, forgot the question. That's a good opportunity for key rotation,
no? Right now there are very few certs (hence key pairs) in place, but
since ACME is free, easy to use, and doesn't support wildcard
certificates, I think generating a dedicated key pair for each service
is more appropriate with Let's Encrypt. If no one objects I'll start
making issuance requests tomorrow evening.

Fine for me, I think keys are not stored client-side anywhere at the
moment. Only thing is, before touching productive services, we should
agree on a maintenance window and inform people beforehand, to avoid
outages before the Christmas period, when everyone is in a rush and
releases are coming up

#13 Updated by Guilhem Moulin about 1 year ago

Florian Effenberger wrote:

Only thing is, before touching productive services, we should agree
on a maintenance window and inform people beforehand, to avoid outages
before the Christmas period, when everyone is in a rush and releases
are coming up

Good point :-) However we have quite a few certs to roll out, and to avoid
hitting the 20-certs per domain limit I think we should start next week at
the latest if we want to migrate everything before the end of the year.

Where should I announce the planned maintenance? I can do this some night
while everyone is sleeping (on this side of the big pond at least), but I
guess there is also a fixed time for this?

#14 Updated by Florian Effenberger about 1 year ago

Good point :-) However we have quite a few certs to roll out, and to avoid
hitting the 20-certs per domain limit I think we should start next week at
the latest if we want to migrate everything before the end of the year.

Sure, sounds sensible. Can you come up with a list of services to start
with? Ideally, we take those not so important as others - I'd leave out
gerrit, jenkins and bugzilla for the moment, and the pumbaa stuff maybe.

Where should I announce the planned maintenance? I can do this some night
while everyone is sleeping (on this side of the big pond at least), but I
guess there is also a fixed time for this?

Can you sync with Cloph on that? IMHO we sent to projects@, website@ and
affected lists, e.g. l10n@ for pootle, or the dev@ list for
jenkins&gerrit, the qa list for Bugzilla etc.

There's also a Github-hosted status page Cloph knows how to update.

#15 Updated by Guilhem Moulin 11 months ago

So on Dec 25 night I requested and deployed certs for

https://devcentral.libreoffice.org
https://perf.libreoffice.org
https://crashreport.libreoffice.org
https://translations.documentfoundation.org

My ACME client doesn't work out of the box on older Ubuntu, so I'll hurry up and migrate those to Debian 8 first.

Also, the website uses a wildcard in its virtual hostname, but Let's Encrypt won't issue certs for wildcard domains. We therefore need to have the full list of valid hostnames before performing the issuance request. Currently we have

server_name www.documentfoundation.org wwwtest.documentfoundation.org *.documentfoundation.org;

There is also a very long list of server_names I might have to split up due to the limit on the number of SAN in the CSR.

server_name ~?documentliberation.(org|net|info|biz|com) ~(www\.)?document-liberation.(org|net|info|biz|com) ~?documentsliberation.(org|net|info|biz|com) ~(www\.)?documents-liberation.(org|net|info|biz|com);

#16 Updated by Florian Effenberger 11 months ago

We therefore need to have the full list of valid hostnames before
performing the issuance request.

Send you the DNS zones yesterday ;-)

#17 Updated by Florian Effenberger 11 months ago

  • Target version changed from Q4/2016 to Q1/2017

#18 Updated by Florian Effenberger 10 months ago

What's the current status of this?

  • We do have Let's Encrypt for all hosts except the legacy ones that need migrating?
  • These are automatically renewed ~30 days before they expire? (Except for pumbaa, which I handle manually for now)
  • How is renewal handled? Via Cron? Salt?

#19 Updated by Guilhem Moulin 10 months ago

Florian Effenberger wrote:

  • We do have Let's Encrypt for all hosts except the legacy ones that need migrating?

Correct (for all production services in the documentfoundation.org or libreoffice.org domains, at least):

https://crt.sh/?Identity=%25documentfoundation.org&iCAID=16418&exclude=expired
https://crt.sh/?Identity=%25libreoffice.org&iCAID=16418&exclude=expired

I also requested certs for other domains such as tdf.io or documentliberation.org. Generally, all http vhosts served by nginx or apache2 and hosted on salted machines should be covered.

  • These are automatically renewed ~30 days before they expire? (Except for pumbaa, which I handle manually for now)
  • How is renewal handled? Via Cron? Salt?

A cronjob is run each day at 3h07 machine time. Only certs expiring < 10 days (864000 secs) ahead are then automatically renewed, and the relevant service reloaded/restarted if need be. Private key material is always preserved.

#20 Updated by Florian Effenberger 8 months ago

Is this in Salt already? Can this ticket be closed, and we handle it via Salt when legacy hosts are migrated anyways?

#21 Updated by Guilhem Moulin 8 months ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

Yeah, it's in salt since late 2016 :-)

Also available in: Atom PDF