TL;DR The web server software Caddy leaked information on which SSL certificates were on each installation through enumeration. We submitted a bug report, built a proof of concept, submitted a CVE and the developer of Caddy Matt Holt fixed it and released Caddy 0.11.1.
Background
Caddy, also known as the Caddy web server, is an alternative to the classic Apache. It is an open source web server written in Go.
The first release of this web server was in April 2015 and included full support for HTTP/2. Caddy was the first web server that shipped with Let’s Encrypt support and enabled HTTPS for all domains by default. Caddy gained popularity with this “Secure by default” approach.
A few months ago, researchers Paul Buonopane (NamePros) and German Hoeffner (SecurityTrails) discovered that Caddy web server yields a random installed SSL certificate while accessing interfaces not actively configured to serve HTTPS.
A closer look at Caddy’s hostname enumeration problem
The bug was first exposed when we checked the certificate on a hosts IP that had a “vhost”-like configuration, housing multiple domains. Every time we would refresh, we were greeted with a different certificate. Looking for an explanation and proposed solution we found issue #1303 (Dec 19, 2016) on Caddy’s GitHub repository and Paul suggested (Jul 30, 2018) revising this issue.
What’s the problem?!
In issue #1303 Matt Holt stated that “[he is] not convinced that certificates are private — in fact, they are literally public keys”. We do not disagree — a certificate is meant to be public afterall. However, this isn’t the actual problem. Being able to enumerate hosts means that you can easily find relations that you otherwise couldn’t find easily — and that can even be dangerous to people’s lives in some cases. Let’s say someone in a country without free speech hosts his blog and several work or private related websites and puts them on different IPs — using the enumeration bug in Caddy you can prove a relationship.
Another example are administrators that put their websites behind a reverse proxy to avoid receiving DDoS attacks — you are able to quickly find their real IP by probing all Caddy hosts. 19.45% of all Caddy websites we enumerated during our proof of concept were hidden behind Cloudflare.
Vulnerability history
Matt Holt informed us that this bug wasn’t related to what was originally reported on issue #1303 dating back to (Dec 19, 2016). A similar issue from Dec 11, 2018, #423 describing the same behavior we saw also turned out to be unrelated. It is unclear how long it has been persistent in the code, but we believe it has been around for at least 6 months.
The proof of concept
Esteban Borges from SecurityTrails created a general workflow of testing and researching. We describe it in the proof of concept below. All private information has been removed.
A one-line command was needed to fetch different SSLs and host information:
echo | openssl s_client -showcerts -servername IP -connect IP:443 2>/dev/null | openssl x509 -inform pem -noout -text 2>/dev/nullWhere IP is the IP address of the Caddy web server.
To begin, we started testing IPs manually:
[[email protected] ~]$ echo | openssl s_client -showcerts -servername IP -connect IP:443 2>/dev/null | openssl x509 -inform pem -noout -text | grep 'CN\|Alt\|DNS'
Issuer: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
Subject: CN = <redacted domain>
X509v3 Subject Alternative Name:
DNS:<redacted domain>Let’s try again, with the same command against the same IP:
[[email protected] ~]$ echo | openssl s_client -showcerts -servername IP -connect IP:443 2>/dev/null | openssl x509 -inform pem -noout -text | grep 'CN\|Alt\|DNS'
Issuer: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
Subject: CN = <different redacted domain>
X509v3 Subject Alternative Name:
DNS:<different redacted domain>And we got another different SSL hostname. Let’s try for a third time now:
[[email protected] ~]$ echo | openssl s_client -showcerts -servername IP -connect IP:443 2>/dev/null | openssl x509 -inform pem -noout -text | grep 'CN\|Alt\|DNS'
Issuer: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
Subject: CN = <another redacted domain>
X509v3 Subject Alternative Name:
DNS:<another redacted domain>As you can see, we were able to reproduce the bug.
Using the SecurityTrails Feed of technology data of all public servers running on port 443 we filtered 19,710 different IPs that were running Caddy web server at the time of probing. Using this data we would be able to enumerate all hosts rather quickly, as an enumeration usually takes less than a second in perfect conditions.
Once we found a way to reproduce the bug, we used what was basically the same command, with a few modifications to integrate it in a “for” loop bash script that was querying all the 19,710 server IP addresses.
for i in $(cat caddyips.txt); do
echo | timeout 5 openssl s_client -showcerts -servername $i -connect $i:443 2>/dev/null | openssl x509 -inform pem -noout -text 2>/dev/null | grep 'CN|Alt|DNS' >> /tmp/results.txt
doneAfter working with tr, sed, and awk to remove duplicates, the result was discovering 38,907 hosts that are actually using Caddy web server:
[[email protected]:~]wc -l results.txt
38907 results.txtSome interesting sites we discovered by analyzing the hosts
Of course such scans always yield some interesting results. The scan showed that government websites from Argentina, Brazil, Taiwan, Dominican Republic, Zambia and the US are hosted on Caddy web server (7 results). We also found educational institutions like MIT, Stanford University, University of Oregon, Brown University, Imperial Valley College using Caddy web server (14 results).
Also revealed during the research was “phpmyadmin” installations on “phpmyadmin” subdomains as well as Caddy’s own website, telemetry and staging telemetry hostnames.
The workaround
We found a workaround shortly after discovering the issue. By explicitly defining a self-signed certificate for :443 Caddy wouldn’t reveal random certificates anymore. This was as easy as defining the following in the Caddyfile:
:443 {
root /srv/www/_default_/public_html
# Self-signed cert to not expose hostname
tls self_signed
}However, in our eyes this wasn’t a feasible solution for a “Secure by Default” web server. That’s why we discussed the issue with further researchers on Nov 9th, 2018 and ultimately got CVE-2018-19148 assigned for it. We calculated a CVSS v3.0 base score of 5.3 (medium) for this vulnerability.
The solution
Matt Holt committed a fix to the GitHub repository (Nov 12, 2018). A new version of caddy, 0.11.1 has been released and can be downloaded from Caddy’s website.
Thanks to Paul Buonopane from NamePros for his research and contributions. A very special thanks to Matt Holt and Toby Allen for verifying and fixing the bug and their contributions to open source.
You can find further information below
- GitHub Issue #1303
- CVE-2018-19148 (on nvd.nist.gov)
- Fix in commit f6e50890b3d81cb04146bd7b2c2b59e99830849a
