Subdomain-Enum: Enumerating Subdomains with the SecurityTrails API™
Reading time: 11 minutes
One of the core products at SecurityTrails is our API. And to ensure that its usage is simple and user-friendly, we strive to follow industry best practices and standardization that the user base will be familiar with.
We take this even further by offering extensive documentation that is easy to understand as well as interactive. On the documentation page, you can test API results in the browser to see the outcome.
The interactive API calls can also show beginners how to construct the correct URL and call the needed API in order to fetch the desired data. Once you understand how the different API calls work, you can browse through code snippets for your favorite language (Python, Java, PHP, Go, Ruby, Node, and JavaScript are supported) and use these snippets locally or build on them to create your own OSINT investigative tool.
What is Subdomain-Enum?
The above outline of our API leads us to an independent developer (Chaitanya Krishna), building his own script to enumerate subdomains using the SecurityTrails API™. This tool is called: Subdomain-Enum.
This interactive script can find all associated subdomains with a given domain and saves the results to a CSV. Written in Python, the script is very easy to understand for anybody familiar with writing code. We'll demonstrate this by attempting to extend or modify the script for different purposes, using the SecurityTrails API™.
Who should use it?
Subdomain enumeration is a great technique for anyone looking to find subdomains quickly during a reconnaissance or data-gathering task.
Whether you're on the offensive or defensive side of your target, enumerating subdomains can help in getting a broader idea about commonly used hosts but also it may discover long-forgotten hosts that are still connected and silently listening.
To name one notorious case affected by this kind of deception technique, here's the one perpetrated on PricewaterhouseCoopers (PwC) subdomain amyza-devapi.pwc.com as the following image shows:
As you can see in the Google search above, this subdomain was filled with fake content that actually had malware hosted. This was possible due to a broken CNAME record that pointed to an expired domain name registered by the deceiver. This is also known as subdomain takeover.
By accomplishing this, they managed to use a good standing subdomain name containing a reputable apex domain, and redirected users into a malware trap hosted at amyca-dev-node.azurewebsites.net. These are called stale DNS records.
Installing Subdomain-Enum
Let's now proceed with installing the Python script. And remember, it's always important to use some type of sandboxing environment when installing new software. You could opt for a virtual machine (VM), container, or a remote test server. We are using Ubuntu 18.04 for this review, and any commands used here should apply to Debian-based distros (and with a few minor tweaks, to other distros as well).
First, we should update our sandbox and install the software we need:
sudo apt update
sudo apt upgrade
sudo apt install git python3-venv
Now we can make a Python virtual environment to install the necessary Python packages:
python3 -m venv strails
cd strails/
source bin/activate
pip install requests
git clone https://github.com/chaitanyakrishna/subdomain-enum
cd subdomain-enum/
We noticed that the repo for Subdomain-Enum does not have a 'requirements.txt' file which specifies the required packages. After taking a look at the script, the only 3rd-party Python library we need is 'requests' (which we installed above).
In terms of Python virtual environments, there are many ways to install and use it. Any installation method that gives you a 'virtual-env' should enable you to use the instructions below as is. If you opt for installing the necessary Python packages via 'apt', you might need to adapt the scripts below (this is as yet unverified).
If you haven't already, get your free API key today. With the API key ready, we can now add it to the script on line 12 of the subdomain-enum.py file:
vim subdomain-enum.py
#!/usr/local/bin/python
import requests
import json
import os
print ("\nSubdomain Enumeration Script\n")
def get_sub_domains(domain,filepath):
url = "https://api.securitytrails.com/v1/domain/"+domain+"/subdomains"
#print(url)
querystring = {"children_only":"true"}
headers = {
'accept': "application/json",
'apikey': "YOUR-SECURITYTRAILS-API-KEY"
}
response = requests.request("GET", url, headers=headers, params=querystring)
result_json=json.loads(response.text)
sub_domains=[i+'.'+domain for i in result_json['subdomains']]
f=open(filepath,'w+')
for i in sub_domains:
f.write(i+'\n')
f.close()
return sub_domains
domain = input("\nEnter Domain name : ")
filepath = input("\nPlease provide a file name to save : ")
get_sub_domains(domain,filepath)
Once your API key is stored and saved in the file, we can now attempt to execute the script.
Starting subdomain mapping
For our first example, we'll try a random domain with a few subdomains:
python subdomain-enum.py
Now we enter the domain name and the file to save the data.
$ python3 subdomain-Enum.py
Subdomain Enumeration Script
Enter Domain name : test-domain.tld
Please provide a file name to save : test-domain-results.txt
And here is the output file:
$ ls
README.md test-domain-results.txt subdomain-Enum.py
Contents will include all results from the API query. In this example, our apex domain is google.com:
r1.sn-jna-i3b6.c.inbox.google.com
mail-vk0-f3.google.com
alt1.gmr-smtp.l.google.com
alt26668073040084382733097124.xmpp.l.google.com
rate-limited-proxy-108-177-75-220.google.com
alt26680965955904396823799.talk.l.google.com
mail-lf1-f6.google.com
rate-limited-proxy-66-249-87-210.google.com
mail-qk1-f191.google.com
r2---sn-jna-i3bs.c.mail.google.com
mail-lf0-f74.google.com
mail-vs1-f66.google.com
alt266895795879893335663545408.xmpp.l.google.com
alt104067895406740770346679773250063542587989392.xmpp.l.google.com
alt36427104341431437098937.xmpp.l.google.com
alt10400855903762802608362207.xmpp.l.google.com
alt44113938101489339614143461787.xmpp.l.google.com
mail-vk0-f110.google.com
alt104008433566897037360.talk.l.google.com
alt104005740506433560120.xmpp.l.google.com
rate-limited-proxy-74-125-150-10.google.com
alt266689573994688059488.talk.l.google.com
r1---sn-q4f7sn7s.c.drive.google.com
alt182540.xmpp.l.google.com
r2---sn-q4flrnek.c.docs.google.com
alt184571.xmpp.l.google.com
r5.sn-vgqs7nes.c.docs.google.com
da-cbf-8.da.ext.google.com
r4.sn-p5qlsnz6.c.drive.google.com
alt1143321512.xmpp.l.google.com
r4---sn-i3belnez.c.drive.google.com
alt10400879043906.xmpp.l.google.com
r4---sn-npoe7n76.c.offline.maps.google.com
r4---sn-nx5s7n7s.c.drive.google.com
alt1112721.xmpp.l.google.com
alt331107.xmpp.l.google.com
[...]
Analyzing stale DNS records
Today we want to take subdomain enumeration to the next level by using our SecurityTrails API™ to build a useful proof-of-concept (PoC) tool that could list all available subdomains of a certain apex domain, and check to see if:
- the subdomains point to a CNAME record
- the CNAME record points to an external hostname
- the external hostname is composed of an expired apex domain
If these three criteria are matched, then we could be facing a CNAME that could be hijacked if the final domain name is free to be registered.
The image above illustrates this same concept. There is a hostname that has a CNAME record, which points to an expired domain name. In the aforementioned case of the Deloitte attack, the malicious 3rd party registered the expired-and-free domain name (which should not be possible to accomplish during the domain name's redemption period) and pointed this previously null-pointed hostname to a different server in their control.
As shown, this recovered domain name can later be pointed to a service specially crafted to conduct all sorts of malicious behaviors.
To make this concept more real, we built a PoC script that allows us to scan the whole subdomain space of a desired apex domain, then check for null-pointed CNAME's.
As with the above examples, we'll query with the API URL
https://api.securitytrails.com/v1/domain/domain-name.tld/subdomains
and obtain a list of subdomains found for a certain apex domain. Once this is done we'll iterate and check to see if the hostname is pointing to a CNAME.
for subdomain in subdomains:
try:
result = dns.resolver.resolve(subdomain, 'CNAME')
except:
print("No CNAME record for hostname: " + subdomain)
pass
else:
print("Checking subdomain: " + subdomain)
check_subdomain(result, subdomain)
Once the check is done we'll send the resulting CNAME, along with the original subdomain, through an additional check to conduct a WHOIS query on the founded CNAME's apex domain.
Results are as follows:
Additional API functions
Although we could have ended the article here (Subdomain-Enum is that easy to use!), we thought we’d have some more fun by extending the script to show how well it works with Python and our API.
The first experiment we'll run is with the WHOIS History endpoint (which requires a Prototyper account). How should our query to the API look? We can figure that out by using the interactive documentation page.
Querying tiktok.com and fetching 100 results will look like this:
We need the URL to look like this in the query: 'https://api.securitytrails.com/v1/domain/tiktok.com/associated?page=1'
Now, we'll create a new script, one that's a modification of 'subdomain-enum.py' to get the data we seek:
touch whois.py
The source code follows:
#!/usr/local/bin/python
import requests
import json
import os
print ("\nSubdomain Enumeration Script\n")
def get_whois_history(domain,filepath):
url = "https://api.securitytrails.com/v1/history/" + domain + "/whois"
querystring = {"page":"1"}
headers = {
'accept': "application/json",
'apikey': "YOUR-SECURITYTRAILS-API-KEY"
}
response = requests.request("GET", url, headers=headers, params=querystring)
result_json = json.loads(response.text)
f = open(filepath,'w+')
f.write(json_dumps(result_json))
f.close()
print("Done!")
domain = input("\nEnter Domain name : ")
filepath = input("\nPlease provide a file name to save : ")
get_whois_history(domain,filepath)
In the script above, we modified the original script to fetch the WHOIS history of the tiktok.com domain. The 'result_json' contains a "nested dictionary" which is saved in JSON format inside the chosen file name.
See the results below:
We'll now modify the script again to test the IP Neighbors feature. This API request takes an IP address and returns the IP neighbors in any given IP level range. We'll check the IP neighbors of one of the malvertising domains we're tracking.
Let's see how the URL should look for this API request:
We can now modify the 'whois.py' script to get the IP neighbors we're seeking:
#!/usr/local/bin/python
import requests
import json
import os
print ("\nSubdomain Enumeration Script\n")
def get_ip_neighbor(ip_add,filepath):
url = "https://api.securitytrails.com/v1/ips/nearby/"+ip_add
#print(url)
querystring = {}
headers = {
'accept': "application/json",
'apikey': "my_secret_api_key"
}
response = requests.request("GET", url, headers=headers, params=querystring)
result_json=json.loads(response.text)
f=open(filepath,'w+')
for x in result_json['blocks']:
f.write("Entry: " + str(entry) + "\n")
for key, value in x.items():
f.write(str(key))
f.write("\n")
f.write(str(value))
f.write("\n")
f.write("--- \n")
entry +=1
f.close()
print("Done!")
# print(result_json)
ip_address=input("\nEnter IP address : ")
filepath=input("\nPlease provide a file name to save : ")
get_ip_neighbor(ip_address,filepath)
The script looks almost identical, except for the parts where we had to apply a few modifications to print the contents to a file. We wrote the scripts in a way that beginners will find easy to comprehend, so the scripts should be self-explanatory (even the Pythonic parts).
The results show us the following:
It is possible to build onto this script by looking at what's available regarding IP data here. The hostnames could also be used to perform subdomain/WHOIS lookups, which is another method of extending the script.
Summary
One of the drawbacks of single-use scripts is that you tend to reinvent the wheel whenever you use them. Many of the OSINT tools out there accommodate full integration of the SecurityTrails API™. However, scripts do provide two major benefits:
- Teaching a person how to use different programming languages.
- Obtaining results quickly without spending a lot of time configuring a new tool.
In the examples above we took the Subdomain-Enum script, added a few modifications, and were able to perform two other OSINT investigations using the SecurityTrails API™. Using the knowledge above, the scripts can be modified even further to explore different aspects of our API.
Get to know our current API integrations and discover more ways to utilize SecurityTrails products and services within your own apps!
