Host Discovery Tips for Bug Bounty Hunters with the SecurityTrails API
Reading time: 10 minutes
Despite a growing corpus of dire predictions and research surrounding the state of information security at large, companies continue to expand their digital footprint to encompass a vast array of cutting edge, yet often dissimilar, architectures.
From a historical perspective, there is nothing new under the sun at play here; after all, information technology patent citations have steadily dominated those of adjacent industries at least since the 1970s, creating a constant influx of knowledge spillover and innovation that accounts for the exponential growth.
However, the organizational mechanisms designed to manage and secure these environments have remained anything but disciplined. In fact, a fair amount of data breaches can still be attributed to the gross misidentification of sensitive endpoints and critical services—a practice that never sits well with adequate leadership as it undoubtedly takes corporate risk to new heights and greatly exacerbates other aspects such as regulatory concerns.
In this blog post, we will examine the role of asset discovery applications—the collective open source intelligence (OSINT) approach at mapping an organization's IP address space as it relates to penetration testing, bug bounty hunting, or simply as part of any sound security assessment program. We will briefly analyze the unique set of circumstances and roadblocks that can impede proper host identification and how leveraging tools like SecurityTrails' own REST API can make all the difference.
Let's have a look.
- What host discovery is and why it's important
- What subdomain mapping is and why it's important
- Host discovery and subdomain mapping with SecurityTrails API
- Further thoughts
- Summary
What host discovery is and why it's important
Security researchers and vendors alike will agree that the answer to the challenges found in obtaining continual (and effective) asset classification lies somewhere between tighter security controls and effective tooling. Thus, inventory and control of both hardware and software assets are considered the top two essential processes towards adequate cybersecurity readiness and defense.
This includes the ability to deploy passive and active asset discovery tools, maintain host inventory and logging information, address any unauthorized, lost, or long-forgotten network resources, as well as the capacity to detect any rogue and/or shadow IT elements-in short, managing your attack surface.
When it comes to red teaming, bug bounty hunting, or general vulnerability assessments, asset discovery entails the ability to search every nook and cranny of the network spread hoping to get a granular representation, with the added possibility of finding the proverbial blind spots in search of any significant, or otherwise inadvertently exposed infrastructure on the target ecosystem.
A caveat: given enough time and limitless resources, one can generate an almost perfect picture of the organization in question. Determining the accuracy of these results, however, can always be a tricky task; one that is conditioned not only to the ever-changing network conditions but also to the power and visibility of your tools. In short, having a robust toolkit at your disposal, such as the SecurityTrails API, with all its powerful mapping features, can make the difference between a successful and an unsuccessful engagement.
What subdomain mapping is and why it's important
Another aspect of the initial OSINT collection procedure that is closely related to host discovery is domain footprinting. Having identified and compiled a list of all "live" assets and their respective services, this next step will ensure a more complete and thorough transition to threat modelling and profiling.
The methodologies here can greatly vary in scope and purpose. For example, given a particular domain as a target, it can be customary to continue the reconnaissance process by performing vertical and horizontal domain correlation via DNS records—the idea behind this is to associate any additional domains to the base root (see above) as well as to find any second-level domain names linked to the target. Vertical domain correlation is commonly referred to as subdomain enumeration.
As previously mentioned, these techniques (host and domain discovery) are not necessarily siloed during the initial stages. We know that companies whose network collection gets sufficiently large are attributed an autonomous system number (ASN) to refer to their CIDR IP ranges. Determining this kind of target's infrastructure using host discovery processes is slightly easier when compared to smaller organizations, with the latter heavily relying on domain names gathered from the former procedure.
There are, however, some roadblocks and inconveniences to account for. As an example, share hosting and content delivery network (CDN) providers, like Cloudflare, can certainly generate some false positives when looking for a specific target host behind them, requiring you to resort to additional methods that involve historical data when it comes to web properties powered behind these internet infrastructure and security giants.
Host discovery and subdomain mapping with SecurityTrails API
Let's take the SecurityTrails API for a spin by creating a number of calls in search for host and domain information as if we were tasked with the preliminary discovery phase. Our goal is to gather as much intelligence about our target domain as possible and to determine what our attack surface looks like.
The SecurityTrails API allows you to programmatically access all IP, DNS, WHOIS, and other company-related information available in the SecurityTrails Web Platform and beyond. It has been designed from the ground up to meet the standards and specifications established by REST principles, allowing you to fetch data mainly using HTTP GET and POST methods. In a nutshell, the API uses a form of internal domain-specific language (DSL) capable of building flexible, yet complex, queries across all datasets similar to the syntax used for SQL WHERE predicates.
Here's an example of a POST request using cURL to retrieve all associated IPs linked to the twitch.tv domain:
curl --request POST \
--url https://api.securitytrails.com/v1/ips/list \
--header 'APIKEY: your_api_key_here' \
--header 'Content-Type: application/json' \
--data '{"query":"ptr_part = '\''twitch.tv'\''"}'
In this case, the attribute "ptr_part" matches the end of the ptr record—the direct association of a given IP address to a domain name as in reverse DNS (RDNS) lookups. A sample of the results is shown below:
If for some reason you cannot include HTTP headers in the request you can also provide the API key as a query parameter. Assuming your API key is your_api_key, you would query our API like so:
https://api.securitytrails.com/v1/ping?apikey=your_api_key
However, because query strings are often logged in clear-text form, we highly discourage you from using the query-string method to supply the API key.
Let's run a somewhat similar DSL API endpoint query—this time we will focus on IP statistics like RDNS pattern identification, which will be grouped and displayed along with ports (number of open ports found) and total results per query.
curl --request POST \
--url https://api.securitytrails.com/v1/ips/stats \
--header 'APIKEY: your_api_key_here' \
--header 'Content-Type: application/json' \
--data '{"query":"ptr_part = '\''twitch.tv'\''"}'
Sample output:
{
"top_ptr_patterns": [
{
"key": "de-cix#.mad.twitch.tv",
"count": 4
},
{
"key": "br#.hel#.twitch.tv",
"count": 2
},
{
"key": "ipv#.de-cix.fra.de.as#.twitch.tv",
"count": 2
},
{
"key": "lo#-br#.sfo#.twitch.tv",
"count": 2
},
{
"key": "lonap#-lhr#.twitch.tv",
"count": 2
},
{
"key": "mail.twitch.tv",
"count": 2
},
{
"key": "netnod-ix-ge-a-sth-#.twitch.tv",
"count": 2
},
{
"key": "netnod-ix-ge-b-sth-#.twitch.tv",
"count": 2
},
{
"key": "six#-sea#.twitch.tv",
"count": 2
},
{
"key": "v#-br.mia#.twitch.tv",
"count": 2
},
{
"key": "v#-mtu#k-six#-sea#.twitch.tv",
"count": 2
},
{
"key": "vpn-corp.twitch.tv",
"count": 2
},
{
"key": "api-anycast.twitch.tv",
"count": 1
},
{
"key": "api.twitch.tv",
"count": 1
},
{
"key": "click.marketing.twitch.tv",
"count": 1
}
],
"ports": [
{
"key": 80,
"count": 5
},
{
"key": 443,
"count": 5
},
{
"key": 22,
"count": 1
}
],
"total": {
"value": 50,
"relation": "eq"
},
"endpoint": "/v1/ips/stats"
}
"ptr_part" is simply one of four powerful IP-based query type attributes available for host discovery purposes. The rest include:
| ptr | matches the full ptr record exactly | |
| port | matches open ports. These are numeric so operators like between gt lt > >= <= etc. are all supported. e.g. port between 1000 and 4000 or port <= 100. | |
| ip | matches the ip address. network masks are supported. e.g. 1.1.1.1/24 |
Of course, not all requests need to rely on these fine-grained operators. The API documentation contains several real-world examples to get you started quickly without resorting to DSL-specific queries. For a list of common use cases and additional DSL fields, you can always check the How to use the DSL guide.
Another very useful search feature is the ability to find associated IPs based on a given domain—the results are based on WHOIS data with the names matched to the domain:
curl --request GET \
--url https://api.securitytrails.com/v1/company/twitch.tv/associated-ips \
--header 'APIKEY: your_api_key_here \
--header 'Accept: application/json'
The results will look something like this:
Finally, let's turn our attention to subdomain mapping—the following query will return child and sibling subdomains for a given hostname:
curl --request GET \
--url 'https://api.securitytrails.com/v1/domain/twitch.tv/subdomains?children_only=false' \
--header 'APIKEY: your_api_key_here' \
--header 'Accept: application/json'
Sample output:
Further thoughts
Data enrichment and consumption requirements, such as those from applications that seek to integrate IP, DNS, WHOIS, or company data, or related security automation products commensurate with more programmatic approaches, can certainly take advantage of the API's array of supported open source languages like Python, Java, or Go.
With only a few lines of code, you can be up and running in no time—just keep in mind, each account is rate-limited to a specific amount of requests per second. If your account is throttled, its rate limit will be reset after a one second interval. Your implementation should expect and be able to handle this by waiting a second before making additional requests.
To showcase how easy it is to work with the API, we wrote a simple Python base class and driver code that lets you leverage the endpoint to check for connectivity status, look for company-related IPs and subdomains (as explained above for the case of twitch.tv), and check your usage statistics for the current month. We welcome you to use it in your tool of choice or next engagement!
Base class (stapimain.py):
#!/usr/bin/env python3
# Author: Gianni Perez
# SecurityTrails 2021
# Requirements #
# python 3+
# pip3 install requests
import requests
class QueryBuilder:
def __init__(self, apikey):
self.apikey = apikey
self.headers = {
"Content-Type": "application/json",
"APIKEY": self.apikey
}
self.url = 'https://api.securitytrails.com/v1/'
def ping(self):
url = self.url + "/ping"
response = requests.request("GET", url, headers=self.headers)
return response.text
def search_for_ips(self, target):
url = self.url + "/ips/list"
payload = {"query": f"ptr_part = '{target}'"}
response = requests.request("POST", url, json=payload, headers=self.headers)
return response.text
def subdomains(self, target):
url = self.url + f"domain/{target}/subdomains"
response = requests.request("GET", url, headers=self.headers)
return response.text
def usage(self):
url = self.url + "/account/usage"
response = requests.request("GET", url, headers=self.headers)
return response.text
Driver code:
import stapimain
api_key = 'your_api_key_here'
if __name__ == "__main__":
new_query = stapimain.QueryBuilder(api_key)
# Test methods
print(new_query.ping())
print(new_query.search_for_ips("twitch.tv"))
print(new_query.subdomains('twitch.tv'))
print(new_query.usage())
Summary
With a steady rise in the number of on-prem, cloud, and mobile device platforms and applications, now commonplace across all modern enterprises, detecting the presence of both prevalent and unknown security vulnerabilities will continue to be an uphill battle. While red teamers are engaged by the need of these organizations to remain competitive in the face of so many threats, others, like bug bounty hunters, are driven by the economics of it all—a vibrant cadence that is sure to last for the foreseeable future.
Whether or not you belong to a particular team, the tools you choose will largely enhance any methodology or strategy you may have prior to an engagement. Moreover, adding automation opportunities to your discovery phase can give you that extra edge you are seeking, especially if you are under a tight deadline. This is why having resources like the SecurityTrails API at your disposal can be an excellent addition to your playbook—go ahead and give it a try during this Bug Bounty Hunting Month at SecurityTrails, and let us know what you think!
