bug bounty hunting

SecurityTrails Blog · May 20 · by Luke Stephens

5 minutes to Build a Basic Monitoring and Alerting System for New Subdomains

Reading time: 1 minutes
Listen to this article

I spent a very long time automating my recon for bug bounties. I collaborated with a couple of friends for about 12 months to build out an automation beast. We had a custom framework, and constant recon scanning with good distribution (at times we scaled up to 100+ servers). We stored data on millions of targets and had Slack notifications for vulnerability detection. It was the third iteration of our automation and we thought it was great. I mean, it was pretty great, and it definitely helped us earn some cash on a few popular bounty programs.

I learned a lot while I was coding this but ultimately, looking back I wish I had done things differently. Namely, I wish I had decoupled each component of the automation into small chainable tools instead of building everything as one giant framework. In other words, I wish I had followed the Unix philosophy…

What’s the Unix philosophy? Wikipedia puts it best:

The Unix philosophy emphasizes building simple, short, clear, modular, and extensible code that can be easily maintained and repurposed by developers other than its creators.

In this blog I’m going to show you how you can use three basic, free, open source tools to implement continuous monitoring for new subdomains in just five minutes. The three tools are:


Youtube Video


Setting up your environment

Let’s see what the necessary steps are to build a basic monitoring and alerting system for detecting new subdomains.

Picking a VPS

This is the kind of automation that you probably want to run 24/7/365. The easiest way to do this is to set up a VPS. You can use whatever Linux distribution but I tend to use Ubuntu because that’s where I feel at home. You can set up an Ubuntu VPS on any popular hosting provider. Linode and Digital Ocean have them starting at $5 per month. If you’re a new AWS customer you could also use a free tier EC2 instance which will be free for the first 12 months.

Installing Golang and Tmux

Once you’ve set up your VPS, you’ll need to install Golang. If you’re using Ubuntu, this should be as simple as:

sudo apt update && sudo apt install golang

You probably also want Tmux so that you can leave the automation running without stopping every time your SSH session ends:

apt install tmux

Installing the tools

Now that you have Golang installed, you can use the built-in Go package manager to install the tools that we’ll be using. These commands work at the time that this blog post was written, but installation instructions for these tools may change so it’s always best to check the installation instructions on the repositories directly.

GO111MODULE=on go get -v github.com/projectdiscovery/notify/cmd/notify
go get -u github.com/hakluke/haktrails
go get -u github.com/tomnomnom/anew

Adding GOBIN to your $PATH

In order to run these new tools by typing the name of the tool:

haktrails

Instead of the full path:

~/go/bin/haktrails

We need to modify our bash $PATH to include ~/go/bin/. We can do this by adding the following line to our ~/.bashrc file :

export PATH="$PATH:~/go/bin/"

Then restart your terminal, or run:

source ~/.bashrc

Setting up your config files

To use haktrails, you’ll need to set up your config file containing your SecurityTrails API key. You can find instructions to do that here.

To use notify, you’ll need to set up your config file with your Discord/Slack/Telegram webhook(s). You can find instructions here.

Gathering existing subdomains

For the purpose of this blog, let’s monitor securitytrails.com. The first step is to gather the existing subdomains for that domain and pop them into a file. You can achieve this by running the following command:

echo securitytrails.com | haktrails subdomains | tee subdomains.txt

Setting up constant monitoring

There are a bunch of ways to do this. My personal preference would be to use hakcron, but you can easily achieve the same thing with some simple bash, so let’s do that.

while :
do
    echo securitytrails.com | haktrails subdomains | tee subdomains.txt
    sleep 3600
done

To use the code above:

  • Copy the code above into a file called monitor.sh
  • Run chmod +x ./monitor.sh
  • Run ./monitor.sh

The bash script runs the command we created above, then does nothing for 3600 seconds (one hour). It repeats this process indefinitely. If you want to leave this running in the long term, you may want to execute it in tmux or screen on a VPS so that it will run indefinitely while you lay on the beach sipping pineapple juice, even when your SSH session dies.

Only printing new subdomains

Remember, the goal is to be alerted whenever a new subdomain pops up. The problem with our current solution is that every subdomain is printed every hour. If we send this straight to Slack/Discord webhooks it’s pretty useless because we would need to trawl through potentially thousands of subdomains every hour—and try to remember which ones we’ve seen before and which ones are new.

Thankfully, there’s a really easy way to solve this using TomNomNom’s anew. We can simply replace “tee” with “anew” in our script.

while :
do
    echo securitytrails.com | haktrails subdomains | anew subdomains.txt
    sleep 3600
done

Now nothing will be printed to stdout unless a new subdomain appears.

Sending the results to Slack/Discord

ProjectDiscovery released a nice and simple little tool called Notify. We can use this to send the output of our bash script to Slack/Discord by piping to it. This is as simple as running:

./monitor.sh | notify

Now, we have a system that checks a domain for new subdomains every hour, and notifies us of any new ones via our favourite chat application. Awesome!

Next steps

Part of the purpose of this blog post is to demonstrate the power of the Unix philosophy. The three tools we used are very simple and they have distinct purposes. By chaining them together, we can create very powerful automation in a few minutes.

It’s easy to see how this system could be expanded to gather URLs using gau, scan for vulnerabilities using nuclei, find XSS using dalfox, and much more. I’d encourage you to think of different ways that you can use these simple tools to glue together high quality data sources such as SecurityTrails and put them to use. The possibilities are endless.

Luke Stephens Blog Author
LUKE STEPHENS

Luke Stephens (hakluke) is a proud dad, husband and hacker of things. Previously he has worked as a musician, web developer, penetration tester, bug bounty hunter and a manager at Bugcrowd. Today he is managing his own cybersecurity consultancy Haksec.