5 minutes to Build a Basic Monitoring and Alerting System for New Subdomains
Reading time: 6 minutes
I spent a very long time automating my recon for bug bounties. I collaborated with a couple of friends for about 12 months to build out an automation beast.
We had a custom framework, and constant recon scanning with good distribution (at times we scaled up to 100+ servers). We stored data on millions of targets and had Slack notifications for vulnerability detection. It was the third iteration of our automation and we thought it was great. I mean, it was pretty great, and it definitely helped us earn some cash on a few popular bounty programs.
I learned a lot while I was coding this but ultimately, looking back I wish I had done things differently. Namely, I wish I had decoupled each component of the automation into small chainable tools instead of building everything as one giant framework. In other words, I wish I had followed the Unix philosophy...
What's the Unix philosophy? Wikipedia puts it best:
The Unix philosophy emphasizes building simple, short, clear, modular, and extensible code that can be easily maintained and repurposed by developers other than its creators.
In this blog I'm going to show you how you can use three basic, free, open source tools to implement continuous monitoring for new subdomains in just five minutes. The three tools are:
- Anew by tomnomnom
- Haktrails by me (you could use any subdomain enumeration tool here, like Subfinder or OWASP-Amass)
- Notify by ProjectDiscovery
- Setting up your environment
- Gathering existing subdomains
- Setting up constant monitoring
- Only printing new subdomains
- Sending the results to Slack/Discord
- Next steps
Setting up your environment
Let's see what the necessary steps are to build a basic monitoring and alerting system for detecting new subdomains.
Picking a VPS
This is the kind of automation that you probably want to run 24/7/365. The easiest way to do this is to set up a VPS. You can use whatever Linux distribution but I tend to use Ubuntu because that's where I feel at home. You can set up an Ubuntu VPS on any popular hosting provider. Linode and Digital Ocean have them starting at $5 per month. If you're a new AWS customer you could also use a free tier EC2 instance which will be free for the first 12 months.
Installing Golang and Tmux
Once you've set up your VPS, you'll need to install Golang. If you're using Ubuntu, this should be as simple as:
sudo apt update && sudo apt install golang
You probably also want Tmux so that you can leave the automation running without stopping every time your SSH session ends:
apt install tmux
Installing the tools
Now that you have Golang installed, you can use the built-in Go package manager to install the tools that we'll be using. These commands work at the time that this blog post was written, but installation instructions for these tools may change so it's always best to check the installation instructions on the repositories directly.
GO111MODULE=on go get -v github.com/projectdiscovery/notify/cmd/notify
go get -u github.com/hakluke/haktrails
go get -u github.com/tomnomnom/anew
Adding GOBIN to your $PATH
In order to run these new tools by typing the name of the tool:
haktrails
Instead of the full path:
~/go/bin/haktrails
We need to modify our bash $PATH to include ~/go/bin/. We can do this by adding the following line to our ~/.bashrc file :
export PATH="$PATH:~/go/bin/"
Then restart your terminal, or run:
source ~/.bashrc
Setting up your config files
To use haktrails, you'll need to set up your config file containing your SecurityTrails API key. You can find instructions to do that here.
To use notify, you'll need to set up your config file with your Discord/Slack/Telegram webhook(s). You can find instructions here.
Gathering existing subdomains
For the purpose of this blog, let's monitor securitytrails.com. The first step is to gather the existing subdomains for that domain and pop them into a file. You can achieve this by running the following command:
echo securitytrails.com | haktrails subdomains | tee subdomains.txt
Setting up constant monitoring
There are a bunch of ways to do this. My personal preference would be to use hakcron, but you can easily achieve the same thing with some simple bash, so let's do that.
while :
do
echo securitytrails.com | haktrails subdomains | tee subdomains.txt
sleep 3600
done
To use the code above:
- Copy the code above into a file called monitor.sh
- Run
chmod +x ./monitor.sh - Run
./monitor.sh
The bash script runs the command we created above, then does nothing for 3600 seconds (one hour). It repeats this process indefinitely. If you want to leave this running in the long term, you may want to execute it in tmux or screen on a VPS so that it will run indefinitely while you lay on the beach sipping pineapple juice, even when your SSH session dies.
Only printing new subdomains
Remember, the goal is to be alerted whenever a new subdomain pops up. The problem with our current solution is that every subdomain is printed every hour. If we send this straight to Slack/Discord webhooks it's pretty useless because we would need to trawl through potentially thousands of subdomains every hour—and try to remember which ones we've seen before and which ones are new.
Thankfully, there's a really easy way to solve this using TomNomNom's anew. We can simply replace "tee" with "anew" in our script.
while :
do
echo securitytrails.com | haktrails subdomains | anew subdomains.txt
sleep 3600
done
Now nothing will be printed to stdout unless a new subdomain appears.
Sending the results to Slack/Discord
ProjectDiscovery released a nice and simple little tool called Notify. We can use this to send the output of our bash script to Slack/Discord by piping to it. This is as simple as running:
./monitor.sh | notify
Now, we have a system that checks a domain for new subdomains every hour, and notifies us of any new ones via our favourite chat application. Awesome!
Next steps
Part of the purpose of this blog post is to demonstrate the power of the Unix philosophy. The three tools we used are very simple and they have distinct purposes. By chaining them together, we can create very powerful automation in a few minutes.
It's easy to see how this system could be expanded to gather URLs using gau, scan for vulnerabilities using nuclei, find XSS using dalfox, and much more. I'd encourage you to think of different ways that you can use these simple tools to glue together high quality data sources such as SecurityTrails and put them to use. The possibilities are endless.
