I learned a lot while I was coding this but ultimately, looking back I wish I had done things differently. Namely, I wish I had decoupled each component of the automation into small chainable tools instead of building everything as one giant framework. In other words, I wish I had followed the Unix philosophy…
What’s the Unix philosophy? Wikipedia puts it best:
The Unix philosophy emphasizes building simple, short, clear, modular, and extensible code that can be easily maintained and repurposed by developers other than its creators.
In this blog I’m going to show you how you can use three basic, free, open source tools to implement continuous monitoring for new subdomains in just five minutes. The three tools are:
- Anew by tomnomnom
- Haktrails by me (you could use any subdomain enumeration tool here, like Subfinder or OWASP-Amass)
- Notify by ProjectDiscovery
Setting up your environment
Let’s see what the necessary steps are to build a basic monitoring and alerting system for detecting new subdomains.
Picking a VPS
This is the kind of automation that you probably want to run 24/7/365. The easiest way to do this is to set up a VPS. You can use whatever Linux distribution but I tend to use Ubuntu because that’s where I feel at home. You can set up an Ubuntu VPS on any popular hosting provider. Linode and Digital Ocean have them starting at $5 per month. If you’re a new AWS customer you could also use a free tier EC2 instance which will be free for the first 12 months.
Installing Golang and Tmux
Once you’ve set up your VPS, you’ll need to install Golang. If you’re using Ubuntu, this should be as simple as:
sudo apt update && sudo apt install golang
You probably also want Tmux so that you can leave the automation running without stopping every time your SSH session ends:
apt install tmux
Installing the tools
Now that you have Golang installed, you can use the built-in Go package manager to install the tools that we’ll be using. These commands work at the time that this blog post was written, but installation instructions for these tools may change so it’s always best to check the installation instructions on the repositories directly.
GO111MODULE=on go get -v github.com/projectdiscovery/notify/cmd/notify go get -u github.com/hakluke/haktrails go get -u github.com/tomnomnom/anew
Adding GOBIN to your $PATH
In order to run these new tools by typing the name of the tool:
Instead of the full path:
We need to modify our bash $PATH to include ~/go/bin/. We can do this by adding the following line to our ~/.bashrc file :
Then restart your terminal, or run:
Setting up your config files
To use haktrails, you’ll need to set up your config file containing your SecurityTrails API key. You can find instructions to do that here.
To use notify, you’ll need to set up your config file with your Discord/Slack/Telegram webhook(s). You can find instructions here.
Gathering existing subdomains
For the purpose of this blog, let’s monitor securitytrails.com. The first step is to gather the existing subdomains for that domain and pop them into a file. You can achieve this by running the following command:
echo securitytrails.com | haktrails subdomains | tee subdomains.txt
Setting up constant monitoring
There are a bunch of ways to do this. My personal preference would be to use hakcron, but you can easily achieve the same thing with some simple bash, so let’s do that.
while : do echo securitytrails.com | haktrails subdomains | tee subdomains.txt sleep 3600 done
To use the code above:
- Copy the code above into a file called monitor.sh
chmod +x ./monitor.sh
The bash script runs the command we created above, then does nothing for 3600 seconds (one hour). It repeats this process indefinitely. If you want to leave this running in the long term, you may want to execute it in tmux or screen on a VPS so that it will run indefinitely while you lay on the beach sipping pineapple juice, even when your SSH session dies.
Only printing new subdomains
Remember, the goal is to be alerted whenever a new subdomain pops up. The problem with our current solution is that every subdomain is printed every hour. If we send this straight to Slack/Discord webhooks it’s pretty useless because we would need to trawl through potentially thousands of subdomains every hour—and try to remember which ones we’ve seen before and which ones are new.
Thankfully, there’s a really easy way to solve this using TomNomNom’s anew. We can simply replace “tee” with “anew” in our script.
while : do echo securitytrails.com | haktrails subdomains | anew subdomains.txt sleep 3600 done
Now nothing will be printed to stdout unless a new subdomain appears.
Sending the results to Slack/Discord
ProjectDiscovery released a nice and simple little tool called Notify. We can use this to send the output of our bash script to Slack/Discord by piping to it. This is as simple as running:
./monitor.sh | notify
Now, we have a system that checks a domain for new subdomains every hour, and notifies us of any new ones via our favourite chat application. Awesome!
Part of the purpose of this blog post is to demonstrate the power of the Unix philosophy. The three tools we used are very simple and they have distinct purposes. By chaining them together, we can create very powerful automation in a few minutes.
It’s easy to see how this system could be expanded to gather URLs using gau, scan for vulnerabilities using nuclei, find XSS using dalfox, and much more. I’d encourage you to think of different ways that you can use these simple tools to glue together high quality data sources such as SecurityTrails and put them to use. The possibilities are endless.