Spiderfoot, an Open Source Intelligence Automation Tool
Reading time: 5 minutesSome time ago we mentioned the cool integration of our security platform into Spiderfoot, one of the best reconnaissance and threat intelligence tools around. Today, we are going to tell you a little bit more about this fantastic security tool.
Steve (the creator or Spiderfoot) wrote the Spiderfoot SecurityTrails plugin doing an amazing job so Spiderfoot users can use the full potential of SecurityTrails from his software. Spiderfoot was also mentioned in our previous post about the Top 20 OSINT tools. Today, we are going to tell you a little bit more about what this fantastic security tool is capable of, some of its main features, and how can it help you gather more information about any network.
Before we start talking and playing with Spiderfoot, let's go back to the basics to answer a simple but crucial question.
What is OSINT?
OSINT stands for Open Source Intelligence and refers to the data that can be fetched from public sources, like IP addresses, domain names, websites, HTTP, DNS, FTP and other sources like social networks, forums, profile pages, videos, etc.
OSINT is one of the best ways to get information about any target and has been utilized by many private researchers, corporations, and government intelligence agencies.
What is SpiderFoot?
Spiderfoot is a tool written by Steve Micallef that automates the whole OSINT process.
As there are tons of data available on the web on different services, networks, and protocols... gathering all this information from every single place and one at the time becomes a pretty time-consuming task.
That's when SpiderFoot comes to help, as it can be used to automate OSINT gathering process to find anything about your target, centralized in one single tool.
To automate OSINT, Spiderfoot queries over 100 public information sources and process all the intelligence data from domain names, email addresses, names, IP addresses, DNS servers, and much more.
Specify the target, choose the modules to run, and Spiderfoot will perform the whole job for you, collecting all the data to build a full profile of anything you are investigating.
OSINT tools like Spiderfoot are especially useful to relate information about any target, unveil possible data leaks or discover full vulnerabilities present in their network or applications.
This information can be helpful while you are running a penetration testing, auditing your own network, or a 3rd party authorized network.
Spiderfoot main features
- Open Source: this security tool was written in Python, and it's hosted on Github. The best part is that it is open source, which means anyone can contribute to make it better.
- Multi-platform: Spiderfoot can be run on both Linux and Windows operating systems.
- Web interface: by default, Spiderfoot can be run from a CLI interface, however, it also supports a cool web interface for those who want ease of use, fancy icons, and rich graphic visualizations.
- Module support: it works including more than 100+ modules, which can help to run almost any kind of test against the target network. SpiderFoot modules were programmed to interact with each other, allowing all related modules to share the same data about the target.
- Documentation: unlike other OSINT tools, Spiderfoot was not only well written in terms of code, it has a brilliant documentation area that will allow you to discover, read, and understand how everything works including installation process, usage, modules, etc.
- Spiderfoot HX: while the standard version will work from any environment, you can also choose to run Spiderfoot from its own self-hosted cloud platform, which includes more advanced features than the self-hosted version.
Installing Spiderfoot
First things first: install the following required python modules in order to have Spiderfoot running:
pip install lxml netaddr M2Crypto cherrypy mako requests bs4
Grab the link from Spiderfoot downloads page, then download and extract:
wget http://www.spiderfoot.net/files/spiderfoot-2.12.0-src.tar.gz
tar zxvf spiderfoot-2.12.0-src.tar.gz
cd spiderfoot-2.12
Starting Spiderfoot is easy, just type:
python ./sf.py
As you see below:
[root@localhost spiderfoot-2.12]# python ./sf.py
Starting web server at http://127.0.0.1:5001 ...
\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*
Use SpiderFoot by starting your web browser of choice and
browse to http://127.0.0.1:5001
\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*
[19/Apr/2018:10:58:53] ENGINE Listening for SIGHUP.
[19/Apr/2018:10:58:53] ENGINE Listening for SIGTERM.
[19/Apr/2018:10:58:53] ENGINE Listening for SIGUSR1.
[19/Apr/2018:10:58:53] ENGINE Bus STARTING
[19/Apr/2018:10:58:53] ENGINE Serving on http://127.0.0.1:5001
[19/Apr/2018:10:58:53] ENGINE Bus STARTED
This will start a web server running at http://127.0.0.1:5001/ , from where you can launch your scans and work from a GUI based screen. From this interface you will be able to launch your first scan and choose three models: by use case , by required data or by module.
If you don't like the GUI interface and prefer old school nerd action you can also use spiderfoot from the console by running:
python ./sf.py
Let's go back to the GUI interface and try to launch your first scan against any target.
Once you click 'Run Scan', you will be redirected to the results page where the scan processes will start appearing in real time:
That screen will show you graphs and clickable bars displaying the entire data gathered from the Spiderfoot modules, along with internal log messages from each scan.
Once the scan finishes, you can start browsing the results to view and analyze the data, as you see below:
OSINT is very important for data collection when you are investigating anything on the internet, it could be a person, a company, or an application. The main issue often is how much time you can spend on that task.
Both, Spiderfoot and our own intelligence platform SecurityTrails can help you speeding up your investigation and become a huge help for your organization.
In the same way, as SecurityTrails was fully integrated as a Spiderfoot module, you can also work on your own apps integrating SecurityTrails features for IP, DNS, Domain, and WHOIS data discovery, signup for free forensics API access today!
