GOSINT: A Framework for Collecting, Processing, and Exporting Indicators of Compromise (IoC)
Reading time: 9 minutesTools are an important mechanism for any professional wanting to simplify their work and be more productive. As a chef might use a whisk to beat eggs faster and prepare more food in less time, information security professionals also depend on tools to make their work easier and quicker to complete.
We previously reviewed one such tool in our Jok3r, a Framework for Automated Network and Web Penetration Testing article.
Today we'll review another open source tool: GOSINT, a web-based utility built by the Computer Security Incident Response Team (CSIRT) at Cisco, that can help researchers and blue-teamers filter events from threat intelligence feeds.
What is GOSINT?
GOSINT was designed to help security analysts collect and review security intelligence data from structured and unstructured third-party threat intelligence feeds. It also scans events in this data against cybersecurity APIs that might show similar events containing malicious behavior.
GOSINT helps researchers investigate security incidents by comparing the incident data against third-party sources like Twitter, VirusTotal or AlienVault, to see if this data has been documented on those third-party sources.
Main features:
- Web based: GOSINT uses a web application as the primary interface. You can adjust settings and run the entire suite of tools through a click-and-use web app.
- Multiple APIs: Multiple third-party APIs are supported for analyzing threat data. These include: Twitter, AlienVault, Cisco Umbrella and VirusTotal.
- Export to CRITs: It is capable of exporting indicators of compromise into a CRITs server.
- Whitelists: Domains can be whitelisted manually or from Alexa rankings to reject indicators from popular domains (that generally aren't malicious). IP addresses can also be whitelisted through ISP whitelisting.
Who is GOSINT for?
GOSINT is primarily geared towards use by people in blue or purple team scenarios. More specifically, GOSINT users and scenarios would include:
- Blue-team/DevOps users - It can be used for proactive monitoring and filtering of threat intelligence data and detecting possible network threats.
- Forensic investigators - When a compromise has occurred, forensic specialists can use GOSINT as part of their toolset to evaluate whether similar incidents have occurred elsewhere.
Installing GOSINT on Linux
This tool offers 3 different ways to be installed.
- The quick installation method uses 2 bash scripts to install all necessary software.
- The second option is a user-contributed Dockerfile that sets GOSINT up in a Docker container.
- The third option is the manual version of the bash script in option 1.
If you're comfortable using the Linux command line, you should be able to interpret the bash script files. We recommend using the manual method for installation. Up to a point, it is fairly straightforward, and when things might get confusing, we've addressed them in this blog post.
Always remember to use some type of sandboxing environment when installing new software. We don't recommend the usage of a remote web server because of the way GOSINT works as a web application, unless you're comfortable with securely configuring web applications and all the software that goes with it (web server, database, etc.).
In our tests for this review, we used Ubuntu 18.04 and any commands used here should apply to Debian-based distros (and with a few minor tweaks, to other distros as well).
The first step is to install some necessary software:
sudo apt update
sudo apt upgrade
sudo apt install mongodb php-fpm nginx git wget
The official documentation recommends that we verify that MongoDB is only listening on the local loopback interface (127.0.0.1/localhost). We can verify this is the case by running:
cat /etc/mongodb.conf
And looking for the line of code that says:
bind_ip = 127.0.0.1
This should confirm that MongoDB is listening only on the local loopback interface.
Now we can proceed to create a 'gosint' user, and then connect to that user:
sudo useradd -m gosint
sudo useradd gosint sudo #adding gosint as a sudo user
sudo su gosint
cd
One of the drawbacks we found with GOSINT is that it looks like the framework was last updated 2-3 years ago. The official instructions recommend using Go 1.8 but the latest version is Go 1.14, so we proceeded by installing Go 1.8 to avoid any compiling or other compatibility issues:
wget https://storage.googleapis.com/golang/go1.8.linux-amd64.tar.gz
tar zxvf go1.8.linux-amd64.tar.gz
You should now have the decompressed folder containing the Go code visible in the current directory. You can verify it is there by running ls. Now we proceed with creating a workspace and setting up a few environment variables:
mkdir ~/projects
export GOROOT=$HOME/go
export PATH=$PATH:$GOROOT/bin
export GOPATH=$HOME/projects
export GOBIN=$GOPATH/bin
export PATH=$GOPATH:$GOBIN:$PATH
Now we install the dependency tool for Go:
go get github.com/tools/godep
go install github.com/tools/godep
A further indication that GOSINT may not be actively developed is the fact that the documentation recommends using 'godep', which is currently archived and recommends users to use other tools.
We will now clone the GOSINT repo and build the binary:
cd ~/projects/src
git clone https://github.com/ciscocsirt/GOSINT
cd GOSINT
godep go build -o gosint
chmod +x gosint
If you've followed the instructions up to this point exactly as shown above, your installation should be working.
Let's now move to the Nginx configuration as this is the most ambiguous part of the installation. Although the official documentation provides an Nginx config file, it doesn't explain where to place it. We also had to modify the config file because the PHP version we used was 7.2 and the provided config supported 7.0 only.
First we navigate to the site-specific config section and we move the default config file somewhere else (instead of deleting it):
cd /etc/nginx/sites-enabled/
sudo mv default /home/gosint/
At this point we can create our own config file and add the contents from the official documentation (with our tweaks):
cd /etc/nginx/sites-enabled/
sudo touch mysite.conf
sudo vim mysite.conf
And the content to add to mysite.conf file:
server {
listen 80;
root /home/gosint/projects/src/GOSINT/website;
index index.php index.html index.htm;
try_files $uri $uri/ @apachesite;
server_name someserver.yourcompany.com;
gzip on;
gzip_proxied any;
gzip_types
text/css
text/javascript
text/xml
text/plain
application/javascript
application/x-javascript
application/json;
#location / {
# try_files $uri $uri/ =404;
#}
error_page 404 /404.html;
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
location @apachesite {
proxy_pass http://localhost:8000;
}
location ~ \.php$ {
try_files $uri =404;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
# PHP 7 - SECURITYTRAILS ADJUSTMENT HERE - CHANGE PHP TO 7.2
fastcgi_pass unix:/var/run/php/php7.2-fpm.sock;
# PHP 5
# fastcgi_pass unix:/run/php5-fpm.sock;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
}
Now we can restart Nginx:
sudo /etc/init.d/nginx reload
Depending on whether you installed GOSINT on your host or a container, you can either start the application normally or by binding to 0.0.0.0. We'll assume that the setup used was a container and proceed in that manner.
Let's start GOSINT:
cd ~/projects/src/GOSINT/
./gosint -serve 0.0.0.0
You should see a result like this in your command line output:
Now we navigate to the container URL (in our case 192.168.5.119/index.php) and you'll see an output like the one below:
We can now add the various API keys through the web dashboard by navigating to "Settings" and adding those keys.
Testing
Our first step toward using GOSINT is to create a data feed. Let's navigate to the Settings page and under the "Indicator Feeds" tab, look for the following:
We added four data feeds for testing purposes, as shown below:
We set all the cron jobs to 1 hour so that we can obtain data quickly. Otherwise, it is recommended to set your cron jobs to daily or another infrequent time (unless the data feed is updated more than once daily). The command line should output some data too.
After the cron jobs have run, you can navigate to the Pre-Processing page to see output that might look like this:
The results are broken down into the date/time of the scan, the indicator (URL/IP address/other), the type (IP/URL/other), the source (obtained from the feed title) and the context.
We can then proceed to click on the 'Everything' option which will run one of the results against the APIs we've enabled, which will be VirusTotal and AlienVault. A popup will appear showing the results of the scan. You can then opt to delete the indicator, move it to Post-Processing or close the popup.
Post-Processing is where indicators of interest would be saved. You can also export the results of Post-Processing indicators via the Transfer Station option, which will enable you to export the data as a CSV or into a CRITs server.
If you want to track a URL or any other type of data, you can choose to do an Ad Hoc operation and run an Ad Hoc Input. We ran an Ad Hoc Investigation against a malvertising domain we are tracking.
You can also mix and match Sources, Operators and Destinations under the Recipe Manager option. Due to the limited number of operators supported, we actually didn't find this section very useful.
Conclusion
While we found GOSINT to be a decent tool and easy to use, it was also limited in many ways.
Based on the last code update, the tool may have been really useful two or three years ago when it was launched. Since then, there has been a lot of development in the "threat intelligence" space and we found a number of freemium tools that do a lot more than GOSINT does.
One of its most limiting factors is the need to track down your own threat-intel feeds (which the freemium tools have already done). Also, we mentioned that one of its features is support of multiple APIs. However, these might be too few, as many more APIs now exist that offer better attack surface analysis features for possible network threats.
As for the Nginx configuration, it was also very limited, but this tutorial we've written should fill in the gaps not covered by the official documentation.
If you're looking for constantly updated and highly enriched data feeds for threat intelligence purposes, start exploring the SecurityTrails Feeds that can be easily integrated with your own apps, or used for offline analysis. Schedule a call with our sales team today.
