osint tools reconnaissance

SecurityTrails Blog · Last updated on Apr 22 2020 · by Esteban Borges

What is OSINT? How can I make use of it?

Reading time: 10 minutes

In the past months, we have reviewed a couple interesting OSINT utilities. In fact, a few weeks ago, we also published the Best OSINT Tools as a great resource for everyone starting an information security investigation. But one thing is missing for all those who have just been introduced to the fascinating world of cybersecurity: the key concept of OSINT.

We are talking and writing about OSINT, Infosec, Tools, Cybersecurity, and Open Source. But do you really know the definition of OSINT? That's what we are going to explore today. What is OSINT? How can I make use of it? What are the main benefits for my company? And which are the best-recommended OSINT techniques? Read on…

What is OSINT?

OSINT stands for Open Source Intelligence, and it is one of the key aspects in understanding the cybersecurity that rules the Internet these days.

The term OSINT comes from many decades ago, in fact, US military agencies started using the term OSINT in the late 1980's as they were re-evaluating the nature of information requirements in tactical levels under battlefields. Then in 1992, the Intelligence Reorganization Act determined the main goals of intel gathering included key concepts like:

  • Must be objective intelligence free of bias
  • Data must be available on public and non-public sources

While the concept of OSINT has evolved since then, as it does not include the non-public sources, the concept originates from that time.

Open source intelligence (OSINT) is information collected from public sources such as those available on the Internet, although the term isn't strictly limited to the internet, but rather means all publicly available sources.

"OS" (from OSINT) means Open Source. In this case, it is not related to the famous open source movement, but to any publicly available source where the user can obtain the information in their intelligence data collection.

The key word behind OSINT concept is information, and most importantly, information that can be obtained for free. It doesn't matter if it is located inside newspapers, blogs, web pages, tweets, social media cards, images, podcasts, or videos as long as it is public, free and legal.

With the right information in your hands, you can get a great advantage over your competition, or speed up any company/people investigation you are in charge of.

OSINT Advantages

OSINT Examples

But OSINT is even simpler, you know; many of us associate OSINT to cyber war, cyber attacks, cybersecurity, etc. And while those things are a part of it, OSINT is much more explicit and uncomplicated.

OSINT examples include:

  • Asking questions on any search engine.
  • Research public forums on how to fix your computer.
  • Watch a youtube video on how to make a birthday cake.

As you see, you don't need to be a hacker to use OSINT in your daily life: you're already using it, you just might have not known it.

However, since we are focused on modern OSINT for the cybersecurity fields, we will now take a look at how your company or project can benefit from it.

How Is Open Source Intelligence Used in Cybersecurity?

Companies and individuals use OSINT all day long, as we've shown before, and yet they don't consciously know it.

Sales, Marketing, and Product management teams also use OSINT to increase conversions or just be more effective while delivering their services to the public.

In the cybersecurity field, using the right utilities for your OSINT investigation can be really effective if you combine it with critical thinking and have a clear OSINT strategy.

Whether you are running a cybersecurity investigation against a company/person or if you are on the opposite side working to identify and mitigate future threats, having pre-defined OSINT techniques and clear goals can save you a lot of time.

Most IT companies do not embrace OSINT to boost their cybersecurity defenses, and sooner or later, this may become a problem as they are not able to identify and detect app, services, and/or cybersecurity threats.

OSINT Techniques and Resources

While there are a lot of OSINT techniques and mechanisms, not all of them will work for your target. First, you will have to ask yourself a couple of questions:

  • What am I looking for?
  • What is my main research goal?
  • What or who is my target?
  • How am I going to conduct my research?

Try to find the answer to these questions, and that will be the first step in your OSINT investigation.

While a lot of OSINT techniques are used by government and military agencies, they can often be applied to your own company, too. Some may work, others may not, but that's part of the OSINT strategy – you will have to identify which sources are good and which ones are irrelevant for your research.

OSINT techniques can be split into two major categories that involve different types of contact with our target.

Active OSINT vs Passive OSINT.jpg

Let's take a look into the most popular OSINT techniques used in cybersecurity:

  • Collect employee full names, job roles, as well as the software they use.
  • Review and monitor search engine information from Google, Bing, Yahoo, and others.
  • Monitoring personal and corporate blogs, as well as review user activity on digital forums.
  • Identify all social networks used by the target user or company.
  • Review content available on social networks like Facebook, Twitter, Google Plus, or Linkedin.
  • Access old cached data from Google – often reveals interesting information.
  • Identify mobile phone numbers, as well as mail addresses from social networks, or Google results.
  • Search for photographs and videos on common social photo sharing sites, such as Flickr, Google Photos, etc.
  • Use Google Maps and other open satellite imagery sources to retrieve images of users' geographic location.

These are some of the most popular techniques you will find. However, after you are done doing OSINT research, you will have a lot of data to analyze. That's when you will have to refine your results, and search in detail for all the really necessary things you need, and discard the rest.

The final step in the OSINT strategy will be to translate all this digital intelligence data into a human-readable format, so it can be understood by non-technical individuals, which are often at the head of most companies.

Risks of performing an OSINT investigation

We touched upon the risks of performing an OSINT investigation when we shared about active/passive techniques, but it’s important to highlight the subject. It’s one of the most commonly overlooked areas of OSINT research.

We divided this concern into three particular risks:

  • Risk of getting detected: This concerns the direct contact made by using active techniques, or third-party services that may give you away as the one who performed the search.

  • Risk of losing access to that information: Once they know you’re tracking their steps or looking for their information, they can start erasing their own trails and shut down public data from social networks, profiles, and the like.

  • Risk of becoming the victim: After all, you can end up being the target of an investigation, or even worse, the organization you belong to can suffer that fate. Great care should be taken when using active OSINT techniques.

These risks can be mitigated by using 3rd party trusted services, such as APIs, proxies, VPN servers, TCP tunneling, and many other advanced techniques.

Taking your OSINT strategy to the next level

OSINT would be nothing if we didn't have the right tools to fetch all this intelligence data.

That's why we will now take a look at the most popular open source intelligence data collection tools available.

Quite simply, it all starts with Google. And when it comes to open source intelligence, it's also one of the most useful scripts and programs around.

The hacking techniques commonly referred to as "Google Dorks" are simple yet effective ways to use the most popular search engine on earth for OSINT purposes. This is done thanks to users exposing sensitive information by accident, leaving unprotected data, variables, databases and codes ready for crawling by Google.

Therefore, the only thing we need is to begin querying Google with a few basic search operators. For example if we use "site:anysite.com", we'll find all the results related to a specific website. If we use "filetype:", it will show only the results from specific types of files.

You can also combine both searches. To search for .sql databases, for example, use this:

filetype:sql site:anysite.com

You'll find more valuable information on how to search for sensitive information on Google in our article about Google Dorks.

Apart from searching on Google, there are a lot of other useful applications when it comes to OSINT.

Wappalyzer (previously covered in our CMS Detector article) is another great resource for technical data about the technologies running on any website, including software name and versions.

This intel data can later be used to search for active CVEs, to find potential threats behind those running technologies.

What other OSINT apps and scripts can you use? Literally hundreds of utilities, including:

  • Personal data collection tools like Pipl, which can reveal a lot of information about individuals, all in one place

  • The Wayback Machine, a site that explores old versions of websites to reveal important information

  • GeoCreepy, which tracks down geographic location information to provide a clear picture of users' current locations

  • Automated OSINT apps for retrieving information, like Spiderfoot or the Phantom + SecurityTrails integration

  • AMASS is another great tool for information gathering and network mapping that you should keep in mind.

  • Popular OSINT browser extensions that include useful sources, like OSINT Browser

  • Running port scanners against the target company's server's infrastructure to find running services.

  • OSINT tools like Shodan, to search for internet-connected devices used by your target

  • Our own SecurityTrails toolkit, which explores DNS services as well as domains, subdomains and IP addresses

Artificial Intelligence: The future of OSINT?

When we talk about AI we often think about simulating human intelligence by using high performance computer systems. This leads to a logical question… can AI help OSINT research?

If AI is the future of OSINT, how can machine vision, learning, natural language processing (NLP), autonomous machines and robotics help the OSINT evolution? These questions are timely, because whether it’s for cybersecurity, military purposes, home or even health, AI may be the perfect ally for boosting OSINT processes when it comes to reconnaissance, information collection, analysis and filtering large amounts of data.

Government and intelligence agencies are already using AI to help with their social collection efforts. Military forces in particular rely on AI to help them succeed in the fights against terrorism, cyber attacks, fake propaganda and national security, among many others.

However, it’s important to remember: “OSINT AI” should not be confused with “OSINT automation”. For many decades already, numerous automated scripts, apps and services have been developed in the open source intelligence world.

The AI approach should not only be called upon for help during the recon phase and data collection, it should also be focused on data analysis, one of the most essential parts of the OSINT process. Filtering the noise and choosing the right data from all the raw logs and hundreds of pages of information is something that can really only be done by an OSINT analyst.

Summary

Now you know what OSINT is, and how you can make use of it to boost your cybersecurity investigations, as well as to prevent attacks into your own network by hiding crucial information from your company, people, as well as domain names, servers, IP addresses and much more.

Knowing this leads to the next question: are you ready to unveil the real power of data intelligence as an advantage against your competition? If so, start testing SecurityTrails cybersecurity platform, or sign up for a free API account to integrate our almighty security platform with your own web applications.

ESTEBAN BORGES

Esteban Borges is a security researcher and technical writer specialized in Linux security. He has been working in the cybersecurity industry for more than 15 years, with a focus on technical server security and open source intelligence.