OSINT is Maturing: Our Interview with Steve Micallef from SpiderFoot
Reading time: 14 minutesOnly a few years back, OSINT wasn’t that widespread, but today advancements in the OSINT field have encouraged the constant growth of, not just the tools that are available, but also the community around it.
Whether you’re experienced in it and use it regularly for reconnaissance, digital investigations and threat intelligence, or you’re new to the concept, there are many tools out there that make the automation of OSINT possible.
One such tool is SpiderFoot.
SpiderFoot is an Open Source Intelligence tool written in Python that was developed by Steve Micallef in his free time. The tool queries over 100 public information services and provides you with intelligence data about domain names, email addresses, names, IP addresses, DNS servers and much more. SpiderFoot has over 100 modules so anyone interested in security, from beginners to professionals can understand their security perimeter.
A year ago, we [announced][blog_spiderfoot] the [SecurityTrails SpiderFoot plugin][integrations] and wrote about all the ways in which you can use this amazing security tool, and today, we have the pleasure of talking to the author himself, Steve Micallef. He shares first-hand information about growing the OSINT community, surprising things he’s uncovered in the process, how he manages to stay on track while having a regular, 9-to-5 job, and how he made his way from Australia to Zurich, Switzerland, a city thriving in digital, medical and financial technology innovations.
SecurityTrails: You're from Australia but currently living in Switzerland. How did the decision to relocate affect your business, and the way you go about things?
Steve Micallef: Well, living in Australia I tended to feel somewhat cut off from the rest of the world, due to the time zone difference and of course the physical distance from Europe and the US. It takes over 24 hours to fly from Melbourne to Zurich! Working for a global bank at the time didn't help either, since I was often having to attend late-night conference calls.
So when the opportunity to relocate to Zurich came up, I jumped at it and landed here a few months later with my family in mid-2010. At that stage, SpiderFoot was actually still the open source version from 2005 that I had developed in C# purely as a self-education exercise, so it was only years later after feeling settled in Switzerland and with my kids a bit older that I found I had some free time for a hobby. Some people I guess would take up hiking or something, but I wanted to learn Python, so off I went and re-wrote it all in Python.
I did also actually learn how to ski, so I'm not a complete "kellerkind" as they might call me in Switzerland (Swiss slang for someone who works in the dark in front of their computer all the time).
Project Created: 2005 Hosted version: SpiderfootHX Number of Modules: 101 Language: Python Example Modules: · Abuse.ch ASN / IP abuse checker · Searching TOR via Ahmia
What's the startup scene like in Zurich, and the cybersecurity community in general, in comparison to Australia?
Steve: I can't really comment on the start-up scene, since aside from currently working for one in Zurich for the past 5 years, I wasn't so connected to any beforehand—neither in Switzerland or Australia. The security scenes in both Melbourne (where I'm from) and Zurich are probably comparable, I'd say. Both have their own security conferences, their own BSides, and many cool people who are passionate about the field. Google also has a big presence in Zurich, where a lot of big names in security work so I think that attracts a lot of talent here.
You've been in the industry for over 15 years, but your interest in information security started even before that. Do you remember the first time that your interest in infosec was sparked?
Steve: Going back to 1995, when I was 15 years old, the Internet was becoming this big thing and I consider myself quite lucky because I was getting into technology at a time when the Internet was really taking off and there was an explosion of raw technical information that was available.
I come from the times of Bulletin Board Systems, I used to dial into them and I also ran one myself. On these bulletin boards there were a lot of text files explaining how to do programming, how to hack and phreak. I did the research and in that time I learned a number of programming languages and was just really into computers. I remember on one of those BBS's, they mentioned a chat room of like-minded people that were talking about technology in general. I joined the chat room, and the discussion started from there. I think I realized very quickly how out of my depth I was because these guys were just on another level but in the same time it forced me to lift my game and to keep the conversation going I had to learn a lot about programming, networking and Linux, which was a new operating system coming onto the scene back then
One of the first things I did was to install Linux myself and set up a shell server as a free service to offer to people in that chat room. One of them got on it and completely compromised my machine, but not in a bad way! He showed me what I did wrong and how he got access to it. I did a lot of research and realized that there are a lot of ways in which things can go wrong if you don't do things properly. Everything got sparked from there—I realized that a well-designed and well-implemented system is kind of a thing of beauty and that you really tend to respect when something is well secured.
You're the author of SpiderFoot, but you also work as the Head of IT and Security at a mobile-banking startup. How do you manage to stay on top of both?
Steve: It's fairly simple for me really: during business hours and whenever needed outside of business hours, I focus on my job. Outside of this, I make time to develop SpiderFoot by either getting up at 5 a.m. and spending a few hours on it before heading to the office, or carving out a day from my weekend to focus purely on it without distractions.
I can easily spend hours coding away at new features, re-designing core aspects or creating content, but because I have a full-time job, I use my "SpiderFoot time" in a very focused and concentrated way since I know it's limited. That limitation forces me to be focused and use the time wisely so I'm always asking myself "What is the most important thing to be doing right now?," "What would add real value to users?," "What still keeps users going to other OSINT tools?"...and this helps me focus on what matters. The constraint actually feeds my creativity and drive.
You've once said that when you first created SpiderFoot in 2005, you didn't even realize it was an OSINT tool. What was the reason behind starting SpiderFoot and making it an open source project at that time?
Steve: Creating SpiderFoot didn't really have any grand plan behind it, I just wanted to learn C#.
I did a training course by a couple of guys from SensePost and one of them was Roelof Temmingh who went on to build Maltego. They did a demo of a tool that they built that was similar to SpiderFoot, meaning you would give it a domain name, it would go and do a bunch of stuff and then return some basic information. This was around 2004–2005 so there wasn't a great amount of information it could obtain but it was still pretty cool. After that training I thought that I could build something like it, and I also used it as an excuse to learn C#.
From there I created the first version of SpiderFoot and put it out there as open source. That was in 2005 and it was version 0.01 beta. It wasn't popular in the beginning, that came later, but I think I didn't do a single update or anything to promote it. It stayed untouched from 2005 until 2012 when I released a version written in Python, which was the beginning of the version we have today.
Population: 8.42M Largest city: Zurich Top level domain: .ch Top Industry: Banking Inventions: WWW by Tim Berners-Lee @ CERN in 1989.
From your perspective, what influenced the OSINT community, and OSINT ecosystem in general, to grow and mature as much it has in the past few years? Do you think there was a change in people's awareness or is something else happening?
Steve: I think it's probably a combination of a few factors.
First, so much information these days is online and mostly open and queryable via APIs. Probably a lot of people think about Facebook or LinkedIn when I say that, but it's so much more than this when you look at what companies like SecurityTrails or SHODAN are doing by also amassing system/network data and making that easily accessible.
Second, it's possibly not an obvious link to make, but I think one of the main drivers behind the growth of OSINT is the availability and ease-of-use in cloud computing and the ecosystem around it. This means a much lower barrier to entry, to set up and run the services I mentioned above.
And finally I think that organizations of today look technically very different from those before cloud computing. When you couple this reality with the barrage of news about data breaches, I think people are catching on to the fact that what they need to secure is distributed all over the place. Out of this, OSINT has emerged as a methodology of finding what is where and what might be exposed, so it's become a go-to approach for penetration testers, red teamers and security analysts.
Successfully building a community around your open source project is more than just putting your code on GitHub. How did you grow your open source community?
Steve: Well it is and it isn't. Of course it takes more than just putting code on GitHub to build a community, as there has to be some "marketing" done to build awareness of what you're building and why. But all the marketing in the world won't help you if you have a crappy unmaintained product. That means good documentation, extensible code, an easy-to-use interface and regular updates. Also, quality has to be a real priority or you won't stand out amongst the random scripts people have hacked together and put online which don't work because they only were ever tested locally.
What is the best advice you can give to someone who's looking into launching their open source project and growing a community around it?
Steve: I can break it down to three fundamental points:
First and foremost is write something you find useful. If you find it useful there is a chance that there is a ton of people out there who will use it. Just write some good quality and functional code—it doesn't have to be perfect—and get it out there.
Kind of embedded in that first point is the second point. Ship, and ship often. When you get it out there, make improvements, continually do so as often as possible.
The third point is that you have to do some type of marketing as well. Marketing may have a dirty sound to it for a lot of people, especially engineers. What it means is that you need to have people that know about the product, because if no one knows, then it's not useful. It's just sitting there on GitHub and nobody is benefiting from it. It's important to, for example, blog about it, mention it to people at conferences, do whatever you need to do for people to know about this cool thing you've made.
Organisations of today look technically very different from those before cloud computing. When you couple this reality with the barrage of news about data breaches, I think people are catching on to the fact that what they need to secure is distributed all over the place.
As for growing a community around it, that's something that comes naturally, if you stick to the points about getting it out there and telling people about it and of course, regularly updating it. With SpiderFoot, I haven't actively tried to build a community around it, I've just followed those three principles for a fairly long time and gradually, as the tool grew in functionality and as I listened to what people wanted and needed, it built a momentum through that.
Not a lot of people know this, but there is a hosted version of SpiderFoot called SpiderFoot HX. Could you tell us more about it?
Steve: SpiderFoot HX is a representation of all the things I would like to do with the SpiderFoot open-source version but can't do because it would be a huge hassle to get it set up and documented in a way that anyone can download it and get it set up themselves.
For the open source version of SpiderFoot I always had the idea that it should be very simple to install and get it running, and for it to be available for Windows, Linux and Mac as well. Now that comes with a set of trade-offs; when you make those decisions about that kind of ease of use and installation, you don't typically opt for using a large-scale database platform or a queuing system or even a very large distributed set of components. For the open source version I've intentionally gone with a fairly monolithic setup where it's just one tool you run and everything is done. It has a built-in SQL-like database and it's very easy to use.
Now, with SpiderFoot HX, it's not like that. SpiderFoot HX has a bunch more functionality which is enabled through the use of a different database platform, cloud-based environment with much more computing power that is tailored to the use cases of SpiderFoot, and a distributed platform so it can scale along with the growth of scanning volume. It's really a different goal that it's trying to achieve; both versions are fundamentally doing the same thing in terms of getting open source intelligence and presenting that in a useful way, but obviously HX has a next level of functionality because of the technology and architecture behind it that I wasn't able to achieve with the open-source version.
Tell us about some future plans for SpiderFoot. What are your module and plugin goals?
Steve: The main focus is on SpiderFoot HX. I spend 80% of my spare time on it right now.
Real focus there being to scale the platform; the user base has grown to almost 2000 members of the private beta and that's growing everyday, so having to have the platform grow and remain stable on a shoestring budget is my challenge at the moment.
Beyond SpiderFoot HX, I'm working on integrating with Hunchly and GraphXR. There will also be new modules which will of course also go into the open-source version as well. Looking into investigation capabilities, I want to introduce more functionality so you can view results by data point, slowly and methodically since there are many use cases for SpiderFoot, so this will tailor to those types of investigations where you don't need to get every piece of data possible but you just want to navigate through the OSINT data piece by piece.
There are still more things on the list that tends to grow, especially with users and their requests. I still have this big backlog of data feeds I would like to integrate with. Fortunately, some people from the OSINT community have been contributing integrations as well, so if there are people out there who can do that, I can provide a list and you can go nuts. I and the growing list of SpiderFoot users would really appreciate that.
If you still haven't tried out SpiderFoot, go to their website and see for yourself its immense data and use cases, or follow Steve on Twitter to be the first to hear all the latest news regarding this amazing reconnaissance tool.
Both SpiderFoot and our own intelligence platform SecurityTrails can help you with speeding up your investigation and taking it to the next level. Signup for free API access today!
