From Scuba and Submarines to DDoS: Diving in with Jose Hernandez from Splunk
Reading time: 14 minutesAfter a long day at work, what do you do when you come home? You might catch up on reading the latest tech trends and research, work on learning new programming languages, or any other activities that will advance your career. While this is all helpful for your professional life, having hobbies can be beneficial to every aspect of your life, including your career.
Whatever hobby you decide to take on, it will help you develop a new skill set in one area that will be transferable to other fields. Even if they seem unrelated to your profession, having diverse interests and an uncommon blend of skills can help you view already established fields from a new perspective and produce breakthrough ideas. This is exactly what Jose Enrique Hernandez did.
Jose is currently a security researcher at Splunk, and his experiences range from building DDoS protection systems, improving organizations' cyber-defense strategies, and developing new security tools, which he documents on his blog. Led by his passion for scuba diving and his hardware knowledge, he designed and programmed an underwater remote controlled vehicle — the hacker submarine! He is also a very active member of the security and open source community and is a frequent speaker at industry conferences, showing us how important it is to contribute to your community.
Jose's interview is special in that we will learn you can use your hobbies, combined with your nerd powers to bring innovations to your industry. He lives in Miami, where he enjoys scuba diving regularly, so we decided to join him on one of his diving expeditions. We also wanted to have some fun with it.
Media might portray hackers as nerds sitting in dark rooms at their laptops, but we're sure they didn't know that the true nerds also bring their laptops underwater.
SecurityTrails: You live in Miami now, but you also lived in the Bay area. What made you leave the tech mecca for The Magic City?
Jose Hernandez: I was presented the opportunity to co-found Zenedge as VP of Technology. The engineering team was based in Santa Monica, and it didn't take much to convince my wife to move. It did not feel like a tremendous change because Silicon Beach is there, which has a small, growing computer security community. After living in Santa Monica for about a year and a half, my wife became pregnant and we decided to move back to Miami to raise our son. It also helped that by that time Zenedge had expanded their headquarters to Aventura, FL, which gave us additional incentive to move back.
Your favorite hobby is scuba diving and it actually inspired you for one amazing project you built — an underwater remote controlled vehicle used to test and measure toxicity in Miami's waters! Tell us a little bit more about that project; how did you build it?
Jose: It all started out as a dare during a Hack Miami meetup up in an old cafe called "Planet Linux Caffe." I gave a presentation about underwater operated vehicles (specifically openROV.com) and how I wished there was one to see how good a dive spot was before jumping in the water. One thing led to the next, and someone dared me to make one. Although I did not know much about electronics, low level programming, or anything nearly close to robotics, for that matter, I decided to accept the challenge. At the time, I was working as a Systems Engineer at Prolexic under the leadership of Wadih Khairallah. Wadih was a very knowledgeable and skilled maker and taught me everything I needed to get started on the submarine project. He helped me purchase parts, soder, and even get started programming arduinos. After going through a few whiteboarding sessions with him and another colleague named Berto, I drafted a few general designs and then a more refined one.
At that point, my wife jumped on the bandwagon and helped me assemble the whole thing, as well as test it over the course of a few months. After being encouraged by Robert Beatty to showcase the ROV in Lab Miami I met Rebekah Monson, a local journalist and friend whom helped me get sponsored by Awesome Foundation, which allowed me to add water quality testing capabilities to the submarine to test the Miami waterways. Although I would have loved to continue working on that project, I was given the opportunity to take my first job with Splunk as a Security Architect and had to relocate to San Francisco. As a result, I had to shut down the project. Today its memories live at sensorsub.com.
How did you get into scuba diving and what drove you to it the most?
Jose: It's a bit of a boring story. I needed credits to graduate with my second college degree, and I had a choice between taking a Scuba Diving or a Wine Tasting class. That was an easy choice for me. I have always been very drawn to the mysteries of the ocean. Diving has allowed me to immerse myself in a world that is completely unknown and foreign, making me feel like I am an alien in someone else's world. The exploration aspect of discovering new things that are unlike what we are used to is very stimulating for me. It also helps me practice mindfulness, as I am forced to be present in the moment while underwater. Without the use of electronics, it is an escape from the terminal.
You found a great way to connect passion with profession. How beneficial do you think it is to have hobbies alongside your regular job?
Jose: Hobbies help you explore and stimulate your creativity. It is a blessing to be able to express yourself through creation. Hobbies help us engage in activities that not only further our growth, but also expand our unique abilities and interests. Without hobbies, we are just focusing our mental efforts on work, which can plague us with boredom if we do not shift our intellect to a variation of tasks. I believe there should be a balance between work and our individual passions.
You frequent industry conferences and are active in the infosec and open source software communities. What is the importance of being an active member of your community?
Jose: I think it is key to be out there networking and learning from others. Having a community you can identify with, rely on, and share ideas with is valuable for reaching exponential growth. Communities are proven to amplify messages.The more value you provide to your peers, the more chances you have to amplify your messages. But as Uncle Ben from Spider Man once said, "With great power comes great responsibility." Always make sure your contributions to any community are pure hearted and seeking to help and improve a fellow peer. I personally like to create pull requests for projects and help when I can, even if it is just a broken feature, improving documentation, or fixing minor errors. Many opensource projects are a labor of love and when a developer receives pull requests, it sparks joy, especially because it shows them that others care enough about the tool to contribute.
You work at Splunk as a Security Researcher. On what projects are you currently working?
Jose: I am currently working on a couple of projects but the ones that are keeping me the most occupied are:
- A tool chain built to replicate, test and create detections. More specifically, security research, as Splunk is currently shipping these detections as a free security update called Enterprise Security Content Update (ESCU) to the Splunk Enterprise Security SIEM. At a high level, the three projects that help us ultimately create these detections can be broken down into:
- An attack simulation tool, which allows us to not only run Atomic Red Canary kind of attack simulations, but also our very own custom ones.
- An attack range that has a set of vulnerable hosts with pre-built splunk agents/telemetry and configuration, in order to collect attack data from these simulations and ultimately index them in splunk (specifically into Data Models).
- A security content rule set, which has tools to validate our detections, convert them into the appropriate configuration for each splunk product, and also builds and tests the ESCU Splunk application.
- A cloud security attack detection and posture assessment focused on auditing the underlying technologies like terraform, Kubernetes clusters, and Docker containers. Much of the initial experimentation can be seen in the recent DEFCON27 talk "Using Splunk/ELK for Auditing AWS/GCP/Azure Security Posture" that Rod Soto and I delivered earlier this year.
- While studying certificate transparency data, we have been experimenting with cert-change-watcher to monitor how Splunk's certificate material gets treated, and we have been slowly adding features to enrich those behaviors. This has opened a world of opportunities for us to detect proactive phishing campaigns against us, as well as helped us discover exposed vulnerabilities.
So, about that certificate monitoring tool you built to monitor whenever a domain has an update to their certificate. For who can this tool be useful?
Jose: The original intent of cert-change-watcher was to give us proactive visibility of an attacker attempting to phish us by registering a certificate or squad on one of our certificates. The tool analyzes all certificate material and changes via certificate transparency logs. We are currently using it to send alerts when a change matches a given parent domain and its subdomains. Today the tool performs two functions once a change is detected:
- Send an alert to a configured slack channel. We use this as a passive form of monitoring. Our SOC team, as well as various corporate security teams, keep an eye on ongoing changes for anything fishy . See an example of an alert below.
- Generates an audit log (in JSON), which is picked up by one of our forwarders and indexed by Splunk, where the data is used for further investigation and trend mapping.
Recently we added a new feature to the tool, which gets me excited about its potential and further usability. The tool can now be configured to automatically execute a Shodan scan of any domain that experiences a change in their certificate. The Shodan scan enriches the change log with information on open ports, any obvious vulnerabilities, and technologies used (PHP, WordPress, Javascript libraries etc.) by the domain. The code for the project can be found at https://github.com/d1vious/cert-change-watcher.
You mentioned that you are working on cloud security attack detection at Splunk. With everything transitioning to cloud, it's important to monitor cloud environments more effectively and study attacks there. What are some of your favorite cloud monitoring tools?
Jose: Thus far, I have been enjoying adding JSON logging capabilities to CS-Suite and using it to scan our internal cloud environments, also the data coming out of Sysdig+Falco, Greynoise, and SecurityTrails SurfaceBrowser has been extremely valuable.
You're experienced in the DDoS field and you've built DDoS protection systems before. What approach do you take to DDoS protection?
Jose: You can break most DDoS attacks into two categories, network based or application based attacks. Each of these attacks has their unique features and ways to mitigate it and should be treated differently.
- Network based floods are usually seen as protocol amplification attacks like UDP, or NTP Amplification, etc.. that are built to flood your network pipe and consume all your bandwidth. Or protocol abuse attacks like SYN floods, ACK floods, or RESET floods that look to misuse TCP in order to consume all your resources at a network level. A good way to easily address these is to leverage a CDN (Cloudflare, Fastly, Akamai, etc.). A CDN usually has plenty of bandwidth on hand and is able to absorb many bandwidth or network abuse just by the nature that they are proxying traffic for your site they will take care of that first connection from clients. Thus CDN is the party that terminates those initial TCP connections from your clients and drop all weird protocols (SSDP, NTP, DNS, ICMP etc..) at their border before passing traffic on to your web service. Again this is assuming you are not hosting a DNS service, which then this advice does not apply to you and clearly a CDN does you no good here. Moreover CDNs tend to deploy hardware purposely built to drop out of state packets, malformed packets, and handle connections spike as part of their general infrastructure design. Something that if a business is not intending to do might add unnecessary cost to IT budgets.
- Application based attacks are usually targeting to exploit certain functions of your application stack and web servers. Common attacks are GET floods and POST floods
The more advanced ones tend to target site search functionality, or authentication pages and usually can handle encryption and javascript when making requests. These are tricky to identify and even mitigate as they can successfully cause damage even at a very low request rate since they aim to make server side process intensive requests. Best way to tackle application level attacks is to employ a combination of challenges like captcha pages (highly disruptive) or more subtle ones like javascript cookie challenges, and request rate base rate limits (amount of allowed POST/GETs per second on a site).
What are the most interesting DDoS attacks you've seen?
Jose: Oh so many, the ones that are always highlights in my mind are:
- Original SSL POST floods using an API for gaming clients.
- Anonymous HOIC's, which is more fun than interesting, mainly because they would put obscenities into the user-agent which made it very easy to block.
- Two online sports betting companies out of Costa Rica attacking each other during the streaming of the Latin Cup soccer game.
- API Abuse against Pokemon Go's.
What would you say makes an effective cyber-attack defense strategy?
Jose: Without knowing who your attackers are and what you are defending, the answer to this question can be quite complex. From my experience, the first thing companies need is visibility. Visibility comes in many forms — infrastructure logs, network captures, threat intelligence, etc. One specific aspect of cybersecurity visibility is, being able to see and study all interactions that touch customer data or critical databases. This is crucial for measuring the impact of cyber attacks when they do occur. Running scans internally and externally can help gain insight on possible vectors across your attack surface. Additionally, making sure you focus your efforts on identity access management, and automating as much as possible can help the process of onboarding and terminating user access.
From your experience, what has shown itself as the weakest link in an organization's defense?
Jose: At this point, it seems like an undisputed truism that humans are the weakest link in an organization's cyber security strategy. More specifically human error, from common vectors like phishing to more advanced ones like exposed secrets or publicly facing vulnerable services. The most common vulnerabilities can always be traced back to some sort of human error, or employee behavior.
And now last, but not least — did you have fun on the photoshoot?! (after the photos are done you can answer this one)
Jose: It was a blast. Fernando the photographer really did an amazing job, right down to the planning of the dive, prop selection, to his direction underwater. The shoot was completed in the course of two immersions via the Florida Key Largo's Rainbow Dive Center diving company. Specifically we dove the French Reef and Christmas Tree cave dive locations both of which were rich with wildlife such as with parrot fish, groupers, yellow tails, and lobsters. During our first immersion we struggled a bit with the props as we did not have enough weight on us to keep everything from floating around. It was an interesting experience to try and keep a desk from floating, and a fake antenna and lamp all at once while attempting to sit in a floating chair against an underwater current. It was definitely one of the most unique dives I have experienced thus far.
If you want to keep up with Jose's projects and new security tools he is developing, follow his blog and Twitter.
We hope you enjoyed learning about Jose's projects, and how you can combine your passion with your profession to create something innovative. Let us know what you thought about the photos and who you'd like to see next in our interview series by shooting us an email at [email protected].
