How to Use the SecurityTrails API™ to Detect User Agents on IP Addresses
Reading time: 6 minutesLocated in the HTTP header, the user agent is crucial in identifying the device accessing the site while presenting an optimal version of itself to that specific device.
For example, if you were using Google Chrome on Windows 10, any website you would visit would see the following user-agent:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36
As shown above, the exact version of Chrome being utilized can be found; in this case, “112.0.0.0” along with the operating system version “Windows 10” and system architecture “x64” as well. This user agent is commonly used to detect the device type, then display a customized website version to the visitor to enhance their experience. Looking further, this user agent can also protect your digital assets by detecting and blocking unknown versions and noting out-of-date operating systems or browsers, often seen as part of compromised systems used to perform DDoS attacks.
Today we’ll explore the impact of user agents in cybersecurity, classic methods of detecting user agents, and the step-by-step process of utilizing the SecurityTrails API™ to effectively detect user agents on IP addresses, empowering you to enhance your cyber security defenses and ensure the safety of your digital assets. Keep reading to learn more!
- Why determining user agents is essential for cybersecurity?
- Classic methods for detecting user agents from IP addresses
- Detecting user agents on specific IP addresses with the SecurityTrails API™
- Summary
Why determining user agents is essential for cybersecurity?
Determining the user agent plays an essential part in not only improving your web application's usability, wherein one can provide different versions of a web application depending on the user-agent accessing it for a near-perfect usability experience, but it also helps increase the security of a web application.
Blocking uncommon user agents, some of which can be part of attack swarms or botnets, is an essential step for a web application. These days, however, simply safelisting only commonly seen user agents such as web browsers is often seen as a more straightforward solution.
For example, if you own a website that is under attack, it becomes easier to block specific user agents compared to millions of IP addresses. Botnets using compromised older devices are often seen as part of DDoS attack swarms, so blocking away these older versions can help offer a layer of security to your website.
While user agents can be “faked” (for example, a Python script can be programmed to appear as Google Chrome), as is often seen with scraping programs used to scrape information from websites, these are easy to block away. This is done by presenting the user-agent with specific challenges, such as Javascript-based puzzles, which the Python script wouldn’t be able to solve when it exceeds a certain number of requests per second.
When it comes to detecting user agents running on certain IP addresses, the importance relies on many different layers, such as estimating the number of users on specific IP addresses and seeing which ones are running old vulnerable software by inspecting the user-agent data.
Classic methods for detecting user agents from IP addresses
Finding the user agent running on a certain IP address can be achieved in many different ways, as various programming languages offer an inbuilt method to grab the user agent from the HTTP headers sent by a visitor's web browser. For example, if your web application was written in PHP, you could use:
echo $_SERVER['HTTP_USER_AGENT'];
This would then return the operating system, architecture, and web browser version.
Similarly, the following code in Golang can be used to set up an HTTP server, then gather the user-agent of a visitor:
package main
import (
"fmt"
"log"
"net/http"
)
func handler(w http.ResponseWriter, r *http.Request) {
// Retrieve the User-Agent header from the request
userAgent := r.UserAgent()
// Print the User-Agent
fmt.Println("User-Agent:", userAgent)
// You can also write the User-Agent to the response
w.Write([]byte("User-Agent: " + userAgent))
}
func main() {
http.HandleFunc("/", handler)
log.Fatal(http.ListenAndServe(":80", nil))
}
Which would then display the following output with the user agent in your console as well as the browser:
While these methods may work effectively to capture the user-agent of the current user visiting your website, you will need more insight into the other user agents seen on any specific IP address.
And here’s where this network user-agent information can become truly important: Cyber criminals will often mask or fake user agents depending on the website they are trying to access. This rapid changing of user agents (sometimes numbering in the hundreds) seen on a single IP address can lead you to proactively determine whether an IP address belongs to a large public Wi-Fi hotspot or to a cyber criminal constantly changing their application’s user agents to gain access to a website. It seems like a tricky situation... However (spoiler alert), there is always light at the end of the tunnel. Keep reading.
Detecting user agents on specific IP addresses with the SecurityTrails API™
Let’s try gathering the user agents detected for the IP address “12.22.33.44” by using the following CURL command, which queries the SecurityTrails user-agent API endpoint:
curl --request GET \
--url 'https://api.securitytrails.com/v1/ips/12.22.33.44/useragents?page=1' \
--header 'APIKEY: YOUR-API-KEY'
Note: Replace “YOUR-API-KEY” with your SecurityTrails API™ Key
Doing so returns the following result, which contains four different user agents detected for this IP address, including two smartphones and two desktop devices (one can assume this is a household environment):
{
"endpoint": "/v1/ips/11.22.33.44/useragents",
"meta": {
"max_page": 1,
"page": 1,
"query": "11.22.33.44",
"total_pages": 1
},
"record_count": 4,
"records": [
{
"browser_family": "Chrome",
"client": {
"engine": "Blink",
"engine_version": "n/a",
"name": "Chrome Mobile",
"type": "browser",
"version": "111.0.0.0"
},
"device": {
"brand": "Tecno Mobile",
"model": "KG5m",
"type": "smartphone"
},
"lastseen": 1679389437,
"os": {
"name": "Android",
"platform": "n/a",
"version": "11"
},
"os_family": "Android",
"user_agent": "Mozilla/5.0 (Linux; Android 11; TECNO KG5m) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Mobile Safari/537.36"
},
{
"browser_family": "Chrome",
"client": {
"engine": "Blink",
"engine_version": "n/a",
"name": "Chrome Webview",
"type": "browser",
"version": "109.0.5414.117"
},
"device": {
"brand": "Tecno Mobile",
"model": "KG5m",
"type": "smartphone"
},
"lastseen": 1679147332,
"os": {
"name": "Android",
"platform": "n/a",
"version": "11"
},
"os_family": "Android",
"user_agent": "Mozilla/5.0 (Linux; Android 11; TECNO KG5m Build/RP1A.200720.011; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/109.0.5414.117 Mobile Safari/537.36"
},
{
"browser_family": "n/a",
"client": {
"engine": "Blink",
"engine_version": "n/a",
"name": "Yandex Browser",
"type": "browser",
"version": "23.1.4.776"
},
"device": {
"brand": "n/a",
"model": "n/a",
"type": "desktop"
},
"lastseen": 1679033575,
"os": {
"name": "Windows",
"platform": "x64",
"version": "10"
},
"os_family": "Windows",
"user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 YaBrowser/23.1.4.776 Yowser/2.5 Safari/537.36"
},
{
"browser_family": "n/a",
"client": {
"engine": "Blink",
"engine_version": "n/a",
"name": "Yandex Browser",
"type": "browser",
"version": "23.1.3.949"
},
"device": {
"brand": "n/a",
"model": "n/a",
"type": "desktop"
},
"lastseen": 1678426847,
"os": {
"name": "Windows",
"platform": "x64",
"version": "10"
},
"os_family": "Windows",
"user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 YaBrowser/23.1.3.949 Yowser/2.5 Safari/537.36"
}
]
}
Beyond its ease of use, another essential benefit of using the SecurityTrails API™ is the additional user agents seen on the IP address information revealed for the IP address being queried. While the manual gathering of user agents from HTTP headers or browser extensions is an option, accurately determining whether bots are using an IP address, or whether it’s a residential environment or an office space, etc, becomes nearly impossible without having an extensive collection of neighboring IP address user-agent data.
Summary
To maintain your website’s security, it’s important to be aware of user agents and take necessary precautions to prevent potential security threats. Blocking away out-of-date browsers’ and operating systems’ user agents can help thwart usually compromised system botnets from accessing and exploiting your web application.
Determining user agents can help from a security and usability perspective, as you can redirect users running out-of-date browsers and operating systems toward updating their software for a better experience when using your web application.
Using the SecurityTrails API™ makes determining your web application's visitors’ user agents easy and accurate. This API endpoint requires a custom subscription, please contact us for more information.
