Beyond Your Borders: Proactive Threat Hunting with Shodan and Open Source Intel

If you’re in the trenches of IT analysis or cybersecurity, you know the threat landscape isn’t just about what’s happening inside your network. The connections your internal systems make to the outside world can be a significant, often overlooked, attack vector. Today, we’re diving into how you can extend your threat hunting beyond your perimeter using powerful open-source tools, specifically focusing on Shodan.

We’ll explore how to leverage Shodan to identify vulnerable external hosts that are actively communicating with systems within your network. Why is this crucial? Because, as we all know, trust relationships are everywhere. An external host your systems trust could be compromised, providing a backdoor right into your environment. Tracking vulnerabilities even “one hop out” is a vital part of a proactive threat hunting strategy.

Setting the Stage: Intelligence-Driven Threat Hunting

Recently, CISA (Cybersecurity and Infrastructure Security Agency) released an advisory highlighting common CVEs exploited by People’s Republic of China (PRC) state-sponsored actors since 2020. This kind of intelligence is gold, but the real question is: how do you operationalize it? How do you take a list of CVEs and actively hunt for them in a way that’s relevant to your organization?

That’s where a tool like Shodan shines. It allows us to take this CISA report, or any similar threat intelligence, and see if our network is touching any external systems known to be vulnerable to these specific exploits.

Why External Vulnerability Matters: The Trust Factor

Think about it: your firewall might allow traffic from a specific external IP because it’s a trusted partner, a cloud service, or a remote management interface. But what if that trusted external IP is running an unpatched, vulnerable service?

• Compromised External Assets: If an external host your network communicates with is compromised (perhaps due to one of those CISA-highlighted CVEs), any data exchanged or trust relationship could be abused.
• Firewall & Allow Lists: Many organizations have allow lists for specific external IPs. If one of those IPs becomes vulnerable, it’s a direct line of sight for attackers.
• Supply Chain & Partner Risks: Your security is also dependent on the security of your partners and vendors whose systems interact with yours.

Identifying these vulnerable external touchpoints allows you to reassess those trust relationships. Maybe you need to implement stricter filtering, apply more scrutiny to traffic from that source, or alert the owner of the external asset.

The Manual Slog: Why We Need Automation

Imagine you have a list of concerning CVEs. To manually check for exposure, you’d typically:

1. Collect network traffic logs (PCAPs, Zeek/Bro logs, firewall logs).
2. Filter out all your internal (RFC 1918) IP addresses to focus on external IPs.
3. Individually look up each external IP in Shodan.
4. Manually check the Shodan record for each IP against your list of CVEs.
5. Compile your findings, probably in a spreadsheet.

This process is incredibly time-consuming, especially for even medium-sized networks. What if you have thousands of external IP communications a day? And what if, after all that effort, the CVE isn’t even something Shodan tracks, or your specific network isn’t communicating with any vulnerable hosts for that CVE? It’s a prime candidate for burnout and missed threats.

A Quick Look at Shodan

For those unfamiliar, Shodan is a search engine for Internet-connected devices. You can search for specific IP addresses and get a wealth of information:

• Network Info: ISP, ASN, geolocation.
• Open Ports: What services are listening.
• Banner Information: Software versions (e.g., DropBear SSH).
• Vulnerabilities: Shodan often lists detected CVEs for a host. You can use the vuln: filter in Shodan (requires a paid account for full functionality) to search directly for CVEs.

For example, when we cross-referenced the CISA advisory’s 16 CVEs with Shodan, we found three that had a significant number of publicly vulnerable hosts:

• A Cisco vulnerability (around 3,000 hosts)
• A Citrix vulnerability (around 300 hosts)
• A Pulse Secure vulnerability (around 300 hosts)

These became our focus for a Shodan-based hunt, as Shodan had visibility into them. Other CVEs, while still important, might require different tools or approaches if Shodan doesn’t have data on them. It’s all about understanding your tools’ capabilities and limitations.

Automating the Hunt: Introducing “Shift”

Because the manual process is so painful, we developed and open-sourced a tool called Shift to automate this exact workflow. We actually released it during our talk at the RSA conference.

You can find Shift on GitHub: https://github.com/Insane-Forensics/Shodan_SHIFT

Shift automates the search for vulnerable external hosts (based on CVEs you define) that are present in your PCAP files. Here’s how it works:

1. PCAP Ingestion: It uses Tshark to process your packet capture files (PCAP or PCAPNG).
2. IP Extraction: It pulls out all unique external source and destination IP addresses from the IP layer. (Currently focused on TCP/IP; DNS and other layers are future considerations).
3. Shodan Queries: Using your Shodan API key, it queries Shodan for information on each extracted public IP.
4. CVE Cross-Referencing: It checks the Shodan host records against a list of CVEs you provide.
5. Output: Results are delivered in JSON, CSV, or can be sent directly to Elasticsearch.

This automation makes it feasible to perform these checks regularly, even in larger environments.

Getting Started with Shift: A Quick Guide

Ready to give it a try? Here’s the rundown:

1. Prerequisites & Installation:

   • Ensure Tshark and Python 3 are installed on your system.
   • Clone the Shift repository from GitHub.
   • Install the necessary Python dependencies: pip3 install -r requirements.txt (we provide a python-requirements.txt in the repo).

2. Create Your CVE Definition File:

   • This is a simple text file (e.g., cves.txt) containing a comma-separated list of the CVEs you want to hunt for.
   • Example (based on the CISA report focus): CVE-2021-22893,CVE-2020-5902,CVE-2019-19781

3. Gather Your PCAPs:

   • Collect PCAP or PCAPNG files from network segments you want to analyze. These can come from tcpdump, Wireshark, your firewall (like Palo Alto Networks), or any other packet capture utility.

4. Run Shift:

   • Execute the script from your terminal. The basic syntax is:
     

     Bash

      

     python3 main.py --pcap <your_pcap_file.pcap> --cve-file <cves.txt> --shodan-key <YOUR_SHODAN_API_KEY> --csv <output_results.csv>
     

   • Replace placeholders with your actual file names and API key.
   • Other output options include --json <output.json> or ELK-specific arguments (check main.py --help for details, including authentication for ELK).

   You’ll see terminal output as Shift extracts IPs, queries Shodan, and processes results. The output file (e.g., your CSV) will list the external IPs from your PCAP that Shodan identifies as having one of your specified CVEs.

   Sample CSV Output:

   Code snippet

    

   IPAddress,CVE
   198.51.100.10,CVE-2021-22893
   203.0.113.25,CVE-2020-5902
   

From Data to Action

With this output, you’ve successfully bridged the gap between general threat intelligence (like a CISA report) and specific, actionable insights for your network. You now have a list of internal communications to potentially compromised external hosts.

The next steps involve traditional incident response and analysis:

• Investigate the internal hosts making these connections.
• Examine the nature and volume of traffic.
• Determine if the trust relationship with the external vulnerable IP is necessary.
• Consider implementing tighter firewall rules or other compensating controls.
• If appropriate, notify the owner of the external vulnerable asset.

Wrapping Up

Proactive threat hunting means looking for trouble before it finds you. By leveraging external intelligence and tools like Shodan, automated by scripts like Shift, you can gain crucial visibility into risks that lie just beyond your network perimeter. We hope this approach and our open-source tool, Shift, prove valuable in your defensive efforts.

We’re always keen to hear your thoughts and experiences. If you try out Shift, have ideas for improvement, or other topics you’d like to see covered, please don’t hesitate to reach out!