Hey there, Osint detectives!
Ever wanted to uncover the secrets behind a website, like who owns it, what it’s connected to, or if it’s up to no good? Welcome to the world of Open-Source Intelligence (OSINT), where domain investigation is your ticket to digging up valuable insights without needing a hacker’s toolkit. Whether you’re a cybersecurity newbie, a curious journalist, or just someone who loves solving puzzles, this tutorial is designed to guide you through the process of investigating domains in a fun, approachable way.
I’ve been hooked on OSINT for years, and I’m thrilled to share this guide to help you become a domain detective.
We’ll start with the basics, move into advanced techniques, and even dive into some cool Linux tools to automate the heavy lifting. No matter your skill level, you’ll walk away with practical skills to investigate any website like a pro.
Let's break it down:
- Manual Domain Investigation – Learn what makes up a URL and how to use free web tools and browser extensions to gather intel.
- Advanced OSINT Techniques – Discover stealthy ways to uncover hidden connections and map a domain’s digital footprint.
- Linux Tools for Automation – Use command-line tools to speed up your investigations, even if you’re new to the terminal.
Let’s grab our magnifying glass and get started!
Part 1: Manual Domain Investigation
Step 1: Understanding a URL
Before we dive into investigating domains, let’s break down what a URL (that web address you type into your browser) is made of. Think of it like a house address—it has parts that tell you exactly where to go. Here’s the anatomy of a URL:
- Protocol: This is the http:// or https:// part, telling your browser how to connect to the site.
- Subdomain: A prefix like blog.example.com, where blog is the subdomain.
- Domain Name: The main part, like example in example.com.
- Top-Level Domain (TLD): The ending, like .com, .org, or country codes like .uk.
- Path: The specific page or resource, like /about in example.com/about.
Knowing these parts is like learning the rules of a board game—it makes everything else easier. For example, when we use tools later, you’ll know whether you’re targeting the domain name or a specific subdomain.
Quick Tip: Don’t worry if you mix these up at first. Just bookmark this list, and you’ll get the hang of it!
Step 2: Using Web Services to Gather Intel
Now, let’s start investigating! Domain investigation (or reconnaissance) is about collecting data like who owns a domain, what servers it uses, or what other sites it’s linked to. The best part? There are tons of free web tools to help you. Here’s a rundown of my favorites and how to use them:
Whois Lookup (whois.domaintools.com): This tool shows who registered a domain, including their name, email, and registration date. Just type the domain (e.g., example.com) into the search bar. Note: Some country-specific TLDs (like .co.uk) might need a different WHOIS service.
Quick Tip: Don’t worry if you mix these up at first. Just bookmark this list, and you’ll get the hang of it!
Step 2: Using Web Services to Gather Intel
Now, let’s start investigating! Domain investigation (or reconnaissance) is about collecting data like who owns a domain, what servers it uses, or what other sites it’s linked to. The best part? There are tons of free web tools to help you. Here’s a rundown of my favorites and how to use them:
Whois Lookup (whois.domaintools.com): This tool shows who registered a domain, including their name, email, and registration date. Just type the domain (e.g., example.com) into the search bar. Note: Some country-specific TLDs (like .co.uk) might need a different WHOIS service.
- Whoxy (whoxy.com): Like Whois Lookup but with a superpower—it shows historical ownership and lets you search for other domains tied to the same owner. Try the “Reverse WHOIS” feature to find related domains.
- VirusTotal (virustotal.com): Checks if a domain is flagged for malware and lists subdomains or linked IPs. Paste the URL and explore the “Relations” tab for connections.
- SecurityTrails (securitytrails.com): Perfect for finding subdomains (like shop.example.com) and historical DNS records. Search the domain and check the “Subdomains” tab.
- Crt.sh (crt.sh): Searches SSL/TLS certificates, which often reveal hidden subdomains. Enter the domain and browse the certificate list.
- BuiltWith (builtwith.com): Shows what tech a site uses (e.g., WordPress, Google Analytics) and finds linked domains via shared tags. Search a domain and check “Relationships.”
- DNSlytics (dnslytics.com): A one-stop shop for domain, IP, and provider info. Free searches are limited, so prioritize key domains.
- WebArchive (archive.org): View past versions of a website to see how it’s changed. Search the domain and pick a snapshot date.
- Threat Intelligence Platform (threatintelligenceplatform.com): Combines WHOIS, DNS, and subdomain data for a quick overview. Just plug in the domain.
- MXtoolbox (mxtoolbox.com): Great for passive DNS analysis, showing historical IP addresses, linked domains and txt records. (!!! Hint)
How to Use Them: Pick a domain (say, example.com), visit each site, and enter the domain in the search bar. Spend a few minutes exploring each tool’s tabs or reports to see what stands out. For example, VirusTotal might flag a domain as suspicious, while BuiltWith could reveal it’s linked to 50 other sites via Google Analytics.
Quick Tip: Bookmark these tools in a folder called “OSINT Tools” for easy access. Trust me, you’ll use them a lot!
Step 3: Turbocharge with Browser Extensions
Browser extensions are like cheat codes for OSINT—they make gathering data faster and easier. If you use Chrome or Firefox, here are some must-have extensions and how to use them:
Quick Tip: Bookmark these tools in a folder called “OSINT Tools” for easy access. Trust me, you’ll use them a lot!
Step 3: Turbocharge with Browser Extensions
Browser extensions are like cheat codes for OSINT—they make gathering data faster and easier. If you use Chrome or Firefox, here are some must-have extensions and how to use them:
- Link Gopher: Click the extension on any webpage, and it lists all links, directories, and paths (e.g., /blog, /products). Try it on example.com to see its structure without clicking around.
- Instant Data Scraper: Scrapes data like tables or lists from a website and saves it as a CSV or Excel file. For example, on BuiltWith, you can scrape a list of domains linked to your target in one click.
- Open Multiple URLs: Got a list of links from Link Gopher or Instant Data Scraper? Paste them into this extension, and it opens them all at once. Set it to skip duplicates to save time.
- Wappalyzer: Shows the tech stack of a website (e.g., CMS, servers) right in your browser. Click the icon while visiting a site to see the details.
- Hunter.io: Finds email addresses tied to a domain. Click the extension on a website to see associated emails, great for verifying ownership.
How to Install: Go to the Chrome Web Store or Firefox Add-ons site, search for these extensions, and click “Add to Browser.” Once installed, their icons appear in your toolbar, ready to use.
Quick Tip: Test these on a familiar site like tesla.com to practice. For example, use Link Gopher to see all its directories, then scrape related domains with Instant Data Scraper.
Quick Tip: Test these on a familiar site like tesla.com to practice. For example, use Link Gopher to see all its directories, then scrape related domains with Instant Data Scraper.
Part 2: Advanced OSINT Techniques
You’ve got the basics down, so let’s level up! These advanced techniques will help you uncover hidden connections and dig deeper without alerting your target. Think of this as moving from a magnifying glass to a microscope.
Step 4: Passive Reconnaissance
Passive reconnaissance means gathering intel without directly interacting with the target, keeping you under the radar. Here’s how to do it:
- Find Subdomains: Use SecurityTrails or Crt.sh to discover subdomains like api.example.com or test.example.com. These can reveal hidden services or vulnerabilities. Just enter the domain and check the subdomain list.
- Reverse WHOIS Lookups: On Whoxy, use the “Reverse WHOIS” feature to find other domains registered by the same person or company. Enter an email or name from a WHOIS record to start.
- Analyze DNS Records: Tools like DNSlytics show DNS records (A, MX, CNAME, TXT), revealing mail servers or third-party services. Search the domain and look at the “DNS Records” section.
- Check Historical Data: Use WebArchive to view old versions of a website or SecurityTrails for past DNS records. This can show if a site switched from a blog to a scam page.
Quick Tip: Combine results from multiple tools. For example, find subdomains on Crt.sh, then check their DNS records on DNSlytics for a fuller picture.
Step 5: Mapping the Infrastructure
To really understand a domain, map its digital ecosystem—its IPs, servers, and related domains. Here’s how:
Step 5: Mapping the Infrastructure
To really understand a domain, map its digital ecosystem—its IPs, servers, and related domains. Here’s how:
- Track IP History: VirusTotal or PassiveTotal show which IPs a domain has used over time. This can reveal hosting changes or shared servers.
- Find Shared Infrastructure: Look for other domains on the same IP using VirusTotal’s “Relations” tab. Shared IPs might point to related sites.
- Check SSL Certificates: Crt.sh lists certificates, which can uncover subdomains or misconfigured security settings. Search the domain and review the certificate details.
Quick Tip: Save your findings in a spreadsheet to track connections. For example, note IPs and linked domains to spot patterns.
Step 6: Social Media and Content Analysis
Domains often connect to social media or external content. Here’s how to explore those links:
Step 6: Social Media and Content Analysis
Domains often connect to social media or external content. Here’s how to explore those links:
- Search for Mentions: On X, search from:example.com or site:x.com example.com to find posts or profiles tied to the domain. Google works too (e.g., "example.com" site:twitter.com).
- Analyze Metadata: If a domain hosts images or PDFs, use ExifTool (online version at exif.tools) to check metadata for author names or locations. For documents, try FOCA to extract metadata.
- Check Backlinks: Use Ahrefs or Moz (free tiers) to see which sites link to your target. This shows its influence and network.
Quick Tip: Set up Google Alerts for your target domain to get real-time updates on new mentions.
Step 7: Advanced WHOIS Analysis
WHOIS data can reveal more than you think. Try these tricks:
Step 7: Advanced WHOIS Analysis
WHOIS data can reveal more than you think. Try these tricks:
- Cross-Reference Details: Compare emails, phone numbers, or organization names across WHOIS records to find patterns or aliases.
- Bypass Privacy Protection: Many domains use privacy services to hide details. Whoisology or historical WHOIS records on Whoxy can sometimes uncover the real owner.
- Map Locations: If WHOIS includes an address, plug it into Google Maps to check if it’s legit or a proxy service.
Quick Tip: Save WHOIS data as a text file for easy reference later.
Example: Investigating a Suspicious Domain
Let’s walk through a real-world scenario. Say you’re checking out shady-site.com:
Let’s walk through a real-world scenario. Say you’re checking out shady-site.com:
- Run a WHOIS lookup on whois.domaintools.com to get the registrant’s email and registration date.
- Check VirusTotal for malware flags and subdomains like login.shady-site.com.
- Use Crt.sh to find SSL certificates, which might reveal hidden subdomains.
- Browse WebArchive to see if the site’s content has changed suspiciously (e.g., from a blog to a phishing page).
- Use BuiltWith to check for tech like Cloudflare and find linked domains via shared analytics tags.
- Search X for mentions of shady-site.com to see if users are complaining about scams.
By combining these steps, you’ll get a clear picture of whether shady-site.com is legit or trouble.
Part 3: Linux Tools for Automation
Ready to feel like a tech wizard? Linux command-line tools can automate your investigations, saving you time and uncovering details you might miss manually. Don’t worry if the terminal feels intimidating—these tools are beginner-friendly with a little practice.
Step 8: Must-Have Linux Tools
Here’s a lineup of powerful tools and how to use them. You’ll need a Linux system (like Ubuntu) or a virtual machine to try these out.
Step 8: Must-Have Linux Tools
Here’s a lineup of powerful tools and how to use them. You’ll need a Linux system (like Ubuntu) or a virtual machine to try these out.
- dig: Queries DNS records (A, MX, NS, TXT).
dig example.com ANY
Shows IPs, mail servers, and more.
- nslookup: Another DNS query tool.
nslookup -type=ANY example.com
- whois: Gets WHOIS data from the terminal.
whois example.com
- Amass: Finds subdomains and maps networks.
amass enum -d example.com -o subdomains.txt
- Subfinder: Fast subdomain enumeration.
subfinder -d example.com -o subdomains.txt
- theHarvester: Collects emails, subdomains, and hosts.
theHarvester -d example.com -b all -f results.html
- dnsrecon: Enumerates DNS records and subdomains.
dnsrecon -d example.com -t std
- Gowitness: Takes screenshots of websites.
gowitness single --url https://example.com
- Wget: Downloads website content.
wget -r -l 1 https://example.com
- Recon-ng: A modular OSINT framework.
recon-ng > workspaces create example_investigation > modules load recon/domains-hosts/google_site_web > set SOURCE example.com > run
How to Install: On Ubuntu, use sudo apt install for tools like dig, whois, and wget. For others, use pip (e.g., pip install theHarvester) or follow the tool’s GitHub instructions. Search online for “install [tool] Ubuntu” if you get stuck.
Quick Tip: Practice these commands on a test domain like tesla.com to build confidence.
Step 9: Automate with a Bash Script
To make things super easy, I’ve created a Bash script that runs multiple tools and saves the results. Here’s how to use it:
#!/bin/bash Define the target domain DOMAIN=$1 Create output directory mkdir -p $DOMAINcd $DOMAIN WHOIS lookup whois $DOMAIN > whois.txt DNS enumeration dig $DOMAIN ANY > dig.txtnslookup -type=ANY $DOMAIN > nslookup.txt Subdomain enumeration amass enum -d $DOMAIN -o amass_subdomains.txtsubfinder -d $DOMAIN -o subfinder_subdomains.txt Collect emails and hosts theHarvester -d $DOMAIN -b all -f theharvester_results.html Take screenshots of main domain gowitness single --url https://$DOMAIN Download website content wget -r -l 1 https://$DOMAIN echo "Investigation complete. Results saved in $DOMAIN directory."
How to Use the Script:
- Open a text editor and paste the script.
- Save it as domain_investigation.sh.
- Make it executable:
chmod +x domain_investigation.sh
- Run it with your target domain:
./domain_investigation.sh example.com
Quick Tip: Customize the script by adding more tools or commands as you get comfortable.
And there you have it! You’re now equipped with the knowledge and tools to investigate domains like a pro. Remember, OSINT is all about curiosity and persistence. The more you practice, the better you’ll get at uncovering hidden details and connecting the dots.