Pivot map
Pivot from here
Outputs from Web Crawler can become inputs for the next tool. These are the most relevant follow-on pivots in the library.
A global, open, stable, and secure Internet that serves the entire Asia Pacific community
ARIN is a nonprofit, member-based organization that administers IP addresses & ASNs in support of the operation and growth of the Internet.
AbuseIPDB provides IP reputation data and community abuse reports for identifying malicious hosts in network and threat investigations.
AccountKiller provides direct deletion links and step-by-step instructions for removing accounts on hundreds of websites and social platforms.
Investigator Use
Hakrawler is an open-source web crawler and endpoint discovery tool written in Go, designed for fast, automated discovery of URLs, links, and endpoints within web applications. For penetration testers, bug bounty hunters, and OSINT investigators conducting web reconnaissance, Hakrawler provides rapid JavaScript-aware crawling to map the full surface area of a target web application.
Unlike simple HTTP-based crawlers, Hakrawler uses headless browser rendering to execute JavaScript — critical for discovering endpoints in modern single-page applications (SPAs) built with React, Angular, or Vue, where content and navigation URLs are generated dynamically. This means Hakrawler surfaces URLs that traditional crawlers miss entirely.
Key capabilities include: recursive link following to configurable depth, discovery of JavaScript bundle imports and API endpoints referenced in scripts, extraction of form action URLs, identification of embedded subdomains, and output formatting compatible with other tools in the reconnaissance pipeline (such as Subfinder, Amass, and FFuf).
For bug bounty research, Hakrawler dramatically reduces the manual effort required to map an application's endpoint landscape before beginning vulnerability testing. Running Hakrawler against an in-scope target produces a URL list that can be fed into fuzz testing, parameter enumeration, and vulnerability scanners — accelerating the reconnaissance phase substantially.
For OSINT investigators mapping organizational web presence, Hakrawler can reveal internal-facing paths accidentally exposed, API documentation pages, staging environments, and administrative interfaces that are linked from within the main application but not prominently surfaced in navigation menus.
Installation requires Go: run "go install github.com/hakluke/hakrawler@latest". The tool reads target URLs from stdin, enabling integration with other tools via piping. Common usage: "echo https://example.com | hakrawler -depth 3 -plain".
Limitations include rate limiting and bot detection on sites that actively block automated crawling. Crawling without authorization is illegal and unethical — Hakrawler is for authorized security assessments, bug bounty programs operating within defined scope, or your own applications only.
Document the target, scope authorization, and crawl parameters when using Hakrawler in security assessments.
Before You Pivot
Capture the target, search terms, and why this source is relevant before you leave the page.
Archive volatile pages, save screenshots, and keep timestamps for anything that may change.
Treat one tool as a lead source. Confirm important findings with independent sources.
Related tools
ArchiveBox is self-hosted open-source web archiving for preserving websites, social posts, and online evidence for investigations.
Web technology information profiler tool. Find out what a website is built with.
CheckShortURL expands shortened URLs to reveal the final destination before clicking, supporting safe analysis of potentially malicious links.
Cutestat provides website analytics including traffic estimates, Alexa rank, server details, WHOIS data, and SEO metrics for any domain.
Down For Everyone Or Just Me confirms whether a website is globally offline or unavailable locally during OSINT investigations.
Photon is a fast OSINT crawler extracting URLs, emails, files, subdomains, and metadata from any target website for investigators.