Tools

Link Dumper - Web Crawler for Extracting Links and JavaScript Files

, and Dark Web Informer - Cyber Threat Intelligence

24 February 2025 . 12:41 AM

2 min read

📌 Overview

Link Dumper is a powerful Python-based web crawler designed for pentesting and reconnaissance. It scans websites for URLs and extracts JavaScript (.js), text (.txt), JSON (.json), and XML (.xml) files. This is useful for subdomain enumeration, API key discovery, and security analysis.

🚀 Features

✅ Extracts URLs from <a> and <script> tags
✅ Finds JavaScript files that might contain sensitive data
✅ Crawls additional links recursively for deeper analysis
✅ Extracts API keys & version numbers from JavaScript files
✅ Saves results in JSON format for further processing
✅ Handles relative and absolute links automatically
✅ Multi-threaded crawling for fast performance

🛠️ Installation

1️⃣ Clone the Repository

git clone https://github.com/walidzitouni/Link_dumper.git
cd Link-Dumper

2️⃣ Install Dependencies

pip install -r requirements.txt

📌 Usage

Basic Usage

Run the script and enter a target URL and Select [1]:

bash main.sh

python3 Link_dumper.py

For version 2, you can also provide a file with a list of URLs and Select [2]:

bash main.sh

Example Output

The tool will generate:

file_links.json → Contains extracted links
file_js.json → Contains extracted JavaScript file URLs

🔥 Features in Detail

1️⃣ Crawling & Link Extraction

Parses all <a> and <script> tags on a given website.
Extracts internal & external URLs.
Saves extracted URLs in file_links.json.

2️⃣ JavaScript File Detection

Finds all JavaScript files (.js).
Useful for discovering API endpoints, hidden parameters, and sensitive data.
Saves .js file URLs in file_js.json.