PIP-INTEL - OSINT and Cyber Intelligence Tool

Pip-Intel is a powerful tool designed for OSINT (Open Source Intelligence) and cyber intelligence gathering activities. It consolidates various open-source tools into a single user-friendly interface simplifying the data collection and analysis processes for researchers and cybersecurity professionals.

Pip-Intel utilizes Python-written pip packages to gather information from various data points. This tool is equipped with the capability to collect detailed information through email addresses, phone numbers, IP addresses, and social media accounts. It offers a wide range of functionalities including email-based OSINT operations, phone number-based inquiries, geolocating IP addresses, social media and user analyses, and even dark web searches.

Uscrapper - Powerful OSINT Webscraper For Personal Data Collection

Introducing Uscrapper 2.0, A powerfull OSINT webscrapper that allows users to extract various personal information from a website. It leverages web scraping techniques and regular expressions to extract email addresses, social media links, author names, geolocations, phone numbers, and usernames from both hyperlinked and non-hyperlinked sources on the webpage, supports multithreading to make this process faster, Uscrapper 2.0 is equipped with advanced Anti-webscrapping bypassing modules and supports webcrawling to scrape from various sublinks within the same domain. The tool also provides an option to generate a report containing the extracted details.

Extracted Details:

Uscrapper extracts the following details from the provided website:

• Email Addresses: Displays email addresses found on the website.
• Social Media Links: Displays links to various social media platforms found on the website.
• Author Names: Displays the names of authors associated with the website.
• Geolocations: Displays geolocation information associated with the website.
• Non-Hyperlinked Details: Displays non-hyperlinked details found on the website including email addresses phone numbers and usernames.

Whats New?:

Uscrapper 2.0:

• Introduced multiple modules to bypass anti-webscrapping techniques.
• Introducing Crawl and scrape: an advanced crawl and scrape module to scrape the websites from within.
• Implemented Multithreading to make these processes faster.

Installation Steps:

git clone https://github.com/z0m31en7/Uscrapper.git

cd Uscrapper/install/ 
chmod +x ./install.sh && ./install.sh      #For Unix/Linux systems

Usage:

To run Uscrapper, use the following command-line syntax:

python Uscrapper-v2.0.py [-h] [-u URL] [-c (INT)] [-t THREADS] [-O] [-ns]

Arguments:

• -h, --help: Show the help message and exit.
• -u URL, --url URL: Specify the URL of the website to extract details from.
• -c INT, --crawl INT: Specify the number of links to crawl
• -t INT, --threads INT: Specify the number of threads to use while crawling and scraping.
• -O, --generate-report: Generate a report file containing the extracted details.
• -ns, --nonstrict: Display non-strict usernames during extraction.

Note:

• Uscrapper relies on web scraping techniques to extract information from websites. Make sure to use it responsibly and in compliance with the website's terms of service and applicable laws.

• The accuracy and completeness of the extracted details depend on the structure and content of the website being analyzed.

• To bypass some Anti-Webscrapping methods we have used selenium which can make the overall process slower.

Contribution:

• Make a pull request with all the necessary details and it will be merged after a review.
• You can contribute by making the regular expressions more efficient and accurate, or by suggesting some more features that can be added.

CloakQuest3r - Uncover The True IP Address Of Websites Safeguarded By Cloudflare

CloakQuest3r is a powerful Python tool meticulously crafted to uncover the true IP address of websites safeguarded by Cloudflare, a widely adopted web security and performance enhancement service. Its core mission is to accurately discern the actual IP address of web servers that are concealed behind Cloudflare's protective shield. Subdomain scanning is employed as a key technique in this pursuit. This tool is an invaluable resource for penetration testers, security professionals, and web administrators seeking to perform comprehensive security assessments and identify vulnerabilities that may be obscured by Cloudflare's security measures.

Key Features:

• Real IP Detection: CloakQuest3r excels in the art of discovering the real IP address of web servers employing Cloudflare's services. This crucial information is paramount for conducting comprehensive penetration tests and ensuring the security of web assets.

• Subdomain Scanning: Subdomain scanning is harnessed as a fundamental component in the process of finding the real IP address. It aids in the identification of the actual server responsible for hosting the website and its associated subdomains.

• Threaded Scanning: To enhance efficiency and expedite the real IP detection process, CloakQuest3r utilizes threading. This feature enables scanning of a substantial list of subdomains without significantly extending the execution time.

• Detailed Reporting: The tool provides comprehensive output, including the total number of subdomains scanned, the total number of subdomains found, and the time taken for the scan. Any real IP addresses unveiled during the process are also presented, facilitating in-depth analysis and penetration testing.

With CloakQuest3r, you can confidently evaluate website security, unveil hidden vulnerabilities, and secure your web assets by disclosing the true IP address concealed behind Cloudflare's protective layers.

Limitation

- Still in the development phase, sometimes it can't detect the real Ip.

- CloakQuest3r combines multiple indicators to uncover real IP addresses behind Cloudflare. While subdomain scanning is a part of the process, we do not assume that all subdomains' A records point to the target host. The tool is designed to provide valuable insights but may not work in every scenario. We welcome any specific suggestions for improvement. 

1. False Negatives: CloakReveal3r may not always accurately identify the real IP address behind Cloudflare, particularly for websites with complex network configurations or strict security measures.

2. Dynamic Environments: Websites' infrastructure and configurations can change over time. The tool may not capture these changes, potentially leading to outdated information.

3. Subdomain Variation: While the tool scans subdomains, it doesn't guarantee that all subdomains' A records will point to the pri   mary host. Some subdomains may also be protected by Cloudflare.

This tool is a Proof of Concept and is for Educational Purposes Only.

How to Use:

1. Run CloudScan with a single command-line argument: the target domain you want to analyze.

    git clone https://github.com/spyboy-productions/CloakQuest3r.git
   

   cd CloakQuest3r
   

   pip3 install -r requirements.txt
   

   python cloakquest3r.py example.com
   

2. The tool will check if the website is using Cloudflare. If not, it will inform you that subdomain scanning is unnecessary.

3. If Cloudflare is detected, CloudScan will scan for subdomains and identify their real IP addresses.

4. You will receive detailed output, including the number of subdomains scanned, the total number of subdomains found, and the time taken for the scan.

5. Any real IP addresses found will be displayed, allowing you to conduct further analysis and penetration testing.

CloudScan simplifies the process of assessing website security by providing a clear, organized, and informative report. Use it to enhance your security assessments, identify potential vulnerabilities, and secure your web assets.

Run It Online:

Run it online on replit.com : https://replit.com/@spyb0y/CloakQuest3r

Holehe - Tool To Check If The Mail Is Used On Different Sites Like Twitter, Instagram And Will Retrieve Information On Sites With The Forgotten Password Function

Holehe Online Version

Summary

Efficiently finding registered accounts from emails.

Holehe checks if an email is attached to an account on sites like twitter, instagram, imgur and more than 120 others.

• Retrieves information using the forgotten password function.
• Does not alert the target email.
• Runs on Python 3.

Installation

With PyPI

pip3 install holehe

With Github

git clone https://github.com/megadose/holehe.git
cd holehe/
python3 setup.py install

Quick Start

Holehe can be run from the CLI and rapidly embedded within existing python applications.

 CLI Example

holehe test@gmail.com

 Python Example

import trio
import httpx

from holehe.modules.social_media.snapchat import snapchat


async def main():
    email = "test@gmail.com"
    out = []
    client = httpx.AsyncClient()

    await snapchat(email, client, out)

    print(out)
    await client.aclose()

trio.run(main)

Module Output

For each module, data is returned in a standard dictionary with the following json-equivalent format :

{
  "name": "example",
  "rateLimit": false,
  "exists": true,
  "emailrecovery": "ex****e@gmail.com",
  "phoneNumber": "0*******78",
  "others": null
}

• rateLitmit : Lets you know if you've been rate-limited.
• exists : If an account exists for the email on that service.
• emailrecovery : Sometimes partially obfuscated recovery emails are returned.
• phoneNumber : Sometimes partially obfuscated recovery phone numbers are returned.
• others : Any extra info.

Rate limit? Change your IP.

Maltego Transform : Holehe Maltego

Thank you to :

• navlys
• Chris
• socialscan
• UhOh365
• soxoj
• mxrch (and for the logo)
• novitae

Donations

For BTC Donations : 1FHDM49QfZX6pJmhjLE5tB2K6CaTLMZpXZ

 License

GNU General Public License v3.0

Built for educational purposes only.

Modules

Xsubfind3R - A CLI Utility To Find Domain'S Known Subdomains From Curated Passive Online Sources

xsubfind3r is a command-line interface (CLI) utility to find domain's known subdomains from curated passive online sources.

Features

• Fetches domains from curated passive sources to maximize results.

• Supports stdin and stdout for easy integration into workflows.

• Cross-Platform (Windows, Linux & macOS).

Installation

Install release binaries (Without Go Installed)

Visit the releases page and find the appropriate archive for your operating system and architecture. Download the archive from your browser or copy its URL and retrieve it with wget or curl:

• ...with wget:

   wget https://github.com/hueristiq/xsubfind3r/releases/download/v<version>/xsubfind3r-<version>-linux-amd64.tar.gz

• ...or, with curl:

   curl -OL https://github.com/hueristiq/xsubfind3r/releases/download/v<version>/xsubfind3r-<version>-linux-amd64.tar.gz

...then, extract the binary:

tar xf xsubfind3r-<version>-linux-amd64.tar.gz

TIP: The above steps, download and extract, can be combined into a single step with this onliner

curl -sL https://github.com/hueristiq/xsubfind3r/releases/download/v<version>/xsubfind3r-<version>-linux-amd64.tar.gz | tar -xzv

NOTE: On Windows systems, you should be able to double-click the zip archive to extract the xsubfind3r executable.

...move the xsubfind3r binary to somewhere in your PATH. For example, on GNU/Linux and OS X systems:

sudo mv xsubfind3r /usr/local/bin/

NOTE: Windows users can follow How to: Add Tool Locations to the PATH Environment Variable in order to add xsubfind3r to their PATH.

Install source (With Go Installed)

Before you install from source, you need to make sure that Go is installed on your system. You can install Go by following the official instructions for your operating system. For this, we will assume that Go is already installed.

go install ...

go install -v github.com/hueristiq/xsubfind3r/cmd/xsubfind3r@latest

go build ... the development Version

• Clone the repository

   git clone https://github.com/hueristiq/xsubfind3r.git 

• Build the utility

   cd xsubfind3r/cmd/xsubfind3r && \
   go build .

• Move the xsubfind3r binary to somewhere in your PATH. For example, on GNU/Linux and OS X systems:

   sudo mv xsubfind3r /usr/local/bin/

  NOTE: Windows users can follow How to: Add Tool Locations to the PATH Environment Variable in order to add xsubfind3r to their PATH.

NOTE: While the development version is a good way to take a peek at xsubfind3r's latest features before they get released, be aware that it may have bugs. Officially released versions will generally be more stable.

Post Installation

xsubfind3r will work right after installation. However, BeVigil, Chaos, Fullhunt, Github, Intelligence X and Shodan require API keys to work, URLScan supports API key but not required. The API keys are stored in the $HOME/.hueristiq/xsubfind3r/config.yaml file - created upon first run - and uses the YAML format. Multiple API keys can be specified for each of these source from which one of them will be used.

Example config.yaml:

version: 0.3.0
sources:
    - alienvault
    - anubis
    - bevigil
    - chaos
    - commoncrawl
    - crtsh
    - fullhunt
    - github
    - hackertarget
    - intelx
    - shodan
    - urlscan
    - wayback
keys:
    bevigil:
        - awA5nvpKU3N8ygkZ
    chaos:
        - d23a554bbc1aabb208c9acfbd2dd41ce7fc9db39asdsd54bbc1aabb208c9acfb
    fullhunt:
        - 0d9652ce-516c-4315-b589-9b241ee6dc24
    github:
        - d23a554bbc1aabb208c9acfbd2dd41ce7fc9db39
        - asdsd54bbc1aabb208c9acfbd2dd41ce7fc9db39
    intelx:
        - 2.intelx.io:00000000-0000-0000-0000-000000000000
    shodan:
        - AAAAClP1bJJSRMEYJazgwhJKrggRwKA
    urlscan:
        - d4c85d34-e425-446e-d4ab-f5a3412acbe8

Usage

To display help message for xsubfind3r use the -h flag:

xsubfind3r -h

help message:

                _      __ _           _ _____      
__  _____ _   _| |__  / _(_)_ __   __| |___ / _ __ 
\ \/ / __| | | | '_ \| |_| | '_ \ / _` | |_ \| '__|
 >  <\__ \ |_| | |_) |  _| | | | | (_| |___) | |   
/_/\_\___/\__,_|_.__/|_| |_|_| |_|\__,_|____/|_| v0.3.0

USAGE:
  xsubfind3r [OPTIONS]

INPUT:
 -d, --domain string[]                 target domains
 -l, --list string                     target domains' list file path

SOURCES:
      --sources bool                   list supported sources
 -u,  --sources-to-use string[]        comma(,) separeted sources to use
 -e,  --sources-to-exclude string[]    comma(,) separeted sources to exclude

OPTIMIZATION:
 -t,  --threads int                    number of threads (default: 50)

OUTPUT:
     --no-color bool                   disable colored output
 -o, --output string                   output subdomains' file path
    -O, --output-directory string         output subdomains' directory path
 -v, --verbosity string                debug, info, warning, error, fatal or silent (default: info)

CONFIGURATION:
 -c,  --configuration string           configuration file path (default: ~/.hueristiq/xsubfind3r/config.yaml)

Contribution

Issues and Pull Requests are welcome! Check out the contribution guidelines.

Licensing

This utility is distributed under the MIT license.

Xurlfind3R - A CLI Utility To Find Domain'S Known URLs From Curated Passive Online Sources

xurlfind3r is a command-line interface (CLI) utility to find domain's known URLs from curated passive online sources.

Features

• Fetches URLs from curated passive sources to maximize results:
  • AlienVault's OTX
  • BeVigil
  • Common Crawl
  • URLScan
  • Github
  • Intelligence X
  • Wayback Machine
• With Wayback Machine, Parses URLs from robots.txt snapshots.
• With Wayback Machine, Parses URLs from webpages' snapshots.
• Cross-Platform (Windows, Linux & macOS)
• Supports URLs match and filter

Installation

Install release binaries (Without Go Installed)

Visit the releases page and find the appropriate archive for your operating system and architecture. Download the archive from your browser or copy its URL and retrieve it with wget or curl:

• ...with wget:

   wget https://github.com/hueristiq/xurlfind3r/releases/download/v<version>/xurlfind3r-<version>-linux-amd64.tar.gz

• ...or, with curl:

   curl -OL https://github.com/hueristiq/xurlfind3r/releases/download/v<version>/xurlfind3r-<version>-linux-amd64.tar.gz

...then, extract the binary:

tar xf xurlfind3r-<version>-linux-amd64.tar.gz

TIP: The above steps, download and extract, can be combined into a single step with this onliner

curl -sL https://github.com/hueristiq/xurlfind3r/releases/download/v<version>/xurlfind3r-<version>-linux-amd64.tar.gz | tar -xzv

NOTE: On Windows systems, you should be able to double-click the zip archive to extract the xurlfind3r executable.

...move the xurlfind3r binary to somewhere in your PATH. For example, on GNU/Linux and OS X systems:

sudo mv xurlfind3r /usr/local/bin/

NOTE: Windows users can follow How to: Add Tool Locations to the PATH Environment Variable in order to add xurlfind3r to their PATH.

Install source (With Go Installed)

Before you install from source, you need to make sure that Go is installed on your system. You can install Go by following the official instructions for your operating system. For this, we will assume that Go is already installed.

go install ...

go install -v github.com/hueristiq/xurlfind3r/cmd/xurlfind3r@latest

go build ... the development Version

• Clone the repository

   git clone https://github.com/hueristiq/xurlfind3r.git 

• Build the utility

   cd xurlfind3r/cmd/xurlfind3r && \
   go build .

• Move the xurlfind3r binary to somewhere in your PATH. For example, on GNU/Linux and OS X systems:

   sudo mv xurlfind3r /usr/local/bin/

  NOTE: Windows users can follow How to: Add Tool Locations to the PATH Environment Variable in order to add xurlfind3r to their PATH.

NOTE: While the development version is a good way to take a peek at xurlfind3r's latest features before they get released, be aware that it may have bugs. Officially released versions will generally be more stable.

Post Installation

xurlfind3r will work right after installation. However, BeVigil, Github and Intelligence X require API keys to work, URLScan supports API key but not required. The API keys are stored in the $HOME/.hueristiq/xurlfind3r/config.yaml file - created upon first run - and uses the YAML format. Multiple API keys can be specified for each of these source from which one of them will be used.

Example config.yaml:

version: 0.2.0
sources:
    - bevigil
    - commoncrawl
    - github
    - intelx
    - otx
    - urlscan
    - wayback
keys:
    bevigil:
        - awA5nvpKU3N8ygkZ
    github:
        - d23a554bbc1aabb208c9acfbd2dd41ce7fc9db39
        - asdsd54bbc1aabb208c9acfbd2dd41ce7fc9db39
    intelx:
        - 2.intelx.io:00000000-0000-0000-0000-000000000000
    urlscan:
        - d4c85d34-e425-446e-d4ab-f5a3412acbe8

Usage

To display help message for xurlfind3r use the -h flag:

xurlfind3r -h

help message:

                 _  __ _           _ _____      
__  ___   _ _ __| |/ _(_)_ __   __| |___ / _ __ 
\ \/ / | | | '__| | |_| | '_ \ / _` | |_ \| '__|
 >  <| |_| | |  | |  _| | | | | (_| |___) | |
/_/\_\\__,_|_|  |_|_| |_|_| |_|\__,_|____/|_| v0.2.0

USAGE:
  xurlfind3r [OPTIONS]

TARGET:
 -d, --domain string              (sub)domain to match URLs

SCOPE:
     --include-subdomains bool    match subdomain's URLs

SOURCES:
 -s,  --sources bool              list sources
 -u,  --use-sources string        sources to use (default: bevigil,commoncrawl,github,intelx,otx,urlscan,wayback)
      --skip-wayback-robots bool  with wayback, skip parsing robots.txt snapshots
      --skip-wayback-source bool  with wayback   , skip parsing source code snapshots

FILTER & MATCH:
 -f, --filter string              regex to filter URLs
 -m, --match string               regex to match URLs

OUTPUT:
     --no-color bool              no color mode
 -o, --output string              output URLs file path
 -v, --verbosity string           debug, info, warning, error, fatal or silent (default: info)

CONFIGURATION:
 -c,  --configuration string      configuration file path (default: ~/.hueristiq/xurlfind3r/config.yaml)

Examples

Basic

xurlfind3r -d hackerone.com --include-subdomains

Filter Regex

# filter images
xurlfind3r -d hackerone.com --include-subdomains -f '`^https?://[^/]*?/.*\.(jpg|jpeg|png|gif|bmp)(\?[^\s]*)?$`'

Match Regex

# match js URLs
xurlfind3r -d hackerone.com --include-subdomains -m '^https?://[^/]*?/.*\.js(\?[^\s]*)?$'

Contributing

Issues and Pull Requests are welcome! Check out the contribution guidelines.

Licensing

This utility is distributed under the MIT license.

SpiderSuite - Advance Web Spider/Crawler For Cyber Security Professionals

An advance cross-platform and multi-feature GUI web spider/crawler for cyber security proffesionals. Spider Suite can be used for attack surface mapping and analysis. For more information visit SpiderSuite's website.

Installation and Usage

Spider Suite is designed for easy installation and usage even for first timers.

• First, download the package of your choice.

• Then install the downloaded SpiderSuite package.

• See First time crawling with SpiderSuite article for tutorial on how to get started.

For complete documentation of Spider Suite see wiki.

Contributing

Can you translate?

Visit SpiderSuite's translation project to make translations to your native language.

Not a developer?

You can help by reporting bugs, requesting new features, improving the documentation, sponsoring the project & writing articles.

For More information see contribution guide.

Contributers

• 3nock (main developer)

Credits

This product includes software developed by the following open source projects:

• Google's Gumbo HTML Parser
• Google's Protocal bufffers
• SQLite database library
• Graphviz library

Seekr - A Multi-Purpose OSINT Toolkit With A Neat Web-Interface

A multi-purpose toolkit for gathering and managing OSINT-Data with a neat web-interface.

Introduction

Seekr is a multi-purpose toolkit for gathering and managing OSINT-data with a sleek web interface. The backend is written in Go and offers a wide range of features for data collection, organization, and analysis. Whether you're a researcher, investigator, or just someone looking to gather information, seekr makes it easy to find and manage the data you need. Give it a try and see how it can streamline your OSINT workflow!

Check the wiki for setup guide, etc.

Why use seekr over my current tool ?

Seekr combines note taking and OSINT in one application. Seekr can be used alongside your current tools. Seekr is desingned with OSINT in mind and optimized for real world usecases.

Key features

• Database for OSINT targets
• GitHub to email
• Account cards for each person in the database
• Account discovery intigrating with the account cards
• Pre defined commonly used fields in the database

Getting Started - Installation

Windows

Download the latest exe here

Linux (stable)

Download the latest stable binary here

Linux (unstable)

To install seekr on linux simply run:

git clone https://github.com/seekr-osint/seekr
cd seekr
go run main.go

Now open the web interface in your browser of choice.

Run on NixOS

Seekr is build with NixOS in mind and therefore supports nix flakes. To run seekr on NixOS run following commands.

nix shell github:seekr-osint/seekr
seekr

Intigrating seekr into your current workflow

journey
	title How to Intigrate seekr into your current workflow.
	section Initial Research
		Create a person in seekr: 100: seekr
    Simple web research: 100: Known tools
		Account scan: 100: seekr
	section Deeper account investigation
		Investigate the accounts: 100: seekr, Known tools
		Keep notes: 100: seekr
  section Deeper Web research
    Deep web research: 100: Known tools
    Keep notes: 100: seekr
	section Finishing the report
		Export the person with seekr: 100: seekr
		Done.: 100

Feedback

We would love to hear from you. Tell us about your opinions on seekr. Where do we need to improve?... You can do this by just opeing up an issue or maybe even telling others in your blog or somewhere else about your experience.

Legal Disclaimer

This tool is intended for legitimate and lawful use only. It is provided for educational and research purposes, and should not be used for any illegal or malicious activities, including doxxing. Doxxing is the practice of researching and broadcasting private or identifying information about an individual, without their consent and can be illegal. The creators and contributors of this tool will not be held responsible for any misuse or damage caused by this tool. By using this tool, you agree to use it only for lawful purposes and to comply with all applicable laws and regulations. It is the responsibility of the user to ensure compliance with all relevant laws and regulations in the jurisdiction in which they operate. Misuse of this tool may result in criminal and/or civil prosecut ion.