Login
The proliferation of new top-level domains (TLDs) has exacerbated a well-known security weakness: Many organizations set up their internal Microsoft authentication systems years ago using domain names in TLDs that didn’t exist at the time. Meaning, they are continuously sending their Windows usernames and passwords to domain names they do not control and which are freely available for anyone to register. Here’s a look at one security researcher’s efforts to map and shrink the size of this insidious problem.
At issue is a well-known security and privacy threat called “namespace collision,” a situation where domain names intended to be used exclusively on an internal company network end up overlapping with domains that can resolve normally on the open Internet.
Windows computers on a private corporate network validate other things on that network using a Microsoft innovation called Active Directory, which is the umbrella term for a broad range of identity-related services in Windows environments. A core part of the way these things find each other involves a Windows feature called “DNS name devolution,” a kind of network shorthand that makes it easier to find other computers or servers without having to specify a full, legitimate domain name for those resources.
Consider the hypothetical private network internalnetwork.example.com: When an employee on this network wishes to access a shared drive called “drive1,” there’s no need to type “drive1.internalnetwork.example.com” into Windows Explorer; entering “\\drive1\” alone will suffice, and Windows takes care of the rest.
But problems can arise when an organization has built their Active Directory network on top of a domain they don’t own or control. While that may sound like a bonkers way to design a corporate authentication system, keep in mind that many organizations built their networks long before the introduction of hundreds of new top-level domains (TLDs), like .network, .inc, and .llc.
For example, a company in 2005 builds their Microsoft Active Directory service around the domain company.llc, perhaps reasoning that since .llc wasn’t even a routable TLD, the domain would simply fail to resolve if the organization’s Windows computers were ever used outside of its local network.
Alas, in 2018, the .llc TLD was born and began selling domains. From then on, anyone who registered company.llc would be able to passively intercept that organization’s Microsoft Windows credentials, or actively modify those connections in some way — such as redirecting them somewhere malicious.
Philippe Caturegli, founder of the security consultancy Seralys, is one of several researchers seeking to chart the size of the namespace collision problem. As a professional penetration tester, Caturegli has long exploited these collisions to attack specific targets that were paying to have their cyber defenses probed. But over the past year, Caturegli has been gradually mapping this vulnerability across the Internet by looking for clues that appear in self-signed security certificates (e.g. SSL/TLS certs).
Caturegli has been scanning the open Internet for self-signed certificates referencing domains in a variety of TLDs likely to appeal to businesses, including .ad, .associates, .center, .cloud, .consulting, .dev, .digital, .domains, .email, .global, .gmbh, .group, .holdings, .host, .inc, .institute, .international, .it, .llc, .ltd, .management, .ms, .name, .network, .security, .services, .site, .srl, .support, .systems, .tech, .university, .win and .zone, among others.
Seralys found certificates referencing more than 9,000 distinct domains across those TLDs. Their analysis determined many TLDs had far more exposed domains than others, and that about 20 percent of the domains they found ending .ad, .cloud and .group remain unregistered.
“The scale of the issue seems bigger than I initially anticipated,” Caturegli said in an interview with KrebsOnSecurity. “And while doing my research, I have also identified government entities (foreign and domestic), critical infrastructures, etc. that have such misconfigured assets.”
Some of the above-listed TLDs are not new and correspond to country-code TLDs, like .it for Italy, and .ad, the country-code TLD for the tiny nation of Andorra. Caturegli said many organizations no doubt viewed a domain ending in .ad as a convenient shorthand for an internal Active Directory setup, while being unaware or unworried that someone could actually register such a domain and intercept all of their Windows credentials and any unencrypted traffic.
When Caturegli discovered an encryption certificate being actively used for the domain memrtcc.ad, the domain was still available for registration. He then learned the .ad registry requires prospective customers to show a valid trademark for a domain before it can be registered.
Undeterred, Caturegli found a domain registrar that would sell him the domain for $160, and handle the trademark registration for another $500 (on subsequent .ad registrations, he located a company in Andorra that could process the trademark application for half that amount).
Caturegli said that immediately after setting up a DNS server for memrtcc.ad, he began receiving a flood of communications from hundreds of Microsoft Windows computers trying to authenticate to the domain. Each request contained a username and a hashed Windows password, and upon searching the usernames online Caturegli concluded they all belonged to police officers in Memphis, Tenn.
“It looks like all of the police cars there have a laptop in the cars, and they’re all attached to this memrtcc.ad domain that I now own,” Caturegli said, noting wryly that “memrtcc” stands for “Memphis Real-Time Crime Center.”
Caturegli said setting up an email server record for memrtcc.ad caused him to begin receiving automated messages from the police department’s IT help desk, including trouble tickets regarding the city’s Okta authentication system.
Mike Barlow, information security manager for the City of Memphis, confirmed the Memphis Police’s systems were sharing their Microsoft Windows credentials with the domain, and that the city was working with Caturegli to have the domain transferred to them.
“We are working with the Memphis Police Department to at least somewhat mitigate the issue in the meantime,” Barlow said.
Domain administrators have long been encouraged to use .local for internal domain names, because this TLD is reserved for use by local networks and cannot be routed over the open Internet. However, Caturegli said many organizations seem to have missed that memo and gotten things backwards — setting up their internal Active Directory structure around the perfectly routable domain local.ad.
Caturegli said he knows this because he “defensively” registered local.ad, which he said is currently used by multiple large organizations for Active Directory setups — including a European mobile phone provider, and the City of Newcastle in the United Kingdom.
Caturegli said he has now defensively registered a number of domains ending in .ad, such as internal.ad and schema.ad. But perhaps the most dangerous domain in his stable is wpad.ad. WPAD stands for Web Proxy Auto-Discovery Protocol, which is an ancient, on-by-default feature built into every version of Microsoft Windows that was designed to make it simpler for Windows computers to automatically find and download any proxy settings required by the local network.
Trouble is, any organization that chose a .ad domain they don’t own for their Active Directory setup will have a whole bunch of Microsoft systems constantly trying to reach out to wpad.ad if those machines have proxy automated detection enabled.
Security researchers have been beating up on WPAD for more than two decades now, warning time and again how it can be abused for nefarious ends. At this year’s DEF CON security conference in Las Vegas, for example, a researcher showed what happened after they registered the domain wpad.dk: Immediately after switching on the domain, they received a flood of WPAD requests from Microsoft Windows systems in Denmark that had namespace collisions in their Active Directory environments.
Image: Defcon.org.
For his part, Caturegli set up a server on wpad.ad to resolve and record the Internet address of any Windows systems trying to reach Microsoft Sharepoint servers, and saw that over one week it received more than 140,000 hits from hosts around the world attempting to connect.
The fundamental problem with WPAD is the same with Active Directory: Both are technologies originally designed to be used in closed, static, trusted office environments, and neither was built with today’s mobile devices or workforce in mind.
Probably one big reason organizations with potential namespace collision problems don’t fix them is that rebuilding one’s Active Directory infrastructure around a new domain name can be incredibly disruptive, costly, and risky, while the potential threat is considered comparatively low.
But Caturegli said ransomware gangs and other cybercrime groups could siphon huge volumes of Microsoft Windows credentials from quite a few companies with just a small up-front investment.
“It’s an easy way to gain that initial access without even having to launch an actual attack,” he said. “You just wait for the misconfigured workstation to connect to you and send you their credentials.”
If we ever learn that cybercrime groups are using namespace collisions to launch ransomware attacks, nobody can say they weren’t warned. Mike O’Connor, an early domain name investor who registered a number of choice domains such as bar.com, place.com and television.com, warned loudly and often back in 2013 that then-pending plans to add more than 1,000 new TLDs would massively expand the number of namespace collisions.
Mr. O’Connor’s most famous domain is corp.com, because for several decades he watched in horror as hundreds of thousands of Microsoft PCs continuously blasted his domain with credentials from organizations that had set up their Active Directory environment around the domain corp.com.
It turned out that Microsoft had actually used corp.com as an example of how one might set up Active Directory in some editions of Windows NT. Worse, some of the traffic going to corp.com was coming from Microsoft’s internal networks, indicating some part of Microsoft’s own internal infrastructure was misconfigured. When O’Connor said he was ready to sell corp.com to the highest bidder in 2020, Microsoft agreed to buy the domain for an undisclosed amount.
“I kind of imagine this problem to be something like a town [that] knowingly built a water supply out of lead pipes, or vendors of those projects who knew but didn’t tell their customers,” O’Connor told KrebsOnSecurity. “This is not an inadvertent thing like Y2K where everybody was surprised by what happened. People knew and didn’t care.”
Invisible protocol sniffer for finding vulnerabilities in the network. Designed for pentesters and security engineers.
FreshRSS 
Above: Invisible network protocol sniffer
Designed for pentesters and security engineers
Author: Magama Bazarov, <caster@exploit.org>
Pseudonym: Caster
Version: 2.6
Codename: Introvert
All information contained in this repository is provided for educational and research purposes only. The author is not responsible for any illegal use of this tool.
It is a specialized network security tool that helps both pentesters and security professionals.
Above is a invisible network sniffer for finding vulnerabilities in network equipment. It is based entirely on network traffic analysis, so it does not make any noise on the air. He's invisible. Completely based on the Scapy library.
Above allows pentesters to automate the process of finding vulnerabilities in network hardware. Discovery protocols, dynamic routing, 802.1Q, ICS Protocols, FHRP, STP, LLMNR/NBT-NS, etc.
Detects up to 27 protocols:
MACSec (802.1X AE)
EAPOL (Checking 802.1X versions)
ARP (Passive ARP, Host Discovery)
CDP (Cisco Discovery Protocol)
DTP (Dynamic Trunking Protocol)
LLDP (Link Layer Discovery Protocol)
802.1Q Tags (VLAN)
S7COMM (Siemens)
OMRON
TACACS+ (Terminal Access Controller Access Control System Plus)
ModbusTCP
STP (Spanning Tree Protocol)
OSPF (Open Shortest Path First)
EIGRP (Enhanced Interior Gateway Routing Protocol)
BGP (Border Gateway Protocol)
VRRP (Virtual Router Redundancy Protocol)
HSRP (Host Standby Redundancy Protocol)
GLBP (Gateway Load Balancing Protocol)
IGMP (Internet Group Management Protocol)
LLMNR (Link Local Multicast Name Resolution)
NBT-NS (NetBIOS Name Service)
MDNS (Multicast DNS)
DHCP (Dynamic Host Configuration Protocol)
DHCPv6 (Dynamic Host Configuration Protocol v6)
ICMPv6 (Internet Control Message Protocol v6)
SSDP (Simple Service Discovery Protocol)
MNDP (MikroTik Neighbor Discovery Protocol)
Above works in two modes:
The tool is very simple in its operation and is driven by arguments:
.pcap as input and looks for protocols in it.pcap file, its name you specify yourselfusage: above.py [-h] [--interface INTERFACE] [--timer TIMER] [--output OUTPUT] [--input INPUT] [--passive-arp]
options:
-h, --help show this help message and exit
--interface INTERFACE
Interface for traffic listening
--timer TIMER Time in seconds to capture packets, if not set capture runs indefinitely
--output OUTPUT File name where the traffic will be recorded
--input INPUT File name of the traffic dump
--passive-arp Passive ARP (Host Discovery)
The information obtained will be useful not only to the pentester, but also to the security engineer, he will know what he needs to pay attention to.
When Above detects a protocol, it outputs the necessary information to indicate the attack vector or security issue:
Impact: What kind of attack can be performed on this protocol;
Tools: What tool can be used to launch an attack;
Technical information: Required information for the pentester, sender MAC/IP addresses, FHRP group IDs, OSPF/EIGRP domains, etc.
Mitigation: Recommendations for fixing the security problems
Source/Destination Addresses: For protocols, Above displays information about the source and destination MAC addresses and IP addresses
You can install Above directly from the Kali Linux repositories
caster@kali:~$ sudo apt update && sudo apt install above
Or...
caster@kali:~$ sudo apt-get install python3-scapy python3-colorama python3-setuptools
caster@kali:~$ git clone https://github.com/casterbyte/Above
caster@kali:~$ cd Above/
caster@kali:~/Above$ sudo python3 setup.py install
# Install python3 first
brew install python3
# Then install required dependencies
sudo pip3 install scapy colorama setuptools
# Clone the repo
git clone https://github.com/casterbyte/Above
cd Above/
sudo python3 setup.py install
Don't forget to deactivate your firewall on macOS!
Above requires root access for sniffing
Above can be run with or without a timer:
caster@kali:~$ sudo above --interface eth0 --timer 120
To stop traffic sniffing, press CTRL + С
WARNING! Above is not designed to work with tunnel interfaces (L3) due to the use of filters for L2 protocols. Tool on tunneled L3 interfaces may not work properly.
Example:
caster@kali:~$ sudo above --interface eth0 --timer 120
-----------------------------------------------------------------------------------------
[+] Start sniffing...
[*] After the protocol is detected - all necessary information about it will be displayed
--------------------------------------------------
[+] Detected SSDP Packet
[*] Attack Impact: Potential for UPnP Device Exploitation
[*] Tools: evil-ssdp
[*] SSDP Source IP: 192.168.0.251
[*] SSDP Source MAC: 02:10:de:64:f2:34
[*] Mitigation: Ensure UPnP is disabled on all devices unless absolutely necessary, monitor UPnP traffic
--------------------------------------------------
[+] Detected MDNS Packet
[*] Attack Impact: MDNS Spoofing, Credentials Interception
[*] Tools: Responder
[*] MDNS Spoofing works specifically against Windows machines
[*] You cannot get NetNTLMv2-SSP from Apple devices
[*] MDNS Speaker IP: fe80::183f:301c:27bd:543
[*] MDNS Speaker MAC: 02:10:de:64:f2:34
[*] Mitigation: Filter MDNS traffic. Be careful with MDNS filtering
--------------------------------------------------
If you need to record the sniffed traffic, use the --output argument
caster@kali:~$ sudo above --interface eth0 --timer 120 --output above.pcap
If you interrupt the tool with CTRL+C, the traffic is still written to the file
If you already have some recorded traffic, you can use the --input argument to look for potential security issues
caster@kali:~$ above --input ospf-md5.cap
Example:
caster@kali:~$ sudo above --input ospf-md5.cap
[+] Analyzing pcap file...
--------------------------------------------------
[+] Detected OSPF Packet
[+] Attack Impact: Subnets Discovery, Blackhole, Evil Twin
[*] Tools: Loki, Scapy, FRRouting
[*] OSPF Area ID: 0.0.0.0
[*] OSPF Neighbor IP: 10.0.0.1
[*] OSPF Neighbor MAC: 00:0c:29:dd:4c:54
[!] Authentication: MD5
[*] Tools for bruteforce: Ettercap, John the Ripper
[*] OSPF Key ID: 1
[*] Mitigation: Enable passive interfaces, use authentication
--------------------------------------------------
[+] Detected OSPF Packet
[+] Attack Impact: Subnets Discovery, Blackhole, Evil Twin
[*] Tools: Loki, Scapy, FRRouting
[*] OSPF Area ID: 0.0.0.0
[*] OSPF Neighbor IP: 192.168.0.2
[*] OSPF Neighbor MAC: 00:0c:29:43:7b:fb
[!] Authentication: MD5
[*] Tools for bruteforce: Ettercap, John the Ripper
[*] OSPF Key ID: 1
[*] Mitigation: Enable passive interfaces, use authentication
The tool can detect hosts without noise in the air by processing ARP frames in passive mode
caster@kali:~$ sudo above --interface eth0 --passive-arp --timer 10
[+] Host discovery using Passive ARP
--------------------------------------------------
[+] Detected ARP Reply
[*] ARP Reply for IP: 192.168.1.88
[*] MAC Address: 00:00:0c:07:ac:c8
--------------------------------------------------
[+] Detected ARP Reply
[*] ARP Reply for IP: 192.168.1.40
[*] MAC Address: 00:0c:29:c5:82:81
--------------------------------------------------
I wrote this tool because of the track "A View From Above (Remix)" by KOAN Sound. This track was everything to me when I was working on this sniffer.
Telegram Nearby Map uses OpenStreetMap and the official Telegram library to find the position of nearby users.
Please note: Telegram's API was updated a while ago to make nearby user distances less precise, preventing exact location calculations. Therefore, Telegram Nearby Map displays users nearby, but does not show their exact location.
Inspired by Ahmed's blog post and a Hacker News discussion. Developed by github.com/tejado.
Every 25 seconds all nearby users will be received with TDLib from Telegram. This includes the distance of every nearby user to "my" location. With three distances from three different points, it is possible to calculate the position of the nearby user.
This only finds Telegram users which have activated the nearby feature. Per default it is deactivated.
Requirements: node.js and an Telegram account
Associated-Threat-Analyzer detects malicious IPv4 addresses and domain names associated with your web application using local malicious domain and IPv4 lists.
git clone https://github.com/OsmanKandemir/associated-threat-analyzer.git
cd associated-threat-analyzer && pip3 install -r requirements.txt
python3 analyzer.py -d target-web.com
You can run this application on a container after build a Dockerfile.
docker build -t osmankandemir/threatanalyzer .
docker run osmankandemir/threatanalyzer -d target-web.com
docker pull osmankandemir/threatanalyzer
docker run osmankandemir/threatanalyzer -d target-web.com
-d DOMAIN , --domain DOMAIN Input Target. --domain target-web1.com
-t DOMAINSFILE, --DomainsFile Malicious Domains List to Compare. -t SampleMaliciousDomains.txt
-i IPSFILE, --IPsFile Malicious IPs List to Compare. -i SampleMaliciousIPs.txt
-o JSON, --json JSON JSON output. --json
https://github.com/OsmanKandemir/indicator-intelligence
https://github.com/stamparm/blackbook
https://github.com/stamparm/ipsum
xsubfind3r is a command-line interface (CLI) utility to find domain's known subdomains from curated passive online sources.
Fetches domains from curated passive sources to maximize results.
Supports stdin and stdout for easy integration into workflows.
Cross-Platform (Windows, Linux & macOS).
Visit the releases page and find the appropriate archive for your operating system and architecture. Download the archive from your browser or copy its URL and retrieve it with wget or curl:
...with wget:
wget https://github.com/hueristiq/xsubfind3r/releases/download/v<version>/xsubfind3r-<version>-linux-amd64.tar.gz...or, with curl:
curl -OL https://github.com/hueristiq/xsubfind3r/releases/download/v<version>/xsubfind3r-<version>-linux-amd64.tar.gz...then, extract the binary:
tar xf xsubfind3r-<version>-linux-amd64.tar.gzTIP: The above steps, download and extract, can be combined into a single step with this onliner
curl -sL https://github.com/hueristiq/xsubfind3r/releases/download/v<version>/xsubfind3r-<version>-linux-amd64.tar.gz | tar -xzv
NOTE: On Windows systems, you should be able to double-click the zip archive to extract the xsubfind3r executable.
...move the xsubfind3r binary to somewhere in your PATH. For example, on GNU/Linux and OS X systems:
sudo mv xsubfind3r /usr/local/bin/NOTE: Windows users can follow How to: Add Tool Locations to the PATH Environment Variable in order to add xsubfind3r to their PATH.
Before you install from source, you need to make sure that Go is installed on your system. You can install Go by following the official instructions for your operating system. For this, we will assume that Go is already installed.
go install ...go install -v github.com/hueristiq/xsubfind3r/cmd/xsubfind3r@latestgo build ... the development VersionClone the repository
git clone https://github.com/hueristiq/xsubfind3r.git Build the utility
cd xsubfind3r/cmd/xsubfind3r && \
go build .Move the xsubfind3r binary to somewhere in your PATH. For example, on GNU/Linux and OS X systems:
sudo mv xsubfind3r /usr/local/bin/NOTE: Windows users can follow How to: Add Tool Locations to the PATH Environment Variable in order to add xsubfind3r to their PATH.
NOTE: While the development version is a good way to take a peek at xsubfind3r's latest features before they get released, be aware that it may have bugs. Officially released versions will generally be more stable.
xsubfind3r will work right after installation. However, BeVigil, Chaos, Fullhunt, Github, Intelligence X and Shodan require API keys to work, URLScan supports API key but not required. The API keys are stored in the $HOME/.hueristiq/xsubfind3r/config.yaml file - created upon first run - and uses the YAML format. Multiple API keys can be specified for each of these source from which one of them will be used.
Example config.yaml:
version: 0.3.0
sources:
- alienvault
- anubis
- bevigil
- chaos
- commoncrawl
- crtsh
- fullhunt
- github
- hackertarget
- intelx
- shodan
- urlscan
- wayback
keys:
bevigil:
- awA5nvpKU3N8ygkZ
chaos:
- d23a554bbc1aabb208c9acfbd2dd41ce7fc9db39asdsd54bbc1aabb208c9acfb
fullhunt:
- 0d9652ce-516c-4315-b589-9b241ee6dc24
github:
- d23a554bbc1aabb208c9acfbd2dd41ce7fc9db39
- asdsd54bbc1aabb208c9acfbd2dd41ce7fc9db39
intelx:
- 2.intelx.io:00000000-0000-0000-0000-000000000000
shodan:
- AAAAClP1bJJSRMEYJazgwhJKrggRwKA
urlscan:
- d4c85d34-e425-446e-d4ab-f5a3412acbe8To display help message for xsubfind3r use the -h flag:
xsubfind3r -hhelp message:
_ __ _ _ _____
__ _____ _ _| |__ / _(_)_ __ __| |___ / _ __
\ \/ / __| | | | '_ \| |_| | '_ \ / _` | |_ \| '__|
> <\__ \ |_| | |_) | _| | | | | (_| |___) | |
/_/\_\___/\__,_|_.__/|_| |_|_| |_|\__,_|____/|_| v0.3.0
USAGE:
xsubfind3r [OPTIONS]
INPUT:
-d, --domain string[] target domains
-l, --list string target domains' list file path
SOURCES:
--sources bool list supported sources
-u, --sources-to-use string[] comma(,) separeted sources to use
-e, --sources-to-exclude string[] comma(,) separeted sources to exclude
OPTIMIZATION:
-t, --threads int number of threads (default: 50)
OUTPUT:
--no-color bool disable colored output
-o, --output string output subdomains' file path
-O, --output-directory string output subdomains' directory path
-v, --verbosity string debug, info, warning, error, fatal or silent (default: info)
CONFIGURATION:
-c, --configuration string configuration file path (default: ~/.hueristiq/xsubfind3r/config.yaml)
Issues and Pull Requests are welcome! Check out the contribution guidelines.
This utility is distributed under the MIT license.
xcrawl3r is a command-line interface (CLI) utility to recursively crawl webpages i.e systematically browse webpages' URLs and follow links to discover linked webpages' URLs.
.js, .json, .xml, .csv, .txt & .map).robots.txt.Visit the releases page and find the appropriate archive for your operating system and architecture. Download the archive from your browser or copy its URL and retrieve it with wget or curl:
...with wget:
wget https://github.com/hueristiq/xcrawl3r/releases/download/v<version>/xcrawl3r-<version>-linux-amd64.tar.gz...or, with curl:
curl -OL https://github.com/hueristiq/xcrawl3r/releases/download/v<version>/xcrawl3r-<version>-linux-amd64.tar.gz...then, extract the binary:
tar xf xcrawl3r-<version>-linux-amd64.tar.gzTIP: The above steps, download and extract, can be combined into a single step with this onliner
curl -sL https://github.com/hueristiq/xcrawl3r/releases/download/v<version>/xcrawl3r-<version>-linux-amd64.tar.gz | tar -xzv
NOTE: On Windows systems, you should be able to double-click the zip archive to extract the xcrawl3r executable.
...move the xcrawl3r binary to somewhere in your PATH. For example, on GNU/Linux and OS X systems:
sudo mv xcrawl3r /usr/local/bin/NOTE: Windows users can follow How to: Add Tool Locations to the PATH Environment Variable in order to add xcrawl3r to their PATH.
Before you install from source, you need to make sure that Go is installed on your system. You can install Go by following the official instructions for your operating system. For this, we will assume that Go is already installed.
go install ...go install -v github.com/hueristiq/xcrawl3r/cmd/xcrawl3r@latestgo build ... the development VersionClone the repository
git clone https://github.com/hueristiq/xcrawl3r.git Build the utility
cd xcrawl3r/cmd/xcrawl3r && \
go build .Move the xcrawl3r binary to somewhere in your PATH. For example, on GNU/Linux and OS X systems:
sudo mv xcrawl3r /usr/local/bin/NOTE: Windows users can follow How to: Add Tool Locations to the PATH Environment Variable in order to add xcrawl3r to their PATH.
NOTE: While the development version is a good way to take a peek at xcrawl3r's latest features before they get released, be aware that it may have bugs. Officially released versions will generally be more stable.
To display help message for xcrawl3r use the -h flag:
xcrawl3r -hhelp message:
_ _____
__ _____ _ __ __ ___ _| |___ / _ __
\ \/ / __| '__/ _` \ \ /\ / / | |_ \| '__|
> < (__| | | (_| |\ V V /| |___) | |
/_/\_\___|_| \__,_| \_/\_/ |_|____/|_| v0.1.0
A CLI utility to recursively crawl webpages.
USAGE:
xcrawl3r [OPTIONS]
INPUT:
-d, --domain string domain to match URLs
--include-subdomains bool match subdomains' URLs
-s, --seeds string seed URLs file (use `-` to get from stdin)
-u, --url string URL to crawl
CONFIGURATION:
--depth int maximum depth to crawl (default 3)
TIP: set it to `0` for infinite recursion
--headless bool If true the browser will be displayed while crawling.
-H, --headers string[] custom header to include in requests
e.g. -H 'Referer: http://example.com/'
TIP: use multiple flag to set multiple headers
--proxy string[] Proxy URL (e.g: http://127.0.0.1:8080)
TIP: use multiple flag to set multiple proxies
--render bool utilize a headless chrome instance to render pages
--timeout int time to wait for request in seconds (default: 10)
--user-agent string User Agent to use (default: web)
TIP: use `web` for a random web user-agent,
`mobile` for a random mobile user-agent,
or you can set your specific user-agent.
RATE LIMIT:
-c, --concurrency int number of concurrent fetchers to use (default 10)
--delay int delay between each request in seconds
--max-random-delay int maximux extra randomized delay added to `--dalay` (default: 1s)
-p, --parallelism int number of concurrent URLs to process (default: 10)
OUTPUT:
--debug bool enable debug mode (default: false)
-m, --monochrome bool coloring: no colored output mode
-o, --output string output file to write found URLs
-v, --verbosity string debug, info, warning, error, fatal or silent (default: debug)
Issues and Pull Requests are welcome! Check out the contribution guidelines.
This utility is distributed under the MIT license.
Alternatives - Check out projects below, that may fit in your workflow:
xurlfind3r is a command-line interface (CLI) utility to find domain's known URLs from curated passive online sources.
robots.txt snapshots.Visit the releases page and find the appropriate archive for your operating system and architecture. Download the archive from your browser or copy its URL and retrieve it with wget or curl:
...with wget:
wget https://github.com/hueristiq/xurlfind3r/releases/download/v<version>/xurlfind3r-<version>-linux-amd64.tar.gz...or, with curl:
curl -OL https://github.com/hueristiq/xurlfind3r/releases/download/v<version>/xurlfind3r-<version>-linux-amd64.tar.gz...then, extract the binary:
tar xf xurlfind3r-<version>-linux-amd64.tar.gzTIP: The above steps, download and extract, can be combined into a single step with this onliner
curl -sL https://github.com/hueristiq/xurlfind3r/releases/download/v<version>/xurlfind3r-<version>-linux-amd64.tar.gz | tar -xzv
NOTE: On Windows systems, you should be able to double-click the zip archive to extract the xurlfind3r executable.
...move the xurlfind3r binary to somewhere in your PATH. For example, on GNU/Linux and OS X systems:
sudo mv xurlfind3r /usr/local/bin/NOTE: Windows users can follow How to: Add Tool Locations to the PATH Environment Variable in order to add xurlfind3r to their PATH.
Before you install from source, you need to make sure that Go is installed on your system. You can install Go by following the official instructions for your operating system. For this, we will assume that Go is already installed.
go install ...go install -v github.com/hueristiq/xurlfind3r/cmd/xurlfind3r@latestgo build ... the development VersionClone the repository
git clone https://github.com/hueristiq/xurlfind3r.git Build the utility
cd xurlfind3r/cmd/xurlfind3r && \
go build .Move the xurlfind3r binary to somewhere in your PATH. For example, on GNU/Linux and OS X systems:
sudo mv xurlfind3r /usr/local/bin/NOTE: Windows users can follow How to: Add Tool Locations to the PATH Environment Variable in order to add xurlfind3r to their PATH.
NOTE: While the development version is a good way to take a peek at xurlfind3r's latest features before they get released, be aware that it may have bugs. Officially released versions will generally be more stable.
xurlfind3r will work right after installation. However, BeVigil, Github and Intelligence X require API keys to work, URLScan supports API key but not required. The API keys are stored in the $HOME/.hueristiq/xurlfind3r/config.yaml file - created upon first run - and uses the YAML format. Multiple API keys can be specified for each of these source from which one of them will be used.
Example config.yaml:
version: 0.2.0
sources:
- bevigil
- commoncrawl
- github
- intelx
- otx
- urlscan
- wayback
keys:
bevigil:
- awA5nvpKU3N8ygkZ
github:
- d23a554bbc1aabb208c9acfbd2dd41ce7fc9db39
- asdsd54bbc1aabb208c9acfbd2dd41ce7fc9db39
intelx:
- 2.intelx.io:00000000-0000-0000-0000-000000000000
urlscan:
- d4c85d34-e425-446e-d4ab-f5a3412acbe8To display help message for xurlfind3r use the -h flag:
xurlfind3r -hhelp message:
_ __ _ _ _____
__ ___ _ _ __| |/ _(_)_ __ __| |___ / _ __
\ \/ / | | | '__| | |_| | '_ \ / _` | |_ \| '__|
> <| |_| | | | | _| | | | | (_| |___) | |
/_/\_\\__,_|_| |_|_| |_|_| |_|\__,_|____/|_| v0.2.0
USAGE:
xurlfind3r [OPTIONS]
TARGET:
-d, --domain string (sub)domain to match URLs
SCOPE:
--include-subdomains bool match subdomain's URLs
SOURCES:
-s, --sources bool list sources
-u, --use-sources string sources to use (default: bevigil,commoncrawl,github,intelx,otx,urlscan,wayback)
--skip-wayback-robots bool with wayback, skip parsing robots.txt snapshots
--skip-wayback-source bool with wayback , skip parsing source code snapshots
FILTER & MATCH:
-f, --filter string regex to filter URLs
-m, --match string regex to match URLs
OUTPUT:
--no-color bool no color mode
-o, --output string output URLs file path
-v, --verbosity string debug, info, warning, error, fatal or silent (default: info)
CONFIGURATION:
-c, --configuration string configuration file path (default: ~/.hueristiq/xurlfind3r/config.yaml)
xurlfind3r -d hackerone.com --include-subdomains# filter images
xurlfind3r -d hackerone.com --include-subdomains -f '`^https?://[^/]*?/.*\.(jpg|jpeg|png|gif|bmp)(\?[^\s]*)?$`'# match js URLs
xurlfind3r -d hackerone.com --include-subdomains -m '^https?://[^/]*?/.*\.js(\?[^\s]*)?$'Issues and Pull Requests are welcome! Check out the contribution guidelines.
This utility is distributed under the MIT license.
fingerprintx is a utility similar to httpx that also supports fingerprinting services like as RDP, SSH, MySQL, PostgreSQL, Kafka, etc. fingerprintx can be used alongside port scanners like Naabu to fingerprint a set of ports identified during a port scan. For example, an engineer may wish to scan an IP range and then rapidly fingerprint the service running on all the discovered ports.
| SERVICE | TRANSPORT | SERVICE | TRANSPORT |
|---|---|---|---|
| HTTP | TCP | REDIS | TCP |
| SSH | TCP | MQTT3 | TCP |
| MODBUS | TCP | VNC | TCP |
| TELNET | TCP | MQTT5 | TCP |
| FTP | TCP | RSYNC | TCP |
| SMB | TCP | RPC | TCP |
| DNS | TCP | OracleDB | TCP |
| SMTP | TCP | RTSP | TCP |
| PostgreSQL | TCP | MQTT5 | TCP (TLS) |
| RDP | TCP | HTTPS | TCP (TLS) |
| POP3 | TCP | SMTPS | TCP (TLS) |
| KAFKA | TCP | MQTT3 | TCP (TLS) |
| MySQL | TCP | RDP | TCP (TLS) |
| MSSQL | TCP | POP3S | TCP (TLS) |
| LDAP | TCP | LDAPS | TCP (TLS) |
| IMAP | TCP | IMAPS | TCP (TLS) |
| SNMP | UDP | Kafka | TCP (TLS) |
| OPENVPN | UDP | NETBIOS-NS | UDP |
| IPSEC | UDP | DHCP | UDP |
| STUN | UDP | NTP | UDP |
| DNS | UDP |
From Github
go install github.com/praetorian-inc/fingerprintx/cmd/fingerprintx@latestFrom source (go version > 1.18)
$ git clone git@github.com:praetorian-inc/fingerprintx.git
$ cd fingerprintx
# with go version > 1.18
$ go build ./cmd/fingerprintx
$ ./fingerprintx -hDocker
$ git clone git@github.com:praetorian-inc/fingerprintx.git
$ cd fingerprintx
# build
docker build -t fingerprintx .
# and run it
docker run --rm fingerprintx -h
docker run --rm fingerprintx -t praetorian.com:80 --jsonfingerprintx -hThe -h option will display all of the supported flags for fingerprintx.
Usage:
fingerprintx [flags]
TARGET SPECIFICATION:
Requires a host and port number or ip and port number. The port is assumed to be open.
HOST:PORT or IP:PORT
EXAMPLES:
fingerprintx -t praetorian.com:80
fingerprintx -l input-file.txt
fingerprintx --json -t praetorian.com:80,127.0.0.1:8000
Flags:
--csv output format in csv
-f, --fast fast mode
-h, --help help for fingerprintx
--json output format in json
-l, --list string input file containing targets
-o, --output string output file
-t, --targets strings target or comma separated target list
-w, --timeout int timeout (milliseconds) (default 500)
-U, --udp run UDP plugins
-v, --verbose verbose modeThe fast mode will only attempt to fingerprint the default service associated with that port for each target. For example, if praetorian.com:8443 is the input, only the https plugin would be run. If https is not running on praetorian.com:8443, there will be NO output. Why do this? It's a quick way to fingerprint most of the services in a large list of hosts (think the 80/20 rule).
With one target:
$ fingerprintx -t 127.0.0.1:8000
http://127.0.0.1:8000By default, the output is in the form: SERVICE://HOST:PORT. To get more detailed service output specify JSON with the --json flag:
$ fingerprintx -t 127.0.0.1:8000 --json
{"ip":"127.0.0.1","port":8000,"service":"http","transport":"tcp","metadata":{"responseHeaders":{"Content-Length":["1154"],"Content-Type":["text/html; charset=utf-8"],"Date":["Mon, 19 Sep 2022 18:23:18 GMT"],"Server":["SimpleHTTP/0.6 Python/3.10.6"]},"status":"200 OK","statusCode":200,"version":"SimpleHTTP/0.6 Python/3.10.6"}}Pipe in output from another program (like naabu):
$ naabu 127.0.0.1 -silent 2>/dev/null | fingerprintx
http://127.0.0.1:8000
ftp://127.0.0.1:21Run with an input file:
$ cat input.txt | fingerprintx
http://praetorian.com:80
telnet://telehack.com:23
# or if you prefer
$ fingerprintx -l input.txt
http://praetorian.com:80
telnet://telehack.com:23
With more metadata output:
Nmap is the standard for network scanning. Why use fingerprintx instead of nmap? The main two reasons are:
fingerprintx works smarter, not harder: the first plugin run against a server with port 8080 open is the http plugin. The default service approach cuts down scanning time in the best case. Most of the time the services running on port 80, 443, 22 are http, https, and ssh -- so that's what fingerprintx checks first.fingerprintx supports json output with the --json flag. Nmap supports numerous output options (normal, xml, grep), but they are often hard to parse and script appropriately. fingerprintx supports json output which eases integration with other tools in processing pipelines.third_party folder that imports the Go cryptography libraries? ssh fingerprinting module identifies the various cryptographic options supported by the server when collecting metadata during the handshake process. This makes use of a few unexported functions, which is why the Go cryptography libraries are included here with an export.go file.target:port input is open. If none of the ports are open there will be no output as there are no services running on the targets.zgrab2 command line usage (and use case) is slightly different than fingerprintx. For zgrab2, the protocol must be specified ahead of time: echo praetorian.com | zgrab2 http -p 8000, which assumes you already know what is running there. For fingerprintx, that is not the case: echo praetorian.com:8000 | fingerprintx. The "application layer" protocol scanning approach is very similar.fingerprintx is the work of a lot of people, including our great intern class of 2022. Here is a list of contributors so far:
