PIP-INTEL - OSINT and Cyber Intelligence Tool

Pip-Intel is a powerful tool designed for OSINT (Open Source Intelligence) and cyber intelligence gathering activities. It consolidates various open-source tools into a single user-friendly interface simplifying the data collection and analysis processes for researchers and cybersecurity professionals.

Pip-Intel utilizes Python-written pip packages to gather information from various data points. This tool is equipped with the capability to collect detailed information through email addresses, phone numbers, IP addresses, and social media accounts. It offers a wide range of functionalities including email-based OSINT operations, phone number-based inquiries, geolocating IP addresses, social media and user analyses, and even dark web searches.

C2-Tracker - Live Feed Of C2 Servers, Tools, And Botnets

Free to use IOC feed for various tools/malware. It started out for just C2 tools but has morphed into tracking infostealers and botnets as well. It uses shodan.io/">Shodan searches to collect the IPs. The most recent collection is always stored in data; the IPs are broken down by tool and there is an all.txt.

The feed should update daily. Actively working on making the backend more reliable

Honorable Mentions

Many of the Shodan queries have been sourced from other CTI researchers:

• BushidoToken
• Michael Koczwara
• ViriBack
• Gi7W0rm
• @Glacius_

Huge shoutout to them!

Thanks to BertJanCyber for creating the KQL query for ingesting this feed

And finally, thanks to Y_nexro for creating C2Live in order to visualize the data

What do I track?

• C2's
  • Cobalt Strike
  • Metasploit Framework
  • Covenant
  • Mythic
  • Brute Ratel C4
  • Posh
  • Sliver
  • Deimos
  • PANDA
  • NimPlant C2
  • Havoc C2
  • Caldera
  • Empire
  • Ares
• Malware
  • AcidRain Stealer
  • Misha Stealer (AKA Grand Misha)
  • Patriot Stealer
  • RAXNET Bitcoin Stealer
  • Titan Stealer
  • Collector Stealer
  • Mystic Stealer
  • Gotham Stealer
  • Meduza Stealer
  • Quasar RAT
  • ShadowPad
  • AsyncRAT
  • DcRat
  • BitRAT
  • DarkComet Trojan
  • XtremeRAT Trojan
  • NanoCore RAT Trojan
  • Gh0st RAT Trojan
  • DarkTrack RAT Trojan
  • njRAT Trojan
  • Remcos Pro RAT Trojan
  • Poison Ivy Trojan
  • Orcus RAT Trojan
  • ZeroAccess Trojan
  • HOOKBOT Trojan
• Tools
  • XMRig Monero Cryptominer
  • GoPhish
• Botnets
  • 7777 Botnet

Running Locally

If you want to host a private version, put your Shodan API key in an environment variable called SHODAN_API_KEY

echo SHODAN_API_KEY=API_KEY >> ~/.bashrc
bash
python3 -m pip install -r requirements.txt
python3 tracker.py

Contributing

I encourage opening an issue/PR if you know of any additional Shodan searches for identifying adversary infrastructure. I will not set any hard guidelines around what can be submitted, just know, fidelity is paramount (high true/false positive ratio is the focus).

References

• Hunting C2 with Shodan by Michael Koczwara
• Hunting Cobalt Strike C2 with Shodan by Michael Koczwara
• https://twitter.com/MichalKoczwara/status/1591750513238118401?cxt=HHwWgsDUiZGqhJcsAAAA
• BushidoToken's OSINT-SearchOperators
• https://twitter.com/MichalKoczwara/status/1641119242618650653
• https://twitter.com/MichalKoczwara/status/1641676761283850241
• https://twitter.com/_montysecurity/status/1643164749599834112
• https://twitter.com/ViriBack/status/1713714868564394336
• https://gi7w0rm.medium.com/the-curious-case-of-the-7777-botnet-86e3464c3ffd
• https://twitter.com/Glacius_/status/1731699013873799209

DarkGPT - An OSINT Assistant Based On GPT-4-200K Designed To Perform Queries On Leaked Databases, Thus Providing An Artificial Intelligence Assistant That Can Be Useful In Your Traditional OSINT Processes

DarkGPT is an artificial intelligence assistant based on GPT-4-200K designed to perform queries on leaked databases. This guide will help you set up and run the project on your local environment.

Prerequisites

Before starting, make sure you have Python installed on your system. This project has been tested with Python 3.8 and higher versions.

Environment Setup

1. Clone the Repository

First, you need to clone the GitHub repository to your local machine. You can do this by executing the following command in your terminal:

git clone https://github.com/luijait/DarkGPT.git cd DarkGPT

1. Configure Environment Variables

You will need to set up some environment variables for the script to work correctly. Copy the .env.example file to a new file named .env:

DEHASHED_API_KEY="your_dehashed_api_key_here"

1. Install Dependencies

This project requires certain Python packages to run. Install them by running the following command:

pip install -r requirements.txt 4. Then Run the project: python3 main.py

SwaggerSpy - Automated OSINT On SwaggerHub

SwaggerSpy is a tool designed for automated Open Source Intelligence (OSINT) on SwaggerHub. This project aims to streamline the process of gathering intelligence from APIs documented on SwaggerHub, providing valuable insights for security researchers, developers, and IT professionals.

Swagger is an open-source framework that allows developers to design, build, document, and consume RESTful web services. It simplifies API development by providing a standard way to describe REST APIs using a JSON or YAML format. Swagger enables developers to create interactive documentation for their APIs, making it easier for both developers and non-developers to understand and use the API.

SwaggerHub is a collaborative platform for designing, building, and managing APIs using the Swagger framework. It offers a centralized repository for API documentation, version control, and collaboration among team members. SwaggerHub simplifies the API development lifecycle by providing a unified platform for API design and testing.

Performing OSINT on SwaggerHub is crucial because developers, in their pursuit of efficient API documentation and sharing, may inadvertently expose sensitive information. Here are key reasons why OSINT on SwaggerHub is valuable:

1. Developer Oversights: Developers might unintentionally include secrets, credentials, or sensitive information in API documentation on SwaggerHub. These oversights can lead to security vulnerabilities and unauthorized access if not identified and addressed promptly.

2. Security Best Practices: OSINT on SwaggerHub helps enforce security best practices. Identifying and rectifying potential security issues early in the development lifecycle is essential to ensure the confidentiality and integrity of APIs.

3. Preventing Data Leaks: By systematically scanning SwaggerHub for sensitive information, organizations can proactively prevent data leaks. This is especially crucial in today's interconnected digital landscape where APIs play a vital role in data exchange between services.

4. Risk Mitigation: Understanding that developers might forget to remove or obfuscate sensitive details in API documentation underscores the importance of continuous OSINT on SwaggerHub. This proactive approach mitigates the risk of unintentional exposure of critical information.

5. Compliance and Privacy: Many industries have stringent compliance requirements regarding the protection of sensitive data. OSINT on SwaggerHub ensures that APIs adhere to these regulations, promoting a culture of compliance and safeguarding user privacy.

6. Educational Opportunities: Identifying oversights in SwaggerHub documentation provides educational opportunities for developers. It encourages a security-conscious mindset, fostering a culture of awareness and responsible information handling.

By recognizing that developers can inadvertently expose secrets, OSINT on SwaggerHub becomes an integral part of the overall security strategy, safeguarding against potential threats and promoting a secure API ecosystem.

SwaggerSpy obtains information from SwaggerHub and utilizes regular expressions to inspect API documentation for sensitive information, such as secrets and credentials.

To use SwaggerSpy, follow these steps:

1. Installation: Clone the SwaggerSpy repository and install the required dependencies.

git clone https://github.com/UndeadSec/SwaggerSpy.git
cd SwaggerSpy
pip install -r requirements.txt

1. Usage: Run SwaggerSpy with the target search terms (more accurate with domains).

python swaggerspy.py searchterm

1. Results: SwaggerSpy will generate a report containing OSINT findings, including information about the API, endpoints, and secrets.

SwaggerSpy is intended for educational and research purposes only. Users are responsible for ensuring that their use of this tool complies with applicable laws and regulations.

Contributions to SwaggerSpy are welcome! Feel free to submit issues, feature requests, or pull requests to help improve this tool.

SwaggerSpy is developed and maintained by Alisson Moretto (UndeadSec)

I'm a passionate cyber threat intelligence pro who loves sharing insights and crafting cybersecurity tools.

• [ ] Review and improve existing regular expressions.
• [ ] Ensure that regular expressions adhere to best practices.
• [ ] Check for any potential optimizations in the regex patterns.
• [ ] Test regular expressions with various input scenarios for accuracy.
• [ ] Document any complex or non-trivial regex patterns for better understanding.
• [ ] Explore opportunities to modularize or break down complex patterns.
• [ ] Verify the regular expressions against the latest specifications or requirements.
• [ ] Update documentation to reflect any changes made to the regular expressions.

SwaggerSpy is licensed under the MIT License. See the LICENSE file for details.

Special thanks to @Liodeus for providing project inspiration through swaggerHole.

Melee - Tool To Detect Infections In MySQL Instances

MELEE: A Tool to Detect Ransomware Infections in MySQL Instances

Attackers are abusing MySQL instances for conducting nefarious operations on the Internet. The cybercriminals are targeting exposed MySQL instances and triggering infections at scale to exfiltrate data, destruct data, and extort money via ransom. For example one of the significant threats MySQL deployments face is ransomware. We have authored a tool named "MELEE" to detect potential infections in MySQL instances. The tool allows security researchers, penetration testers, and threat intelligence experts to detect compromised and infected MySQL instances running malicious code. The tool also enables you to conduct efficient research in the field of malware targeting cloud databases. In this release of the tool, the following modules are supported:

• MySQL instance information gathering and reconnaissance
• MySQL instance exposure to the Internet
• MySQL access permissions for assessing remote command execution
• MySQL user enumeration
• MySQL ransomware infections
• Basic assessment checks for detecting ransomware infections
• Extensive assessment checks for extracting insidious details about potential ransomware infections
• MySQL ransomware detection and scanning for both unauthenticated and authenticated deployments

Tool Usage

Douglas-042 - Powershell Script To Help Speed ​​Up Threat Hunting Incident Response Processes

DOUGLAS-042 stands as an ingenious embodiment of a PowerShell script meticulously designed to expedite the triage process and facilitate the meticulous collection of crucial evidence derived from both forensic artifacts and the ephemeral landscape of volatile data. Its fundamental mission revolves around providing indispensable aid in the arduous task of pinpointing potential security breaches within Windows ecosystems. With an overarching focus on expediency, DOUGLAS-042 orchestrates the efficient prioritization and methodical aggregation of data, ensuring that no vital piece of information eludes scrutiny when investigating a possible compromise. As a testament to its organized approach, the amalgamated data finds its sanctuary within the confines of a meticulously named text file, bearing the nomenclature of the host system's very own hostname. This practice of meticulous data archival emerges not just as a systematic convention, but as a cornerstone that paves the way for seamless transitions into subsequent stages of the Forensic journey.

Content Queries

• General information
• Accountand group information
• Network
• Process Information
• OS Build and HOTFIXE
• Persistence
• HARDWARE Information
• Encryption information
• FIREWALL INFORMATION
• Services
• History
• SMB Queries
• Remoting queries
• REGISTRY Analysis
• LOG queries
• Instllation of Software
• User activity

Advanced Queries

• Prefetch file information
• DLL List
• WMI filters and consumers
• Named pipes

Usage

Using administrative privileges, just run the script from a PowerShell console, then the results will be saved in the directory as a txt file.

$ PS >./douglas.ps1

Advance usage

$ PS >./douglas.ps1 -a

OSINT-Framework - OSINT Framework

OSINT framework focused on gathering information from free tools or resources. The intention is to help people find free OSINT resources. Some of the sites included might require registration or offer more data for $$$, but you should be able to get at least a portion of the available information for no cost.

I originally created this framework with an information security point of view. Since then, the response from other fields and disciplines has been incredible. I would love to be able to include any other OSINT resources, especially from fields outside of infosec. Please let me know about anything that might be missing!

Please visit the framework at the link below and good hunting!

https://osintframework.com

Legend

(T) - Indicates a link to a tool that must be installed and run locally
(D) - Google Dork, for more information: Google Hacking
(R) - Requires registration
(M) - Indicates a URL that contains the search term and the URL itself must be edited manually

For Update Notifications

Follow me on Twitter: @jnordine - https://twitter.com/jnordine
Watch or star the project on Github: https://github.com/lockfale/osint-framework

Suggestions, Comments, Feedback

Feedback or new tool suggestions are extremely welcome! Please feel free to submit a pull request or open an issue on github or reach out on Twitter.

Contribute with a GitHub Pull Request

For new resources, please ensure that the site is available for public and free use.

1. Update the arf.json file in the format shown below. If this isn't the first entry for a folder, add a comma to the last closing brace of the previous entry.

Associated-Threat-Analyzer - Detects Malicious IPv4 Addresses And Domain Names Associated With Your Web Application Using Local Malicious Domain And IPv4 Lists

Associated-Threat-Analyzer detects malicious IPv4 addresses and domain names associated with your web application using local malicious domain and IPv4 lists.

Installation

From Git

git clone https://github.com/OsmanKandemir/associated-threat-analyzer.git
cd associated-threat-analyzer && pip3 install -r requirements.txt
python3 analyzer.py -d target-web.com

From Dockerfile

You can run this application on a container after build a Dockerfile.

Warning : If you want to run a Docker container, associated threat analyzer recommends to use your malicious IPs and domains lists, because maintainer may not be update a default malicious IP and domain lists on docker image.

docker build -t osmankandemir/threatanalyzer .
docker run osmankandemir/threatanalyzer -d target-web.com

From DockerHub

docker pull osmankandemir/threatanalyzer
docker run osmankandemir/threatanalyzer -d target-web.com

Usage

-d DOMAIN , --domain DOMAIN Input Target. --domain target-web1.com
-t DOMAINSFILE, --DomainsFile Malicious Domains List to Compare. -t SampleMaliciousDomains.txt
-i IPSFILE, --IPsFile Malicious IPs List to Compare. -i SampleMaliciousIPs.txt
-o JSON, --json JSON  JSON output. --json

DONE

• First-level depth scan your domain address.

TODO list

• Third-level or the more depth static files scanning for target web application.

Other linked github project. You can take a look.

Finds related domains and IPv4 addresses to do threat intelligence after Indicator-Intelligence v1.1.1 collects static files

https://github.com/OsmanKandemir/indicator-intelligence

Default Malicious IPs and Domains Sources

https://github.com/stamparm/blackbook

https://github.com/stamparm/ipsum

Development and Contribution

See; CONTRIBUTING.md

Tiny_Tracer - A Pin Tool For Tracing API Calls Etc

A Pin Tool for tracing:

• API calls, including parameters of selected functions
• selected instructions: RDTSC, CPUID, INT
• inline system calls, including parameters of selected syscalls
• transition between sections of the traced module (helpful in finding OEP of the packed module)

Bypasses the anti-tracing check based on RDTSC.

Generates a report in a .tag format (which can be loaded into other analysis tools):

RVA;traced event

i.e.

345c2;section: .text
58069;called: C:\Windows\SysWOW64\kernel32.dll.IsProcessorFeaturePresent
3976d;called: C:\Windows\SysWOW64\kernel32.dll.LoadLibraryExW
3983c;called: C:\Windows\SysWOW64\kernel32.dll.GetProcAddress
3999d;called: C:\Windows\SysWOW64\KernelBase.dll.InitializeCriticalSectionEx
398ac;called: C:\Windows\SysWOW64\KernelBase.dll.FlsAlloc
3995d;called: C:\Windows\SysWOW64\KernelBase.dll.FlsSetValue
49275;called: C:\Windows\SysWOW64\kernel32.dll.LoadLibraryExW
4934b;called: C:\Windows\SysWOW64\kernel32.dll.GetProcAddress
...

How to build

On Windows

To compile the prepared project you need to use Visual Studio >= 2012. It was tested with Intel Pin 3.28.
Clone this repo into \source\tools that is inside your Pin root directory. Open the project in Visual Studio and build. Detailed description available here.
To build with Intel Pin < 3.26 on Windows, use the appropriate legacy Visual Studio project.

On Linux

For now the support for Linux is experimental. Yet it is possible to build and use Tiny Tracer on Linux as well. Please refer tiny_runner.sh for more information. Detailed description available here.

Usage

 Details about the usage you will find on the project's Wiki.

WARNINGS

• In order for Pin to work correctly, Kernel Debugging must be DISABLED.
• In install32_64 you can find a utility that checks if Kernel Debugger is disabled (kdb_check.exe, source), and it is used by the Tiny Tracer's .bat scripts. This utilty sometimes gets flagged as a malware by Windows Defender (it is a known false positive). If you encounter this issue, you may need to exclude the installation directory from Windows Defender scans.
• Since the version 3.20 Pin has dropped a support for old versions of Windows. If you need to use the tool on Windows < 8, try to compile it with Pin 3.19.

Questions? Ideas? Join Discussions!

Holehe - Tool To Check If The Mail Is Used On Different Sites Like Twitter, Instagram And Will Retrieve Information On Sites With The Forgotten Password Function

Holehe Online Version

Summary

Efficiently finding registered accounts from emails.

Holehe checks if an email is attached to an account on sites like twitter, instagram, imgur and more than 120 others.

• Retrieves information using the forgotten password function.
• Does not alert the target email.
• Runs on Python 3.

Installation

With PyPI

pip3 install holehe

With Github

git clone https://github.com/megadose/holehe.git
cd holehe/
python3 setup.py install

Quick Start

Holehe can be run from the CLI and rapidly embedded within existing python applications.

 CLI Example

holehe test@gmail.com

 Python Example

import trio
import httpx

from holehe.modules.social_media.snapchat import snapchat


async def main():
    email = "test@gmail.com"
    out = []
    client = httpx.AsyncClient()

    await snapchat(email, client, out)

    print(out)
    await client.aclose()

trio.run(main)

Module Output

For each module, data is returned in a standard dictionary with the following json-equivalent format :

{
  "name": "example",
  "rateLimit": false,
  "exists": true,
  "emailrecovery": "ex****e@gmail.com",
  "phoneNumber": "0*******78",
  "others": null
}

• rateLitmit : Lets you know if you've been rate-limited.
• exists : If an account exists for the email on that service.
• emailrecovery : Sometimes partially obfuscated recovery emails are returned.
• phoneNumber : Sometimes partially obfuscated recovery phone numbers are returned.
• others : Any extra info.

Rate limit? Change your IP.

Maltego Transform : Holehe Maltego

Thank you to :

• navlys
• Chris
• socialscan
• UhOh365
• soxoj
• mxrch (and for the logo)
• novitae

Donations

For BTC Donations : 1FHDM49QfZX6pJmhjLE5tB2K6CaTLMZpXZ

 License

GNU General Public License v3.0

Built for educational purposes only.

Modules

HEDnsExtractor - Raw Html Extractor From Hurricane Electric Portal

HEDnsExtractor

Raw html extractor from Hurricane Electric portal

Features

• Automatically identify IPAddr ou Networks through command line parameter or stdin
• Extract networks based on IPAddr.
• Extract domains from networks.

Installation

go install -v github.com/HuntDownProject/hednsextractor/cmd/hednsextractor@latest

Usage

usage -h

nslookup hackerone.com | awk '/Address: / {print $2}' | hednsextractor -silent -only-networks

[INF] [104.16.99.52] 104.16.0.0/12
[INF] [104.16.99.52] 104.16.96.0/20

nslookup hackerone.com | awk '/Address: / {print $2}' | hednsextractor -silent -only-domains | tail -n 10

herllus.com
hezzy.store
hilariostore.com
hiperdrop.com
hippratas.online
hitsstory.com
hobbyshop.site
holyangelstore.com
holzfallerstore.fun
homedescontoo.com

cat $HOME/.config/hednsextractor/config.yaml 

# hednsextractor config file
# generated by https://github.com/projectdiscovery/goflags

# show only domains
#only-domains: false

# show only networks
#only-networks: false

# show virustotal score
#vt: false

# minimum virustotal score to show
#vt-score: 0

# ip address or network to query
#target: 

# show silent output
#silent: false

# show verbose output
#verbose: false

# virustotal api key
vt-api-key: Your API Key goes here

nslookup hackerone.com | awk '/Address: / {print $2}' | hednsextractor -only-domains -vt             

          _______  ______   _        _______  _______          _________ _______  _______  _______ _________ _______  _______ 
|\     /|(  ____ \(  __  \ ( (    /|(  ____ \(  ____ \|\     /|\__   __/(  ____ )(  ___  )(  ____ \\__   __/(  ___  )(  ____ )
| )   ( || (    \/| (  \  )|  \  ( || (    \/| (    \/( \   / )   ) (   | (    )|| (   ) || (    \/   ) (   | (   ) || (    )|
| (___) || (__    | |   ) ||   \ | || (_____ | (__     \ (_) /    | |   | (____)|| (___) || |         | |   | |   | || (____)|
|  ___  ||  __)   | |   | || (\ \) |(_____  )|  __)     ) _ (     | |   |     __)|  ___  || |         | |   | |   | ||     __)
| (   ) || (      | |   ) || | \   |      ) || (       / ( ) \    | |   | (\ (   | (   ) || |         | |   | |   | || (\ (   
| )   ( || (____/\| (__/  )| )  \  |/\____) || (____/\( /   \ )   | |   | ) \ \__| )   ( || (____/\   | |   | (___) || ) \ \__
|/     \|(_______/(______/ |/    )_)\_______)(_______/|/     \|      )_(   |/   \__/|/     \|(_______/   )_(   (_______)|/   \__/

[INF] Current hednsextractor version v1.0.0
[INF] [104.16.0.0/12] domain: ohst.ltd VT Score: 0
[INF] [104.16.0.0/12] domain: jxcraft.net VT Score: 0
[INF] [104.16.0.0/12] domain: teatimegm.com VT Score: 2
[INF] [104.16.0.0/12] domain: debugcheat.com VT Score: 0

SOC-Multitool - A Powerful And User-Friendly Browser Extension That Streamlines Investigations For Security Professionals

Introducing SOC Multi-tool, a free and open-source browser extension that makes investigations faster and more efficient. Now available on the Chrome Web Store and compatible with all Chromium-based browsers such as Microsoft Edge, Chrome, Brave, and Opera.
Now available on Chrome Web Store!

Streamline your investigations

SOC Multi-tool eliminates the need for constant copying and pasting during investigations. Simply highlight the text you want to investigate, right-click, and navigate to the type of data highlighted. The extension will then open new tabs with the results of your investigation.

Modern and feature-rich

The SOC Multi-tool is a modernized multi-tool built from the ground up, with a range of features and capabilities. Some of the key features include:

• IP Reputation Lookup using VirusTotal & AbuseIPDB
• IP Info Lookup using Tor relay checker & WHOIS
• Hash Reputation Lookup using VirusTotal
• Domain Reputation Lookup using VirusTotal & AbuseIPDB
• Domain Info Lookup using Alienvault
• Living off the land binaries Lookup using the LOLBas project
• Decoding of Base64 & HEX using CyberChef
• File Extension & Filename Lookup using fileinfo.com & File.net
• MAC Address manufacturer Lookup using maclookup.com
• Parsing of UserAgent using user-agents.net
• Microsoft Error code Lookup using Microsoft's DB
• Event ID Lookup (Windows, Sharepoint, SQL Server, Exchange, and Sysmon) using ultimatewindowssecurity.com
• Blockchain Address Lookup using blockchain.com
• CVE Info using cve.mitre.org

Easy to install

You can easily install the extension by downloading the release from the Chrome Web Store!
If you wish to make edits you can download from the releases page, extract the folder and make your changes.
To load your edited extension turn on developer mode in your browser's extensions settings, click "Load unpacked" and select the extracted folder!


SOC Multi-tool is a community-driven project and the developer encourages users to contribute and share better resources.

KoodousFinder - A Simple Tool To Allows Users To Search For And Analyze Android Apps For Potential Security Threats And Vulnerabilities

A simple tool to allows users to search for and analyze android apps for potential security threats and vulnerabilities

Account and API Key

Create a Koodous account and get your api key https://koodous.com/settings/developers

Install

$ pip install koodousfinder

Arguments

Examples

koodous.py --package-name "app: Brata AND package: com.brata"
koodous.py --package-name "package: com.google.android.videos AND trusted: true"
koodous.py --package-name "com.metasploit"
python3 koodous.py --app-name "WhatsApp MOD"

Modifiers for advanced search

More information: https://docs.koodous.com/apks.html.
#TODO

• Discord Integration
• Rulesets view

Wafaray - Enhance Your Malware Detection With WAF + YARA (WAFARAY)

WAFARAY is a LAB deployment based on Debian 11.3.0 (stable) x64 made and cooked between two main ingredients WAF + YARA to detect malicious files (e.g. webshells, virus, malware, binaries) typically through web functions (upload files).

Purpose

In essence, the main idea came to use WAF + YARA (YARA right-to-left = ARAY) to detect malicious files at the WAF level before WAF can forward them to the backend e.g. files uploaded through web functions see: https://owasp.org/www-community/vulnerabilities/Unrestricted_File_Upload

When a web page allows uploading files, most of the WAFs are not inspecting files before sending them to the backend. Implementing WAF + YARA could provide malware detection before WAF forwards the files to the backend.

Do malware detection through WAF?

Yes, one solution is to use ModSecurity + Clamav, most of the pages call ClamAV as a process and not as a daemon, in this case, analysing a file could take more than 50 seconds per file. See this resource: https://kifarunix.com/intercept-malicious-file-upload-with-modsecurity-and-clamav/

Do malware detection through WAF + YARA?

:-( A few clues here Black Hat Asia 2019 please continue reading and see below our quick LAB deployment.

WAFARAY: how does it work ?

Basically, It is a quick deployment (1) with pre-compiled and ready-to-use YARA rules via ModSecurity (WAF) using a custom rule; (2) this custom rule will perform an inspection and detection of the files that might contain malicious code, (3) typically web functions (upload files) if the file is suspicious will reject them receiving a 403 code Forbidden by ModSecurity.

Main Paths:

• Yara Compiled rules: /YaraRules/Compiled
• Yara Default rules: /YaraRules/rules
• Yara Scripts: /YaraRules/YaraScripts
• Apache vhosts: /etc/apache2/sites-enabled
• Temporal Files: /temporal
  

Approach

• Blueteamers: Rule enforcement, best alerting, malware detection on files uploaded through web functions.
• Redteamers/pentesters: GreyBox scope , upload and bypass with a malicious file, rule enforcement.
• Security Officers: Keep alerting, threat hunting.
• SOC: Best monitoring about malicious files.
• CERT: Malware Analysis, Determine new IOC.

Building Detection Lab

The Proof of Concept is based on Debian 11.3.0 (stable) x64 OS system, OWASP CRC v3.3.2 and Yara 4.0.5, you will find the automatic installation script here wafaray_install.sh and an optional manual installation guide can be found here: manual_instructions.txt also a PHP page has been created as a "mock" to observe the interaction and detection of malicious files using WAF + YARA.

Installation (recommended) with shell scripts

alex@waf-labs:~$ su root 
root@waf-labs:/home/alex#

# Remember to change YOUR_USER by your username (e.g waf)
root@waf-labs:/home/alex# sed -i 's/^\(# User privi.*\)/\1\nalex ALL=(ALL) NOPASSWD:ALL/g' /etc/sudoers
root@waf-labs:/home/alex# exit
alex@waf-labs:~$ sudo sed -i 's/^\(deb cdrom.*\)/#\1/g' /etc/apt/sources.list
alex@waf-labs:~$ sudo sed -i 's/^# \(deb\-src http.*\)/ \1/g' /etc/apt/sources.list
alex@waf-labs:~$ sudo sed -i 's/^# \(deb http.*\)/ \1/g' /etc/apt/sources.list
alex@waf-labs:~$ echo -ne "\n\ndeb http://deb.debian.org/debian/ bullseye main\ndeb-src http://deb.debian.org/debian/ bullseye main\n" | sudo tee -a  /etc/apt/sources.list
alex@waf-labs:~$ sudo apt-get update
alex@waf-labs:~$ sudo apt-get install sudo -y
alex@waf-labs:~$ sudo apt-get install git vim dos2unix net-tools -y
alex@waf-labs:~$ git clone https://github.com/alt3kx/wafarayalex@waf-labs:~$ cd wafaray
alex@waf-labs:~$ dos2unix wafaray_install.sh
alex@waf-labs:~$ chmod +x wafaray_install.sh
alex@waf-labs:~$ sudo ./wafaray_install.sh >> log_install.log

# Test your LAB environment 
alex@waf-labs:~$ firefox localhost:8080/upload.php

Yara Rules

Once the Yara Rules were downloaded and compiled.

It is similar to when you deploy ModSecurity, you need to customize what kind of rule you need to apply. The following log is an example of when the Web Application Firewall + Yara detected a malicious file, in this case, eicar was detected.

Message: Access denied with code 403 (phase 2). File "/temporal/20220812-184146-YvbXKilOKdNkDfySME10ywAAAAA-file-Wx1hQA" rejected by 
the approver script "/YaraRules/YaraScripts/modsec_yara.py": 0 SUSPECTED [YaraSignature: eicar] 
[file "/etc/apache2/sites-enabled/test.conf"] [line "56"] [id "500002"] 
[msg "Suspected File Upload:eicar.com.txt -> /temporal/20220812-184146-YvbXKilOKdNkDfySME10ywAAAAA-file-Wx1hQA - URI: /upload.php"]

Testing WAFARAY... voilà...

Stop / Start ModSecurity

$ sudo service apache2 stop
$ sudo service apache2 start

Apache Logs

$ cd /var/log
$ sudo tail -f apache2/test_access.log apache2/test_audit.log apache2/test_error.log

Demos

Demo 1 - EICAR

A malicious file is uploaded, and the ModSecurity rules plus Yara denied uploading file to the backend if the file matched with at least one Yara Rule. (Example of Malware: https://secure.eicar.org/eicar.com.txt) NOT EXECUTE THE FILE.

Demo 2 - WebShell.php

For this demo, we disable the rule 933110 - PHP Inject Attack to validate Yara Rules. A malicious file is uploaded, and the ModSecurity rules plus Yara denied uploading file to the backend if the file matched with at least one Yara Rule. (Example of WebShell PHP: https://github.com/drag0s/php-webshell) NOT EXECUTE THE FILE.

Demo 3 - Malware Bazaar (RecordBreaker) Published: 2022-08-13

A malicious file is uploaded, and the ModSecurity rules plus Yara denied uploading file to the backend if the file matched with at least one Yara Rule. (Example of Malware Bazaar (RecordBreaker): https://bazaar.abuse.ch/sample/94ffc1624939c5eaa4ed32d19f82c369333b45afbbd9d053fa82fe8f05d91ac2/) NOT EXECUTE THE FILE.

YARA Rules sources

In case that you want to download more yara rules, you can see the following repositories:

• Yara Signatures Compiled - https://github.com/Yara-Rules/rules
• YARAHub - https://yaraify.abuse.ch/
• Awesome Yara Rules - https://github.com/InQuest/awesome-yara#rules
• Advanced Threat Research Yara Rules - https://github.com/advanced-threat-research/Yara-Rules
• Icewater - https://github.com/SupportIntelligence/Icewater
• Open Source Yara Rules - https://github.com/mikesxrs/Open-Source-YARA-rules
• Bartblaza Yara Rules - https://github.com/bartblaze/Yara-rules
• Cobalstrike - https://github.com/Te-k/cobaltstrike
• Yara Forensic - https://github.com/Xumeiquer/yara-forensics
• Loki - https://github.com/Neo23x0/Loki
• YarGen - https://github.com/Neo23x0/yarGen
• YarAnalyzer - https://github.com/Neo23x0/yarAnalyzer/
• Valhalla - https://www.nextron-systems.com/valhalla/, https://valhalla.nextron-systems.com/
• AlienVault - https://otx.alienvault.com/ (Create an account)

References

• https://portswigger.net/daily-swig/waf-reloaded-modsecurity-3-1-showcased-at-black-hat-asia
• https://yara.readthedocs.io/en/latest/gettingstarted.html
• https://yara.readthedocs.io/en/v3.4.0/yarapython.html
• https://virustotal.github.io/yara/
• https://www.tutorialspoint.com/perl/perl_introduction.htm
• https://malware.expert/scan-every-file-clam-antivirus-scanner-modsecurity/
• https://xael.org/pages/pyclamd-en.html
• https://docs.clamav.net/
• https://www.decalage.info/en/python/pyclamd
• https://opensource.apple.com/source/clamav/clamav-116.2/clamav.Conf/clamd.conf.auto.html
• https://c99.sh/hunting-0days-with-yara-rules/
• https://github.com/claroty/arya
• https://isc.sans.edu/diary/YARA%27s+Console+Module/28288

Roadmap until next release

• Malware Hash Database (MLDBM). The Database stores the MD5 or SHA1 that files were detected as suspicious.
• To be tested CRS Modsecurity v.3.3.3 new rules
• ModSecurity rules improvement to malware detection with Database.
• To be created blacklist and whitelist related to MD5 or SHA1.
• To be tested, run in background if the Yara analysis takes more than 3 seconds.
• To be tested, new payloads, example: Powershell Obfuscasted (WebShells)
• Remarks for live enviroments. (WAF AWS, WAF GCP, ...)

Authors

Alex Hernandez aka (@_alt3kx_)
Jesus Huerta aka @mindhack03d

Contributors

Israel Zeron Medina aka @spk085

Indicator-Intelligence - Finds Related Domains And IPv4 Addresses To Do Threat Intelligence After Indicator-Intelligence Collects Static Files

Finds related domains and IPv4 addresses to do threat intelligence after Indicator-Intelligence collects static files.

Done

• Related domains, IPs collect

Installation

From Source Code

You can use virtualenv for package dependencies before installation.

git clone https://github.com/OsmanKandemir/indicator-intelligence.git
cd indicator-intelligence
python setup.py build
python setup.py install

From Pypi

The script is available on PyPI. To install with pip:

pip install indicatorintelligence

From Dockerfile

You can run this application on a container after build a Dockerfile.

docker build -t indicator .
docker run indicator --domains target-web.com --json

From DockerHub

docker pull osmankandemir/indicator
docker run osmankandemir/indicator --domains target-web.com --json

From Poetry

pip install poetry
poetry install

Usage

-d DOMAINS [DOMAINS], --domains DOMAINS [DOMAINS] Input Targets. --domains target-web1.com target-web2.com
-p PROXY, --proxy PROXY Use HTTP proxy. --proxy 0.0.0.0:8080
-a AGENT, --agent AGENT Use agent. --agent 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)'
-o JSON, --json JSON  JSON output. --json

Function Usage

hardCIDR - Linux Bash Script To Discover The Netblocks, Or Ranges, Owned By The Target Organization

A Linux Bash script to discover the netblocks, or ranges, (in CIDR notation) owned by the target organization during the intelligence gathering phase of a penetration test. This information is maintained by the five Regional Internet Registries (RIRs):

ARIN (North America)
RIPE (Europe/Asia/Middle East)
APNIC (Asia/Pacific)
LACNIC (Latin America)
AfriNIC (Africa)

In addition to netblocks and IP addresses, Autonomous System Numbers (ASNs) are also of interest. ASNs are used as part of the Border Gateway Protocol (BGP) for uniquely identifying each network on the Internet. Target organizations may have their own ASNs due to the size of their network or as a result of redundant service paths from peered service providers. These ASNs will reveal additional netblocks owned by the organization.

Requirements

ipcalc (for RIPE, APNIC, LACNIC, AfriNIC queries)

LACNIC

A note on LACNIC before diving into the usage. LACNIC only allows query of either network range, ASN, Org Handle, or PoC Handle. This does not help us in locating these values based upon the organization name. They do however publish a list of all assigned ranges on a publically accessible FTP server, along with their rate-limiting thresholds. So, there is an accompanying data file, which the script checks for, used to perform LACNIC queries locally. The script includes an update option -r, that can be used to update this data on an interval of your choosing. Approximate run time is just shy of 28 hours.

Usage

The script with no specified options will query ARIN and a pool of BGP route servers. The route server is selected at random at runtime. The -h option lists the help:

The options may be used in any combination, all, or none. Unfortunately, none of the “other” RIRs note the actual CIDR notation of the range, so ipcalc is used to perform this function. If it is not installed on your system, the script will install it for you.

At the prompts, enter the organization name, the email domain, and whether country codes are used as part of the email. If answered Y to country codes, you will be prompted as to whether they precede the domain name or are appended to the TLD. A directory will be created for the output files in /tmp/. If the directory is found to exist, you will be prompted whether to overwrite. If answered N, a time stamp will be appended to the directory name.

The script queries each RIR, as well as a BGP route server, prompting along the way as to whether records were located. Upon completion, three files will be generated: a CSV based on Org Handle, a CSV based on PoC Handle, and a line delimited file of all located raanges in CIDR notation.

Cancelling the script at any time will remove any temporary working files and the directory created for the resultant output files.

It should be noted that, due to similarity in some organization names, you could get back results not related to the target. The CSV files will provide the associated handles and URLs for further validation where necessary. It is also possible that employees of the target organization used their corporate email address to register their own domains. These will be found within the results as well.

Running with Docker

docker build -t hardcidr .

Building the hardcidr image

docker run -v $(pwd):/tmp -it hardcidr

Running the container. Output will be saved to current directory

Additional Information

For more information, check out the blog post on the TrustedSec website: Classy Inter-Domain Routing Enumeration