Uro - Declutters Url Lists For Crawling/Pentesting

Using a URL list for security testing can be painful as there are a lot of URLs that have uninteresting/duplicate content; uro aims to solve that.

It doesn't make any http requests to the URLs and removes: - incremental urls e.g. /page/1/ and /page/2/ - blog posts and similar human written content e.g. /posts/a-brief-history-of-time - urls with same path but parameter value difference e.g. /page.php?id=1 and /page.php?id=2 - images, js, css and other "useless" files

Installation

The recommended way to install uro is as follows:

pipx install uro

Note: If you are using an older version of python, use pip instead of pipx

Basic Usage

The quickest way to include uro in your workflow is to feed it data through stdin and print it to your terminal.

cat urls.txt | uro

Advanced usage

Reading urls from a file (-i/--input)

uro -i input.txt

Writing urls to a file (-o/--output)

If the file already exists, uro will not overwrite the contents. Otherwise, it will create a new file.

uro -i input.txt -o output.txt

Whitelist (-w/--whitelist)

uro will ignore all other extensions except the ones provided.

uro -w php asp html

Note: Extensionless pages e.g. /books/1 will still be included. To remove them too, use --filter hasext.

Blacklist (-b/--blacklist)

uro will ignore the given extensions.

uro -b jpg png js pdf

Note: uro has a list of "useless" extensions which it removes by default; that list will be overridden by whatever extensions you provide through blacklist option. Extensionless pages e.g. /books/1 will still be included. To remove them too, use --filter hasext.

Filters (-f/--filters)

For granular control, uro supports the following filters:

1. hasparams: only output urls that have query parameters e.g. http://example.com/page.php?id=
2. noparams: only output urls that have no query parameters e.g. http://example.com/page.php
3. hasext: only output urls that have extensions e.g. http://example.com/page.php
4. noext: only output urls that have no extensions e.g. http://example.com/page
5. allexts: don't remove any page based on extension e.g. keep .jpg which would be removed otherwise
6. keepcontent: keep human written content e.g. blogs.
7. keepslash: don't remove trailing slash from urls e.g. http://example.com/page/
8. vuln: only output urls with parameters that are know to be vulnerable. More info.

Example: uro --filters hasexts hasparams

CrimsonEDR - Simulate The Behavior Of AV/EDR For Malware Development Training

CrimsonEDR is an open-source project engineered to identify specific malware patterns, offering a tool for honing skills in circumventing Endpoint Detection and Response (EDR). By leveraging diverse detection methods, it empowers users to deepen their understanding of security evasion tactics.

Features

Detection	Description
Direct Syscall	Detects the usage of direct system calls, often employed by malware to bypass traditional API hooks.
NTDLL Unhooking	Identifies attempts to unhook functions within the NTDLL library, a common evasion technique.
AMSI Patch	Detects modifications to the Anti-Malware Scan Interface (AMSI) through byte-level analysis.
ETW Patch	Detects byte-level alterations to Event Tracing for Windows (ETW), commonly manipulated by malware to evade detection.
PE Stomping	Identifies instances of PE (Portable Executable) stomping.
Reflective PE Loading	Detects the reflective loading of PE files, a technique employed by malware to avoid static analysis.
Unbacked Thread Origin	Identifies threads originating from unbacked memory regions, often indicative of malicious activity.
Unbacked Thread Start Address	Detects threads with start addresses pointing to unbacked memory, a potential sign of code injection.
API hooking	Places a hook on the NtWriteVirtualMemory function to monitor memory modifications.
Custom Pattern Search	Allows users to search for specific patterns provided in a JSON file, facilitating the identification of known malware signatures.

Installation

To get started with CrimsonEDR, follow these steps:

1. Install dependancy: bash sudo apt-get install gcc-mingw-w64-x86-64
2. Clone the repository: bash git clone https://github.com/Helixo32/CrimsonEDR
3. Compile the project: bash cd CrimsonEDR; chmod +x compile.sh; ./compile.sh

⚠️ Warning

Windows Defender and other antivirus programs may flag the DLL as malicious due to its content containing bytes used to verify if the AMSI has been patched. Please ensure to whitelist the DLL or disable your antivirus temporarily when using CrimsonEDR to avoid any interruptions.

Usage

To use CrimsonEDR, follow these steps:

1. Make sure the ioc.json file is placed in the current directory from which the executable being monitored is launched. For example, if you launch your executable to monitor from C:\Users\admin\, the DLL will look for ioc.json in C:\Users\admin\ioc.json. Currently, ioc.json contains patterns related to msfvenom. You can easily add your own in the following format:

{
  "IOC": [
    ["0x03", "0x4c", "0x24", "0x08", "0x45", "0x39", "0xd1", "0x75"],
    ["0xf1", "0x4c", "0x03", "0x4c", "0x24", "0x08", "0x45", "0x39"],
    ["0x58", "0x44", "0x8b", "0x40", "0x24", "0x49", "0x01", "0xd0"],
    ["0x66", "0x41", "0x8b", "0x0c", "0x48", "0x44", "0x8b", "0x40"],
    ["0x8b", "0x0c", "0x48", "0x44", "0x8b", "0x40", "0x1c", "0x49"],
    ["0x01", "0xc1", "0x38", "0xe0", "0x75", "0xf1", "0x4c", "0x03"],
    ["0x24", "0x49", "0x01", "0xd0", "0x66", "0x41", "0x8b", "0x0c"],
    ["0xe8", "0xcc", "0x00", "0x00", "0x00", "0x41", "0x51", "0x41"]
  ]
}

1. Execute CrimsonEDRPanel.exe with the following arguments:

   • -d <path_to_dll>: Specifies the path to the CrimsonEDR.dll file.

   • -p <process_id>: Specifies the Process ID (PID) of the target process where you want to inject the DLL.

For example:

.\CrimsonEDRPanel.exe -d C:\Temp\CrimsonEDR.dll -p 1234

Useful Links

Here are some useful resources that helped in the development of this project:

• Windows Processes, Nefarious Anomalies, and You
• MalDev Academy

Contact

For questions, feedback, or support, please reach out to me via:

• Discord : helixo32
• LinkedIn : Matthias Ossard