Uro - Declutters Url Lists For Crawling/Pentesting

Using a URL list for security testing can be painful as there are a lot of URLs that have uninteresting/duplicate content; uro aims to solve that.

It doesn't make any http requests to the URLs and removes: - incremental urls e.g. /page/1/ and /page/2/ - blog posts and similar human written content e.g. /posts/a-brief-history-of-time - urls with same path but parameter value difference e.g. /page.php?id=1 and /page.php?id=2 - images, js, css and other "useless" files

Installation

The recommended way to install uro is as follows:

pipx install uro

Note: If you are using an older version of python, use pip instead of pipx

Basic Usage

The quickest way to include uro in your workflow is to feed it data through stdin and print it to your terminal.

cat urls.txt | uro

Advanced usage

Reading urls from a file (-i/--input)

uro -i input.txt

Writing urls to a file (-o/--output)

If the file already exists, uro will not overwrite the contents. Otherwise, it will create a new file.

uro -i input.txt -o output.txt

Whitelist (-w/--whitelist)

uro will ignore all other extensions except the ones provided.

uro -w php asp html

Note: Extensionless pages e.g. /books/1 will still be included. To remove them too, use --filter hasext.

Blacklist (-b/--blacklist)

uro will ignore the given extensions.

uro -b jpg png js pdf

Note: uro has a list of "useless" extensions which it removes by default; that list will be overridden by whatever extensions you provide through blacklist option. Extensionless pages e.g. /books/1 will still be included. To remove them too, use --filter hasext.

Filters (-f/--filters)

For granular control, uro supports the following filters:

1. hasparams: only output urls that have query parameters e.g. http://example.com/page.php?id=
2. noparams: only output urls that have no query parameters e.g. http://example.com/page.php
3. hasext: only output urls that have extensions e.g. http://example.com/page.php
4. noext: only output urls that have no extensions e.g. http://example.com/page
5. allexts: don't remove any page based on extension e.g. keep .jpg which would be removed otherwise
6. keepcontent: keep human written content e.g. blogs.
7. keepslash: don't remove trailing slash from urls e.g. http://example.com/page/
8. vuln: only output urls with parameters that are know to be vulnerable. More info.

Example: uro --filters hasexts hasparams

WordPress Plugin Exploited to Steal Credit Card Data from E-commerce Sites

Unknown threat actors are abusing lesser-known code snippet plugins for WordPress to insert malicious PHP code in victim sites that are capable of harvesting credit card data. The campaign, observed by Sucuri on May 11, 2024, entails the abuse of a WordPress plugin called Dessky Snippets, which allows users to add custom PHP code. It has over 200 active installations.

WordPress Bricks Theme Under Active Attack: Critical Flaw Impacts 25,000+ Sites

A critical security flaw in the Bricks theme for WordPress is being actively exploited by threat actors to run arbitrary PHP code on susceptible installations. The flaw, tracked as CVE-2024-25600 (CVSS score: 9.8), enables unauthenticated attackers to achieve remote code execution. It impacts all versions of the Bricks up to and including 1.9.6. It has been addressed by the theme developers in&

WordPress Releases Update 6.4.2 to Address Critical Remote Attack Vulnerability

WordPress has released version 6.4.2 with a patch for a critical security flaw that could be exploited by threat actors by combining it with another bug to execute arbitrary PHP code on vulnerable sites. "A remote code execution vulnerability that is not directly exploitable in core; however, the security team feels that there is a potential for high severity when combined with some plugins,

Warning: 3 Critical Vulnerabilities Expose ownCloud Users to Data Breaches

The maintainers of the open-source file-sharing software ownCloud have warned of three critical security flaws that could be exploited to disclose sensitive information and modify files. A brief description of the vulnerabilities is as follows - CVE-2023-49103 (CVSS score: 10.0) - Disclosure of sensitive credentials and configuration in containerized deployments impacting graphapi versions from

PHP Packagist supply chain poisoned by hacker “looking for a job”

I pwned you! Gizza job! You know it makes sense!

SHA-3 code execution bug patched in PHP – check your version!

As everyone waits for news of a bug in OpenSSL, here's a reminder that other cryptographic code in your life may also need patching!

Poisoned Python and PHP packages purloin passwords for AWS access

More supply chain trouble - this time with clear examples so you can learn how to spot this stuff yourself.

Irony alert! PHP fixes security flaw in input validation code

What's wrong with this sequence? 1. Step into the road 2. Check if it's safe 3. Keep on walki...