Scrapling - An Undetectable, Powerful, Flexible, High-Performance Python Library That Makes Web Scraping Simple And Easy Again!

Dealing with failing web scrapers due to anti-bot protections or website changes? Meet Scrapling.

Scrapling is a high-performance, intelligent web scraping library for Python that automatically adapts to website changes while significantly outperforming popular alternatives. For both beginners and experts, Scrapling provides powerful features while maintaining simplicity.

>> from scrapling.defaults import Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher
# Fetch websites' source under the radar!
>> page = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True)
>> print(page.status)
200
>> products = page.css('.product', auto_save=True)  # Scrape data that survives website design changes!
>> # Later, if the website structure changes, pass `auto_match=True`
>> products = page.css('.product', auto_match=True)  # and Scrapling still finds them!

Key Features

Fetch websites as you prefer with async support

• HTTP Requests: Fast and stealthy HTTP requests with the Fetcher class.
• Dynamic Loading & Automation: Fetch dynamic websites with the PlayWrightFetcher class through your real browser, Scrapling's stealth mode, Playwright's Chrome browser, or NSTbrowser's browserless!
• Anti-bot Protections Bypass: Easily bypass protections with StealthyFetcher and PlayWrightFetcher classes.

Adaptive Scraping

• 🔄 Smart Element Tracking: Relocate elements after website changes, using an intelligent similarity system and integrated storage.
• 🎯 Flexible Selection: CSS selectors, XPath selectors, filters-based search, text search, regex search and more.
• 🔍 Find Similar Elements: Automatically locate elements similar to the element you found!
• 🧠 Smart Content Scraping: Extract data from multiple websites without specific selectors using Scrapling powerful features.

High Performance

• 🚀 Lightning Fast: Built from the ground up with performance in mind, outperforming most popular Python scraping libraries.
• 🔋 Memory Efficient: Optimized data structures for minimal memory footprint.
• ⚡ Fast JSON serialization: 10x faster than standard library.

Developer Friendly

• 🛠️ Powerful Navigation API: Easy DOM traversal in all directions.
• 🧬 Rich Text Processing: All strings have built-in regex, cleaning methods, and more. All elements' attributes are optimized dictionaries that takes less memory than standard dictionaries with added methods.
• 📝 Auto Selectors Generation: Generate robust short and full CSS/XPath selectors for any element.
• 🔌 Familiar API: Similar to Scrapy/BeautifulSoup and the same pseudo-elements used in Scrapy.
• 📘 Type hints: Complete type/doc-strings coverage for future-proofing and best autocompletion support.

Getting Started

from scrapling.fetchers import Fetcher

fetcher = Fetcher(auto_match=False)

# Do http GET request to a web page and create an Adaptor instance
page = fetcher.get('https://quotes.toscrape.com/', stealthy_headers=True)
# Get all text content from all HTML tags in the page except `script` and `style` tags
page.get_all_text(ignore_tags=('script', 'style'))

# Get all quotes elements, any of these methods will return a list of strings directly (TextHandlers)
quotes = page.css('.quote .text::text')  # CSS selector
quotes = page.xpath('//span[@class="text"]/text()')  # XPath
quotes = page.css('.quote').css('.text::text')  # Chained selectors
quotes = [element.text for element in page.css('.quote .text')]  # Slower than bulk query above

# Get the first quote element
quote = page.css_first('.quote')  # same as page.css('.quote').first or page.css('.quote')[0]

# Tired of selectors? Use find_all/find
# Get all 'div' HTML tags that one of its 'class' values is 'quote'
quotes = page.find_all('div', {'class': 'quote'})
# Same as
quotes = page.find_all('div', class_='quote')
quotes = page.find_all(['div'], class_='quote')
quotes = page.find_all(class_='quote')  # and so on...

# Working with elements
quote.html_content  # Get Inner HTML of this element
quote.prettify()  # Prettified version of Inner HTML above
quote.attrib  # Get that element's attributes
quote.path  # DOM path to element (List of all ancestors from <html> tag till the element itself)

To keep it simple, all methods can be chained on top of each other!

Parsing Performance

Scrapling isn't just powerful - it's also blazing fast. Scrapling implements many best practices, design patterns, and numerous optimizations to save fractions of seconds. All of that while focusing exclusively on parsing HTML documents. Here are benchmarks comparing Scrapling to popular Python libraries in two tests.

Text Extraction Speed Test (5000 nested elements).

As you see, Scrapling is on par with Scrapy and slightly faster than Lxml which both libraries are built on top of. These are the closest results to Scrapling. PyQuery is also built on top of Lxml but still, Scrapling is 4 times faster.

Extraction By Text Speed Test

Scrapling can find elements with more methods and it returns full element Adaptor objects not only the text like AutoScraper. So, to make this test fair, both libraries will extract an element with text, find similar elements, and then extract the text content for all of them. As you see, Scrapling is still 4.5 times faster at the same task.

All benchmarks' results are an average of 100 runs. See our benchmarks.py for methodology and to run your comparisons.

Installation

Scrapling is a breeze to get started with; Starting from version 0.2.9, we require at least Python 3.9 to work.

pip3 install scrapling

Then run this command to install browsers' dependencies needed to use Fetcher classes

scrapling install

If you have any installation issues, please open an issue.

Fetching Websites

Fetchers are interfaces built on top of other libraries with added features that do requests or fetch pages for you in a single request fashion and then return an Adaptor object. This feature was introduced because the only option we had before was to fetch the page as you wanted it, then pass it manually to the Adaptor class to create an Adaptor instance and start playing around with the page.

Features

You might be slightly confused by now so let me clear things up. All fetcher-type classes are imported in the same way

from scrapling.fetchers import Fetcher, StealthyFetcher, PlayWrightFetcher

All of them can take these initialization arguments: auto_match, huge_tree, keep_comments, keep_cdata, storage, and storage_args, which are the same ones you give to the Adaptor class.

If you don't want to pass arguments to the generated Adaptor object and want to use the default values, you can use this import instead for cleaner code:

from scrapling.defaults import Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher

then use it right away without initializing like:

page = StealthyFetcher.fetch('https://example.com') 

Also, the Response object returned from all fetchers is the same as the Adaptor object except it has these added attributes: status, reason, cookies, headers, history, and request_headers. All cookies, headers, and request_headers are always of type dictionary.

[!NOTE] The auto_match argument is enabled by default which is the one you should care about the most as you will see later.

Fetcher

This class is built on top of httpx with additional configuration options, here you can do GET, POST, PUT, and DELETE requests.

For all methods, you have stealthy_headers which makes Fetcher create and use real browser's headers then create a referer header as if this request came from Google's search of this URL's domain. It's enabled by default. You can also set the number of retries with the argument retries for all methods and this will make httpx retry requests if it failed for any reason. The default number of retries for all Fetcher methods is 3.

Hence: All headers generated by stealthy_headers argument can be overwritten by you through the headers argument

You can route all traffic (HTTP and HTTPS) to a proxy for any of these methods in this format http://username:password@localhost:8030

>> page = Fetcher().get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True)
>> page = Fetcher().post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')
>> page = Fetcher().put('https://httpbin.org/put', data={'key': 'value'})
>> page = Fetcher().delete('https://httpbin.org/delete')

For Async requests, you will just replace the import like below:

>> from scrapling.fetchers import AsyncFetcher
>> page = await AsyncFetcher().get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True)
>> page = await AsyncFetcher().post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')
>> page = await AsyncFetcher().put('https://httpbin.org/put', data={'key': 'value'})
>> page = await AsyncFetcher().delete('https://httpbin.org/delete')

StealthyFetcher

This class is built on top of Camoufox, bypassing most anti-bot protections by default. Scrapling adds extra layers of flavors and configurations to increase performance and undetectability even further.

>> page = StealthyFetcher().fetch('https://www.browserscan.net/bot-detection')  # Running headless by default
>> page.status == 200
True
>> page = await StealthyFetcher().async_fetch('https://www.browserscan.net/bot-detection')  # the async version of fetch
>> page.status == 200
True

Note: all requests done by this fetcher are waiting by default for all JS to be fully loaded and executed so you don't have to :)

This list isn't final so expect a lot more additions and flexibility to be added in the next versions!

PlayWrightFetcher

This class is built on top of Playwright which currently provides 4 main run options but they can be mixed as you want.

>> page = PlayWrightFetcher().fetch('https://www.google.com/search?q=%22Scrapling%22', disable_resources=True)  # Vanilla Playwright option
>> page.css_first("#search a::attr(href)")
'https://github.com/D4Vinci/Scrapling'
>> page = await PlayWrightFetcher().async_fetch('https://www.google.com/search?q=%22Scrapling%22', disable_resources=True)  # the async version of fetch
>> page.css_first("#search a::attr(href)")
'https://github.com/D4Vinci/Scrapling'

Note: all requests done by this fetcher are waiting by default for all JS to be fully loaded and executed so you don't have to :)

Using this Fetcher class, you can make requests with: 1) Vanilla Playwright without any modifications other than the ones you chose. 2) Stealthy Playwright with the stealth mode I wrote for it. It's still a WIP but it bypasses many online tests like Sannysoft's. Some of the things this fetcher's stealth mode does include: * Patching the CDP runtime fingerprint. * Mimics some of the real browsers' properties by injecting several JS files and using custom options. * Using custom flags on launch to hide Playwright even more and make it faster. * Generates real browser's headers of the same type and same user OS then append it to the request's headers. 3) Real browsers by passing the real_chrome argument or the CDP URL of your browser to be controlled by the Fetcher and most of the options can be enabled on it. 4) NSTBrowser's docker browserless option by passing the CDP URL and enabling nstbrowser_mode option.

Hence using the real_chrome argument requires that you have Chrome browser installed on your device

Add that to a lot of controlling/hiding options as you will see in the arguments list below.

This list isn't final so expect a lot more additions and flexibility to be added in the next versions!

Advanced Parsing Features

Smart Navigation

>>> quote.tag
'div'

>>> quote.parent
<data='<div class="col-md-8"> <div class="quote...' parent='<div class="row"> <div class="col-md-8">...'>

>>> quote.parent.tag
'div'

>>> quote.children
[<data='<span class="text" itemprop="text">"The...' parent='<div class="quote" itemscope itemtype="h...'>,
 <data='<span>by <small class="author" itemprop=...' parent='<div class="quote" itemscope itemtype="h...'>,
 <data='<div class="tags"> Tags: <meta class="ke...' parent='<div class="quote" itemscope itemtype="h...'>]

>>> quote.siblings
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
 <data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
...]

>>> quote.next  # gets the next element, the same logic applies to `quote.previous`
<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>

>>> quote.children.css_first(".author::text")
'Albert Einstein'

>>> quote.has_class('quote')
True

# Generate new selectors for any element
>>> quote.generate_css_selector
'body > div > div:nth-of-type(2) > div > div'

# Test these selectors on your favorite browser or reuse them again in the library's methods!
>>> quote.generate_xpath_selector
'//body/div/div[2]/div/div'

If your case needs more than the element's parent, you can iterate over the whole ancestors' tree of any element like below

for ancestor in quote.iterancestors():
    # do something with it...

You can search for a specific ancestor of an element that satisfies a function, all you need to do is to pass a function that takes an Adaptor object as an argument and return True if the condition satisfies or False otherwise like below:

>>> quote.find_ancestor(lambda ancestor: ancestor.has_class('row'))
<data='<div class="row"> <div class="col-md-8">...' parent='<div class="container"> <div class="row...'>

Content-based Selection & Finding Similar Elements

You can select elements by their text content in multiple ways, here's a full example on another website:

>>> page = Fetcher().get('https://books.toscrape.com/index.html')

>>> page.find_by_text('Tipping the Velvet')  # Find the first element whose text fully matches this text
<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>

>>> page.urljoin(page.find_by_text('Tipping the Velvet').attrib['href'])  # We use `page.urljoin` to return the full URL from the relative `href`
'https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html'

>>> page.find_by_text('Tipping the Velvet', first_match=False)  # Get all matches if there are more
[<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>]

>>> page.find_by_regex(r'£[\d\.]+')  # Get the first element that its text content matches my price regex
<data='<p class="price_color">£51.77</p>' parent='<div class="product_price"> <p class="pr...'>

>>> page.find_by_regex(r'£[\d\.]+', first_match=False)  # Get all elements that matches my price regex
[<data='<p class="price_color">£51.77</p>' parent='<div class="product_price"> <p class="pr...'>,
 <data='<p class="price_color">£53.74</p>' parent='<div class="product_price"> <p class="pr...'>,
 <data='<p class="price_color">£50.10</p>' parent='<div class="product_price"> <p class="pr...'>,
 <data='<p class="price_color">£47.82</p>' parent='<div class="product_price"> <p class="pr...'>,
 ...]

Find all elements that are similar to the current element in location and attributes

# For this case, ignore the 'title' attribute while matching
>>> page.find_by_text('Tipping the Velvet').find_similar(ignore_attributes=['title'])
[<data='<a href="catalogue/a-light-in-the-attic_...' parent='<h3><a href="catalogue/a-light-in-the-at...'>,
 <data='<a href="catalogue/soumission_998/index....' parent='<h3><a href="catalogue/soumission_998/in...'>,
 <data='<a href="catalogue/sharp-objects_997/ind...' parent='<h3><a href="catalogue/sharp-objects_997...'>,
...]

# You will notice that the number of elements is 19 not 20 because the current element is not included.
>>> len(page.find_by_text('Tipping the Velvet').find_similar(ignore_attributes=['title']))
19

# Get the `href` attribute from all similar elements
>>> [element.attrib['href'] for element in page.find_by_text('Tipping the Velvet').find_similar(ignore_attributes=['title'])]
['catalogue/a-light-in-the-attic_1000/index.html',
 'catalogue/soumission_998/index.html',
 'catalogue/sharp-objects_997/index.html',
 ...]

To increase the complexity a little bit, let's say we want to get all books' data using that element as a starting point for some reason

>>> for product in page.find_by_text('Tipping the Velvet').parent.parent.find_similar():
        print({
            "name": product.css_first('h3 a::text'),
            "price": product.css_first('.price_color').re_first(r'[\d\.]+'),
            "stock": product.css('.availability::text')[-1].clean()
        })
{'name': 'A Light in the ...', 'price': '51.77', 'stock': 'In stock'}
{'name': 'Soumission', 'price': '50.10', 'stock': 'In stock'}
{'name': 'Sharp Objects', 'price': '47.82', 'stock': 'In stock'}
...

The documentation will provide more advanced examples.

Handling Structural Changes

Let's say you are scraping a page with a structure like this:

<div class="container">
    <section class="products">
        <article class="product" id="p1">
            <h3>Product 1</h3>
            <p class="description">Description 1</p>
        </article>
        <article class="product" id="p2">
            <h3>Product 2</h3>
            <p class="description">Description 2</p>
        </article>
    </section>
</div>

And you want to scrape the first product, the one with the p1 ID. You will probably write a selector like this

page.css('#p1')

When website owners implement structural changes like

<div class="new-container">
    <div class="product-wrapper">
        <section class="products">
            <article class="product new-class" data-id="p1">
                <div class="product-info">
                    <h3>Product 1</h3>
                    <p class="new-description">Description 1</p>
                </div>
            </article>
            <article class="product new-class" data-id="p2">
                <div class="product-info">
                    <h3>Product 2</h3>
                    <p class="new-description">Description 2</p>
                </div>
            </article>
        </section>
    </div>
</div>

The selector will no longer function and your code needs maintenance. That's where Scrapling's auto-matching feature comes into play.

from scrapling.parser import Adaptor
# Before the change
page = Adaptor(page_source, url='example.com')
element = page.css('#p1' auto_save=True)
if not element:  # One day website changes?
    element = page.css('#p1', auto_match=True)  # Scrapling still finds it!
# the rest of the code...

How does the auto-matching work? Check the FAQs section for that and other possible issues while auto-matching.

Real-World Scenario

Let's use a real website as an example and use one of the fetchers to fetch its source. To do this we need to find a website that will change its design/structure soon, take a copy of its source then wait for the website to make the change. Of course, that's nearly impossible to know unless I know the website's owner but that will make it a staged test haha.

To solve this issue, I will use The Web Archive's Wayback Machine. Here is a copy of StackOverFlow's website in 2010, pretty old huh?Let's test if the automatch feature can extract the same button in the old design from 2010 and the current design using the same selector :)

If I want to extract the Questions button from the old design I can use a selector like this #hmenus > div:nth-child(1) > ul > li:nth-child(1) > a This selector is too specific because it was generated by Google Chrome. Now let's test the same selector in both versions

>> from scrapling.fetchers import Fetcher
>> selector = '#hmenus > div:nth-child(1) > ul > li:nth-child(1) > a'
>> old_url = "https://web.archive.org/web/20100102003420/http://stackoverflow.com/"
>> new_url = "https://stackoverflow.com/"
>> 
>> page = Fetcher(automatch_domain='stackoverflow.com').get(old_url, timeout=30)
>> element1 = page.css_first(selector, auto_save=True)
>> 
>> # Same selector but used in the updated website
>> page = Fetcher(automatch_domain="stackoverflow.com").get(new_url)
>> element2 = page.css_first(selector, auto_match=True)
>> 
>> if element1.text == element2.text:
...    print('Scrapling found the same element in the old design and the new design!')
'Scrapling found the same element in the old design and the new design!'

Note that I used a new argument called automatch_domain, this is because for Scrapling these are two different URLs, not the website so it isolates their data. To tell Scrapling they are the same website, we then pass the domain we want to use for saving auto-match data for them both so Scrapling doesn't isolate them.

In a real-world scenario, the code will be the same except it will use the same URL for both requests so you won't need to use the automatch_domain argument. This is the closest example I can give to real-world cases so I hope it didn't confuse you :)

Notes: 1. For the two examples above I used one time the Adaptor class and the second time the Fetcher class just to show you that you can create the Adaptor object by yourself if you have the source or fetch the source using any Fetcher class then it will create the Adaptor object for you. 2. Passing the auto_save argument with the auto_match argument set to False while initializing the Adaptor/Fetcher object will only result in ignoring the auto_save argument value and the following warning message text Argument `auto_save` will be ignored because `auto_match` wasn't enabled on initialization. Check docs for more info. This behavior is purely for performance reasons so the database gets created/connected only when you are planning to use the auto-matching features. Same case with the auto_match argument.

1. The auto_match parameter works only for Adaptor instances not Adaptors so if you do something like this you will get an error python page.css('body').css('#p1', auto_match=True) because you can't auto-match a whole list, you have to be specific and do something like python page.css_first('body').css('#p1', auto_match=True)

Find elements by filters

Inspired by BeautifulSoup's find_all function you can find elements by using find_all/find methods. Both methods can take multiple types of filters and return all elements in the pages that all these filters apply to.

• To be more specific:
• Any string passed is considered a tag name
• Any iterable passed like List/Tuple/Set is considered an iterable of tag names.
• Any dictionary is considered a mapping of HTML element(s) attribute names and attribute values.
• Any regex patterns passed are used as filters to elements by their text content
• Any functions passed are used as filters
• Any keyword argument passed is considered as an HTML element attribute with its value.

So the way it works is after collecting all passed arguments and keywords, each filter passes its results to the following filter in a waterfall-like filtering system.
It filters all elements in the current page/element in the following order:

1. All elements with the passed tag name(s).
2. All elements that match all passed attribute(s).
3. All elements that its text content match all passed regex patterns.
4. All elements that fulfill all passed function(s).

Note: The filtering process always starts from the first filter it finds in the filtering order above so if no tag name(s) are passed but attributes are passed, the process starts from that layer and so on. But the order in which you pass the arguments doesn't matter.

Examples to clear any confusion :)

>> from scrapling.fetchers import Fetcher
>> page = Fetcher().get('https://quotes.toscrape.com/')
# Find all elements with tag name `div`.
>> page.find_all('div')
[<data='<div class="container"> <div class="row...' parent='<body> <div class="container"> <div clas...'>,
 <data='<div class="row header-box"> <div class=...' parent='<div class="container"> <div class="row...'>,
...]

# Find all div elements with a class that equals `quote`.
>> page.find_all('div', class_='quote')
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
 <data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
...]

# Same as above.
>> page.find_all('div', {'class': 'quote'})
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
 <data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
...]

# Find all elements with a class that equals `quote`.
>> page.find_all({'class': 'quote'})
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
 <data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
...]

# Find all div elements with a class that equals `quote`, and contains the element `.text` which contains the word 'world' in its content.
>> page.find_all('div', {'class': 'quote'}, lambda e: "world" in e.css_first('.text::text'))
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>]

# Find all elements that don't have children.
>> page.find_all(lambda element: len(element.children) > 0)
[<data='<html lang="en"><head><meta charset="UTF...'>,
 <data='<head><meta charset="UTF-8"><title>Quote...' parent='<html lang="en"><head><meta charset="UTF...'>,
 <data='<body> <div class="container"> <div clas...' parent='<html lang="en"><head><meta charset="UTF...'>,
...]

# Find all elements that contain the word 'world' in its content.
>> page.find_all(lambda element: "world" in element.text)
[<data='<span class="text" itemprop="text">"The...' parent='<div class="quote" itemscope itemtype="h...'>,
 <data='<a class="tag" href="/tag/world/page/1/"...' parent='<div class="tags"> Tags: <meta class="ke...'>]

# Find all span elements that match the given regex
>> page.find_all('span', re.compile(r'world'))
[<data='<span class="text" itemprop="text">"The...' parent='<div class="quote" itemscope itemtype="h...'>]

# Find all div and span elements with class 'quote' (No span elements like that so only div returned)
>> page.find_all(['div', 'span'], {'class': 'quote'})
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
 <data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
...]

# Mix things up
>> page.find_all({'itemtype':"http://schema.org/CreativeWork"}, 'div').css('.author::text')
['Albert Einstein',
 'J.K. Rowling',
...]

Is That All?

Here's what else you can do with Scrapling:

• Accessing the lxml.etree object itself of any element directly python >>> quote._root <Element div at 0x107f98870>

• Saving and retrieving elements manually to auto-match them outside the css and the xpath methods but you have to set the identifier by yourself.

• To save an element to the database: python >>> element = page.find_by_text('Tipping the Velvet', first_match=True) >>> page.save(element, 'my_special_element')

• Now later when you want to retrieve it and relocate it inside the page with auto-matching, it would be like this python >>> element_dict = page.retrieve('my_special_element') >>> page.relocate(element_dict, adaptor_type=True) [<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>] >>> page.relocate(element_dict, adaptor_type=True).css('::text') ['Tipping the Velvet']

• if you want to keep it as lxml.etree object, leave the adaptor_type argument python >>> page.relocate(element_dict) [<Element a at 0x105a2a7b0>]

• Filtering results based on a function

# Find all products over $50
expensive_products = page.css('.product_pod').filter(
    lambda p: float(p.css('.price_color').re_first(r'[\d\.]+')) > 50
)

• Searching results for the first one that matches a function

# Find all the products with price '53.23'
page.css('.product_pod').search(
    lambda p: float(p.css('.price_color').re_first(r'[\d\.]+')) == 54.23
)

• Doing operations on element content is the same as scrapy python quote.re(r'regex_pattern') # Get all strings (TextHandlers) that match the regex pattern quote.re_first(r'regex_pattern') # Get the first string (TextHandler) only quote.json() # If the content text is jsonable, then convert it to json using `orjson` which is 10x faster than the standard json library and provides more options except that you can do more with them like python quote.re( r'regex_pattern', replace_entities=True, # Character entity references are replaced by their corresponding character clean_match=True, # This will ignore all whitespaces and consecutive spaces while matching case_sensitive= False, # Set the regex to ignore letters case while compiling it ) Hence all of these methods are methods from the TextHandler within that contains the text content so the same can be done directly if you call the .text property or equivalent selector function.

• Doing operations on the text content itself includes

• Cleaning the text from any white spaces and replacing consecutive spaces with single space python quote.clean()
• You already know about the regex matching and the fast json parsing but did you know that all strings returned from the regex search are actually TextHandler objects too? so in cases where you have for example a JS object assigned to a JS variable inside JS code and want to extract it with regex and then convert it to json object, in other libraries, these would be more than 1 line of code but here you can do it in 1 line like this python page.xpath('//script/text()').re_first(r'var dataLayer = (.+);').json()

• Sort all characters in the string as if it were a list and return the new string python quote.sort(reverse=False)

  To be clear, TextHandler is a sub-class of Python's str so all normal operations/methods that work with Python strings will work with it.

• Any element's attributes are not exactly a dictionary but a sub-class of mapping called AttributesHandler that's read-only so it's faster and string values returned are actually TextHandler objects so all operations above can be done on them, standard dictionary operations that don't modify the data, and more :)

• Unlike standard dictionaries, here you can search by values too and can do partial searches. It might be handy in some cases (returns a generator of matches) python >>> for item in element.attrib.search_values('catalogue', partial=True): print(item) {'href': 'catalogue/tipping-the-velvet_999/index.html'}
• Serialize the current attributes to JSON bytes: python >>> element.attrib.json_string b'{"href":"catalogue/tipping-the-velvet_999/index.html","title":"Tipping the Velvet"}'
• Converting it to a normal dictionary python >>> dict(element.attrib) {'href': 'catalogue/tipping-the-velvet_999/index.html', 'title': 'Tipping the Velvet'}

Scrapling is under active development so expect many more features coming soon :)

More Advanced Usage

There are a lot of deep details skipped here to make this as short as possible so to take a deep dive, head to the docs section. I will try to keep it updated as possible and add complex examples. There I will explain points like how to write your storage system, write spiders that don't depend on selectors at all, and more...

Note that implementing your storage system can be complex as there are some strict rules such as inheriting from the same abstract class, following the singleton design pattern used in other classes, and more. So make sure to read the docs first.

[!IMPORTANT] A website is needed to provide detailed library documentation.
I'm trying to rush creating the website, researching new ideas, and adding more features/tests/benchmarks but time is tight with too many spinning plates between work, personal life, and working on Scrapling. I have been working on Scrapling for months for free after all.

If you like Scrapling and want it to keep improving then this is a friendly reminder that you can help by supporting me through the sponsor button.

⚡ Enlightening Questions and FAQs

This section addresses common questions about Scrapling, please read this section before opening an issue.

How does auto-matching work?

1. You need to get a working selector and run it at least once with methods css or xpath with the auto_save parameter set to True before structural changes happen.
2. Before returning results for you, Scrapling uses its configured database and saves unique properties about that element.

3. Now because everything about the element can be changed or removed, nothing from the element can be used as a unique identifier for the database. To solve this issue, I made the storage system rely on two things:

   1. The domain of the URL you gave while initializing the first Adaptor object
   2. The identifier parameter you passed to the method while selecting. If you didn't pass one, then the selector string itself will be used as an identifier but remember you will have to use it as an identifier value later when the structure changes and you want to pass the new selector.

   Together both are used to retrieve the element's unique properties from the database later. 4. Now later when you enable the auto_match parameter for both the Adaptor instance and the method call. The element properties are retrieved and Scrapling loops over all elements in the page and compares each one's unique properties to the unique properties we already have for this element and a score is calculated for each one. 5. Comparing elements is not exact but more about finding how similar these values are, so everything is taken into consideration, even the values' order, like the order in which the element class names were written before and the order in which the same element class names are written now. 6. The score for each element is stored in the table, and the element(s) with the highest combined similarity scores are returned.

How does the auto-matching work if I didn't pass a URL while initializing the Adaptor object?

Not a big problem as it depends on your usage. The word default will be used in place of the URL field while saving the element's unique properties. So this will only be an issue if you used the same identifier later for a different website that you didn't pass the URL parameter while initializing it as well. The save process will overwrite the previous data and auto-matching uses the latest saved properties only.

If all things about an element can change or get removed, what are the unique properties to be saved?

For each element, Scrapling will extract: - Element tag name, text, attributes (names and values), siblings (tag names only), and path (tag names only). - Element's parent tag name, attributes (names and values), and text.

I have enabled the auto_save/auto_match parameter while selecting and it got completely ignored with a warning message

That's because passing the auto_save/auto_match argument without setting auto_match to True while initializing the Adaptor object will only result in ignoring the auto_save/auto_match argument value. This behavior is purely for performance reasons so the database gets created only when you are planning to use the auto-matching features.

I have done everything as the docs but the auto-matching didn't return anything, what's wrong?

It could be one of these reasons: 1. No data were saved/stored for this element before. 2. The selector passed is not the one used while storing element data. The solution is simple - Pass the old selector again as an identifier to the method called. - Retrieve the element with the retrieve method using the old selector as identifier then save it again with the save method and the new selector as identifier. - Start using the identifier argument more often if you are planning to use every new selector from now on. 3. The website had some extreme structural changes like a new full design. If this happens a lot with this website, the solution would be to make your code as selector-free as possible using Scrapling features.

Can Scrapling replace code built on top of BeautifulSoup4?

Pretty much yeah, almost all features you get from BeautifulSoup can be found or achieved in Scrapling one way or another. In fact, if you see there's a feature in bs4 that is missing in Scrapling, please make a feature request from the issues tab to let me know.

Can Scrapling replace code built on top of AutoScraper?

Of course, you can find elements by text/regex, find similar elements in a more reliable way than AutoScraper, and finally save/retrieve elements manually to use later as the model feature in AutoScraper. I have pulled all top articles about AutoScraper from Google and tested Scrapling against examples in them. In all examples, Scrapling got the same results as AutoScraper in much less time.

Is Scrapling thread-safe?

Yes, Scrapling instances are thread-safe. Each Adaptor instance maintains its state.

More Sponsors!

Contributing

Everybody is invited and welcome to contribute to Scrapling. There is a lot to do!

Please read the contributing file before doing anything.

Disclaimer for Scrapling Project

[!CAUTION] This library is provided for educational and research purposes only. By using this library, you agree to comply with local and international laws regarding data scraping and privacy. The authors and contributors are not responsible for any misuse of this software. This library should not be used to violate the rights of others, for unethical purposes, or to use data in an unauthorized or illegal manner. Do not use it on any website unless you have permission from the website owner or within their allowed rules like the robots.txt file, for example.

License

This work is licensed under BSD-3

Acknowledgments

This project includes code adapted from: - Parsel (BSD License) - Used for translator submodule

Thanks and References

• Daijro's brilliant work on both BrowserForge and Camoufox
• Vinyzu's work on Playwright's mock on Botright
• brotector
• fakebrowser
• rebrowser-patches

Known Issues

• In the auto-matching save process, the unique properties of the first element from the selection results are the only ones that get saved. So if the selector you are using selects different elements on the page that are in different locations, auto-matching will probably return to you the first element only when you relocate it later. This doesn't include combined CSS selectors (Using commas to combine more than one selector for example) as these selectors get separated and each selector gets executed alone.

Docf-Sec-Check - DockF-Sec-Check Helps To Make Your Dockerfile Commands More Secure

DockF-Sec-Check helps to make your Dockerfile commands more secure.

Done

• [x] First-level security notification in the Dockerfile

TODO List

• [ ] Correctly detect the Dockerfile.
• [ ] Second-level security notification in the Dockerfile.
• [ ] Security notification in Docker images.
• [ ] ***** (Private Repository)

Installation

From Source Code

You can use virtualenv for package dependencies before installation.

git clone https://github.com/OsmanKandemir/docf-sec-check.git
cd docf-sec-check
python setup.py build
python setup.py install

From Pypi

The application is available on PyPI. To install with pip:

pip install docfseccheck

From Dockerfile

You can run this application on a container after build a Dockerfile. You need to specify a path (YOUR-LOCAL-PATH) to scan the Dockerfile in your local.

docker build -t docfseccheck .
docker run -v <YOUR-LOCAL-PATH>/Dockerfile:/docf-sec-check/Dockerfile docfseccheck -f /docf-sec-check/Dockerfile

From DockerHub

docker pull osmankandemir/docfseccheck:v1.0
docker run -v <YOUR-LOCAL-PATH>/Dockerfile:/docf-sec-check/Dockerfile osmankandemir/docfseccheck:v1.0 -f /docf-sec-check/Dockerfile

Usage

-f DOCKERFILE [DOCKERFILE], --file DOCKERFILE [DOCKERFILE] Dockerfile path. --file Dockerfile

Function Usage

from docfchecker import DocFChecker

#Dockerfile is your file PATH.

DocFChecker(["Dockerfile"]) 

Development and Contribution

See; CONTRIBUTING.md

License

Copyright (c) 2024 Osman Kandemir \ Licensed under the GPL-3.0 License.

Donations

If you like DocF-Sec-Check and would like to show support, you can use Buy A Coffee or Github Sponsors feature for the developer using the button below.

Or

Sponsor me : https://github.com/sponsors/OsmanKandemir 😊

Your support will be much appreciated😊

Secator - The Pentester'S Swiss Knife

secator is a task and workflow runner used for security assessments. It supports dozens of well-known security tools and it is designed to improve productivity for pentesters and security researchers.

Features

• Curated list of commands

• Unified input options

• Unified output schema

• CLI and library usage

• Distributed options with Celery

• Complexity from simple tasks to complex workflows

• Customizable

Supported tools

secator integrates the following tools:

Feel free to request new tools to be added by opening an issue, but please check that the tool complies with our selection criterias before doing so. If it doesn't but you still want to integrate it into secator, you can plug it in (see the dev guide).

Installation

Installing secator

pipx install secator

pip install secator

wget -O - https://raw.githubusercontent.com/freelabz/secator/main/scripts/install.sh | sh

docker run -it --rm --net=host -v ~/.secator:/root/.secator freelabz/secator --help

alias secator="docker run -it --rm --net=host -v ~/.secator:/root/.secator freelabz/secator"

secator --help

git clone https://github.com/freelabz/secator
cd secator
docker-compose up -d
docker-compose exec secator secator --help

Note: If you chose the Bash, Docker or Docker Compose installation methods, you can skip the next sections and go straight to Usage.

Installing languages

secator uses external tools, so you might need to install languages used by those tools assuming they are not already installed on your system.

We provide utilities to install required languages if you don't manage them externally:

secator install langs go

secator install langs ruby

Installing tools

secator does not install any of the external tools it supports by default.

We provide utilities to install or update each supported tool which should work on all systems supporting apt:

secator install tools

secator install tools <TOOL_NAME>

secator install tools httpx

Please make sure you are using the latest available versions for each tool before you run secator or you might run into parsing / formatting issues.

Installing addons

secator comes installed with the minimum amount of dependencies.

There are several addons available for secator:

secator install addons worker

secator install addons google

secator install addons mongodb

secator install addons redis

secator install addons dev

secator install addons trace

secator install addons build

Install CVEs

secator makes remote API calls to https://cve.circl.lu/ to get in-depth information about the CVEs it encounters. We provide a subcommand to download all known CVEs locally so that future lookups are made from disk instead:

secator install cves

Checking installation health

To figure out which languages or tools are installed on your system (along with their version):

secator health

Usage

secator --help

Usage examples

Run a fuzzing task (ffuf):

secator x ffuf http://testphp.vulnweb.com/FUZZ

Run a url crawl workflow:

secator w url_crawl http://testphp.vulnweb.com

Run a host scan:

secator s host mydomain.com

and more... to list all tasks / workflows / scans that you can use:

secator x --help
secator w --help
secator s --help

Learn more

To go deeper with secator, check out: * Our complete documentation * Our getting started tutorial video * Our Medium post * Follow us on social media: @freelabz on Twitter and @FreeLabz on YouTube

Damn-Vulnerable-Drone - An Intentionally Vulnerable Drone Hacking Simulator Based On The Popular ArduPilot/MAVLink Architecture, Providing A Realistic Environment For Hands-On Drone Hacking

The Damn Vulnerable Drone is an intentionally vulnerable drone hacking simulator based on the popular ArduPilot/MAVLink architecture, providing a realistic environment for hands-on drone hacking.

About the Damn Vulnerable Drone

What is the Damn Vulnerable Drone?

The Damn Vulnerable Drone is a virtually simulated environment designed for offensive security professionals to safely learn and practice drone hacking techniques. It simulates real-world ArduPilot & MAVLink drone architectures and vulnerabilities, offering a hands-on experience in exploiting drone systems.

Why was it built?

The Damn Vulnerable Drone aims to enhance offensive security skills within a controlled environment, making it an invaluable tool for intermediate-level security professionals, pentesters, and hacking enthusiasts.

Similar to how pilots utilize flight simulators for training, we can use the Damn Vulnerable Drone simulator to gain in-depth knowledge of real-world drone systems, understand their vulnerabilities, and learn effective methods to exploit them.

The Damn Vulnerable Drone platform is open-source and available at no cost and was specifically designed to address the substantial expenses often linked with drone hardware, hacking tools, and maintenance. Its cost-free nature allows users to immerse themselves in drone hacking without financial concerns. This accessibility makes the Damn Vulnerable Drone a crucial resource for those in the fields of information security and penetration testing, promoting the development of offensive cybersecurity skills in a safe environment.

How does it work?

The Damn Vulnerable Drone platform operates on the principle of Software-in-the-Loop (SITL), a simulation technique that allows users to run drone software as if it were executing on an actual drone, thereby replicating authentic drone behaviors and responses.

ArduPilot's SITL allows for the execution of the drone's firmware within a virtual environment, mimicking the behavior of a real drone without the need for physical hardware. This simulation is further enhanced with Gazebo, a dynamic 3D robotics simulator, which provides a realistic environment and physics engine for the drone to interact with. Together, ArduPilot's SITL and Gazebo lay the foundation for a sophisticated and authentic drone simulation experience.

While the current Damn Vulnerable Drone setup doesn't mirror every drone architecture or configuration, the integrated tactics, techniques and scenarios are broadly applicable across various drone systems, models and communication protocols.

Features

• Docker-based Environment: Runs in a completely virtualized docker-based setup, making it accessible and safe for drone hacking experimentation.
• Simulated Wireless Networking: Simulated Wifi (802.11) interfaces to practice wireless drone attacks.
• Onboard Camera Streaming & Gimbal: Simulated RTSP drone onboard camera stream with gimbal and companion computer integration.
• Companion Computer Web Interface: Companion Computer configuration management via web interface and simulated serial connection to Flight Controller.
• QGroundControl/MAVProxy Integration: One-click QGroundControl UI launching (only supported on x86 architecture) with MAVProxy GCS integration.
• MAVLink Router Integration: Telemetry forwarding via MAVLink Router on the Companion Computer Web Interface.
• Dynamic Flight Logging: Fully dynamic Ardupilot flight bin logs stored on a simulated SD Card.
• Management Web Console: Simple to use simulator management web console used to trigger scenarios and drone flight states.
• Comprehensive Hacking Scenarios: Ideal for practicing a wide range of drone hacking techniques, from basic reconnaissance to advanced exploitation.
• Detailed Walkthroughs: If you need help hacking against a particular scenario you can leverage the detailed walkthrough documentation as a spoiler.

SherlockChain - A Streamlined AI Analysis Framework For Solidity, Vyper And Plutus Contracts

SherlockChain is a powerful smart contract analysis framework that combines the capabilities of the renowned Slither tool with advanced AI-powered features. Developed by a team of security experts and AI researchers, SherlockChain offers unparalleled insights and vulnerability detection for Solidity, Vyper and Plutus smart contracts.

Key Features

• Comprehensive Vulnerability Detection: SherlockChain's suite of detectors identifies a wide range of vulnerabilities, including high-impact issues like reentrancy, unprotected upgrades, and more.
• AI-Powered Analysis: Integrated AI models enhance the accuracy and precision of vulnerability detection, providing developers with actionable insights and recommendations.
• Seamless Integration: SherlockChain seamlessly integrates with popular development frameworks like Hardhat, Foundry, and Brownie, making it easy to incorporate into your existing workflow.
• Intuitive Reporting: SherlockChain generates detailed reports with clear explanations and code snippets, helping developers quickly understand and address identified issues.
• Customizable Analyses: The framework's flexible API allows users to write custom analyses and detectors, tailoring the tool to their specific needs.
• Continuous Monitoring: SherlockChain can be integrated into your CI/CD pipeline, providing ongoing monitoring and alerting for your smart contract codebase.

Installation

To install SherlockChain, follow these steps:

git clone https://github.com/0xQuantumCoder/SherlockChain.git
cd SherlockChain
pip install .

AI-Powered Features

SherlockChain's AI integration brings several advanced capabilities to the table:

1. Intelligent Vulnerability Prioritization: AI models analyze the context and potential impact of detected vulnerabilities, providing developers with a prioritized list of issues to address.
2. Automated Remediation Suggestions: The AI component suggests potential fixes and code modifications to address identified vulnerabilities, accelerating the remediation process.
3. Proactive Security Auditing: SherlockChain's AI models continuously monitor your codebase, proactively identifying emerging threats and providing early warning signals.

4. Natural Language Interaction: Users can interact with SherlockChain using natural language, allowing them to query the tool, request specific analyses, and receive detailed responses. he --help command in the SherlockChain framework provides a comprehensive overview of all the available options and features. It includes information on:

5. Vulnerability Detection: The --detect and --exclude-detectors options allow users to specify which vulnerability detectors to run, including both built-in and AI-powered detectors.

6. Reporting: The --report-format, --report-output, and various --report-* options control how the analysis results are reported, including the ability to generate reports in different formats (JSON, Markdown, SARIF, etc.).
7. Filtering: The --filter-* options enable users to filter the reported issues based on severity, impact, confidence, and other criteria.
8. AI Integration: The --ai-* options allow users to configure and control the AI-powered features of SherlockChain, such as prioritizing high-impact vulnerabilities, enabling specific AI detectors, and managing AI model configurations.
9. Integration with Development Frameworks: Options like --truffle and --truffle-build-directory facilitate the integration of SherlockChain into popular development frameworks like Truffle.
10. Miscellaneous Options: Additional options for compiling contracts, listing detectors, and customizing the analysis process.

The --help command provides a detailed explanation of each option, its purpose, and how to use it, making it a valuable resource for users to quickly understand and leverage the full capabilities of the SherlockChain framework.

Example usage:

sherlockchain --help

This will display the comprehensive usage guide for the SherlockChain framework, including all available options and their descriptions.

usage: sherlockchain [-h] [--version] [--solc-remaps SOLC_REMAPS] [--solc-settings SOLC_SETTINGS]
                    [--solc-version SOLC_VERSION] [--truffle] [--truffle-build-directory TRUFFLE_BUILD_DIRECTORY]
                    [--truffle-config-file TRUFFLE_CONFIG_FILE] [--compile] [--list-detectors]
                    [--list-detectors-info] [--detect DETECTORS] [--exclude-detectors EXCLUDE_DETECTORS]
                    [--print-issues] [--json] [--markdown] [--sarif] [--text] [--zip] [--output OUTPUT]
                    [--filter-paths FILTER_PATHS] [--filter-paths-exclude FILTER_PATHS_EXCLUDE]
                    [--filter-contracts FILTER_CONTRACTS] [--filter-contracts-exclude FILTER_CONTRACTS_EXCLUDE]
                    [--filter-severity FILTER_SEVERITY] [--filter-impact FILTER_IMPACT]
                    [--filter-confidence FILTER_CONFIDENCE] [--filter-check-suicidal]
                    [--filter-check-upgradeable] [--f   ilter-check-erc20] [--filter-check-erc721]
                    [--filter-check-reentrancy] [--filter-check-gas-optimization] [--filter-check-code-quality]
                    [--filter-check-best-practices] [--filter-check-ai-detectors] [--filter-check-all]
                    [--filter-check-none] [--check-all] [--check-suicidal] [--check-upgradeable]
                    [--check-erc20] [--check-erc721] [--check-reentrancy] [--check-gas-optimization]
                    [--check-code-quality] [--check-best-practices] [--check-ai-detectors] [--check-none]
                    [--check-all-detectors] [--check-all-severity] [--check-all-impact] [--check-all-confidence]
                    [--check-all-categories] [--check-all-filters] [--check-all-options] [--check-all]
                    [--check-none] [--report-format {json,markdown,sarif,text,zip}] [--report-output OUTPUT]
                    [--report-severity REPORT_SEVERITY] [--report-impact R   EPORT_IMPACT]
                    [--report-confidence REPORT_CONFIDENCE] [--report-check-suicidal]
                    [--report-check-upgradeable] [--report-check-erc20] [--report-check-erc721]
                    [--report-check-reentrancy] [--report-check-gas-optimization] [--report-check-code-quality]
                    [--report-check-best-practices] [--report-check-ai-detectors] [--report-check-all]
                    [--report-check-none] [--report-all] [--report-suicidal] [--report-upgradeable]
                    [--report-erc20] [--report-erc721] [--report-reentrancy] [--report-gas-optimization]
                    [--report-code-quality] [--report-best-practices] [--report-ai-detectors] [--report-none]
                    [--report-all-detectors] [--report-all-severity] [--report-all-impact]
                    [--report-all-confidence] [--report-all-categories] [--report-all-filters]
                    [--report-all-options] [-   -report-all] [--report-none] [--ai-enabled] [--ai-disabled]
                    [--ai-priority-high] [--ai-priority-medium] [--ai-priority-low] [--ai-priority-all]
                    [--ai-priority-none] [--ai-confidence-high] [--ai-confidence-medium] [--ai-confidence-low]
                    [--ai-confidence-all] [--ai-confidence-none] [--ai-detectors-all] [--ai-detectors-none]
                    [--ai-detectors-specific AI_DETECTORS_SPECIFIC] [--ai-detectors-exclude AI_DETECTORS_EXCLUDE]
                    [--ai-models-path AI_MODELS_PATH] [--ai-models-update] [--ai-models-download]
                    [--ai-models-list] [--ai-models-info] [--ai-models-version] [--ai-models-check]
                    [--ai-models-upgrade] [--ai-models-remove] [--ai-models-clean] [--ai-models-reset]
                    [--ai-models-backup] [--ai-models-restore] [--ai-models-export] [--ai-models-import]
                    [--ai-models-config AI_MODELS_CONFIG]    [--ai-models-config-update] [--ai-models-config-reset]
                    [--ai-models-config-export] [--ai-models-config-import] [--ai-models-config-list]
                    [--ai-models-config-info] [--ai-models-config-version] [--ai-models-config-check]
                    [--ai-models-config-upgrade] [--ai-models-config-remove] [--ai-models-config-clean]
                    [--ai-models-config-reset] [--ai-models-config-backup] [--ai-models-config-restore]
                    [--ai-models-config-export] [--ai-models-config-import] [--ai-models-config-path AI_MODELS_CONFIG_PATH]
                    [--ai-models-config-file AI_MODELS_CONFIG_FILE] [--ai-models-config-url AI_MODELS_CONFIG_URL]
                    [--ai-models-config-name AI_MODELS_CONFIG_NAME] [--ai-models-config-description AI_MODELS_CONFIG_DESCRIPTION]
                    [--ai-models-config-version-major AI_MODELS_CONFIG_VERSION_MAJOR]
                    [--ai-models-config-   version-minor AI_MODELS_CONFIG_VERSION_MINOR]
                    [--ai-models-config-version-patch AI_MODELS_CONFIG_VERSION_PATCH]
                    [--ai-models-config-author AI_MODELS_CONFIG_AUTHOR]
                    [--ai-models-config-license AI_MODELS_CONFIG_LICENSE]
                    [--ai-models-config-url-documentation AI_MODELS_CONFIG_URL_DOCUMENTATION]
                    [--ai-models-config-url-source AI_MODELS_CONFIG_URL_SOURCE]
                    [--ai-models-config-url-issues AI_MODELS_CONFIG_URL_ISSUES]
                    [--ai-models-config-url-changelog AI_MODELS_CONFIG_URL_CHANGELOG]
                    [--ai-models-config-url-support AI_MODELS_CONFIG_URL_SUPPORT]
                    [--ai-models-config-url-website AI_MODELS_CONFIG_URL_WEBSITE]
                    [--ai-models-config-url-logo AI_MODELS_CONFIG_URL_LOGO]
                    [--ai-models-config-url-icon AI_MODELS_CONFIG_URL_ICON]
                       [--ai-models-config-url-banner AI_MODELS_CONFIG_URL_BANNER]
                    [--ai-models-config-url-screenshot AI_MODELS_CONFIG_URL_SCREENSHOT]
                    [--ai-models-config-url-video AI_MODELS_CONFIG_URL_VIDEO]
                    [--ai-models-config-url-demo AI_MODELS_CONFIG_URL_DEMO]
                    [--ai-models-config-url-documentation-api AI_MODELS_CONFIG_URL_DOCUMENTATION_API]
                    [--ai-models-config-url-documentation-user AI_MODELS_CONFIG_URL_DOCUMENTATION_USER]
                    [--ai-models-config-url-documentation-developer AI_MODELS_CONFIG_URL_DOCUMENTATION_DEVELOPER]
                    [--ai-models-config-url-documentation-faq AI_MODELS_CONFIG_URL_DOCUMENTATION_FAQ]
                    [--ai-models-config-url-documentation-tutorial AI_MODELS_CONFIG_URL_DOCUMENTATION_TUTORIAL]
                    [--ai-models-config-url-documentation-guide AI_MODELS_CONFIG_URL_DOCUMENTATION_GUIDE]
                       [--ai-models-config-url-documentation-whitepaper AI_MODELS_CONFIG_URL_DOCUMENTATION_WHITEPAPER]
                    [--ai-models-config-url-documentation-roadmap AI_MODELS_CONFIG_URL_DOCUMENTATION_ROADMAP]
                    [--ai-models-config-url-documentation-blog AI_MODELS_CONFIG_URL_DOCUMENTATION_BLOG]
                    [--ai-models-config-url-documentation-community AI_MODELS_CONFIG_URL_DOCUMENTATION_COMMUNITY]

This comprehensive usage guide provides information on all the available options and features of the SherlockChain framework, including:

• Vulnerability detection options: --detect, --exclude-detectors
• Reporting options: --report-format, --report-output, --report-*
• Filtering options: --filter-*
• AI integration options: --ai-*
• Integration with development frameworks: --truffle, --truffle-build-directory
• Miscellaneous options: --compile, --list-detectors, --list-detectors-info

By reviewing this comprehensive usage guide, you can quickly understand how to leverage the full capabilities of the SherlockChain framework to analyze your smart contracts and identify potential vulnerabilities. This will help you ensure the security and reliability of your DeFi protocol before deployment.

AI-Powered Detectors

Linux-Smart-Enumeration - Linux Enumeration Tool For Pentesting And CTFs With Verbosity Levels

wget "https://github.com/diego-treitos/linux-smart-enumeration/releases/latest/download/lse.sh" -O lse.sh;chmod 700 lse.sh

curl "https://github.com/diego-treitos/linux-smart-enumeration/releases/latest/download/lse.sh" -Lo lse.sh;chmod 700 lse.sh

Use: ./lse.sh [options]

 OPTIONS
  -c           Disable color
  -i           Non interactive mode
  -h           This help
  -l LEVEL     Output verbosity level
                 0: Show highly important results. (default)
                 1: Show interesting results.
                 2: Show all gathered information.
  -s SELECTION Comma separated list of sections or tests to run. Available
               sections:
                 usr: User related tests.
                 sud: Sudo related tests.
                 fst: File system related tests.
                 sys: System related tests.
                 sec: Security measures related tests.
                 ret: Recurren tasks (cron, timers) related tests.
                 net: Network related tests.
                 srv: Services related tests.
                 pro: Processes related tests.
                 sof: Software related tests.
                 ctn: Container (docker, lxc) related tests.
                 cve: CVE related tests.
               Specific tests can be used with their IDs (i.e.: usr020,sud)
  -e PATHS     Comma separated list of paths to exclude. This allows you
               to do faster scans at the cost of completeness
  -p SECONDS   Time that the process monitor will spend watching for
               processes. A value of 0 will disable any watch (default: 60)
  -S           Serve the lse.sh script in this host so it can be retrieved
               from a remote host.

bash <(wget -q -O - "https://github.com/diego-treitos/linux-smart-enumeration/releases/latest/download/lse.sh") -l2 -i

bash <(curl -s "https://github.com/diego-treitos/linux-smart-enumeration/releases/latest/download/lse.sh") -l1 -i

BypassFuzzer - Fuzz 401/403/404 Pages For Bypasses

The original 403fuzzer.py :)

Fuzz 401/403ing endpoints for bypasses

This tool performs various checks via headers, path normalization, verbs, etc. to attempt to bypass ACL's or URL validation.

It will output the response codes and length for each request, in a nicely organized, color coded way so things are reaable.

I implemented a "Smart Filter" that lets you mute responses that look the same after a certain number of times.

You can now feed it raw HTTP requests that you save to a file from Burp.

Follow me on twitter! @intrudir

Usage

usage: bypassfuzzer.py -h

Specifying a request to test

Best method: Feed it a raw HTTP request from Burp!

Simply paste the request into a file and run the script!
- It will parse and use cookies & headers from the request. - Easiest way to authenticate for your requests

python3 bypassfuzzer.py -r request.txt

Using other flags

Specify a URL

python3 bypassfuzzer.py -u http://example.com/test1/test2/test3/forbidden.html

Specify cookies to use in requests:
some examples:

--cookies "cookie1=blah"
-c "cookie1=blah; cookie2=blah"

Specify a method/verb and body data to send

bypassfuzzer.py -u https://example.com/forbidden -m POST -d "param1=blah&param2=blah2"
bypassfuzzer.py -u https://example.com/forbidden -m PUT -d "param1=blah&param2=blah2"

Specify custom headers to use with every request Maybe you need to add some kind of auth header like Authorization: bearer <token>

Specify -H "header: value" for each additional header you'd like to add:

bypassfuzzer.py -u https://example.com/forbidden -H "Some-Header: blah" -H "Authorization: Bearer 1234567"

Smart filter feature!

Based on response code and length. If it sees a response 8 times or more it will automatically mute it.

Repeats are changeable in the code until I add an option to specify it in flag

NOTE: Can't be used simultaneously with -hc or -hl (yet)

# toggle smart filter on
bypassfuzzer.py -u https://example.com/forbidden --smart

Specify a proxy to use

Useful if you wanna proxy through Burp

bypassfuzzer.py -u https://example.com/forbidden --proxy http://127.0.0.1:8080

Skip sending header payloads or url payloads

# skip sending headers payloads
bypassfuzzer.py -u https://example.com/forbidden -sh
bypassfuzzer.py -u https://example.com/forbidden --skip-headers

# Skip sending path normailization payloads
bypassfuzzer.py -u https://example.com/forbidden -su
bypassfuzzer.py -u https://example.com/forbidden --skip-urls

Hide response code/length

Provide comma delimited lists without spaces. Examples:

# Hide response codes
bypassfuzzer.py -u https://example.com/forbidden -hc 403,404,400  

# Hide response lengths of 638
bypassfuzzer.py -u https://example.com/forbidden -hl 638  

TODO

• [x] Automatically check other methods/verbs for bypass
• [x] absolute domain attack
• [ ] Add HTTP/2 support
• [ ] Looking for ideas. Ping me on twitter! @intrudir

PingRAT - Secretly Passes C2 Traffic Through Firewalls Using ICMP Payloads

Features:

• Uses ICMP for Command and Control
• Undetectable by most AV/EDR solutions
• Written in Go

Installation:

Download the binaries

or build the binaries and you are ready to go:

$ git clone https://github.com/Nemesis0U/PingRAT.git
$ go build client.go
$ go build server.go

Usage:

Server:

./server -h
Usage of ./server:
  -d string
        Destination IP address
  -i string
        Listener (virtual) Network Interface (e.g. eth0)

Client:

./client -h
Usage of ./client:
  -d string
        Destination IP address
  -i string
        (Virtual) Network Interface (e.g., eth0)

C2-Cloud - The C2 Cloud Is A Robust Web-Based C2 Framework, Designed To Simplify The Life Of Penetration Testers

The C2 Cloud is a robust web-based C2 framework, designed to simplify the life of penetration testers. It allows easy access to compromised backdoors, just like accessing an EC2 instance in the AWS cloud. It can manage several simultaneous backdoor sessions with a user-friendly interface.

C2 Cloud is open source. Security analysts can confidently perform simulations, gaining valuable experience and contributing to the proactive defense posture of their organizations.

Reverse shells support:

1. Reverse TCP
2. Reverse HTTP
3. Reverse HTTPS (configure it behind an LB)
4. Telegram C2

Demo

C2 Cloud walkthrough: https://youtu.be/hrHT_RDcGj8
Ransomware simulation using C2 Cloud: https://youtu.be/LKaCDmLAyvM
Telegram C2: https://youtu.be/WLQtF4hbCKk

Key Features

🔒 Anywhere Access: Reach the C2 Cloud from any location.
🔄 Multiple Backdoor Sessions: Manage and support multiple sessions effortlessly.
🖱️ One-Click Backdoor Access: Seamlessly navigate to backdoors with a simple click.
📜 Session History Maintenance: Track and retain complete command and response history for comprehensive analysis.

Tech Stack

🛠️ Flask: Serving web and API traffic, facilitating reverse HTTP(s) requests.
🔗 TCP Socket: Serving reverse TCP requests for enhanced functionality.
🌐 Nginx: Effortlessly routing traffic between web and backend systems.
📨 Redis PubSub: Serving as a robust message broker for seamless communication.
🚀 Websockets: Delivering real-time updates to browser clients for enhanced user experience.
💾 Postgres DB: Ensuring persistent storage for seamless continuity.

Architecture

Application setup

• Management port: 9000
  
• Reversse HTTP port: 8000
  

• Reverse TCP port: 8888
  

• Clone the repo

• Optional: Update chait_id, bot_token in c2-telegram/config.yml
• Execute docker-compose up -d to start the containers Note: The c2-api service will not start up until the database is initialized. If you receive 500 errors, please try after some time.

Credits

Inspired by Villain, a CLI-based C2 developed by Panagiotis Chartas.

License

Distributed under the MIT License. See LICENSE for more information.

Contact

• LinkedIn
• Twitter

HackerInfo - Infromations Web Application Security

Infromations Web Application Security

install :

sudo apt install python3 python3-pip

pip3 install termcolor

pip3 install google

pip3 install optioncomplete

pip3 install bs4


pip3 install prettytable

git clone https://github.com/Matrix07ksa/HackerInfo/

cd HackerInfo

chmod +x HackerInfo

./HackerInfo -h

python3 HackerInfo.py -d www.facebook.com -f pdf 
[+] <-- Running Domain_filter_File ....-->
[+] <-- Searching  [www.facebook.com] Files [pdf] ....-->
https://www.facebook.com/gms_hub/share/dcvsda_wf.pdf
https://www.facebook.com/gms_hub/share/facebook_groups_for_pages.pdf
https://www.facebook.com/gms_hub/share/videorequirementschart.pdf
https://www.facebook.com/gms_hub/share/fundraise-on-facebook_hi_in.pdf
https://www.facebook.com/gms_hub/share/bidding-strategy_decision-tree_en_us.pdf
https://www.facebook.com/gms_hub/share/fundraise-on-facebook_es_la.pdf
https://www.facebook.com/gms_hub/share/fundraise-on-facebook_ar.pdf
https://www.facebook.com/gms_hub/share/fundraise-on-facebook_ur_pk.pdf
https://www.facebook.com/gms_hub/share/fundraise-on-facebook_cs_cz.pdf
https://www.facebook.com/gms_hub/share/fundraise-on-facebook_it_it.pdf
https://www.facebook.com/gms_hub/share/fundraise-on-facebook_pl_pl.pdf
h   ttps://www.facebook.com/gms_hub/share/fundraise-on-facebook_nl.pdf
https://www.facebook.com/gms_hub/share/fundraise-on-facebook_pt_br.pdf
https://www.facebook.com/gms_hub/share/creative-best-practices_id_id.pdf
https://www.facebook.com/gms_hub/share/creative-best-practices_fr_fr.pdf
https://www.facebook.com/gms_hub/share/fundraise-on-facebook_tr_tr.pdf
https://www.facebook.com/gms_hub/share/creative-best-practices_hi_in.pdf
https://www.facebook.com/rsrc.php/yA/r/AVye1Rrg376.pdf
https://www.facebook.com/gms_hub/share/creative-best-practices_ur_pk.pdf
https://www.facebook.com/gms_hub/share/creative-best-practices_nl_nl.pdf
https://www.facebook.com/gms_hub/share/creative-best-practices_de_de.pdf
https://www.facebook.com/gms_hub/share/fundraise-on-facebook_de_de.pdf
https://www.facebook.com/gms_hub/share/creative-best-practices_cs_cz.pdf
https://www.facebook.com/gms_hub/share/fundraise-on-facebook_sk_sk.pdf
https://www.facebook.com/gms   _hub/share/creative-best-practices_japanese_jp.pdf
#####################[Finshid]########################

Usage:

Library install hackinfo:

sudo python setup.py install
pip3 install hackinfo

Noia - Simple Mobile Applications Sandbox File Browser Tool

Noia is a web-based tool whose main aim is to ease the process of browsing mobile applications sandbox and directly previewing SQLite databases, images, and more. Powered by frida.re.

Please note that I'm not a programmer, but I'm probably above the median in code-savyness. Try it out, open an issue if you find any problems. PRs are welcome.

Installation & Usage

npm install -g noia
noia

Features

• Explore third-party applications files and directories. Noia shows you details including the access permissions, file type and much more.

• View custom binary files. Directly preview SQLite databases, images, and more.

• Search application by name.

• Search files and directories by name.

• Navigate to a custom directory using the ctrl+g shortcut.

• Download the application files and directories for further analysis.

• Basic iOS support

and more

Setup

Desktop requirements:

• node.js LTS and npm
• Any decent modern desktop browser

Noia is available on npm, so just type the following command to install it and run it:

npm install -g noia
noia

Device setup:

Noia is powered by frida.re, thus requires Frida to run.

Rooted Device

See: * https://frida.re/docs/android/ * https://frida.re/docs/ios/

Non-rooted Device

• https://koz.io/using-frida-on-android-without-root/
• https://github.com/sensepost/objection/wiki/Patching-Android-Applications
• https://nowsecure.com/blog/2020/01/02/how-to-conduct-jailed-testing-with-frida/

Security Warning

This tool is not secure and may include some security vulnerabilities so make sure to isolate the webpage from potential hackers.

LICENCE

MIT

WinFiHack - A Windows Wifi Brute Forcing Utility Which Is An Extremely Old Method But Still Works Without The Requirement Of External Dependencies

WinFiHack is a recreational attempt by me to rewrite my previous project Brute-Hacking-Framework's main wifi hacking script that uses netsh and native Windows scripts to create a wifi bruteforcer. This is in no way a fast script nor a superior way of doing the same hack but it needs no external libraries and just Python and python scripts.

The packages are minimal or nearly none 😅. The package install command is:

pip install rich pyfiglet

Thats it.

So listing the features:

• Overall Features:
• We can use custom interfaces or non-default interfaces to run the attack.
• Well-defined way of using netsh and listing and utilizing targets.
• Upgradeability
• Code-Wise Features:
• Interactive menu-driven system with rich.
• versatility in using interface, targets, and password files.

So this is how the bruteforcer works:

• Provide Interface:

• The user is required to provide the network interface for the tool to use.

• By default, the interface is set to Wi-Fi.

• Search and Set Target:

• The user must search for and select the target network.

• During this process, the tool performs the following sub-steps:

  • Disconnects all active network connections for the selected interface.
  • Searches for all available networks within range.

• Input Password File:

• The user inputs the path to the password file.

• The default path for the password file is ./wordlist/default.txt.

• Run the Attack:

• With the target set and the password file ready, the tool is now prepared to initiate the attack.

• Attack Procedure:

• The attack involves iterating through each password in the provided file.
• For each password, the following steps are taken:
  • A custom XML configuration for the connection attempt is generated and stored.
  • The tool attempts to connect to the target network using the generated XML and the current password.
  • To verify the success of the connection attempt, the tool performs a "1 packet ping" to Google.
  • If the ping is unsuccessful, the connection attempt is considered failed, and the tool proceeds to the next password in the list.
  • This loop continues until a successful ping response is received, indicating a successful connection attempt.

After installing all the packages just run python main.py rest is history 👍 make sure you run this on Windows cause this won't work on any other OS. The interface looks like this:

For contributions: - First Clone: First Clone the repo into your dev env and do the edits. - Comments: I would apprtiate if you could add comments explaining your POV and also explaining the upgrade. - Submit: Submit a PR for me to verify the changes and apprive it if necessary.

SpeedyTest - Command-Line Tool For Measuring Internet Speed

SpeedyTest is a powerful command-line tool for measuring internet speed. With its advanced features and intuitive interface, it provides accurate and comprehensive speed test results. Whether you're a network administrator, developer, or simply want to monitor your internet connection, SpeedyTest is the perfect tool for the job.

• Measure download speed, upload speed, and ping latency.
• Generate detailed reports with graphical representation of speed test results.
• Save and export test results in various formats (CSV, JSON, etc.).
• Customize speed test parameters and server selection.
• Compare speed test results over time to track performance changes.
• Integrate SpeedyTest into your own applications using the provided API.
• track your timeline with saved database

git clone https://github.com/HalilDeniz/SpeedyTest.git

Before you can use SpeedyTest, you need to make sure that you have the necessary requirements installed. You can install these requirements by running the following command:

pip install -r requirements.txt

Run the following command to perform a speed test:

python3 speendytest.py

Receiving data \
Speed test completed!
Speed test time: 20.22 second
Server    : Farknet - Konya
IP Address: speedtest.farknet.com.tr:8080
Country   : Turkey
City      : Konya
Ping      : 20.41 ms
Download  : 90.12 Mbps
Loading   : 20 Mbps

Contributions are welcome! To contribute to SpeedyTest, follow these steps:

1. Fork the repository.
2. Create a new branch for your feature or bug fix.
3. Make your changes and commit them.
4. Push your changes to your forked repository.
5. Open a pull request in the main repository.

If you have any questions, comments, or suggestions about PrivacyNet, please feel free to contact me:

• LinkedIn......: LinkedIn
• TryHackMe.: TryHackMe
• Instagram...: Instagram
• YouTube.....: YouTube
• Email: halildeniz313@gmail.com

SpeedyTest is released under the MIT License. See LICENSE for details.

Pmkidcracker - A Tool To Crack WPA2 Passphrase With PMKID Value Without Clients Or De-Authentication

This program is a tool written in Python to recover the pre-shared key of a WPA2 WiFi network without any de-authentication or requiring any clients to be on the network. It targets the weakness of certain access points advertising the PMKID value in EAPOL message 1.

Program Usage

python pmkidcracker.py -s <SSID> -ap <APMAC> -c <CLIENTMAC> -p <PMKID> -w <WORDLIST> -t <THREADS(Optional)>

NOTE: apmac, clientmac, pmkid must be a hexstring, e.g b8621f50edd9

How PMKID is Calculated

The two main formulas to obtain a PMKID are as follows:

1. Pairwise Master Key (PMK) Calculation: passphrase + salt(ssid) => PBKDF2(HMAC-SHA1) of 4096 iterations
2. PMKID Calculation: HMAC-SHA1[pmk + ("PMK Name" + bssid + clientmac)]

This is just for understanding, both are already implemented in find_pw_chunk and calculate_pmkid.

Obtaining the PMKID

Below are the steps to obtain the PMKID manually by inspecting the packets in WireShark.

*You may use Hcxtools or Bettercap to quickly obtain the PMKID without the below steps. The manual way is for understanding.

To obtain the PMKID manually from wireshark, put your wireless antenna in monitor mode, start capturing all packets with airodump-ng or similar tools. Then connect to the AP using an invalid password to capture the EAPOL 1 handshake message. Follow the next 3 steps to obtain the fields needed for the arguments.

Open the pcap in WireShark:

• Filter with wlan_rsna_eapol.keydes.msgnr == 1 in WireShark to display only EAPOL message 1 packets.
• In EAPOL 1 pkt, Expand IEEE 802.11 QoS Data Field to obtain AP MAC, Client MAC
• In EAPOL 1 pkt, Expand 802.1 Authentication > WPA Key Data > Tag: Vendor Specific > PMKID is below

If access point is vulnerable, you should see the PMKID value like the below screenshot:

Demo Run

Disclaimer

This tool is for educational and testing purposes only. Do not use it to exploit the vulnerability on any network that you do not own or have permission to test. The authors of this script are not responsible for any misuse or damage caused by its use.

WiFi-password-stealer - Simple Windows And Linux Keystroke Injection Tool That Exfiltrates Stored WiFi Data (SSID And Password)

Have you ever watched a film where a hacker would plug-in, seemingly ordinary, USB drive into a victim's computer and steal data from it? - A proper wet dream for some.

Disclaimer: All content in this project is intended for security research purpose only.

Introduction

During the summer of 2022, I decided to do exactly that, to build a device that will allow me to steal data from a victim's computer. So, how does one deploy malware and exfiltrate data? In the following text I will explain all of the necessary steps, theory and nuances when it comes to building your own keystroke injection tool. While this project/tutorial focuses on WiFi passwords, payload code could easily be altered to do something more nefarious. You are only limited by your imagination (and your technical skills).

Setup

After creating pico-ducky, you only need to copy the modified payload (adjusted for your SMTP details for Windows exploit and/or adjusted for the Linux password and a USB drive name) to the RPi Pico.

Prerequisites

• Physical access to victim's computer.

• Unlocked victim's computer.

• Victim's computer has to have an internet access in order to send the stolen data using SMTP for the exfiltration over a network medium.

• Knowledge of victim's computer password for the Linux exploit.

Requirements - What you'll need

• Raspberry Pi Pico (RPi Pico)
• Micro USB to USB Cable
• Jumper Wire (optional)
• pico-ducky - Transformed RPi Pico into a USB Rubber Ducky
• USB flash drive (for the exploit over physical medium only)

Note:

• It is possible to build this tool using Rubber Ducky, but keep in mind that RPi Pico costs about $4.00 and the Rubber Ducky costs $80.00.

• However, while pico-ducky is a good and budget-friedly solution, Rubber Ducky does offer things like stealthiness and usage of the lastest DuckyScript version.

• In order to use Ducky Script to write the payload on your RPi Pico you first need to convert it to a pico-ducky. Follow these simple steps in order to create pico-ducky.

Keystroke injection tool

Keystroke injection tool, once connected to a host machine, executes malicious commands by running code that mimics keystrokes entered by a user. While it looks like a USB drive, it acts like a keyboard that types in a preprogrammed payload. Tools like Rubber Ducky can type over 1,000 words per minute. Once created, anyone with physical access can deploy this payload with ease.

Keystroke injection

The payload uses STRING command processes keystroke for injection. It accepts one or more alphanumeric/punctuation characters and will type the remainder of the line exactly as-is into the target machine. The ENTER/SPACE will simulate a press of keyboard keys.

Delays

We use DELAY command to temporarily pause execution of the payload. This is useful when a payload needs to wait for an element such as a Command Line to load. Delay is useful when used at the very beginning when a new USB device is connected to a targeted computer. Initially, the computer must complete a set of actions before it can begin accepting input commands. In the case of HIDs setup time is very short. In most cases, it takes a fraction of a second, because the drivers are built-in. However, in some instances, a slower PC may take longer to recognize the pico-ducky. The general advice is to adjust the delay time according to your target.

Exfiltration

Data exfiltration is an unauthorized transfer of data from a computer/device. Once the data is collected, adversary can package it to avoid detection while sending data over the network, using encryption or compression. Two most common way of exfiltration are:

• Exfiltration over the network medium.

• This approach was used for the Windows exploit. The whole payload can be seen here.

• Exfiltration over a physical medium.

• This approach was used for the Linux exploit. The whole payload can be seen here.

Windows exploit

In order to use the Windows payload (payload1.dd), you don't need to connect any jumper wire between pins.

Sending stolen data over email

Once passwords have been exported to the .txt file, payload will send the data to the appointed email using Yahoo SMTP. For more detailed instructions visit a following link. Also, the payload template needs to be updated with your SMTP information, meaning that you need to update RECEIVER_EMAIL, SENDER_EMAIL and yours email PASSWORD. In addition, you could also update the body and the subject of the email.

 Note:

• After sending data over the email, the .txt file is deleted.

• You can also use some an SMTP from another email provider, but you should be mindful of SMTP server and port number you will write in the payload.

• Keep in mind that some networks could be blocking usage of an unknown SMTP at the firewall.

Linux exploit

In order to use the Linux payload (payload2.dd) you need to connect a jumper wire between GND and GPIO5 in order to comply with the code in code.py on your RPi Pico. For more information about how to setup multiple payloads on your RPi Pico visit this link.

Storing stolen data to USB flash drive

Once passwords have been exported from the computer, data will be saved to the appointed USB flash drive. In order for this payload to function properly, it needs to be updated with the correct name of your USB drive, meaning you will need to replace USBSTICK with the name of your USB drive in two places.

In addition, you will also need to update the Linux PASSWORD in the payload in three places. As stated above, in order for this exploit to be successful, you will need to know the victim's Linux machine password, which makes this attack less plausible.

Bash script

In order to run the wifi_passwords_print.sh script you will need to update the script with the correct name of your USB stick after which you can type in the following command in your terminal:

echo PASSWORD | sudo -S sh wifi_passwords_print.sh USBSTICK

where PASSWORD is your account's password and USBSTICK is the name for your USB device.

Quick overview of the payload

NetworkManager is based on the concept of connection profiles, and it uses plugins for reading/writing data. It uses .ini-style keyfile format and stores network configuration profiles. The keyfile is a plugin that supports all the connection types and capabilities that NetworkManager has. The files are located in /etc/NetworkManager/system-connections/. Based on the keyfile format, the payload uses the grep command with regex in order to extract data of interest. For file filtering, a modified positive lookbehind assertion was used ((?<=keyword)). While the positive lookbehind assertion will match at a certain position in the string, sc. at a position right after the keyword without making that text itself part of the match, the regex (?<=keyword).* will match any text after the keyword. This allows the payload to match the values after SSID and psk (pre-shared key) keywords.

For more information about NetworkManager here is some useful links:

• Manually creating NetworkManager profiles in keyfile format
• Description of keyfile settings plugin

Exfiltrated data formatting

Below is an example of the exfiltrated and formatted data from a victim's machine in a .txt file.

USB Mass Storage Device Problem

One of the advantages of Rubber Ducky over RPi Pico is that it doesn't show up as a USB mass storage device once plugged in. Once plugged into the computer, all the machine sees it as a USB keyboard. This isn't a default behavior for the RPi Pico. If you want to prevent your RPi Pico from showing up as a USB mass storage device when plugged in, you need to connect a jumper wire between pin 18 (GND) and pin 20 (GPIO15). For more details visit this link.

 Tip:

• Upload your payload to RPi Pico before you connect the pins.
• Don't solder the pins because you will probably want to change/update the payload at some point.

Payload Writer

When creating a functioning payload file, you can use the writer.py script, or you can manually change the template file. In order to run the script successfully you will need to pass, in addition to the script file name, a name of the OS (windows or linux) and the name of the payload file (e.q. payload1.dd). Below you can find an example how to run the writer script when creating a Windows payload.

python3 writer.py windows payload1.dd

Limitations/Drawbacks

• This pico-ducky currently works only on Windows OS.

• This attack requires physical access to an unlocked device in order to be successfully deployed.

• The Linux exploit is far less likely to be successful, because in order to succeed, you not only need physical access to an unlocked device, you also need to know the admins password for the Linux machine.

• Machine's firewall or network's firewall may prevent stolen data from being sent over the network medium.

• Payload delays could be inadequate due to varying speeds of different computers used to deploy an attack.

• The pico-ducky device isn't really stealthy, actually it's quite the opposite, it's really bulky especially if you solder the pins.

• Also, the pico-ducky device is noticeably slower compared to the Rubber Ducky running the same script.

• If the Caps Lock is ON, some of the payload code will not be executed and the exploit will fail.

• If the computer has a non-English Environment set, this exploit won't be successful.

• Currently, pico-ducky doesn't support DuckyScript 3.0, only DuckyScript 1.0 can be used. If you need the 3.0 version you will have to use the Rubber Ducky.

To-Do List

• Fix Caps Lock bug.
• Fix non-English Environment bug.
• Obfuscate the command prompt.
• Implement exfiltration over a physical medium.
• Create a payload for Linux.
• Encode/Encrypt exfiltrated data before sending it over email.
• Implement indicator of successfully completed exploit.
• Implement command history clean-up for Linux exploit.
• Enhance the Linux exploit in order to avoid usage of sudo.

Top 20 Most Popular Hacking Tools in 2023

1. PhoneSploit-Pro - An All-In-One Hacking Tool To Remotely Exploit Android Devices Using ADB And Metasploit-Framework To Get A Meterpreter Session





2. Gmailc2 - A Fully Undetectable C2 Server That Communicates Via Google SMTP To Evade Antivirus Protections And Network Traffic Restrictions





3. Faraday - Open Source Vulnerability Management Platform





4. CloakQuest3r - Uncover The True IP Address Of Websites Safeguarded By Cloudflare





5. Killer - Is A Tool Created To Evade AVs And EDRs Or Security Tools





6. Geowifi - Search WiFi Geolocation Data By BSSID And SSID On Different Public Databases





7. Waf-Bypass - Check Your WAF Before An Attacker Does





8. PentestGPT - A GPT-empowered Penetration Testing Tool





9. Sirius - First Truly Open-Source General Purpose Vulnerability Scanner





10. LSMS - Linux Security And Monitoring Scripts





11. GodPotato - Local Privilege Escalation Tool From A Windows Service Accounts To NT AUTHORITY\SYSTEM





12. Bypass-403 - A Simple Script Just Made For Self Use For Bypassing 403





13. ThunderCloud - Cloud Exploit Framework





14. GPT_Vuln-analyzer - Uses ChatGPT API And Python-Nmap Module To Use The GPT3 Model To Create Vulnerability Reports Based On Nmap Scan Data





15. Kscan - Simple Asset Mapping Tool





16. RedTeam-Physical-Tools - Red Team Toolkit - A Curated List Of Tools That Are Commonly Used In The Field For Physical Security, Red Teaming, And Tactical Covert Entry





17. DNSWatch - DNS Traffic Sniffer and Analyzer





18. IpGeo - Tool To Extract IP Addresses From Captured Network Traffic File





19. TelegramRAT - Cross Platform Telegram Based RAT That Communicates Via Telegram To Evade Network Restrictions





20. XSS-Exploitation-Tool - An XSS Exploitation Tool

NetProbe - Network Probe

NetProbe is a tool you can use to scan for devices on your network. The program sends ARP requests to any IP address on your network and lists the IP addresses, MAC addresses, manufacturers, and device models of the responding devices.

Features

• Scan for devices on a specified IP address or subnet
• Display the IP address, MAC address, manufacturer, and device model of discovered devices
• Live tracking of devices (optional)
• Save scan results to a file (optional)
• Filter by manufacturer (e.g., 'Apple') (optional)
• Filter by IP range (e.g., '192.168.1.0/24') (optional)
• Scan rate in seconds (default: 5) (optional)

Download

You can download the program from the GitHub page.

$ git clone https://github.com/HalilDeniz/NetProbe.git

Installation

To install the required libraries, run the following command:

$ pip install -r requirements.txt

Usage

To run the program, use the following command:

$ python3 netprobe.py [-h] -t  [...] -i  [...] [-l] [-o] [-m] [-r] [-s]

• -h,--help: show this help message and exit
• -t,--target: Target IP address or subnet (default: 192.168.1.0/24)
• -i,--interface: Interface to use (default: None)
• -l,--live: Enable live tracking of devices
• -o,--output: Output file to save the results
• -m,--manufacturer: Filter by manufacturer (e.g., 'Apple')
• -r,--ip-range: Filter by IP range (e.g., '192.168.1.0/24')
• -s,--scan-rate: Scan rate in seconds (default: 5)

Example:

$ python3 netprobe.py -t 192.168.1.0/24 -i eth0 -o results.txt -l

Help Menu

$ python3 netprobe.py --help                      
usage: netprobe.py [-h] -t  [...] -i  [...] [-l] [-o] [-m] [-r] [-s]

NetProbe: Network Scanner Tool

options:
  -h, --help            show this help message and exit
  -t  [ ...], --target  [ ...]
                        Target IP address or subnet (default: 192.168.1.0/24)
  -i  [ ...], --interface  [ ...]
                        Interface to use (default: None)
  -l, --live            Enable live tracking of devices
  -o , --output         Output file to save the results
  -m , --manufacturer   Filter by manufacturer (e.g., 'Apple')
  -r , --ip-range       Filter by IP range (e.g., '192.168.1.0/24')
  -s , --scan-rate      Scan rate in seconds (default: 5)

Default Scan

$ python3 netprobe.py 

Live Tracking

You can enable live tracking of devices on your network by using the -l or --live flag. This will continuously update the device list every 5 seconds.

$ python3 netprobe.py -t 192.168.1.0/24 -i eth0 -l

Save Results

You can save the scan results to a file by using the -o or --output flag followed by the desired output file name.

$ python3 netprobe.py -t 192.168.1.0/24 -i eth0 -l -o results.txt

┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ IP Address   ┃ MAC Address       ┃ Packet Size ┃ Manufacturer                 ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 192.168.1.1  │ **:6e:**:97:**:28 │ 102         │ ASUSTek COMPUTER INC.        │
│ 192.168.1.3  │ 00:**:22:**:12:** │ 102         │ InPro Comm                   │
│ 192.168.1.2  │ **:32:**:bf:**:00 │ 102         │ Xiaomi Communications Co Ltd │
│ 192.168.1.98 │ d4:**:64:**:5c:** │ 102         │ ASUSTek COMPUTER INC.        │
│ 192.168.1.25 │ **:49:**:00:**:38 │ 102         │ Unknown                      │
└──────────────┴───────────────────┴─────────────┴──────────────────────────────┘

Contact

If you have any questions, suggestions, or feedback about the program, please feel free to reach out to me through any of the following platforms:

• Mywebsite: Denizhalil
• LinkedIn: LinkedIn
• TryHackMe: TryHackMe
• Instagram: Instagram
• YouTube: YouTube
• Email: halildeniz313@gmail.com

License

This program is released under the MIT LICENSE. See LICENSE for more information.

PassBreaker - Command-line Password Cracking Tool Developed In Python

PassBreaker is a command-line password cracking tool developed in Python. It allows you to perform various password cracking techniques such as wordlist-based attacks and brute force attacks. 

Features

• Wordlist-based password cracking
• Brute force password cracking
• Support for multiple hash algorithms
• Optional salt value
• Parallel processing option for faster cracking
• Password complexity evaluation
• Customizable minimum and maximum password length
• Customizable character set for brute force attacks

Installation

1. Clone the repository:

   git clone https://github.com/HalilDeniz/PassBreaker.git

2. Install the required dependencies:

   pip install -r requirements.txt

Usage

python passbreaker.py <password_hash> <wordlist_file> [--algorithm]

Replace <password_hash> with the target password hash and <wordlist_file> with the path to the wordlist file containing potential passwords.

Options

• --algorithm <algorithm>: Specify the hash algorithm to use (e.g., md5, sha256, sha512).
• -s, --salt <salt>: Specify a salt value to use.
• -p, --parallel: Enable parallel processing for faster cracking.
• -c, --complexity: Evaluate password complexity before cracking.
• -b, --brute-force: Perform a brute force attack.
• --min-length <min_length>: Set the minimum password length for brute force attacks.
• --max-length <max_length>: Set the maximum password length for brute force attacks.
• --character-set <character_set>: Set the character set to use for brute force attacks.

Elbette! İşte İngilizce olarak yazılmış başlık ve küçük bir bilgi ile daha fazla kullanım örneği:

Usage Examples

Wordlist-based Password Cracking

python passbreaker.py 5f4dcc3b5aa765d61d8327deb882cf99 passwords.txt --algorithm md5

This command attempts to crack the password with the hash value "5f4dcc3b5aa765d61d8327deb882cf99" using the MD5 algorithm and a wordlist from the "passwords.txt" file.

Brute Force Attack

python passbreaker.py 5f4dcc3b5aa765d61d8327deb882cf99 --brute-force --min-length 6 --max-length 8 --character-set abc123

This command performs a brute force attack to crack the password with the hash value "5f4dcc3b5aa765d61d8327deb882cf99" by trying all possible combinations of passwords with a length between 6 and 8 characters, using the character set "abc123".

Password Complexity Evaluation

python passbreaker.py 5f4dcc3b5aa765d61d8327deb882cf99 passwords.txt --algorithm sha256 --complexity

This command evaluates the complexity of passwords in the "passwords.txt" file and attempts to crack the password with the hash value "5f4dcc3b5aa765d61d8327deb882cf99" using the SHA-256 algorithm. It only tries passwords that meet the complexity requirements.

Using Salt Value

python passbreaker.py 5f4dcc3b5aa765d61d8327deb882cf99 passwords.txt --algorithm md5 --salt mysalt123

This command uses a specific salt value ("mysalt123") for the password cracking process. Salt is used to enhance the security of passwords.

Parallel Processing

python passbreaker.py 5f4dcc3b5aa765d61d8327deb882cf99 passwords.txt --algorithm sha512 --parallel

This command performs password cracking with parallel processing for faster cracking. It utilizes multiple processing cores, but it may consume more system resources.

These examples demonstrate different features and use cases of the "PassBreaker" password cracking tool. Users can customize the parameters based on their needs and goals.

Disclaimer

This tool is intended for educational and ethical purposes only. Misuse of this tool for any malicious activities is strictly prohibited. The developers assume no liability and are not responsible for any misuse or damage caused by this tool.

Contributing

Contributions are welcome! To contribute to PassBreaker, follow these steps:

1. Fork the repository.
2. Create a new branch for your feature or bug fix.
3. Make your changes and commit them.
4. Push your changes to your forked repository.
5. Open a pull request in the main repository.

Contact

If you have any questions, comments, or suggestions about PassBreaker, please feel free to contact me:

• LinkedIn: LinkedIn
• TryHackMe: TryHackMe
• Instagram: Instagram
• YouTube: YouTube
• Email: halildeniz313@gmail.com

License

PassBreaker is released under the MIT License. See LICENSE for more information.

TrafficWatch - TrafficWatch, A Packet Sniffer Tool, Allows You To Monitor And Analyze Network Traffic From PCAP Files

TrafficWatch, a packet sniffer tool, allows you to monitor and analyze network traffic from PCAP files. It provides insights into various network protocols and can help with network troubleshooting, security analysis, and more.

• Protocol-specific packet analysis for ARP, ICMP, TCP, UDP, DNS, DHCP, HTTP, SNMP, LLMNR, and NetBIOS.
• Packet filtering based on protocol, source IP, destination IP, source port, destination port, and more.
• Summary statistics on captured packets.
• Interactive mode for in-depth packet inspection.
• Timestamps for each captured packet.
• User-friendly colored output for improved readability.
• Python 3.x
• scapy
• argparse
• pyshark
• colorama

1. Clone the repository:

   git clone https://github.com/HalilDeniz/TrafficWatch.git

2. Navigate to the project directory:

   cd TrafficWatch

3. Install the required dependencies:

   pip install -r requirements.txt

python3 trafficwatch.py --help
usage: trafficwatch.py [-h] -f FILE [-p {ARP,ICMP,TCP,UDP,DNS,DHCP,HTTP,SNMP,LLMNR,NetBIOS}] [-c COUNT]

Packet Sniffer Tool

options:
-h, --help            show this help message and exit
-f FILE, --file FILE  Path to the .pcap file to analyze
-p {ARP,ICMP,TCP,UDP,DNS,DHCP,HTTP,SNMP,LLMNR,NetBIOS}, --protocol {ARP,ICMP,TCP,UDP,DNS,DHCP,HTTP,SNMP,LLMNR,NetBIOS}
                     Filter by specific protocol
-c COUNT, --count COUNT
                     Number of packets to display

To analyze packets from a PCAP file, use the following command:

python trafficwatch.py -f path/to/your.pcap

To specify a protocol filter (e.g., HTTP) and limit the number of displayed packets (e.g., 10), use:

python trafficwatch.py -f path/to/your.pcap -p HTTP -c 10

• -f or --file: Path to the PCAP file for analysis.
• -p or --protocol: Filter packets by protocol (ARP, ICMP, TCP, UDP, DNS, DHCP, HTTP, SNMP, LLMNR, NetBIOS).
• -c or --count: Limit the number of displayed packets.

Contributions are welcome! If you want to contribute to TrafficWatch, please follow our contribution guidelines.

If you have any questions, comments, or suggestions about Dosinator, please feel free to contact me:

• LinkedIn: Halil Ibrahim Deniz
• TryHackMe: Halilovic
• Instagram: deniz.halil333
• YouTube: Halil Deniz
• Email: halildeniz313@gmail.com

This project is licensed under the MIT License.

Thank you for considering supporting me! Your support enables me to dedicate more time and effort to creating useful tools like DNSWatch and developing new projects. By contributing, you're not only helping me improve existing tools but also inspiring new ideas and innovations. Your support plays a vital role in the growth of this project and future endeavors. Together, let's continue building and learning. Thank you!" 

RecycledInjector - Native Syscalls Shellcode Injector

(Currently) Fully Undetected same-process native/.NET assembly shellcode injector based on RecycledGate by thefLink, which is also based on HellsGate + HalosGate + TartarusGate to ensure undetectable native syscalls even if one technique fails.

To remain stealthy and keep entropy on the final executable low, do ensure that shellcode is always loaded externally since most AV/EDRs won't check for signatures on non-executable or DLL files anyway.

Important to also note that the fully undetected part refers to the loading of the shellcode, however, the shellcode will still be subject to behavior monotoring, thus make sure the loaded executable also makes use of defense evasion techniques (e.g., SharpKatz which features DInvoke instead of Mimikatz).

Usage

.\RecycledInjector.exe <path_to_shellcode_file>

Proof of Concept

This proof of concept leverages Terminator by ZeroMemoryEx to kill most security solution/agents present on the system. It is used against Microsoft Defender for Endpoint EDR.

On the left we inject the Terminator shellcode to load the vulnerable driver and kill MDE processes, and on the right is an example of loading and executing Invoke-Mimikatz remotely from memory, which is not stopped as there is no running security solution anymore on the system.

KaliPackergeManager - Kali Packerge Manager

kalipm.sh is a powerful package management tool for Kali Linux that provides a user-friendly menu-based interface to simplify the installation of various packages and tools. It streamlines the process of managing software and enables users to effortlessly install packages from different categories. 

Features

• Interactive Menu: Enjoy an intuitive and user-friendly menu-based interface for easy package selection.
• Categorized Packages: Browse packages across multiple categories, including System, Desktop, Tools, Menu, and Others.
• Efficient Installation: Automatically install selected packages with the help of the apt-get package manager.
• System Updates: Keep your system up to date with the integrated update functionality.

Installation

To install KaliPm, you can simply clone the repository from GitHub:

git clone https://github.com/HalilDeniz/KaliPackergeManager.git

Usage

1. Clone the repository or download the KaliPM.sh script.
2. Navigate to the directory where the script is located.
3. Make the script executable by running the following command:

   chmod +x kalipm.sh
   

4. Execute the script using the following command:

   ./kalipm.sh
   

5. Follow the on-screen instructions to select a category and choose the desired packages for installation.

Categories

• System: Includes essential core items that are always included in the Kali Linux system.
• Desktop: Offers various desktop environments and window managers to customize your Kali Linux experience.
• Tools: Provides a wide range of specialized tools for tasks such as hardware hacking, cryptography, wireless protocols, and more.
• Menu: Consists of packages tailored for information gathering, vulnerability assessments, web application attacks, and other specific purposes.
• Others: Contains additional packages and resources that don't fall into the above categories.

Update

KaliPM.sh also includes an update feature to ensure your system is up to date. Simply select the "Update" option from the menu, and the script will run the necessary commands to clean, update, upgrade, and perform a full-upgrade on your system.

Contributing

Contributions are welcome! To contribute to KaliPackergeManager, follow these steps:

1. Fork the repository.
2. Create a new branch for your feature or bug fix.
3. Make your changes and commit them.
4. Push your changes to your forked repository.
5. Open a pull request in the main repository.

Contact

If you have any questions, comments, or suggestions about Tool Name, please feel free to contact me:

• LinkedIn: https://www.linkedin.com/in/halil-ibrahim-deniz/
• TryHackMe: https://tryhackme.com/p/halilovic
• Instagram: https://www.instagram.com/deniz.halil333/
• YouTube: https://www.youtube.com/c/HalilDeniz
• Email: halildeniz313@gmail.com

DorXNG - Next Generation DorX. Built By Dorks, For Dorks

DorXNG is a modern solution for harvesting OSINT data using advanced search engine operators through multiple upstream search providers. On the backend it leverages a purpose built containerized image of SearXNG, a self-hosted, hackable, privacy focused, meta-search engine.

Our SearXNG implementation routes all search queries over the Tor network while refreshing circuits every ten seconds with Tor's MaxCircuitDirtiness configuration directive. We have also disabled all of SearXNG's client side timeout features. These settings allow for evasion of search engine restrictions commonly encountered while issuing many repeated search queries.

The DorXNG client application is written in Python3, and interacts with the SearXNG API to issue search queries concurrently. It can even issue requests across multiple SearXNG instances. The resulting search results are stored in a SQLite3 database.

We have enabled every supported upstream search engine that allows advanced search operator queries:

• Google
• DuckDuckGo
• Qwant
• Bing
• Brave
• Startpage
• Yahoo

For more information about what search engines SearXNG supports See: Configured Engines

Setup ️

LINUX ONLY ** Sorry Normies **

Install DorXNG

git clone https://github.com/researchanddestroy/dorxng
cd dorxng
pip install -r requirements.txt
./DorXNG.py -h

Download and Run Our Custom SearXNG Docker Container (at least one). Multiple SearXNG instances can be used. Use the --serverlist option with DorXNG. See: server.lst

When starting multiple containers wait at least a few seconds between starting each one.

docker run researchanddestroy/searxng:latest

If you would like to build the container yourself:

git clone https://github.com/researchanddestroy/searxng # The URL must be all lowercase for the build process to complete
cd searxng
DOCKER_BUILDKIT=1 make docker.build
docker images
docker run <image-id>

By default DorXNG has a hard coded server variable in parse_args.py which is set to the IP address that Docker will assign to the first container you run on your machine 172.17.0.2. This can be changed, or overwritten with --server or --serverlist.

Start Issuing Search Queries

./DorXNG.py -q 'search query'

Query the DorXNG Database

./DorXNG.py -D 'regex search string'

Instructions 

-h, --help            show this help message and exit
-s SERVER, --server SERVER
                      DorXNG Server Instance - Example: 'https://172.17.0.2/search'
-S SERVERLIST, --serverlist SERVERLIST
                      Issue Search Queries Across a List of Servers - Format: Newline Delimited
-q QUERY, --query QUERY
                      Issue a Search Query - Examples: 'search query' | '!tch search query' | 'site:example.com intext:example'
-Q QUERYLIST, --querylist QUERYLIST
                      Iterate Through a Search Query List - Format: Newline Delimited
-n NUMBER, --number NUMBER
                      Define the Number of Page Result Iterations
-c CONCURRENT, --concurrent CONCURRENT
                      Define the Number of Concurrent Page Requests
-l LIMITDATABASE, --limitdatabase LIMITDATABASE
                      Set Maximum Database Size Limit - Starts New Database After Exceeded - Example: --   limitdatabase 10 (10k Database Entries) - Suggested Maximum Database Size is 50k
                      when doing Deep Recursion
-L LOOP, --loop LOOP  Define the Number of Main Function Loop Iterations - Infinite Loop with 0
-d DATABASE, --database DATABASE
                      Specify SQL Database File - Default: 'dorxng.db'
-D DATABASEQUERY, --databasequery DATABASEQUERY
                      Issue Database Query - Format: Regex
-m MERGEDATABASE, --mergedatabase MERGEDATABASE
                      Merge SQL Database File - Example: --mergedatabase database.db
-t TIMEOUT, --timeout TIMEOUT
                      Specify Timeout Interval Between Requests - Default: 4 Seconds - Disable with 0
-r NONEWRESULTS, --nonewresults NONEWRESULTS
                      Specify Number of Iterations with No New Results - Default: 4 (3 Attempts) - Disable with 0
-v, --verbose         Enable Verbose Output
-vv, --veryverbose    Enable Very Ver   bose Output - Displays Raw JSON Output

Tips 

Sometimes you will hit a Tor exit node that is already shunted by upstream search providers, causing you to receive a minimal amount of search results. Not to worry... Just keep firing off queries. 

Keep your DorXNG SQL database file and rerun your command, or use the --loop switch to iterate the main function repeatedly. 

Most often, the more passes you make over a search query the more results you'll find. 

Also keep in mind that we have made a sacrifice in speed for a higher degree of data output. This is an OSINT project after all. 

Each search query you make is being issued to 7 upstream search providers... Especially with --concurrent queries this generates a lot of upstream requests... So have patience.

Keep in mind that DorXNG will continue to append new search results to your database file. Use the --database switch to specify a database filename, the default filename is dorxng.db. This probably doesn't matter for most, but if you want to keep your OSINT investigations seperate it's there for you.

Four concurrent search requests seems to be the sweet spot. You can issue more, but the more queries you issue at a time the longer it takes to receive results. It also increases the likelihood you receive HTTP/429 Too Many Requests responses from upstream search providers on that specific Tor circuit.

If you start multiple SearXNG Docker containers too rapidly Tor connections may fail to establish. While initializing a container, a valid response from the Tor Connectivity Check function looks like this:

./DorXNG.py -d ghdb.db -D '^http.*\.sql$'

./DorXNG.py -q 'search query'

./DorXNG.py -q 'search query' -c4

./DorXNG.py -q 'search query' -n4

./DorXNG.py -q 'search query' -c4 -n64

./DorXNG.py -S server.lst -q 'search query' -c4 -n64 -t0

./DorXNG.py -Q query.lst -c4 -n64

./DorXNG.py -S server.lst -Q query.lst -c4 -n64 -t0

./DorXNG.py -S server.lst -Q query.lst -c4 -n64 -t0 -L4

./DorXNG.py -S server.lst -Q query.lst -c4 -n64 -t0 -L0 -l10

./DorXNG.py -d new-database.db -m dorxng.db

for i in `ls *.db`; do ./DorXNG.py -d new-database.db -m $i; done

./DorXNG.py -d new-database.db -D 'regex search string'

Associated-Threat-Analyzer - Detects Malicious IPv4 Addresses And Domain Names Associated With Your Web Application Using Local Malicious Domain And IPv4 Lists

Associated-Threat-Analyzer detects malicious IPv4 addresses and domain names associated with your web application using local malicious domain and IPv4 lists.

Installation

From Git

git clone https://github.com/OsmanKandemir/associated-threat-analyzer.git
cd associated-threat-analyzer && pip3 install -r requirements.txt
python3 analyzer.py -d target-web.com

From Dockerfile

You can run this application on a container after build a Dockerfile.

Warning : If you want to run a Docker container, associated threat analyzer recommends to use your malicious IPs and domains lists, because maintainer may not be update a default malicious IP and domain lists on docker image.

docker build -t osmankandemir/threatanalyzer .
docker run osmankandemir/threatanalyzer -d target-web.com

From DockerHub

docker pull osmankandemir/threatanalyzer
docker run osmankandemir/threatanalyzer -d target-web.com

Usage

-d DOMAIN , --domain DOMAIN Input Target. --domain target-web1.com
-t DOMAINSFILE, --DomainsFile Malicious Domains List to Compare. -t SampleMaliciousDomains.txt
-i IPSFILE, --IPsFile Malicious IPs List to Compare. -i SampleMaliciousIPs.txt
-o JSON, --json JSON  JSON output. --json

DONE

• First-level depth scan your domain address.

TODO list

• Third-level or the more depth static files scanning for target web application.

Other linked github project. You can take a look.

Finds related domains and IPv4 addresses to do threat intelligence after Indicator-Intelligence v1.1.1 collects static files

https://github.com/OsmanKandemir/indicator-intelligence

Default Malicious IPs and Domains Sources

https://github.com/stamparm/blackbook

https://github.com/stamparm/ipsum

Development and Contribution

See; CONTRIBUTING.md

Xsubfind3R - A CLI Utility To Find Domain'S Known Subdomains From Curated Passive Online Sources

xsubfind3r is a command-line interface (CLI) utility to find domain's known subdomains from curated passive online sources.

Features

• Fetches domains from curated passive sources to maximize results.

• Supports stdin and stdout for easy integration into workflows.

• Cross-Platform (Windows, Linux & macOS).

Installation

Install release binaries (Without Go Installed)

Visit the releases page and find the appropriate archive for your operating system and architecture. Download the archive from your browser or copy its URL and retrieve it with wget or curl:

• ...with wget:

   wget https://github.com/hueristiq/xsubfind3r/releases/download/v<version>/xsubfind3r-<version>-linux-amd64.tar.gz

• ...or, with curl:

   curl -OL https://github.com/hueristiq/xsubfind3r/releases/download/v<version>/xsubfind3r-<version>-linux-amd64.tar.gz

...then, extract the binary:

tar xf xsubfind3r-<version>-linux-amd64.tar.gz

TIP: The above steps, download and extract, can be combined into a single step with this onliner

curl -sL https://github.com/hueristiq/xsubfind3r/releases/download/v<version>/xsubfind3r-<version>-linux-amd64.tar.gz | tar -xzv

NOTE: On Windows systems, you should be able to double-click the zip archive to extract the xsubfind3r executable.

...move the xsubfind3r binary to somewhere in your PATH. For example, on GNU/Linux and OS X systems:

sudo mv xsubfind3r /usr/local/bin/

NOTE: Windows users can follow How to: Add Tool Locations to the PATH Environment Variable in order to add xsubfind3r to their PATH.

Install source (With Go Installed)

Before you install from source, you need to make sure that Go is installed on your system. You can install Go by following the official instructions for your operating system. For this, we will assume that Go is already installed.

go install ...

go install -v github.com/hueristiq/xsubfind3r/cmd/xsubfind3r@latest

go build ... the development Version

• Clone the repository

   git clone https://github.com/hueristiq/xsubfind3r.git 

• Build the utility

   cd xsubfind3r/cmd/xsubfind3r && \
   go build .

• Move the xsubfind3r binary to somewhere in your PATH. For example, on GNU/Linux and OS X systems:

   sudo mv xsubfind3r /usr/local/bin/

  NOTE: Windows users can follow How to: Add Tool Locations to the PATH Environment Variable in order to add xsubfind3r to their PATH.

NOTE: While the development version is a good way to take a peek at xsubfind3r's latest features before they get released, be aware that it may have bugs. Officially released versions will generally be more stable.

Post Installation

xsubfind3r will work right after installation. However, BeVigil, Chaos, Fullhunt, Github, Intelligence X and Shodan require API keys to work, URLScan supports API key but not required. The API keys are stored in the $HOME/.hueristiq/xsubfind3r/config.yaml file - created upon first run - and uses the YAML format. Multiple API keys can be specified for each of these source from which one of them will be used.

Example config.yaml:

version: 0.3.0
sources:
    - alienvault
    - anubis
    - bevigil
    - chaos
    - commoncrawl
    - crtsh
    - fullhunt
    - github
    - hackertarget
    - intelx
    - shodan
    - urlscan
    - wayback
keys:
    bevigil:
        - awA5nvpKU3N8ygkZ
    chaos:
        - d23a554bbc1aabb208c9acfbd2dd41ce7fc9db39asdsd54bbc1aabb208c9acfb
    fullhunt:
        - 0d9652ce-516c-4315-b589-9b241ee6dc24
    github:
        - d23a554bbc1aabb208c9acfbd2dd41ce7fc9db39
        - asdsd54bbc1aabb208c9acfbd2dd41ce7fc9db39
    intelx:
        - 2.intelx.io:00000000-0000-0000-0000-000000000000
    shodan:
        - AAAAClP1bJJSRMEYJazgwhJKrggRwKA
    urlscan:
        - d4c85d34-e425-446e-d4ab-f5a3412acbe8

Usage

To display help message for xsubfind3r use the -h flag:

xsubfind3r -h

help message:

                _      __ _           _ _____      
__  _____ _   _| |__  / _(_)_ __   __| |___ / _ __ 
\ \/ / __| | | | '_ \| |_| | '_ \ / _` | |_ \| '__|
 >  <\__ \ |_| | |_) |  _| | | | | (_| |___) | |   
/_/\_\___/\__,_|_.__/|_| |_|_| |_|\__,_|____/|_| v0.3.0

USAGE:
  xsubfind3r [OPTIONS]

INPUT:
 -d, --domain string[]                 target domains
 -l, --list string                     target domains' list file path

SOURCES:
      --sources bool                   list supported sources
 -u,  --sources-to-use string[]        comma(,) separeted sources to use
 -e,  --sources-to-exclude string[]    comma(,) separeted sources to exclude

OPTIMIZATION:
 -t,  --threads int                    number of threads (default: 50)

OUTPUT:
     --no-color bool                   disable colored output
 -o, --output string                   output subdomains' file path
    -O, --output-directory string         output subdomains' directory path
 -v, --verbosity string                debug, info, warning, error, fatal or silent (default: info)

CONFIGURATION:
 -c,  --configuration string           configuration file path (default: ~/.hueristiq/xsubfind3r/config.yaml)

Contribution

Issues and Pull Requests are welcome! Check out the contribution guidelines.

Licensing

This utility is distributed under the MIT license.

HackBot - A Simple Cli Chatbot Having Llama2 As Its Backend Chat AI

Welcome to HackBot, an AI-powered cybersecurity chatbot designed to provide helpful and accurate answers to your cybersecurity-related queries and also do code analysis and scan analysis. Whether you are a security researcher, an ethical hacker, or just curious about cybersecurity, HackBot is here to assist you in finding the information you need.

HackBot utilizes the powerful language model Meta-LLama2 through the "LlamaCpp" library. This allows HackBot to respond to your questions in a coherent and relevant manner. Please make sure to keep your queries in English and adhere to the guidelines provided to get the best results from HackBot.

Features

• AI Cybersecurity Chat: HackBot can answer various cybersecurity-related queries, helping you with penetration testing, security analysis, and more.
• Interactive Interface: The chatbot provides an interactive command-line interface, making it easy to have conversations with HackBot.
• Clear Output: HackBot presents its responses in a well-formatted markdown, providing easily readable and organized answers.
• Static Code Analysis: Utilizes the provided scan data or log file for conducting static code analysis. It thoroughly examines the source code without executing it, identifying potential vulnerabilities, coding errors, and security issues.
• Vulnerability Analysis: Performs a comprehensive vulnerability analysis using the provided scan data or log file. It identifies and assesses security weaknesses, misconfigurations, and potential exploits present in the target system or network.

How it looks

Chat:

Static Code analysis:

Vulnerability analysis:

Installation

Prerequisites

Before you proceed with the installation, ensure you have the following prerequisites:

• Python (3.6 or later)
• pip package manager
• Visual studio Code - Follow the steps in this link llama-cpp-prereq-install-instructions
• cmake

Step 1: Clone the Repository

git clone https://github.com/morpheuslord/hackbot.git
cd hackbot

Step 2: Install Dependencies

pip install -r requirements.txt

Step 3: Download the AI Model

python hackbot.py

The first time you run HackBot, it will check for the AI model required for the chatbot. If the model is not present, it will be automatically downloaded and saved as "llama-2-7b-chat.ggmlv3.q4_0.bin" in the project directory.

Usage

To start a conversation with HackBot, run the following command:

python hackbot.py

HackBot will display a banner and wait for your input. You can ask cybersecurity-related questions, and HackBot will respond with informative answers. To exit the chat, simply type "quit_bot" in the input prompt.

Here are some additional commands you can use:

• clear_screen: Clears the console screen for better readability.
• quit_bot: This is used to quit the chat application
• bot_banner: Prints the default bots banner.
• contact_dev: Provides my contact information.
• save_chat: Saves the current sessions interactions.
• vuln_analysis: Does a Vuln analysis using the scan data or log file.
• static_code_analysis: Does a Static code analysis using the scan data or log file.

Note: I am working on more addons and more such commands to give a more chatGPT experience

Please Note: HackBot's responses are based on the Meta-LLama2 AI model, and its accuracy depends on the quality of the queries and data provided to it.

I am also working on AI training by which I can teach it how to be more accurately tuned to work for hackers on a much more professional level.

Contributing

We welcome contributions to improve HackBot's functionality and accuracy. If you encounter any issues or have suggestions for enhancements, please feel free to open an issue or submit a pull request. Follow these steps to contribute:

1. Fork the repository.
2. Create a new branch with a descriptive name.
3. Make your changes and commit them.
4. Push your changes to your forked repository.
5. Open a pull request to the main branch of this repository.

Please maintain a clean commit history and adhere to the project's coding guidelines.

AI training

If anyone with the know-how of training text generation models can help improve the code.

Contact

For any questions, feedback, or inquiries related to HackBot, feel free to contact the project maintainer:

• Email: morpheuslord@protonmail.com
• Twitter: @morpheuslord2
• LinkedIn: ChiranjeeviG

Xcrawl3R - A CLI Utility To Recursively Crawl Webpages

xcrawl3r is a command-line interface (CLI) utility to recursively crawl webpages i.e systematically browse webpages' URLs and follow links to discover linked webpages' URLs.

Features

• Recursively crawls webpages for URLs.
• Parses URLs from files (.js, .json, .xml, .csv, .txt & .map).
• Parses URLs from robots.txt.
• Parses URLs from sitemaps.
• Renders pages (including Single Page Applications such as Angular and React).
• Cross-Platform (Windows, Linux & macOS)

Installation

Install release binaries (Without Go Installed)

Visit the releases page and find the appropriate archive for your operating system and architecture. Download the archive from your browser or copy its URL and retrieve it with wget or curl:

• ...with wget:

   wget https://github.com/hueristiq/xcrawl3r/releases/download/v<version>/xcrawl3r-<version>-linux-amd64.tar.gz

• ...or, with curl:

   curl -OL https://github.com/hueristiq/xcrawl3r/releases/download/v<version>/xcrawl3r-<version>-linux-amd64.tar.gz

...then, extract the binary:

tar xf xcrawl3r-<version>-linux-amd64.tar.gz

TIP: The above steps, download and extract, can be combined into a single step with this onliner

curl -sL https://github.com/hueristiq/xcrawl3r/releases/download/v<version>/xcrawl3r-<version>-linux-amd64.tar.gz | tar -xzv

NOTE: On Windows systems, you should be able to double-click the zip archive to extract the xcrawl3r executable.

...move the xcrawl3r binary to somewhere in your PATH. For example, on GNU/Linux and OS X systems:

sudo mv xcrawl3r /usr/local/bin/

NOTE: Windows users can follow How to: Add Tool Locations to the PATH Environment Variable in order to add xcrawl3r to their PATH.

Install source (With Go Installed)

Before you install from source, you need to make sure that Go is installed on your system. You can install Go by following the official instructions for your operating system. For this, we will assume that Go is already installed.

go install ...

go install -v github.com/hueristiq/xcrawl3r/cmd/xcrawl3r@latest

go build ... the development Version

• Clone the repository

   git clone https://github.com/hueristiq/xcrawl3r.git 

• Build the utility

   cd xcrawl3r/cmd/xcrawl3r && \
   go build .

• Move the xcrawl3r binary to somewhere in your PATH. For example, on GNU/Linux and OS X systems:

   sudo mv xcrawl3r /usr/local/bin/

  NOTE: Windows users can follow How to: Add Tool Locations to the PATH Environment Variable in order to add xcrawl3r to their PATH.

NOTE: While the development version is a good way to take a peek at xcrawl3r's latest features before they get released, be aware that it may have bugs. Officially released versions will generally be more stable.

Usage

To display help message for xcrawl3r use the -h flag:

xcrawl3r -h

help message:

                             _ _____      
__  _____ _ __ __ ___      _| |___ / _ __ 
\ \/ / __| '__/ _` \ \ /\ / / | |_ \| '__|
 >  < (__| | | (_| |\ V  V /| |___) | |   
/_/\_\___|_|  \__,_| \_/\_/ |_|____/|_| v0.1.0

A CLI utility to recursively crawl webpages.

USAGE:
  xcrawl3r [OPTIONS]

INPUT:
  -d, --domain string               domain to match URLs
      --include-subdomains bool     match subdomains' URLs
  -s, --seeds string                seed URLs file (use `-` to get from stdin)
  -u, --url string                  URL to crawl

CONFIGURATION:
      --depth int                   maximum depth to crawl (default 3)
                                       TIP: set it to `0` for infinite recursion
      --headless bool               If true the browser will be displayed while crawling.
  -H, --headers string[]            custom header to include in requests
                                       e.g. -H 'Referer: http://example.com/'
                                       TIP: use multiple flag to set multiple headers
      --proxy string[]              Proxy URL (e.g: http://127.0.0.1:8080)
                                       TIP: use multiple flag to set multiple proxies
      --render bool                 utilize a headless chrome instance to render pages
      --timeout int                 time to wait for request in seconds (default: 10)
      --user-agent string           User Agent to use (default: web)
                                       TIP: use `web` for a random web user-agent,
                                       `mobile` for a random mobile user-agent,
                                           or you can set your specific user-agent.

RATE LIMIT:
  -c, --concurrency int             number of concurrent fetchers to use (default 10)
      --delay int                   delay between each request in seconds
      --max-random-delay int        maximux extra randomized delay added to `--dalay` (default: 1s)
  -p, --parallelism int             number of concurrent URLs to process (default: 10)

OUTPUT:
      --debug bool                  enable debug mode (default: false)
  -m, --monochrome bool             coloring: no colored output mode
  -o, --output string               output file to write found URLs
  -v, --verbosity string            debug, info, warning, error, fatal or silent (default: debug)

Contributing

Issues and Pull Requests are welcome! Check out the contribution guidelines.

Licensing

This utility is distributed under the MIT license.

Credits

• Alternatives - Check out projects below, that may fit in your workflow:

  katana * gospider * hakrawler * urlgrab

Xurlfind3R - A CLI Utility To Find Domain'S Known URLs From Curated Passive Online Sources

xurlfind3r is a command-line interface (CLI) utility to find domain's known URLs from curated passive online sources.

Features

• Fetches URLs from curated passive sources to maximize results:
  • AlienVault's OTX
  • BeVigil
  • Common Crawl
  • URLScan
  • Github
  • Intelligence X
  • Wayback Machine
• With Wayback Machine, Parses URLs from robots.txt snapshots.
• With Wayback Machine, Parses URLs from webpages' snapshots.
• Cross-Platform (Windows, Linux & macOS)
• Supports URLs match and filter

Installation

Install release binaries (Without Go Installed)

Visit the releases page and find the appropriate archive for your operating system and architecture. Download the archive from your browser or copy its URL and retrieve it with wget or curl:

• ...with wget:

   wget https://github.com/hueristiq/xurlfind3r/releases/download/v<version>/xurlfind3r-<version>-linux-amd64.tar.gz

• ...or, with curl:

   curl -OL https://github.com/hueristiq/xurlfind3r/releases/download/v<version>/xurlfind3r-<version>-linux-amd64.tar.gz

...then, extract the binary:

tar xf xurlfind3r-<version>-linux-amd64.tar.gz

TIP: The above steps, download and extract, can be combined into a single step with this onliner

curl -sL https://github.com/hueristiq/xurlfind3r/releases/download/v<version>/xurlfind3r-<version>-linux-amd64.tar.gz | tar -xzv

NOTE: On Windows systems, you should be able to double-click the zip archive to extract the xurlfind3r executable.

...move the xurlfind3r binary to somewhere in your PATH. For example, on GNU/Linux and OS X systems:

sudo mv xurlfind3r /usr/local/bin/

NOTE: Windows users can follow How to: Add Tool Locations to the PATH Environment Variable in order to add xurlfind3r to their PATH.

Install source (With Go Installed)

Before you install from source, you need to make sure that Go is installed on your system. You can install Go by following the official instructions for your operating system. For this, we will assume that Go is already installed.

go install ...

go install -v github.com/hueristiq/xurlfind3r/cmd/xurlfind3r@latest

go build ... the development Version

• Clone the repository

   git clone https://github.com/hueristiq/xurlfind3r.git 

• Build the utility

   cd xurlfind3r/cmd/xurlfind3r && \
   go build .

• Move the xurlfind3r binary to somewhere in your PATH. For example, on GNU/Linux and OS X systems:

   sudo mv xurlfind3r /usr/local/bin/

  NOTE: Windows users can follow How to: Add Tool Locations to the PATH Environment Variable in order to add xurlfind3r to their PATH.

NOTE: While the development version is a good way to take a peek at xurlfind3r's latest features before they get released, be aware that it may have bugs. Officially released versions will generally be more stable.

Post Installation

xurlfind3r will work right after installation. However, BeVigil, Github and Intelligence X require API keys to work, URLScan supports API key but not required. The API keys are stored in the $HOME/.hueristiq/xurlfind3r/config.yaml file - created upon first run - and uses the YAML format. Multiple API keys can be specified for each of these source from which one of them will be used.

Example config.yaml:

version: 0.2.0
sources:
    - bevigil
    - commoncrawl
    - github
    - intelx
    - otx
    - urlscan
    - wayback
keys:
    bevigil:
        - awA5nvpKU3N8ygkZ
    github:
        - d23a554bbc1aabb208c9acfbd2dd41ce7fc9db39
        - asdsd54bbc1aabb208c9acfbd2dd41ce7fc9db39
    intelx:
        - 2.intelx.io:00000000-0000-0000-0000-000000000000
    urlscan:
        - d4c85d34-e425-446e-d4ab-f5a3412acbe8

Usage

To display help message for xurlfind3r use the -h flag:

xurlfind3r -h

help message:

                 _  __ _           _ _____      
__  ___   _ _ __| |/ _(_)_ __   __| |___ / _ __ 
\ \/ / | | | '__| | |_| | '_ \ / _` | |_ \| '__|
 >  <| |_| | |  | |  _| | | | | (_| |___) | |
/_/\_\\__,_|_|  |_|_| |_|_| |_|\__,_|____/|_| v0.2.0

USAGE:
  xurlfind3r [OPTIONS]

TARGET:
 -d, --domain string              (sub)domain to match URLs

SCOPE:
     --include-subdomains bool    match subdomain's URLs

SOURCES:
 -s,  --sources bool              list sources
 -u,  --use-sources string        sources to use (default: bevigil,commoncrawl,github,intelx,otx,urlscan,wayback)
      --skip-wayback-robots bool  with wayback, skip parsing robots.txt snapshots
      --skip-wayback-source bool  with wayback   , skip parsing source code snapshots

FILTER & MATCH:
 -f, --filter string              regex to filter URLs
 -m, --match string               regex to match URLs

OUTPUT:
     --no-color bool              no color mode
 -o, --output string              output URLs file path
 -v, --verbosity string           debug, info, warning, error, fatal or silent (default: info)

CONFIGURATION:
 -c,  --configuration string      configuration file path (default: ~/.hueristiq/xurlfind3r/config.yaml)

Examples

Basic

xurlfind3r -d hackerone.com --include-subdomains

Filter Regex

# filter images
xurlfind3r -d hackerone.com --include-subdomains -f '`^https?://[^/]*?/.*\.(jpg|jpeg|png|gif|bmp)(\?[^\s]*)?$`'

Match Regex

# match js URLs
xurlfind3r -d hackerone.com --include-subdomains -m '^https?://[^/]*?/.*\.js(\?[^\s]*)?$'

Contributing

Issues and Pull Requests are welcome! Check out the contribution guidelines.

Licensing

This utility is distributed under the MIT license.

CakeFuzzer - Automatically And Continuously Discover Vulnerabilities In Web Applications Created Based On Specific Frameworks

Cake Fuzzer is a project that is meant to help automatically and continuously discover vulnerabilities in web applications created based on specific frameworks with very limited false positives. Currently it is implemented to support the Cake PHP framework.

If you would like to learn more about the research process check out this article series: CakePHP Application Cybersecurity Research

Project goals

Typical approaches to discovering vulnerabilities using automated tools in web applications are:

• Static Application Security Testing (SAST) – Method that involves a scanner detecting vulnerabilities based on the source code without running the application
• Dynamic Application Security Testing (DAST) – Method that incorporates a vulnerability scanner that attacks the running application and identifies the vulnerabilities based on the application responses

Both methods have disadvantages. SAST results in a high percentage of false positives – findings that are either not vulnerabilities or not exploitable vulnerabilities. DAST results in fewer false positives but discovers fewer vulnerabilities due to the limited information. It also requires some knowledge about the application and a security background of a person who runs a scan. This often comes with a custom scan configuration per application to work properly.

The Cake Fuzzer project is meant to combine the advantages of both approaches and eliminate the above-mentioned disadvantages. This approach is called Interactive Application Security Testing (IAST).

The goals of the project are:

• Create an automated process of discovering vulnerabilities in applications based on the CakePHP Framework
• No application knowledge requirement or pre-configuration of the web application
• Result with minimal or close to 0 amount of false positives
• Require minimal security knowledge to run the scanner

Note: Some classes of vulnerabilities are not the target of the Cake Fuzzer, therefore Cake Fuzzer will not be able to detect them. Examples of those classes are business logic vulnerabilities and access control issues.

Architecture

Overview

Drawio: Cake Fuzzer Architecture

Cake Fuzzer consists of 3 main (fairly independent) servers that in total allow for dynamic vulnerability testing of CakePHP allications.

• AttackQueue - Scheduling and execution of attack scenarios.
• Monitors - Monitoring of given entities (executor outputs / file contents / processes / errors ).
• Registry - Storage and classification of found vulnerabilities. They run independently. AttackQueue can add new scanners to monitors, and Monitors can schedule new attacks (eg based on found vulnerability to further attack application).

Other components include:

• Fuzzer - defines and schedules attacks to AttackQueue (serves as entry)
• Configuration - sets up application dependent info (eg. path to CakePHP application)
• Instrumentation - based on configuration defines changes to the application / os to prepare the ground for attacks.

Approach

Cake Fuzzer is based on the concept of Interactive Application Security Testing (IAST). It contains a predefined set of attacks that are randomly modified before the execution. Cake Fuzzer has the knowledge of the application internals thanks to the Cake PHP framework therefore the attacks will be launched on all possible entry points of the application.

During the attack, the Cake Fuzzer monitors various aspects of the application and the underlying system such as:

• network connection,
• file system,
• application response,
• error logs.

These sources of information allow Cake Fuzzer to identify more vulnerabilities and report them with higher certainty.

Requirements

• CakePHP Web Application installed, configured, and running. Example CakePHP web applications: MISP (https://github.com/MISP/MISP), Cerebrate (https://github.com/cerebrate-project/cerebrate)
• PHP CLI

Development environment using MISP on VMWare virtual machine

The following section describes steps to setup a Cake Fuzzer development environment where the target is outdated MISP v2.4.146 that is vulnerable to CVE-2021-41326.

Requirements

• VMWare Workstation (Other virtualization platform can be used as long as they support sharing/mounting directories between host and guest OS)

Steps

Run the following commands on your host operating system to download an outdated MISP VM:

cd ~/Downloads # Or wherever you want to store the MISP VM
wget https://vm.misp-project.org/MISP_v2.4.146@0c25b72/MISP_v2.4.146@0c25b72-VMware.zip -O MISP.zip
unzip MISP.zip
rm MISP.zip
mv VMware/ MISP-2.4.146

Conduct the following actions in VMWare GUI to prepare sharing Cake Fuzzer files between your host OS and MISP:

1. Open virtual machine in VMWare and go to > Settings > Options > Shared Folders > Add.
2. Mount directory where you keep Cake Fuzzer on your host OS and name it cake_fuzzer on the VM.
3. Start the VM.
4. Note the IP address displayed in the VMWare window after MISP fully boots up.

Run the following commands on your host OS (replace MISP_IP_ADDRESS with previously noted IP address):

ssh-copy-id misp@MISP_IP_ADDRESS
ssh misp@MISP_IP_ADDRESS

Once you SSH into the MISP run the following commands (in MISP terminal) to finish setup of sharing Cake Fuzzer files between host OS and MISP:

sudo apt update
sudo apt-get -y install open-vm-tools open-vm-tools-desktop
sudo apt-get -y install build-essential module-assistant linux-headers-virtual linux-image-virtual && sudo dpkg-reconfigure open-vm-tools
sudo mkdir /cake_fuzzer # Note: This path is fixed as it's hardcoded in the instrumentation (one of the patches)
sudo vmhgfs-fuse .host:/cake_fuzzer /cake_fuzzer -o allow_other -o uid=1000
ls -l /cake_fuzzer # If everything went fine you should see content of the Cake Fuzzer directory from your host OS. Any changes on your host OS will be reflected inside the VM and vice-versa.

Prepare MISP for simple testing (in MISP terminal):

CAKE=/var/www/MISP/app/Console/cake
SUDO='sudo -H -u www-data'
$CAKE userInit -q
$SUDO $CAKE Admin setSetting "Security.password_policy_length" 1
$SUDO $CAKE Admin setSetting "Security.password_policy_complexity" '/.*/'
$SUDO $CAKE Password admin@admin.test admin --override_password_change

Finally instal Cake Fuzzer dependencies and prepare the venv (in MISP terminal):

source /cake_fuzzer/precheck.sh

Contribution to Vulnerability Database

Cake Fuzzer scans for vulnerabilities that inside of /cake_fuzzer/strategies folder.

To add a new attack we need to add a new new-attack.json file to strategies folder. Each vulnerability contains 2 major fileds:Scenarios and Scanners. Scenarios where attack payloads base forms stored. Scanners in the other hand detecting regex or pharases for response, stout, sterr, logs, and results.

Creating payload for Scenarios

To create a payload first you need to have the understanding of the vulnerability and how to detect it with as few payloads as possible.

• While constructing the scenario you should think of as most generic payload as possible. However, the more generic payload, the more chances are that it will produce false-positives.

• It is preferable to us a canary value such as__cakefuzzer__new-attack_§CAKEFUZZER_PAYLOAD_GUID§__ in your scenarios. Canary value contains a fixed string (for example: __cakefuzzer__new-attack_) and a dynamic identifier that will be changed dynamically by the fuzzer (GUID part §CAKEFUZZER_PAYLOAD_GUID§). First canary part is used to ensure that payload is detected by Scanners. Second canary part, the GUID is translated to pseudo-random value on every execution of your payload. So whenever your payload will be injected into the a parameter used by the application, the canary will be changed to something like this: __cakefuzzer__new-attack_8383938__, where the 8383938 is unique across all other attacks.

Detecting and generating Scanners

To create a scanner, first you need to understand how may the application behave when the vulnerability is triggered. There are few scanner types that you can use such as response, sterr, logs, files, and processes. Each scanner serves a different purpose.

For example when you building a scanner for an XSS, you will look for the indication of the vulnerability in the HTML response of the application. You can use ResultOutputScanner scanner to look for canary value and payload. In other hand SQL Injection vulnerabilities could be detected via error logs. For that purpose you can use LogFilesContentsScanner and ResultErrorsScanner.

• One of the important points of creating a scanner is finding a regular expression or a pharase that does not catch false-positive values. If you want to contribute a new vulnerability, you should ensure that there is no false-positive by using the new vulnerability in scans.
• Last attribute of these Scanner regular expressions is generating an efficent regex. Avoid using regex that match all cases .* or .+. They are very time consuming and drasticly increase the time required to finish the entire scan.

Efficiency

As mentioned before efficiency is important part of the vulnerabilities. Both Scenarios and Scanners should include as few elements as possible. This is because Cake Fuzzer executes every single scenario in all possible detected paths multiple times. On the other hand, all responses, new log entries, etc. are constantly checked by the Scanners. There should be a lot of parameters, paths, and end-points detected and therefore using more payload or Scanner affects the efficiency quite a lot.

Removing Specific Vulnerability

If do not want to scan a specific vulnerability class, remove specified json file from the strategies folder, clean the database and run the fuzzer again.

For example if you do not want to scan your applicaiton for SQL Injection vulnerabilities, do the following steps:

First of all remove already prepared attack scenarios. To achive this delete all files inside of the /cake_fuzzer/databases folder:

rm  /cake_fuzzer/databases/*

After that remove the sqlinj.json file from the /cake_fuzzer/strategies

rm /cake_fuzzer/strategies/sqlinj.json

Finally re-run the fuzzer and all cake_fuzzer running proccess without any SQL Injection attack executed.

PoC Usage

Installation

Clone respository

git clone https://github.com/Zigrin-Security/CakeFuzzer /cake_fuzzer

Warning Cake Fuzzer won't work properly if it's under different path than /cake_fuzzer. Keep in mind that it has to be placed under the root directory of the file system, next /root, /tmp, and so on.

Change directory to respository

cd /cake_fuzzer

Venv

Enter virtual environment if you are not already in:

source /cake_fuzzer/precheck.sh

OR

source venv/bin/activate

Configuration

cp config/config.example.ini config/config.ini

Configure config/config.ini:

WEBROOT_DIR="/var/www/html"         # Path to the tested applications `webroot` directory
CONCURRENT_QUEUES=5                            # [Optional] Number of attacks executed concurretnly at once
ONLY_PATHS_WITH_PREFIX="/"                      # [Optional] Fuzzer will generates only attacks for attacks starting with this prefix
EXCLUDE_PATHS=""                                # [Optional] Fuzzer will exlude from scanning all paths that match this regular expression. If it's empty, all paths will be processed
PAYLOAD_GUID_PHRASE="§CAKEFUZZER_PAYLOAD_GUID§" # [Optional] Internal keyword that is substituted right before attack with unique payload id
INSTRUMENTATION_INI="config/instrumentation_cake4.ini" # [Optional] Path to custom instrumentations of the application.

Execution

Start component processes

Warning During the Cake Fuzzer scan, multiple functionalities of your application will be invoked in uncontrolled manner multiple times. This may result issuing connections to external services your application is connected to, and pulling or pushing data from/to it. It is highly recommended to run Cake Fuzzer in isolated controlled environment without access to sensitive external services.

Note Cake Fuzzer bypass blackholing, CSRF protections, and authorization. It sends all attacks with privileges of a first user in the database. It is recommended that this user has the highest permissions.

The application consists of several components.

Warning All cake_fuzzer commands have to be executed as root.

Before starting the fuzzer make sure your target application is fully instrumented:

python cake_fuzzer.py instrument check

If there are some unapplied changes apply them with:

python cake_fuzzer.py instrument apply

To run cake fuzzer do the following (It's recommended to use at least 3 separate terminal):

# First Terminal
python cake_fuzzer.py run fuzzer                  # Generates attacks, adds them to the QUEUE and registers new SCANNERS (then exits)
python cake_fuzzer.py run periodic_monitors       # Responsible for monitoring (use CTRL+C to stop & exit at the end of the scan)

# Second terminal
python cake_fuzzer.py run iteration_monitors     # Responsible for monitoring (use CTRL+C to stop & exit at the end of the scan)

# Third terminal
python cake_fuzzer.py run attack_queue            # Starts the ATTACK QUEUE (use CTRL+C to stop & exit at the end of the scan)

# Once all attacks are executed
python cake_fuzzer.py run registry                # Generates `results.json` based on found vulnerabilities

Note: There is currently a bug that can change the owner of logs (or any other dynamically changed filies of the target web app). This may cause errors when normally using the web application or even false-negatives on future Cake Fuzzer executions. For MISP we recommend running the following after every execution of the fuzzer:

sudo chown -R www-data:www-data /var/www/MISP/app/tmp/logs/

Once your scan finishes revert the instrumentation:

python cake_fuzzer.py instrument revert

To Run Again

To run cake fuzzer again, do the following:

Delete Applications Logs (as an example to this, MISP logs stored /var/www/MISP/app/tmp/logs)

rm  /var/www/MISP/app/tmp/logs/*

Delete All Files Inside of /cake_fuzzer/databases folder

rm  /cake_fuzzer/databases/*

Delete cake_fuzzer/results.jsonfile (Firstly do not forget to save or examine previous scan resulst)

rm  /cake_fuzzer/results.json

Finally follow previous running proccess again with 3 terminals

FAQ / Troubleshooting

Attack Queue seems like doing nothing

Attack queue marks executed attacks in the database as 'executed' so to run whole suite again you need to remove the database and add attacks again.

Make sure to kill monitors and attack queues before removing the database.

rm database.db*
python cake_fuzzer.py run fuzzer
python cake_fuzzer.py run attack_queue

Target app does not save logs to log files

This is likely due to the fact that the previous log files were overwritten by root. Cake Fuzzer operates as root so new log files will be created with the root as the owner. Remove them:

chmod -R a+w /var/www/MISP/app/tmp/logs/*

No files in /cake_fuzzer dir of a VM after a reboot

If you use VM with sharing cake fuzzer with your host machine, make sure that the host directory is properly attached to the guest VM:

sudo vmhgfs-fuse .host:/cake_fuzzer /cake_fuzzer -o allow_other -o uid=1000

Target app crashes after running Cake Fuzzer

Cake Fuzzer has to be located under the root directory of the machine and the base directory name should be cake_fuzzer specificaly.

mv CakeFuzzer/ /cake_fuzzer

"Patch" errors while runing instrument apply

Instrumentation proccess is a part of Cake Fuzzer execution flow. When you run instrument apply followed by instrument check, both of these commands should result in the same number of changes.

If you get any "patch" error you could apply patches manually and delete problematic patch file. Patches are located under the /cake_fuzzer/cakefuzzer/instrumentation/pathces directory.

Dependency errors

While installing or running if you have python dependency error, manuallay install dependencies after switching to virtual environment.

First switch to the virtual environment

source venv/bin/activate

After that you can install dependecies with pip3.

pip3 install -r requriments.txt

Credits

Inspiration

This project was inspired by:

• Automatic Detection of Vulnerabilities in Web Applications using Fuzzing by Miguel Filipe Beatriz – https://fenix.tecnico.ulisboa.pt/downloadFile/563345090413029/ExtendedAbstract-MEICA-67039-MiguelBeatriz.pdf
• WPGarlic by Krzysztof Zając – https://github.com/kazet/wpgarlic

Commision

This project was commissioned by:

Initial contributors

• Dawid Czarnecki (original author)
• Adam Bobowski (lead developer)
• @Kattorne
• Ulaş Deniz İlhan

Blackbone - Windows Memory Hacking Library

Windows memory hacking library

Features

• x86 and x64 support

Process interaction

• Manage PEB32/PEB64
• Manage process through WOW64 barrier

Process Memory

• Allocate and free virtual memory
• Change memory protection
• Read/Write virtual memory

Process modules

• Enumerate all (32/64 bit) modules loaded. Enumerate modules using Loader list/Section objects/PE headers methods.
• Get exported function address
• Get the main module
• Unlink module from loader lists
• Inject and eject modules (including pure IL images)
• Inject 64bit modules into WOW64 processes
• Manually map native PE images

Threads

• Enumerate threads
• Create and terminate threads. Support for cross-session thread creation.
• Get thread exit code
• Get main thread
• Manage TEB32/TEB64
• Join threads
• Suspend and resume threads
• Set/Remove hardware breakpoints

Pattern search

• Search for arbitrary pattern in local or remote process

Remote code execution

• Execute functions in remote process
• Assemble own code and execute it remotely
• Support for cdecl/stdcall/thiscall/fastcall conventions
• Support for arguments passed by value, pointer or reference, including structures
• FPU types are supported
• Execute code in new thread or any existing one

Remote hooking

• Hook functions in remote process using int3 or hardware breakpoints
• Hook functions upon return

Manual map features

• x86 and x64 image support
• Mapping into any arbitrary unprotected process
• Section mapping with proper memory protection flags
• Image relocations (only 2 types supported. I haven't seen a single PE image with some other relocation types)
• Imports and Delayed imports are resolved
• Bound import is resolved as a side effect, I think
• Module exports
• Loading of forwarded export images
• Api schema name redirection
• SxS redirection and isolation
• Activation context support
• Dll path resolving similar to native load order
• TLS callbacks. Only for one thread and only with PROCESS_ATTACH/PROCESS_DETACH reasons.
• Static TLS
• Exception handling support (SEH and C++)
• Adding module to some native loader structures(for basic module api support: GetModuleHandle, GetProcAdress, etc.)
• Security cookie initialization
• C++/CLI images are supported
• Image unloading
• Increase reference counter for import libraries in case of manual import mapping
• Cyclic dependencies are handled properly

Driver features

• Allocate/free/protect user memory
• Read/write user and kernel memory
• Disable permanent DEP for WOW64 processes
• Change process protection flag
• Change handle access rights
• Remap process memory
• Hiding allocated user-mode memory
• User-mode dll injection and manual mapping
• Manual mapping of drivers

Requirements

• Visual Studio 2017 15.7 or higher
• Windows SDK 10.0.17134 or higher
• WDK 10.0.17134 or higher (driver only)
• VC++ 2017 Libs for Spectre (x86 and x64)
• Visual C++ ATL (x86/x64) with Spectre Mitigations

Mantra - A Tool Used To Hunt Down API Key Leaks In JS Files And Pages

The tool in question was created in Go and its main objective is to search for API keys in JavaScript files and HTML pages.

It works by checking the source code of web pages and script files for strings that are identical or similar to API keys. These keys are often used for authentication to online services such as third-party APIs and are confidential and should not be shared publicly.

By using this tool, developers can quickly identify if their API keys are leaking and take steps to fix the problem before they are compromised. Furthermore, the tool can be useful for security officers, who can use it to verify that applications and websites that use external APIs are adequately protecting their keys.

In summary, this tool is an efficient and accurate solution to help secure your API keys and prevent sensitive information leaks.

Help

Usage

Install

git clone https://github.com/MrEmpy/Mantra
cd Mantra
make
./build/mantra-amd64-linux -h

Buy me a coffee?

LivePix

ScrapPY - A Python Utility For Scraping Manuals, Documents, And Other Sensitive PDFs To Generate Wordlists That Can Be Utilized By Offensive Security Tools

ScrapPY is a Python utility for scraping manuals, documents, and other sensitive PDFs to generate targeted wordlists that can be utilized by offensive security tools to perform brute force, forced browsing, and dictionary attacks. ScrapPY performs word frequency, entropy, and metadata analysis, and can run in full output modes to craft custom wordlists for targeted attacks. The tool dives deep to discover keywords and phrases leading to potential passwords or hidden directories, outputting to a text file that is readable by tools such as Hydra, Dirb, and Nmap. Expedite initial access, vulnerability discovery, and lateral movement with ScrapPY!

Install:

Download Repository:

$ mkdir ScrapPY
$ cd ScrapPY/
$ sudo git clone https://github.com/RoseSecurity/ScrapPY.git

Install Dependencies:

$ pip3 install -r requirements.txt

ScrapPY Usage:

usage: ScrapPY.py [-h] [-f FILE] [-m {word-frequency,full,metadata,entropy}] [-o OUTPUT]

Output metadata of document:

$ python3 ScrapPY.py -f example.pdf -m metadata

Output top 100 frequently used keywords to a file name Top_100_Keywords.txt:

$ python3 ScrapPY.py -f example.pdf -m word-frequency -o Top_100_Keywords.txt

Output all keywords to default ScrapPY.txt file:

$ python3 ScrapPY.py -f example.pdf

Output top 100 keywords with highest entropy rating:

$ python3 ScrapPY.py -f example.pdf -m entropy

ScrapPY Output:

# ScrapPY outputs the ScrapPY.txt file or specified name file to the directory in which the tool was ran. To view the first fifty lines of the file, run this command:

$ head -50 ScrapPY.txt

# To see how many words were generated, run this command:

$ wc -l ScrapPY.txt

Integration with Offensive Security Tools:

Easily integrate with tools such as Dirb to expedite the process of discovering hidden subdirectories:

root@RoseSecurity:~# dirb http://192.168.1.123/ /root/ScrapPY/ScrapPY.txt

-----------------
DIRB v2.21
By The Dark Raver
-----------------

START_TIME: Fri May 16 13:41:45 2014
URL_BASE: http://192.168.1.123/
WORDLIST_FILES: /root/ScrapPY/ScrapPY.txt

-----------------

GENERATED WORDS: 4592

---- Scanning URL: http://192.168.1.123/ ----
==> DIRECTORY: http://192.168.1.123/vi/
+ http://192.168.1.123/programming (CODE:200|SIZE:2726)
+ http://192.168.1.123/s7-logic/ (CODE:403|SIZE:1122)
==> DIRECTORY: http://192.168.1.123/config/
==> DIRECTORY: http://192.168.1.123/docs/
==> DIRECTORY: http://192.168.1.123/external/

Utilize ScrapPY with Hydra for advanced brute force attacks:

root@RoseSecurity:~# hydra -l root -P /root/ScrapPY/ScrapPY.txt -t 6 ssh://192.168.1.123
Hydra v7.6 (c)2013 by van Hauser/THC & David Maciejak - for legal purposes only

Hydra (http://www.thc.org/thc-hydra) starting at 2014-05-19 07:53:33
[DATA] 6 tasks, 1 server, 1003 login tries (l:1/p:1003), ~167 tries per task
[DATA] attacking service ssh on port 22

Enhance Nmap scripts with ScrapPY wordlists:

nmap -p445 --script smb-brute.nse --script-args userdb=users.txt,passdb=ScrapPY.txt 192.168.1.123

Future Development:

• Allow for custom output file naming and increased verbosity
• Integrate different modes of operation including word frequency analysis
• Allow for metadata analysis
• Search for high-entropy data
• Search for path-like data
• Implement image OCR to enumerate data from images in PDFs
• Allow for processing of multiple PDFs

Polaris - Validation Of Best Practices In Your Kubernetes Clusters

Polaris is an open source policy engine for Kubernetes that validates and remediates resource configuration. It includes 30+ built in configuration policies, as well as the ability to build custom policies with JSON Schema. When run on the command line or as a mutating webhook, Polaris can automatically remediate issues based on policy criteria.

Polaris can be run in three different modes:

• As a dashboard - Validate Kubernetes resources against policy-as-code.
• As an admission controller - Automatically reject or modify workloads that don't adhere to your organization's policies.
• As a command-line tool - Incorporate policy-as-code into the CI/CD process to test local YAML files.

Documentation

Check out the documentation at docs.fairwinds.com

Join the Fairwinds Open Source Community

The goal of the Fairwinds Community is to exchange ideas, influence the open source roadmap, and network with fellow Kubernetes users. Chat with us on Slack or join the user group to get involved!

Other Projects from Fairwinds

Enjoying Polaris? Check out some of our other projects:

• Goldilocks - Right-size your Kubernetes Deployments by compare your memory and CPU settings against actual usage
• Pluto - Detect Kubernetes resources that have been deprecated or removed in future versions
• Nova - Check to see if any of your Helm charts have updates available
• rbac-manager - Simplify the management of RBAC in your Kubernetes clusters

Or check out the full list

Fairwinds Insights

If you're interested in running Polaris in multiple clusters, tracking the results over time, integrating with Slack, Datadog, and Jira, or unlocking other functionality, check out Fairwinds Insights, a platform for auditing and enforcing policy in Kubernetes clusters.