Login
Bugsy is a command-line interface (CLI) tool that provides automatic security vulnerability remediation for your code. It is the community edition version of Mobb, the first vendor-agnostic automated security vulnerability remediation tool. Bugsy is designed to help developers quickly identify and fix security vulnerabilities in their code.
Mobb is the first vendor-agnostic automatic security vulnerability remediation tool. It ingests SAST results from Checkmarx, CodeQL (GitHub Advanced Security), OpenText Fortify, and Snyk and produces code fixes for developers to review and commit to their code.
Bugsy has two modes - Scan (no SAST report needed) & Analyze (the user needs to provide a pre-generated SAST report from one of the supported SAST tools).
Scan
Analyze
This is a community edition version that only analyzes public GitHub repositories. Analyzing private repositories is allowed for a limited amount of time. Bugsy does not detect any vulnerabilities in your code, it uses findings detected by the SAST tools mentioned above.
You can simply run Bugsy from the command line, using npx:
CATSploit is an automated penetration testing tool using Cyber Attack Techniques Scoring (CATS) method that can be used without pentester. Currently, pentesters implicitly made the selection of suitable attack techniques for target systems to be attacked. CATSploit uses system configuration information such as OS, open ports, software version collected by scanner and calculates a score value for capture eVc and detectability eVd of each attack techniques for target system. By selecting the highest score values, it is possible to select the most appropriate attack technique for the target system without hack knack(professional pentesterโs skill) .
CATSploit automatically performs penetration tests in the following sequence:
Information gathering and prior information input First, gathering information of target systems. CATSploit supports nmap and OpenVAS to gather information of target systems. CATSploit also supports prior information of target systems if you have.
Calculating score value of attack techniques Using information obtained in the previous phase and attack techniques database, evaluation values of capture (eVc) and detectability (eVd) of each attack techniques are calculated. For each target computer, the values of each attack technique are calculated.
Selection of attack techniques by using scores and make attack scenario Select attack techniques and create attack scenarios according to pre-defined policies. For example, for a policy that prioritized hard-to-detect, the attack techniques with the lowest eVd(Detectable Score) will be selected.
Execution of attack scenario CATSploit executes the attack techniques according to attack scenario constructed in the previous phase. CATSploit uses Metasploit as a framework and Metasploit API to execute actual attacks.
CATSploit has the following prerequisites:
For Metasploit, Nmap and OpenVAS, it is assumed to be installed with the Kali Distribution.
To install the latest version of CATSploit, please use the following commands:
FreshRSS 
$ git clone https://github.com/catsploit/catsploit.git
$ cd catsploit
$ git clone https://github.com/catsploit/cats-helper.git
$ sudo ./setup.sh
CATSploit is a server-client configuration, and the server reads the configuration JSON file at startup. In config.json, the following fields should be modified for your environment.
(*) Adjust the number according to the specs of your machine.
To start the server, execute the following command:
$ python cats_server.py -c [CONFIG_FILE]
Next, prepare another console, start the client program, and initiate a connection to the server.
$ python catsploit.py -s [SOCKET_PATH]
After successfully connecting to the server and initializing it, the session will start.
_________ ___________ __ _ __
/ ____/ |/_ __/ ___/____ / /___ (_) /_
/ / / /| | / / \__ \/ __ \/ / __ \/ / __/
/ /___/ ___ |/ / ___/ / /_/ / / /_/ / / /_
\____/_/ |_/_/ /____/ .___/_/\____/_/\__/
/_/
[*] Connecting to cats-server
[*] Done.
[*] Initializing server
[*] Done.
catsploit>
The client can execute a variety of commands. Each command can be executed with -h option to display the format of its arguments.
usage: [-h] {host,scenario,scan,plan,attack,post,reset,help,exit} ...
positional arguments:
{host,scenario,scan,plan,attack,post,reset,help,exit}
options:
-h, --help show this help message and exit
I've posted the commands and options below as well for reference.
host list:
show information about the hosts
usage: host list [-h]
options:
-h, --help show this help message and exit
host detail:
show more information about one host
usage: host detail [-h] host_id
positional arguments:
host_id ID of the host for which you want to show information
options:
-h, --help show this help message and exit
scenario list:
show information about the scenarios
usage: scenario list [-h]
options:
-h, --help show this help message and exit
scenario detail:
show more information about one scenario
usage: scenario detail [-h] scenario_id
positional arguments:
scenario_id ID of the scenario for which you want to show information
options:
-h, --help show this help message and exit
scan:
run network-scan and security-scan
usage: scan [-h] [--port PORT] targe t_host [target_host ...]
positional arguments:
target_host IP address to be scanned
options:
-h, --help show this help message and exit
--port PORT ports to be scanned
plan:
planning attack scenarios
usage: plan [-h] src_host_id dst_host_id
positional arguments:
src_host_id originating host
dst_host_id target host
options:
-h, --help show this help message and exit
attack:
execute attack scenario
usage: attack [-h] scenario_id
positional arguments:
scenario_id ID of the scenario you want to execute
options:
-h, --help show this help message and exit
post find-secret:
find confidential information files that can be performed on the pwned host
usage: post find-secret [-h] host_id
positional arguments:
host_id ID of the host for which you want to find confidential information
op tions:
-h, --help show this help message and exit
reset:
reset data on the server
usage: reset [-h] {system} ...
positional arguments:
{system} reset system
options:
-h, --help show this help message and exit
exit:
exit CATSploit
usage: exit [-h]
options:
-h, --help show this help message and exit
In this example, we use CATSploit to scan network, plan the attack scenario, and execute the attack.
catsploit> scan 192.168.0.0/24
Network Scanning ... 100%
[*] Total 2 hosts were discovered.
Vulnerability Scanning ... 100%
[*] Total 14 vulnerabilities were discovered.
catsploit> host list
โโโโโโโโโโโโณโโโโโโโโโโโโโโโโโณโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโ
โ hostID โ IP โ Hostname โ Platform โ Pwned โ
โกโโโโโโ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ attacker โ 0.0.0.0 โ kali โ kali 2022.4 โ True โ
โ h_exbiy6 โ 192.168.0.10 โ โ Linux 3.10 - 4.11 โ False โ
โ h_nhqyfq โ 192.168.0.20 โ โ Microsoft Windows 7 SP1 โ False โ
โโโโโโโโโโโโด โโโโโโโโโโโโโโโโดโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโ
catsploit> host detail h_exbiy6
โโโโโโโโโโโโณโโโโโโโโโโโโโโโณโโโโโโโโโโโณโโโโโโโโโโโโโโโณโโโโโโโโ
โ hostID โ IP โ Hostname โ Platform โ Pwned โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ h_exbiy6 โ 192.168.0.10 โ ubuntu โ ubuntu 14.04 โ False โ
โโโโโโโโโโโโดโโโโโโโโโโโโโโโดโโโโโโโโโโโดโโโโโโโโโโโโโโโดโ โโโโโโ
[IP address]
โโโโโโโโโโโโโโโโณโโโโโโโโโโโณโโโโโโโณโโโโโโโโโโโโโ
โ ipv4 โ ipv4mask โ ipv6 โ ipv6prefix โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ 192.168.0.10 โ โ โ โ
โโโโโโโโโโโโโ โโดโโโโโโโโโโโดโโโโโโโดโโโโโโโโโโโโโ
[Open ports]
โโโโโโโโโโโโโโโโณโโโโโโโโณโโโโโโโณโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ ip โ proto โ port โ service โ product โ version โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ 192.168.0.10 โ tcp โ 21 โ ftp โ ProFTPD โ 1.3.5 โ
โ 192.168.0.10 โ tcp โ 22 โ ssh โ OpenSSH โ 6.6.1p1 Ubuntu 2ubuntu2.10 โ
โ 192.168.0.10 โ tcp โ 80 โ http โ Apache httpd โ 2.4.7 โ
โ 192.168.0.10 โ tcp โ 445 โ netbios-ssn โ Samba smbd โ 3.X - 4.X โ
โ 192.168.0.10 โ tcp โ 631 โ ipp โ CUPS โ 1.7 โ
โโโโโโโโโโโโโโโโดโโโโโโโโดโโโโโโโดโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
[Vulnerabilities]
โโโโโโโโโโโโโโโโณโโโโโโโโณโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโ
โ ip โ proto โ port โ vuln_name โ cve โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ 192.168.0.10 โ tcp โ 0 โ TCP Timestamps Information Disclosure โ N/A โ
โ 192.168.0.10 โ tcp โ 21 โ FTP Unencrypted Cleartext Login โ N/A โ
โ 192.168.0.10 โ tcp โ 22 โ Weak MAC Algorithm(s) Supported (SSH) โ N/A โ
โ 192.168.0.10 โ tcp โ 22 โ Weak Encryption Algorithm(s) Supported (SSH) โ N/A โ
โ 192.168.0.10 โ tcp โ 22 โ Weak Host Key Algorithm(s) (SSH) โ N/A โ
โ 192.168.0.10 โ tcp โ 22 โ Weak Key Exchange (KEX) Algorithm(s) Supported (SSH) โ N/A โ
โ 192.168.0.10 โ tcp โ 80 โ Test HTTP dangerous methods โ N/A โ
โ 192.168.0.10 โ tcp โ 80 โ Drupal Core SQLi Vulnerability (SA-CORE-2014-005) - Active Check โ CVE-2014-3704 โ
โ 192.168.0.10 โ tcp โ 80 โ Drupal Coder RCE Vulnerability (SA-CONTRIB-2016-039) - Active Check โ N/A โ
โ 192.168.0.10 โ tcp โ 80 โ Sensitive File Disclosure (HTTP) โ N/A โ
โ 192.168.0.10 โ tcp โ 80 โ Unprotected Web App / Device Installers (HTTP) โ N/A โ
โ 192.168.0.10 โ tcp โ 80 โ Cleartext Transmission of Sensitive Information via HTTP โ N/A โ
โ 192.168.0.10 โ tcp โ 80 โ jQuery < 1.9.0 XSS Vulnerability โ CVE-2012-6708 โ
โ 192.168.0.10 โ tcp โ 80 โ jQuery < 1.6.3 XSS Vulnerability โ CVE-2011-4969 โ
โ 192.168.0.10 โ tcp โ 80 โ Drupal 7.0 Information Disclosure Vulnerability - Active Check โ CVE-2011-3730 โ
โ 192.168.0.10 โ tcp โ 631 โ SSL/TLS: Report Vulnerable Cipher Suites for HTTPS โ CVE-2016-2183 โ
โ 192.168.0.10 โ tcp โ 631 โ SSL/TLS: Report Vulnerable Cipher Suites for HTTPS โ CVE-2016-6329 โ
โ 192.168.0.10 โ tcp โ 631 โ SSL/TLS: Report Vulnerable Cipher Suites for HTTPS โ CVE-2020-12872 โ
โ 192.168.0.10 โ tcp โ 631 โ SSL/TLS: Deprecated TLSv1.0 and TLSv1.1 Protocol Detection โ CVE-2011-3389 โ
โ 192.168.0.10 โ tcp โ 631 โ SSL/TLS: Deprecated TLSv1.0 and TLSv1.1 Protocol Detection โ CVE-2015-0204 โ
โโโโโโโโโโโโโโโโดโโโโโโโโดโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโ& #9472;โโโโโโโโโโโโโ
[Users]
โโโโโโโโโโโโโณโโโโโโโโ
โ user name โ group โ
โกโโโโโโโโโโโโโโโโโโโโฉ
โโโโโโโโโโโโโดโโโโโโโโ
catsploit> plan attacker h_exbiy6
Planning attack scenario...100%
[*] Done. 15 scenarios was planned.
[*] To check each scenario, try 'scenario list' and/or 'scenario detail'.
catsploit> scenario list
โโโโโโโโโโโโโโโณโโโโโ โโโโโโโโณโโโโโโโโโโโโโโโโโณโโโโโโโโณโโโโโโโโณโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ scenario id โ src host ip โ target host ip โ eVc โ eVd โ steps โ first attack step โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโγ 3;โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ 3d3ivc โ 0.0.0.0 โ 192.168.0.10 โ 1.0 โ 32.0 โ 1 โ exploit/multi/http/jenkins_sโฆ โ
โ 5gnsvh โ 0.0.0.0 โ 192.168.0.10 โ 1.0 โ 53.76 โ 2 โ exploit/multi/http/jenkins_sโฆ โ
โ 6nlxyc โ 0.0.0.0 โ 192.168.0.10 โ 0.0 โ 48.32 โ 2 โ exploit/multi/http/jenkins_sโฆ โ
โ 8jos4z โ 0.0.0.0 โ 192.168.0.1 0 โ 0.7 โ 72.8 โ 2 โ exploit/multi/http/jenkins_sโฆ โ
โ 8kmmts โ 0.0.0.0 โ 192.168.0.10 โ 0.0 โ 32.0 โ 1 โ exploit/multi/elasticsearch/โฆ โ
โ agjmma โ 0.0.0.0 โ 192.168.0.10 โ 0.0 โ 24.0 โ 1 โ exploit/windows/http/manageeโฆ โ
โ joglhf โ 0.0.0.0 โ 192.168.0.10 โ 70.0 โ 60.0 โ 1 โ auxiliary/scanner/ssh/ssh_loโฆ โ
โ rmgrof โ 0.0.0.0 โ 192.168.0.10 โ 100.0 โ 32.0 โ 1 โ exploit/multi/http/drupal_drโฆ โ
โ xuowzk โ 0.0.0.0 โ 192.168.0.10 โ 0.0 โ 24.0 โ 1 โ exploit/multi/http/struts_dmโฆ โ
โ yttv51 โ 0.0.0.0 โ 192.168.0.10 โ 0.01 โ 53.76 โ 2 โ exploit/multi/http/jenkins_sโฆ โ
โ znv76x โ 0.0.0.0 โ 192.168.0.10 โ 0.01 โ 53.76 โ 2 โ exploit/multi/http/jenkins_sโฆ โ
โโโโโโโโโโโโโโโดโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโดโโโโโโโโดโโโโโโโโดโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
catsploit> scenario detail rmgrof
โโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโณโโโโโโโโณโโโโโโโ
โ src host ip โ target host ip โ eVc โ eVd โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ 0.0.0.0 โ 192.168.0.10 โ 100.0 โ 32.0 โ
โโโโโโโโโโโโโโโดโโโโโโโโ โโโโโโโโดโโโโโโโโดโโโโโโโ
[Steps]
โโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโ
โ # โ step โ params โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ 1 โ exploit/multi/http/drupal_drupageddon โ RHOSTS: 192.168.0.10 โ
โ โ โ LHOST: 192.168.10.100 โ
โโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโ
catsploit> attack rmgrof
> ~> ~
> Metasploit Console Log
> ~
> ~
[+] Attack scenario succeeded!
catsploit> exit
Bye.
All informations and codes are provided solely for educational purposes and/or testing your own systems.
For any inquiry, please contact the email address as follows:
catsploit@nk.MitsubishiElectric.co.jp
MetaHub is an automated contextual security findings enrichment and impact evaluation tool for vulnerability management. You can use it with AWS Security Hub or any ASFF-compatible security scanner. Stop relying on useless severities and switch to impact scoring definitions based on YOUR context.
MetaHub is an open-source security tool for impact-contextual vulnerability management. It can automate the process of contextualizing security findings based on your environment and your needs: YOUR context, identifying ownership, and calculate an impact scoring based on it that you can use for defining prioritization and automation. You can use it with AWS Security Hub or any ASFF security scanners (like Prowler).
MetaHub describe your context by connecting to your affected resources in your affected accounts. It can describe information about your AWS account and organization, the affected resources tags, the affected CloudTrail events, your affected resource configurations, and all their associations: if you are contextualizing a security finding affecting an EC2 Instance, MetaHub will not only connect to that instance itself but also its IAM Roles; from there, it will connect to the IAM Policies associated with those roles. It will connect to the Security Groups and analyze all their rules, the VPC and the Subnets where the instance is running, the Volumes, the Auto Scaling Groups, and more.
After fetching all the information from your context, MetaHub will evaluate certain important conditions for all your resources: exposure, access, encryption, status, environment and application. Based on those calculations and in addition to the information from the security findings affecting the resource all together, MetaHub will generate a Scoring for each finding.
Check the following dashboard generated by MetaHub. You have the affected resources, grouping all the security findings affecting them together and the original severity of the finding. After that, you have the Impact Score and all the criteria MetaHub evaluated to generate that score. All this information is filterable, sortable, groupable, downloadable, and customizable.
You can rely on this Impact Score for prioritizing findings (where should you start?), directing attention to critical issues, and automating alerts and escalations.
MetaHub can also filter, deduplicate, group, report, suppress, or update your security findings in automated workflows. It is designed for use as a CLI tool or within automated workflows, such as AWS Security Hub custom actions or AWS Lambda functions.
The following is the JSON output for a an EC2 instance; see how MetaHub organizes all the information about its context together, under associations, config, tags, account cloudtrail, and impact
In MetaHub, context refers to information about the affected resources like their configuration, associations, logs, tags, account, and more.
MetaHub doesn't stop at the affected resource but analyzes any associated or attached resources. For instance, if there is a security finding on an EC2 instance, MetaHub will not only analyze the instance but also the security groups attached to it, including their rules. MetaHub will examine the IAM roles that the affected resource is using and the policies attached to those roles for any issues. It will analyze the EBS attached to the instance and determine if they are encrypted. It will also analyze the Auto Scaling Groups that the instance is associated with and how. MetaHub will also analyze the VPC, Subnets, and other resources associated with the instance.
The Context module has the capability to retrieve information from the affected resources, affected accounts, and every associated resources. The context module has five main parts: config (which includes associations as well), tags, cloudtrail, and account. By default config and tags are enabled, but you can change this behavior using the option --context (for enabling all the context modules you can use --context config tags cloudtrail account). The output of each enabled key will be added under the affected resource.
Under the config key, you can find anyting related to the configuration of the affected resource. For example, if the affected resource is an EC2 Instance, you will see keys like private_ip, public_ip, or instance_profile.
You can filter your findings based on Config outputs using the option: --mh-filters-config <key> {True/False}. See Config Filtering.
Under the associations key, you will find all the associated resources of the affected resource. For example, if the affected resource is an EC2 Instance, you will find resources like: Security Groups, IAM Roles, Volumes, VPC, Subnets, Auto Scaling Groups, etc. Each time MetaHub finds an association, it will connect to the associated resource again and fetch its own context.
Associations are key to understanding the context and impact of your security findings as their exposure.
You can filter your findings based on Associations outputs using the option: --mh-filters-config <key> {True/False}. See Config Filtering.
MetaHub relies on AWS Resource Groups Tagging API to query the tags associated with your resources.
Note that not all AWS resource type supports this API. You can check supported services.
Tags are a crucial part of understanding your context. Tagging strategies often include:
If you follow a proper tagging strategy, you can filter and generate interesting outputs. For example, you could list all findings related to a specific team and provide that data directly to that team.
You can filter your findings based on Tags outputs using the option: --mh-filters-tags TAG=VALUE. See Tags Filtering
Under the key cloudtrail, you will find critical Cloudtrail events related to the affected resource, such as creating events.
The Cloudtrail events that we look for are defined by resource type, and you can add, remove or change them by editing the configuration file resources.py.
For example for an affected resource of type Security Group, MetaHub will look for the following events:
CreateSecurityGroup: Security Group Creation eventAuthorizeSecurityGroupIngress: Security Group Rule Authorization event.Under the key account, you will find information about the account where the affected resource is runnning, like if it's part of an AWS Organizations, information about their contacts, etc.
MetaHub also focuses on ownership detection. It can determine the owner of the affected resource in various ways. This information can be used to automatically assign a security finding to the correct owner, escalate it, or make decisions based on this information.
An automated way to determine the owner of a resource is critical for security teams. It allows them to focus on the most critical issues and escalate them to the right people in automated workflows. But automating workflows this way, it is only viable if you have a reliable way to define the impact of a finding, which is why MetaHub also focuses on impact.
The impact module in MetaHub focuses on generating a score for each finding based on the context of the affected resource and all the security findings affecting them. For the context, we define a series of evaluated criteria; you can add, remove, or modify these criteria based on your needs. The Impact criteria are combined with a metric generated based on all the Security Findings affecting the affected resource and their severities.
The following are the impact criteria that MetaHub evaluates by default:
Exposure evaluates the how the the affected resource is exposed to other networks. For example, if the affected resource is public, if it is part of a VPC, if it has a public IP or if it is protected by a firewall or a security group.
| Possible Statuses | Value | Description |
|---|---|---|
| ๏ด effectively-public | 100% | The resource is effectively public from the Internet. |
| ๏ restricted-public | 40% | The resource is public, but there is a restriction like a Security Group. |
| ๏ unrestricted-private | 30% | The resource is private but unrestricted, like an open security group. |
| ๏ launch-public | 10% | These are resources that can launch other resources as public. For example, an Auto Scaling group or a Subnet. |
| ๏ข restricted | 0% | The resource is restricted. |
| ๏ต unknown | - | The resource couldn't be checked |
Access evaluates the resource policy layer. MetaHub checks every available policy including: IAM Managed policies, IAM Inline policies, Resource Policies, Bucket ACLS, and any association to other resources like IAM Roles which its policies are also analyzed . An unrestricted policy is not only an itsue itself of that policy, it afected any other resource which is using it.
| Possible Statuses | Value | Description |
|---|---|---|
| ๏ด unrestricted | 100% | The principal is unrestricted, without any condition or restriction. |
| ๏ด untrusted-principal | 70% | The principal is an AWS Account, not part of your trusted accounts. |
| ๏ unrestricted-principal | 40% | The principal is not restricted, defined with a wildcard. It could be conditions restricting it or other restrictions like s3 public blocks. |
| ๏ cross-account-principal | 30% | The principal is from another AWS account. |
| ๏ unrestricted-actions | 30% | The actions are defined using wildcards. |
| ๏ dangerous-actions | 30% | Some dangerous actions are defined as part of this policy. |
| ๏ unrestricted-service | 10% | The policy allows an AWS service as principal without restriction. |
| ๏ข restricted | 0% | The policy is restricted. |
| ๏ต unknown | - | The policy couldn't be checked. |
Encryption evaluate the different encryption layers based on each resource type. For example, for some resources it evaluates if at_rest and in_transit encryption configuration are both enabled.
| Possible Statuses | Value | Description |
|---|---|---|
| ๏ด unencrypted | 100% | The resource is not fully encrypted. |
| ๏ข encrypted | 0% | The resource is fully encrypted including any of it's associations. |
| ๏ต unknown | - | The resource encryption couldn't be checked. |
Status evaluate the status of the affected resource in terms of attachment or functioning. For example, for an EC2 Instance we evaluate if the resource is running, stopped, or terminated, but for resources like EBS Volumes and Security Groups, we evaluate if those resources are attached to any other resource.
| Possible Statuses | Value | Description |
|---|---|---|
| ๏ attached | 100% | The resource supports attachment and is attached. |
| ๏ running | 100% | The resource supports running and is running. |
| ๏ enabled | 100% | The resource supports enabled and is enabled. |
| ๏ข not-attached | 0% | The resource supports attachment, and it is not attached. |
| ๏ข not-running | 0% | The resource supports running and it is not running. |
| ๏ข not-enabled | 0% | The resource supports enabled and it is not enabled. |
| ๏ต unknown | - | The resource couldn't be checked for status. |
Environment evaluates the environment where the affected resource is running. By default, MetaHub defines 3 environments: production, staging, and development, but you can add, remove, or modify these environments based on your needs. MetaHub evaluates the environment based on the tags of the affected resource, the account id or the account alias. You can define your own environemnts definitions and strategy in the configuration file (See Customizing Configuration).
| Possible Statuses | Value | Description |
|---|---|---|
| ๏ production | 100% | It is a production resource. |
| ๏ข staging | 30% | It is a staging resource. |
| ๏ข development | 0% | It is a development resource. |
| ๏ต unknown | - | The resource couldn't be checked for enviroment. |
Application evaluates the application that the affected resource is part of. MetaHub relies on the AWS myApplications feature, which relies on the Tag awsApplication, but you can extend this functionality based on your context for example by defining other tags you use for defining applications or services (like Service or any other), or by relying on account id or alias. You can define your application definitions and strategy in the configuration file (See Customizing Configuration).
| Possible Statuses | Value | Description |
|---|---|---|
| ๏ต unknown | - | The resource couldn't be checked for application. |
As part of the impact score calculation, we also evaluate the total ammount of security findings and their severities affecting the resource. We use the following formula to calculate this metric:
(SUM of all (Finding Severity / Highest Severity) with a maximum of 1)For example, if the affected resource has two findings affecting it, one with HIGH and another with LOW severity, the Impact Findings Score will be:
SUM(HIGH (3) / CRITICAL (4) + LOW (0.5) / CRITICAL (4)) = 0.875MetaHub reads your security findings from AWS Security Hub or any ASFF-compatible security scanner. It then queries the affected resources directly in the affected account to provide additional context. Based on that context, it calculates it's impact. Finally, it generates different outputs based on your needs.
Some use cases for MetaHub include:
MetaHub provides a range of ways to list and manage security findings for investigation, suppression, updating, and integration with other tools or alerting systems. To avoid Shadowing and Duplication, MetaHub organizes related findings together when they pertain to the same resource. For more information, refer to Findings Aggregation
MetaHub queries the affected resources directly in the affected account to provide additional context using the following options:
MetaHub supports filters on top of these context* outputs to automate the detection of other resources with the same issues. You can filter security findings affecting resources tagged in a certain way (e.g., Environment=production) and combine this with filters based on Config or Associations, like, for example, if the resource is public, if it is encrypted, only if they are part of a VPC, if they are using a specific IAM role, and more. For more information, refer to Config filters and Tags filters for more information.
But that's not all. If you are using MetaHub with Security Hub, you can even combine the previous filters with the Security Hub native filters (AWS Security Hub filtering). You can filter the same way you would with the AWS CLI utility using the option --sh-filters, but in addition, you can save and re-use your filters as YAML files using the option --sh-template.
If you prefer, With MetaHub, you can back enrich your findings directly in AWS Security Hub using the option --enrich-findings. This action will update your AWS Security Hub findings using the field UserDefinedFields. You can then create filters or Insights directly in AWS Security Hub and take advantage of the contextualization added by MetaHub.
When investigating findings, you may need to update security findings altogether. MetaHub also allows you to execute bulk updates to AWS Security Hub findings, such as changing Workflow Status using the option --update-findings. As an example, you identified that you have hundreds of security findings about public resources. Still, based on the MetaHub context, you know those resources are not effectively public as they are protected by routing and firewalls. You can update all the findings for the output of your MetaHub query with one command. When updating findings using MetaHub, you also update the field Note of your finding with a custom text for future reference.
MetaHub supports different Output Modes, some of them json based like json-inventory, json-statistics, json-short, json-full, but also powerfull html, xlsx and csv. These outputs are customizable; you can choose which columns to show. For example, you may need a report about your affected resources, adding the tag Owner, Service, and Environment and nothing else. Check the configuration file and define the columns you need.
MetaHub supports multi-account setups. You can run the tool from any environment by assuming roles in your AWS Security Hub master account and your child/service accounts where your resources live. This allows you to fetch aggregated data from multiple accounts using your AWS Security Hub multi-account implementation while also fetching and enriching those findings with data from the accounts where your affected resources live based on your needs. Refer to Configuring Security Hub for more information.
MetaHub uses configuration files that let you customize some checks behaviors, default filters, and more. The configuration files are located in lib/config/.
Things you can customize:
lib/config/configuration.py: This file contains the default configuration for MetaHub. You can change the default filters, the default output modes, the environment definitions, and more.
lib/config/impact.py: This file contains the values and it's weights for the impact formula criteria. You can modify the values and the weights based on your needs.
lib/config/reources.py: This file contains definitions for every resource type, like which CloudTrail events to look for.
MetaHub is a Python3 program. You need to have Python3 installed in your system and the required Python modules described in the file requirements.txt.
Requirements can be installed in your system manually (using pip3) or using a Python virtual environment (suggested method).
git clone git@github.com:gabrielsoltz/metahub.git
cd metahub
python3 -m venv venv/metahub
source venv/metahub/bin/activate
pip3 install -r requirements.txt
./metahub -h
deactivate
Next time, you only need steps 4 and 6 to use the program.
Alternatively, you can run this tool using Docker.
MetaHub is also available as a Docker image. You can run it directly from the public Docker image or build it locally.
The available tagging for MetaHub containers are the following:
latest: in sync with master branch<x.y.z>: you can find the releases here
stable: this tag always points to the latest release.For running from the public registry, you can run the following command:
docker run -ti public.ecr.aws/n2p8q5p4/metahub:latest ./metahub -hIf you are already logged into the AWS host machine, you can seamlessly use the same credentials within a Docker container. You can achieve this by either passing the necessary environment variables to the container or by mounting the credentials file.
For instance, you can run the following command:
docker run -e AWS_DEFAULT_REGION -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN -ti public.ecr.aws/n2p8q5p4/metahub:latest ./metahub -hOn the other hand, if you are not logged in on the host machine, you will need to log in again from within the container itself.
Or you can also build it locally:
git clone git@github.com:gabrielsoltz/metahub.git
cd metahub
docker build -t metahub .
docker run -ti metahub ./metahub -hMetaHub is Lambda/Serverless ready! You can run MetaHub directly on an AWS Lambda function without any additional infrastructure required.
Running MetaHub in a Lambda function allows you to automate its execution based on your defined triggers.
Terraform code is provided for deploying the Lambda function and all its dependencies.
The terraform code for deploying the Lambda function is provided under the terraform/ folder.
Just run the following commands:
cd terraform
terraform init
terraform applyThe code will create a zip file for the lambda code and a zip file for the Python dependencies. It will also create a Lambda function and all the required resources.
You can customize MetaHub options for your lambda by editing the file lib/lambda.py. You can change the default options for MetaHub, such as the filters, the Meta* options, and more.
Terraform will create the minimum required permissions for the Lambda function to run locally (in the same account). If you want your Lambda to assume a role in other accounts (for example, you will need this if you are executing the Lambda in the Security Hub master account that is aggregating findings from other accounts), you will need to specify the role to assume, adding the option --mh-assume-role in the Lambda function configuration (See previous step) and adding the corresponding policy to allow the Lambda to assume that role in the lambda role.
MetaHub can be run as a Security Hub Custom Action. This allows you to run MetaHub directly from the Security Hub console for a selected finding or for a selected set of findings.
The custom action will then trigger a Lambda function that will run MetaHub for the selected findings. By default, the Lambda function will run MetaHub with the option --enrich-findings, which means that it will update your finding back with MetaHub outputs. If you want to change this, see Customize Lambda behavior
You need first to create the Lambda function and then create the custom action in Security Hub.
For creating the lambda function, follow the instructions in the Run with Lambda section.
For creating the AWS Security Hub custom action:
For example, you can use aws configure option.
aws configureOr you can export your credentials to the environment.
export AWS_DEFAULT_REGION="us-east-1"
export AWS_ACCESS_KEY_ID= "ASXXXXXXX"
export AWS_SECRET_ACCESS_KEY= "XXXXXXXXX"
export AWS_SESSION_TOKEN= "XXXXXXXXX"If you are running MetaHub for a single AWS account setup (AWS Security Hub is not aggregating findings from different accounts), you don't need to use any additional options; MetaHub will use the credentials in your environment. Still, if your IAM design requires it, it is possible to log in and assume a role in the same account you are logged in. Just use the options --sh-assume-role to specify the role and --sh-account with the same AWS Account ID where you are logged in.
--sh-region: The AWS Region where Security Hub is running. If you don't specify a region, it will use the one configured in your environment. If you are using AWS Security Hub Cross-Region aggregation, you should use that region as the --sh-region option so that you can fetch all findings together.
--sh-account and --sh-assume-role: The AWS Account ID where Security Hub is running and the AWS IAM role to assume in that account. These options are helpful when you are logged in to a different AWS Account than the one where AWS Security Hub is running or when running AWS Security Hub in a multiple AWS Account setup. Both options must be used together. The role provided needs to have enough policies to get and update findings in AWS Security Hub (if needed). If you don't specify a --sh-account, MetaHub will assume the one you are logged in.
--sh-profile: You can also provide your AWS profile name to use for AWS Security Hub. When using this option, you don't need to specify --sh-account or --sh-assume-role as MetaHub will use the credentials from the profile. If you are using --sh-account and --sh-assume-role, those options take precedence over --sh-profile.
This is the minimum IAM policy you need to read and write from AWS Security Hub. If you don't want to update your findings with MetaHub, you can remove the securityhub:BatchUpdateFindings action.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"security hub:GetFindings",
"security hub:ListFindingAggregators",
"security hub:BatchUpdateFindings",
"iam:ListAccountAliases"
],
"Resource": [
"*"
]
}
]
}If you are running MetaHub for a multiple AWS Account setup (AWS Security Hub is aggregating findings from multiple AWS Accounts), you must provide the role to assume for Context queries because the affected resources are not in the same AWS Account that the AWS Security Hub findings. The --mh-assume-role will be used to connect with the affected resources directly in the affected account. This role needs to have enough policies for being able to describe resources.
The minimum policy needed for context includes the managed policy arn:aws:iam::aws:policy/SecurityAudit and the following actions:
tag:GetResourceslambda:GetFunctionlambda:GetFunctionUrlConfigcloudtrail:LookupEventsaccount:GetAlternateContactorganizations:DescribeAccountiam:ListAccountAliasesMetaHub can read security findings directly from AWS Security Hub using its API. If you don't use Security Hub, you can use any ASFF-based scanner. Most cloud security scanners support the ASFF format. Check with them or leave an issue if you need help.
If you want to read from an input ASFF file, you need to use the options:
./metahub.py --inputs file-asff --input-asff path/to/the/file.json.asff path/to/the/file2.json.asffYou also can combine AWS Security Hub findings with input ASFF files specifying both inputs:
./metahub.py --inputs file-asff securityhub --input-asff path/to/the/file.json.asffWhen using a file as input, you can't use the option --sh-filters for filter findings, as this option relies on AWS API for filtering. You can't use the options --update-findings or --enrich-findings as those findings are not in the AWS Security Hub. If you are reading from both sources at the same time, only the findings from AWS Security Hub will be updated.
MetaHub can generate different programmatic and visual outputs. By default, all output modes are enabled: json-short, json-full, json-statistics, json-inventory, html, csv, and xlsx.
The outputs will be saved in the outputs/ folder with the execution date.
If you want only to generate a specific output mode, you can use the option --output-modes with the desired output mode.
For example, if you only want to generate the output json-short, you can use:
./metahub.py --output-modes json-shortIf you want to generate json-short, json-full and html outputs, you can use:
./metahub.py --output-modes json-short json-full htmlShow all findings titles together under each affected resource and the AwsAccountId, Region, and ResourceType:
Show all findings with all data. Findings are organized by ResourceId (ARN). For each finding, you will also get: SeverityLabel, Workflow, RecordState, Compliance, Id, and ProductArn:
Show a list of all resources with their ARN.
Show statistics for each field/value. In the output, you will see each field/value and the number of occurrences; for example, the following output shows statistics for six findings.
You can create rich HTML reports of your findings, adding your context as part of them.
HTML Reports are interactive in many ways:
You can create CSV reports of your findings, adding your context as part of them.
ย
Similar to CSV but with more formatting options.
You can customize which Context keys to unroll as columns for your HTML, CSV, and XLSX outputs using the options --output-tag-columns and --output-config-columns (as a list of columns). If the keys you specified don't exist for the affected resource, they will be empty. You can also configure these columns by default in the configuration file (See Customizing Configuration).
For example, you can generate an HTML output with Tags and add "Owner" and "Environment" as columns to your report using the:
./metahub --output-modes html --output-tag-columns Owner EnvironmentYou can filter the security findings and resources that you get from your source in different ways and combine all of them to get exactly what you are looking for, then re-use those filters to create alerts.
MetaHub supports filtering AWS Security Hub findings in the form of KEY=VALUE filtering for AWS Security Hub using the option --sh-filters, the same way you would filter using AWS CLI but limited to the EQUALS comparison. If you want another comparison, use the option --sh-template Security Hub Filtering using YAML templates.
You can check available filters in AWS Documentation
./metahub --sh-filters <KEY=VALUE>If you don't specify any filters, default filters are applied: RecordState=ACTIVE WorkflowStatus=NEW
Passing filters using this option resets the default filters. If you want to add filters to the defaults, you need to specify them in addition to the default ones. For example, adding SeverityLabel to the default filters:
./metahub --sh-filters RecordState=ACTIVE WorkflowStatus=NEWIf a value contains spaces, you should specify it using double quotes: "ProductName="Security Hub"
You can add how many different filters you need to your query and also add the same filter key with different values:
Examples:
./metaHub --sh-filters RecordState=ACTIVE WorkflowStatus=NEW SeverityLabel=CRITICAL./metaHub --sh-filters RecordState=ACTIVE WorkflowStatus=NEW SeverityLabel=CRITICAL SeverityLabel=HIGH./metaHub --sh-filters RecordState=ACTIVE WorkflowStatus=NEW SeverityLabel=CRITICAL AwsAccountId=1234567890./metahub --sh-filters RecordState=ACTIVE WorkflowStatus=NEW Title="EC2.22 Unused EC2 security groups should be removed"./metahub --sh-filters RecordState=ACTIVE WorkflowStatus=NEW ResourceType=AwsEc2SecurityGroup./metahub --sh-filters RecordState=ACTIVE WorkflowStatus=NEW ResourceId="arn:aws:ec2:eu-west-1:01234567890:security-group/sg-01234567890"./metahub --sh-filters Id="arn:aws:security hub:eu-west-1:01234567890:subscription/aws-foundational-security-best-practices/v/1.0.0/EC2.19/finding/01234567890-1234-1234-1234-01234567890"./metahub --sh-filters ComplianceStatus=FAILEDMetaHub lets you create complex filters using YAML files (templates) that you can re-use when needed. YAML templates let you write filters using any comparison supported by AWS Security Hub like "EQUALS' | 'PREFIX' | 'NOT_EQUALS' | 'PREFIX_NOT_EQUALS". You can call your YAML file using the option --sh-template <<FILE>>.
You can find examples under the folder templates
./metaHub --sh-template templates/default.ymlMetaHub supports Config filters (and associations) using KEY=VALUE where the value can only be True or False using the option --mh-filters-config. You can use as many filters as you want and separate them using spaces. If you specify more than one filter, you will get all resources that match all filters.
Config filters only support True or False values:
True or with data.False or without data.Config filters run after AWS Security Hub filters:
--sh-filters (or the default ones).--mh-filters-config, so it's a subset of the resources from point 1.Examples:
ResourceType=AwsEc2SecurityGroup) with AWS Security Hub findings that are ACTIVE and NEW (RecordState=ACTIVE WorkflowStatus=NEW) only if they are associated to Network Interfaces (network_interfaces=True):./metahub --sh-filters RecordState=ACTIVE WorkflowStatus=NEW ResourceType=AwsEc2SecurityGroup --mh-filters-config network_interfaces=TrueResourceType=AwsS3Bucket) only if they are public (public=True):./metahub --sh-filters ResourceType=AwsS3Bucket --mh-filters-config public=FalseMetaHub supports Tags filters in the form of KEY=VALUE where KEY is the Tag name and value is the Tag Value. You can use as many filters as you want and separate them using spaces. Specifying multiple filters will give you all resources that match at least one filter.
Tags filters run after AWS Security Hub filters:
--sh-filters (or the default ones).--mh-filters-tags, so it's a subset of the resources from point 1.Examples:
ResourceType=AwsEc2SecurityGroup) with AWS Security Hub findings that are ACTIVE and NEW (RecordState=ACTIVE WorkflowStatus=NEW) only if they are tagged with a tag Environment and value Production:./metahub --sh-filters RecordState=ACTIVE WorkflowStatus=NEW ResourceType=AwsEc2SecurityGroup --mh-filters-tags Environment=ProductionYou can use MetaHub to update your AWS Security Hub Findings workflow status (NOTIFIED, NEW, RESOLVED, SUPPRESSED) with a single command. You will use the --update-findings option to update all the findings from your MetaHub query. This means you can update one, ten, or thousands of findings using only one command. AWS Security Hub API is limited to 100 findings per update. Metahub will split your results into 100 items chucks to avoid this limitation and update your findings beside the amount.
For example, using the following filter: ./metahub --sh-filters ResourceType=AwsSageMakerNotebookInstance RecordState=ACTIVE WorkflowStatus=NEW I found two affected resources with three finding each making six Security Hub findings in total.
Running the following update command will update those six findings' workflow status to NOTIFIED with a Note:
./metahub --update-findings Workflow=NOTIFIED Note="Enter your ticket ID or reason here as a note that you will add to the finding as part of this update."
The --update-findings will ask you for confirmation before updating your findings. You can skip this confirmation by using the option --no-actions-confirmation.
You can use MetaHub to enrich back your AWS Security Hub Findings with Context outputs using the option --enrich-findings. Enriching your findings means updating them directly in AWS Security Hub. MetaHub uses the UserDefinedFields field for this.
By enriching your findings directly in AWS Security Hub, you can take advantage of features like Insights and Filters by using the extra information not available in Security Hub before.
For example, you want to enrich all AWS Security Hub findings with WorkflowStatus=NEW, RecordState=ACTIVE, and ResourceType=AwsS3Bucket that are public=True with Context outputs:
./metahub --sh-filters RecordState=ACTIVE WorkflowStatus=NEW ResourceType=AwsS3Bucket --mh-filters-checks public=True --enrich-findings
The --enrich-findings will ask you for confirmation before enriching your findings. You can skip this confirmation by using the option --no-actions-confirmation.
Working with Security Findings sometimes introduces the problem of Shadowing and Duplication.
Shadowing is when two checks refer to the same issue, but one in a more generic way than the other one.
Duplication is when you use more than one scanner and get the same problem from more than one.
Think of a Security Group with port 3389/TCP open to 0.0.0.0/0. Let's use Security Hub findings as an example.
If you are using one of the default Security Standards like AWS-Foundational-Security-Best-Practices, you will get two findings for the same issue:
EC2.18 Security groups should only allow unrestricted incoming traffic for authorized portsEC2.19 Security groups should not allow unrestricted access to ports with high riskIf you are also using the standard CIS AWS Foundations Benchmark, you will also get an extra finding:
4.2 Ensure no security groups allow ingress from 0.0.0.0/0 to port 3389Now, imagine that SG is not in use. In that case, Security Hub will show an additional fourth finding for your resource!
EC2.22 Unused EC2 security groups should be removedSo now you have in your dashboard four findings for one resource!
Suppose you are working with multi-account setups and many resources. In that case, this could result in many findings that refer to the same thing without adding any extra value to your analysis.
MetaHub aggregates security findings under the affected resource.
This is how MetaHub shows the previous example with output-mode json-short:
"arn:aws:ec2:eu-west-1:01234567890:security-group/sg-01234567890": {
"findings": [
"EC2.19 Security groups should not allow unrestricted access to ports with high risk",
"EC2.18 Security groups should only allow unrestricted incoming traffic for authorized ports",
"4.2 Ensure no security groups allow ingress from 0.0.0.0/0 to port 3389",
"EC2.22 Unused EC2 security groups should be removed"
],
"AwsAccountId": "01234567890",
"Region": "eu-west-1",
"ResourceType": "AwsEc2SecurityGroup"
}This is how MetaHub shows the previous example with output-mode json-full:
"arn:aws:ec2:eu-west-1:01234567890:security-group/sg-01234567890": {
"findings": [
{
"EC2.19 Security groups should not allow unrestricted access to ports with high risk": {
"SeverityLabel": "CRITICAL",
"Workflow": {
"Status": "NEW"
},
"RecordState": "ACTIVE",
"Compliance": {
"Status": "FAILED"
},
"Id": "arn:aws:security hub:eu-west-1:01234567890:subscription/aws-foundational-security-best-practices/v/1.0.0/EC2.22/finding/01234567890-1234-1234-1234-01234567890",
"ProductArn": "arn:aws:security hub:eu-west-1::product/aws/security hub"
}
},
{
"EC2.18 Security groups should only allow unrestricted incoming traffic for authorized ports": {
"SeverityLabel": "HIGH",
"Workflow": {
"Status": "NEW"
},
"RecordState": "ACTIVE",< br/> "Compliance": {
"Status": "FAILED"
},
"Id": "arn:aws:security hub:eu-west-1:01234567890:subscription/aws-foundational-security-best-practices/v/1.0.0/EC2.22/finding/01234567890-1234-1234-1234-01234567890",
"ProductArn": "arn:aws:security hub:eu-west-1::product/aws/security hub"
}
},
{
"4.2 Ensure no security groups allow ingress from 0.0.0.0/0 to port 3389": {
"SeverityLabel": "HIGH",
"Workflow": {
"Status": "NEW"
},
"RecordState": "ACTIVE",
"Compliance": {
"Status": "FAILED"
},
"Id": "arn:aws:security hub:eu-west-1:01234567890:subscription/aws-foundational-security-best-practices/v/1.0.0/EC2.22/finding/01234567890-1234-1234-1234-01234567890",
"ProductArn": "arn:aws:security hub:eu-west-1::product/aws/security hub"
}
},
{
"EC2.22 Unused EC2 security groups should be removed": {
"SeverityLabel": "MEDIUM",
"Workflow": {
"Status": "NEW"
},
"RecordState": "ACTIVE",
"Compliance": {
"Status": "FAILED"
},
"Id": "arn:aws:security hub:eu-west-1:01234567890:subscription/aws-foundational-security-best-practices/v/1.0.0/EC2.22/finding/01234567890-1234-1234-1234-01234567890",
"ProductArn": "arn:aws:security hub:eu-west-1::product/aws/security hub"
}
}
],
"AwsAccountId": "01234567890",
"AwsAccountAlias": "obfuscated",
"Region": "eu-west-1",
"ResourceType": "AwsEc2SecurityGroup"
}Your findings are combined under the ARN of the resource affected, ending in only one result or one non-compliant resource.
You can now work in MetaHub with all these four findings together as if they were only one. For example, you can update these four Workflow Status findings using only one command: See Updating Workflow Status
You can follow this guide if you want to contribute to the Context module guide.
APIDetector is a powerful and efficient tool designed for testing exposed Swagger endpoints in various subdomains with unique smart capabilities to detect false-positives. It's particularly useful for security professionals and developers who are engaged in API testing and vulnerability scanning.
Before running APIDetector, ensure you have Python 3.x and pip installed on your system. You can download Python here.
Clone the APIDetector repository to your local machine using:
git clone https://github.com/brinhosa/apidetector.git
cd apidetector
pip install requests Run APIDetector using the command line. Here are some usage examples:
Common usage, scan with 30 threads a list of subdomains using a Chrome user-agent and save the results in a file:
python apidetector.py -i list_of_company_subdomains.txt -o results_file.txt -t 30 -ua "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"To scan a single domain:
python apidetector.py -d example.comTo scan multiple domains from a file:
python apidetector.py -i input_file.txtTo specify an output file:
python apidetector.py -i input_file.txt -o output_file.txtTo use a specific number of threads:
python apidetector.py -i input_file.txt -t 20To scan with both HTTP and HTTPS protocols:
python apidetector.py -m -d example.comTo run the script in quiet mode (suppress verbose output):
python apidetector.py -q -d example.comTo run the script with a custom user-agent:
python apidetector.py -d example.com -ua "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"-d, --domain: Single domain to test.-i, --input: Input file containing subdomains to test.-o, --output: Output file to write valid URLs to.-t, --threads: Number of threads to use for scanning (default is 10).-m, --mixed-mode: Test both HTTP and HTTPS protocols.-q, --quiet: Disable verbose output (default mode is verbose).-ua, --user-agent: Custom User-Agent string for requests.Exposing Swagger or OpenAPI documentation endpoints can present various risks, primarily related to information disclosure. Here's an ordered list based on potential risk levels, with similar endpoints grouped together APIDetector scans:
'/swagger-ui.html', '/swagger-ui/', '/swagger-ui/index.html', '/api/swagger-ui.html', '/documentation/swagger-ui.html', '/swagger/index.html', '/api/docs', '/docs', '/api/swagger-ui', '/documentation/swagger-ui'
'/openapi.json', '/swagger.json', '/api/swagger.json', '/swagger.yaml', '/swagger.yml', '/api/swagger.yaml', '/api/swagger.yml', '/api.json', '/api.yaml', '/api.yml', '/documentation/swagger.json', '/documentation/swagger.yaml', '/documentation/swagger.yml'
'/v2/api-docs', '/v3/api-docs', '/api/v2/swagger.json', '/api/v3/swagger.json', '/api/v1/documentation', '/api/v2/documentation', '/api/v3/documentation', '/api/v1/api-docs', '/api/v2/api-docs', '/api/v3/api-docs', '/swagger/v2/api-docs', '/swagger/v3/api-docs', '/swagger-ui.html/v2/api-docs', '/swagger-ui.html/v3/api-docs', '/api/swagger/v2/api-docs', '/api/swagger/v3/api-docs'
'/swagger-resources', '/swagger-resources/configuration/ui', '/swagger-resources/configuration/security', '/api/swagger-resources', '/api.html'
Contributions to APIDetector are welcome! Feel free to fork the repository, make changes, and submit pull requests.
The use of APIDetector should be limited to testing and educational purposes only. The developers of APIDetector assume no liability and are not responsible for any misuse or damage caused by this tool. It is the end user's responsibility to obey all applicable local, state, and federal laws. Developers assume no responsibility for unauthorized or illegal use of this tool. Before using APIDetector, ensure you have permission to test the network or systems you intend to scan.
This project is licensed under the MIT License.
Service that scans your Infrastructure as Code for common vulnerabilities.
| Aspect | Information |
|---|---|
| Tool name | IaC Scan Runner |
| Docker image | xscanner/runner |
| PyPI package | iac-scan-runner |
| Documentation | docs |
| Contact us | xopera@xlab.si |
The IaC Scan Runner is a REST API service used to scan IaC (Infrastructure as Code) package and perform various code checks in order to find possible vulnerabilities and improvements. Explore the docs for more info.
This section explains how to run the REST API.
You can run the REST API using a public xscanner/runner Docker image as follows:
# run IaC Scan Runner REST API in a Docker container and
# navigate to localhost:8080/swagger or localhost:8080/redoc
$ docker run --name iac-scan-runner -p 8080:80 xscanner/runnerOr you can build the image locally and run it as follows:
# build Docker container (it will take some time)
$ docker build -t iac-scan-runner .
# run IaC Scan Runner REST API in a Docker container and
# navigate to localhost:8080/swagger or localhost:8080/redoc
$ docker run --name iac-scan-runner -p 8080:80 iac-scan-runnerTo run using the IaC Scan Runner CLI:
# install the CLI
$ python3 -m venv .venv && . .venv/bin/activate
(.venv) $ pip install iac-scan-runner
# print OpenAPI specification
(.venv) $ iac-scan-runner openapi
# install prerequisites
(.venv) $ iac-scan-runner install
# run IaC Scan Runner REST API
(.venv) $ iac-scan-runner runTo run locally from source:
# Export env variables
export MONGODB_CONNECTION_STRING=mongodb://localhost:27017
export SCAN_PERSISTENCE=enabled
export USER_MANAGEMENT=enabled
# Setup MongoDB
$ docker run --name mongodb -p 27017:27017 mongo
# install prerequisites
$ python3 -m venv .venv && . .venv/bin/activate
(.venv) $ pip install -r requirements.txt
(.venv) $ ./install-checks.sh
# run IaC Scan Runner REST API (add --reload flag to apply code changes on the way)
(.venv) $ uvicorn src.iac_scan_runner.api:appThis part will show one of the possible deployments and short examples on how to use API calls.
Firstly we will clone the iac scan runner repository and run the API.
$ git clone https://github.com/xlab-si/iac-scan-runner.git
$ docker compose upAfter this is done you can use different API endpoints by calling localhost:8000. You can also navigate to localhost:8000/swagger or localhost:8000/redoc and test all the API endpoints there. In this example, we will use curl for calling API endpoints.
curl -X 'POST' \
'http://0.0.0.0/project?creator_id=test' \
-H 'accept: application/json' \
-d ''project id will be returned to us. For this example project id is 1e7b2a91-2896-40fd-8d53-83db56088026.
curl -X 'PUT' \
'http://0.0.0.0:8000/projects/1e7b2a91-2896-40fd-8d53-83db56088026/checks/ansible-lint/disable' \
-H 'accept: application/json'curl -X 'POST' \
'http://0.0.0.0:8000/projects/1e7b2a91-2896-40fd-8d53-83db56088026/scan?scan_response_type=json' \
-H 'accept: application/json' \
-H 'Content-Type: multipart/form-data' \
-F 'iac=@YOUR.zip;type=application/zip'That is it.
At certain point, it might be required to include new check tools within the scan workflow, with aim to provide wider coverage of IaC standards and project types. Therefore, in this subsection, a sequence of required steps for that purpose is identified and described. However, the steps have to be performed manually as it will be described, but it is planned to automatize this procedure in future via API and provide user-friendly interface that will aid the user while importing new tools that will become part of the available catalogue that makes the scan workflow. Figure 16 depicts the required steps which have to be taken in order to extend the scan workflow with a new tool.
Step 1 โ Adding tool-specific class to checks directory First, it is required to add a new tool-specific Python class to the checks directory inside IaC Scan Runnerโs source code: iac-scan-runner/src/iac_scan_runner/checks/new_tool.py
The class of a new tool inherits the existing Check class, which provides generalization of scan workflow tools. Moreover, it is necessary to provide implementation of the following methods:
Step 2 โ Adding the check tool class instance within ScanRunner constructor Once the new class derived from Check is added to the IaC Scan Runnerโs source code, it is also required to modify the source code of its main class, called ScanRunner. When it comes to modifications of this class, it is required first to import the tool-specific class, create a new check tool-specific class instance and adding it to the dictionary of IaC checks inside def init_checks(self). A. Importing the check tool class from iac_scan_runner.checks.tfsec import TfsecCheck B. Creating new instance of check tool object inside init_checks """Initiate predefined check objects""" new_tool = NewToolCheck() C. Adding it to self.iac_checks dictionary inside init_checks
self.iac_checks = {
new_tool.name: new_tool,
โฆ
}Step 3 โ Adding the check tool to the compatibility matrix inside Compatibility class On the other side, inside file src/iac_scan_runner/compatibility.py, the dictionary which represents compatibility matrix should be extended as well. There are two possible cases: a) new file type should be added as a key, together with list of relevant tools as value b) new tool should be added to the compatibility list for the existing file type.
compatibility_matrix = {
"new_type": ["new_tool_1", "new_tool_2"],
โฆ
"old_typeK": ["tool_1", โฆ "tool_N", "new_tool_3"]
}Step 4 โ Providing the support for result summarization Finally, the last step in sequence of required modifications for scan workflow extension is to modify class ResultsSummary (src/iac_scan_runner/results_summary.py). Precisely, it is required to append a part of the code to its method summarize_outcome that will look for specific strings which are tool-specific and can be used to identify whether the check passed or failed. Inside the loop that traverses the compatible checks, for each new tool the following structure of if-else should be included:
if check == "new_tool":
if outcome.find("Check pass string") > -1:
self.outcomes[check]["status"] = "Passed"
return "Passed"
else:
self.outcomes[check]["status"] = "Problems"
return "Problems"You can contact the xOpera team by sending an email to xopera@xlab.si.
This project has received funding from the European Unionโs Horizon 2020 research and innovation programme under Grant Agreement No. 101000162 (PIACERE).
