FreshRSS

πŸ”’
❌ Secure Planet Training Courses Updated For 2019 - Click Here
There are new available articles, click to refresh the page.
Before yesterdayTools

Hfinger - Fingerprinting HTTP Requests

By: Zion3R


Tool for Fingerprinting HTTP requests of malware. Based on Tshark and written in Python3. Working prototype stage :-)

Its main objective is to provide unique representations (fingerprints) of malware requests, which help in their identification. Unique means here that each fingerprint should be seen only in one particular malware family, yet one family can have multiple fingerprints. Hfinger represents the request in a shorter form than printing the whole request, but still human interpretable.

Hfinger can be used in manual malware analysis but also in sandbox systems or SIEMs. The generated fingerprints are useful for grouping requests, pinpointing requests to particular malware families, identifying different operations of one family, or discovering unknown malicious requests omitted by other security systems but which share fingerprint.

An academic paper accompanies work on this tool, describing, for example, the motivation of design choices, and the evaluation of the tool compared to p0f, FATT, and Mercury.


    The idea

    The basic assumption of this project is that HTTP requests of different malware families are more or less unique, so they can be fingerprinted to provide some sort of identification. Hfinger retains information about the structure and values of some headers to provide means for further analysis. For example, grouping of similar requests - at this moment, it is still a work in progress.

    After analysis of malware's HTTP requests and headers, we have identified some parts of requests as being most distinctive. These include: * Request method * Protocol version * Header order * Popular headers' values * Payload length, entropy, and presence of non-ASCII characters

    Additionally, some standard features of the request URL were also considered. All these parts were translated into a set of features, described in details here.

    The above features are translated into varying length representation, which is the actual fingerprint. Depending on report mode, different features are used to fingerprint requests. More information on these modes is presented below. The feature selection process will be described in the forthcoming academic paper.

    Installation

    Minimum requirements needed before installation: * Python >= 3.3, * Tshark >= 2.2.0.

    Installation available from PyPI:

    pip install hfinger

    Hfinger has been tested on Xubuntu 22.04 LTS with tshark package in version 3.6.2, but should work with older versions like 2.6.10 on Xubuntu 18.04 or 3.2.3 on Xubuntu 20.04.

    Please note that as with any PoC, you should run Hfinger in a separated environment, at least with Python virtual environment. Its setup is not covered here, but you can try this tutorial.

    Usage

    After installation, you can call the tool directly from a command line with hfinger or as a Python module with python -m hfinger.

    For example:

    foo@bar:~$ hfinger -f /tmp/test.pcap
    [{"epoch_time": "1614098832.205385000", "ip_src": "127.0.0.1", "ip_dst": "127.0.0.1", "port_src": "53664", "port_dst": "8080", "fingerprint": "2|3|1|php|0.6|PO|1|us-ag,ac,ac-en,ho,co,co-ty,co-le|us-ag:f452d7a9/ac:as-as/ac-en:id/co:Ke-Al/co-ty:te-pl|A|4|1.4"}]

    Help can be displayed with short -h or long --help switches:

    usage: hfinger [-h] (-f FILE | -d DIR) [-o output_path] [-m {0,1,2,3,4}] [-v]
    [-l LOGFILE]

    Hfinger - fingerprinting malware HTTP requests stored in pcap files

    optional arguments:
    -h, --help show this help message and exit
    -f FILE, --file FILE Read a single pcap file
    -d DIR, --directory DIR
    Read pcap files from the directory DIR
    -o output_path, --output-path output_path
    Path to the output directory
    -m {0,1,2,3,4}, --mode {0,1,2,3,4}
    Fingerprint report mode.
    0 - similar number of collisions and fingerprints as mode 2, but using fewer features,
    1 - representation of all designed features, but a little more collisions than modes 0, 2, and 4,
    2 - optimal (the default mode),
    3 - the lowest number of generated fingerprints, but the highest number of collisions,
    4 - the highest fingerprint entropy, but slightly more fingerprints than modes 0-2
    -v, --verbose Report information about non-standard values in the request
    (e.g., non-ASCII characters, no CRLF tags, values not present in the configuration list).
    Without --logfile (-l) will print to the standard error.
    -l LOGFILE, --logfile LOGFILE
    Output logfile in the verbose mode. Implies -v or --verbose switch.

    You must provide a path to a pcap file (-f), or a directory (-d) with pcap files. The output is in JSON format. It will be printed to standard output or to the provided directory (-o) using the name of the source file. For example, output of the command:

    hfinger -f example.pcap -o /tmp/pcap

    will be saved to:

    /tmp/pcap/example.pcap.json

    Report mode -m/--mode can be used to change the default report mode by providing an integer in the range 0-4. The modes differ on represented request features or rounding modes. The default mode (2) was chosen by us to represent all features that are usually used during requests' analysis, but it also offers low number of collisions and generated fingerprints. With other modes, you can achieve different goals. For example, in mode 3 you get a lower number of generated fingerprints but a higher chance of a collision between malware families. If you are unsure, you don't have to change anything. More information on report modes is here.

    Beginning with version 0.2.1 Hfinger is less verbose. You should use -v/--verbose if you want to receive information about encountered non-standard values of headers, non-ASCII characters in the non-payload part of the request, lack of CRLF tags (\r\n\r\n), and other problems with analyzed requests that are not application errors. When any such issues are encountered in the verbose mode, they will be printed to the standard error output. You can also save the log to a defined location using -l/--log switch (it implies -v/--verbose). The log data will be appended to the log file.

    Using hfinger in a Python application

    Beginning with version 0.2.0, Hfinger supports importing to other Python applications. To use it in your app simply import hfinger_analyze function from hfinger.analysis and call it with a path to the pcap file and reporting mode. The returned result is a list of dicts with fingerprinting results.

    For example:

    from hfinger.analysis import hfinger_analyze

    pcap_path = "SPECIFY_PCAP_PATH_HERE"
    reporting_mode = 4
    print(hfinger_analyze(pcap_path, reporting_mode))

    Beginning with version 0.2.1 Hfinger uses logging module for logging information about encountered non-standard values of headers, non-ASCII characters in the non-payload part of the request, lack of CRLF tags (\r\n\r\n), and other problems with analyzed requests that are not application errors. Hfinger creates its own logger using name hfinger, but without prior configuration log information in practice is discarded. If you want to receive this log information, before calling hfinger_analyze, you should configure hfinger logger, set log level to logging.INFO, configure log handler up to your needs, add it to the logger. More information is available in the hfinger_analyze function docstring.

    Fingerprint creation

    A fingerprint is based on features extracted from a request. Usage of particular features from the full list depends on the chosen report mode from a predefined list (more information on report modes is here). The figure below represents the creation of an exemplary fingerprint in the default report mode.

    Three parts of the request are analyzed to extract information: URI, headers' structure (including method and protocol version), and payload. Particular features of the fingerprint are separated using | (pipe). The final fingerprint generated for the POST request from the example is:

    2|3|1|php|0.6|PO|1|us-ag,ac,ac-en,ho,co,co-ty,co-le|us-ag:f452d7a9/ac:as-as/ac-en:id/co:Ke-Al/co-ty:te-pl|A|4|1.4

    The creation of features is described below in the order of appearance in the fingerprint.

    Firstly, URI features are extracted: * URI length represented as a logarithm base 10 of the length, rounded to an integer, (in the example URI is 43 characters long, so log10(43)β‰ˆ2), * number of directories, (in the example there are 3 directories), * average directory length, represented as a logarithm with base 10 of the actual average length of the directory, rounded to an integer, (in the example there are three directories with total length of 20 characters (6+6+8), so log10(20/3)β‰ˆ1), * extension of the requested file, but only if it is on a list of known extensions in hfinger/configs/extensions.txt, * average value length represented as a logarithm with base 10 of the actual average value length, rounded to one decimal point, (in the example two values have the same length of 4 characters, what is obviously equal to 4 characters, and log10(4)β‰ˆ0.6).

    Secondly, header structure features are analyzed: * request method encoded as first two letters of the method (PO), * protocol version encoded as an integer (1 for version 1.1, 0 for version 1.0, and 9 for version 0.9), * order of the headers, * and popular headers and their values.

    To represent order of the headers in the request, each header's name is encoded according to the schema in hfinger/configs/headerslow.json, for example, User-Agent header is encoded as us-ag. Encoded names are separated by ,. If the header name does not start with an upper case letter (or any of its parts when analyzing compound headers such as Accept-Encoding), then encoded representation is prefixed with !. If the header name is not on the list of the known headers, it is hashed using FNV1a hash, and the hash is used as encoding.

    When analyzing popular headers, the request is checked if they appear in it. These headers are: * Connection * Accept-Encoding * Content-Encoding * Cache-Control * TE * Accept-Charset * Content-Type * Accept * Accept-Language * User-Agent

    When the header is found in the request, its value is checked against a table of typical values to create pairs of header_name_representation:value_representation. The name of the header is encoded according to the schema in hfinger/configs/headerslow.json (as presented before), and the value is encoded according to schema stored in hfinger/configs directory or configs.py file, depending on the header. In the above example Accept is encoded as ac and its value */* as as-as (asterisk-asterisk), giving ac:as-as. The pairs are inserted into fingerprint in order of appearance in the request and are delimited using /. If the header value cannot be found in the encoding table, it is hashed using the FNV1a hash.
    If the header value is composed of multiple values, they are tokenized to provide a list of values delimited with ,, for example, Accept: */*, text/* would give ac:as-as,te-as. However, at this point of development, if the header value contains a "quality value" tag (q=), then the whole value is encoded with its FNV1a hash. Finally, values of User-Agent and Accept-Language headers are directly encoded using their FNV1a hashes.

    Finally, in the payload features: * presence of non-ASCII characters, represented with the letter N, and with A otherwise, * payload's Shannon entropy, rounded to an integer, * and payload length, represented as a logarithm with base 10 of the actual payload length, rounded to one decimal point.

    Report modes

    Hfinger operates in five report modes, which differ in features represented in the fingerprint, thus information extracted from requests. These are (with the number used in the tool configuration): * mode 0 - producing a similar number of collisions and fingerprints as mode 2, but using fewer features, * mode 1 - representing all designed features, but producing a little more collisions than modes 0, 2, and 4, * mode 2 - optimal (the default mode), representing all features which are usually used during requests' analysis, but also offering a low number of collisions and generated fingerprints, * mode 3 - producing the lowest number of generated fingerprints from all modes, but achieving the highest number of collisions, * mode 4 - offering the highest fingerprint entropy, but also generating slightly more fingerprints than modes 0-2.

    The modes were chosen in order to optimize Hfinger's capabilities to uniquely identify malware families versus the number of generated fingerprints. Modes 0, 2, and 4 offer a similar number of collisions between malware families, however, mode 4 generates a little more fingerprints than the other two. Mode 2 represents more request features than mode 0 with a comparable number of generated fingerprints and collisions. Mode 1 is the only one representing all designed features, but it increases the number of collisions by almost two times comparing to modes 0, 1, and 4. Mode 3 produces at least two times fewer fingerprints than other modes, but it introduces about nine times more collisions. Description of all designed features is here.

    The modes consist of features (in the order of appearance in the fingerprint): * mode 0: * number of directories, * average directory length represented as an integer, * extension of the requested file, * average value length represented as a float, * order of headers, * popular headers and their values, * payload length represented as a float. * mode 1: * URI length represented as an integer, * number of directories, * average directory length represented as an integer, * extension of the requested file, * variable length represented as an integer, * number of variables, * average value length represented as an integer, * request method, * version of protocol, * order of headers, * popular headers and their values, * presence of non-ASCII characters, * payload entropy represented as an integer, * payload length represented as an integer. * mode 2: * URI length represented as an integer, * number of directories, * average directory length represented as an integer, * extension of the requested file, * average value length represented as a float, * request method, * version of protocol, * order of headers, * popular headers and their values, * presence of non-ASCII characters, * payload entropy represented as an integer, * payload length represented as a float. * mode 3: * URI length represented as an integer, * average directory length represented as an integer, * extension of the requested file, * average value length represented as an integer, * order of headers. * mode 4: * URI length represented as a float, * number of directories, * average directory length represented as a float, * extension of the requested file, * variable length represented as a float, * average value length represented as a float, * request method, * version of protocol, * order of headers, * popular headers and their values, * presence of non-ASCII characters, * payload entropy represented as a float, * payload length represented as a float.



    PurpleOps - An Open-Source Self-Hosted Purple Team Management Web Application

    By: Zion3R


    An open-source self-hosted purple team management web application.


    Key Features

    • Template engagements and testcases
    • Framework friendly
    • Role-based Access Control & MFA
    • Inbuilt DOCX reporting + custom template support

    How PurpleOps is different:

    • No attribution needed
    • Hackable, no "no-reversing" clauses
    • No over complications with tomcat, redis, manual database transplanting and an obtuce permission model

    Installation

    mongodb -d -p 27017:27017 mongo $ pip3 install -r requirements.txt $ python3 seeder.py $ python3 purpleops.py" dir="auto">
    # Clone this repository
    $ git clone https://github.com/CyberCX-STA/PurpleOps

    # Go into the repository
    $ cd PurpleOps

    # Alter PurpleOps settings (if you want to customize anything but should work out the box)
    $ nano .env

    # Run the app with docker
    $ sudo docker compose up

    # PurpleOps should now by available on http://localhost:5000, it is recommended to add a reverse proxy such as nginx or Apache in front of it if you want to expose this to the outside world.

    # Alternatively
    $ sudo docker run --name mongodb -d -p 27017:27017 mongo
    $ pip3 install -r requirements.txt
    $ python3 seeder.py
    $ python3 purpleops.py

    Contact Us

    We would love to hear back from you, if something is broken or have and idea to make it better add a ticket or ping us pops@purpleops.app | @_w_m__

    Credits



    Nuclearpond - A Utility Leveraging Nuclei To Perform Internet Wide Scans For The Cost Of A Cup Of Coffee


    Nuclear Pond is used to leverage Nuclei in the cloud with unremarkable speed, flexibility, and perform internet wide scans for far less than a cup of coffee.

    It leverages AWS Lambda as a backend to invoke Nuclei scans in parallel, choice of storing json findings in s3 to query with AWS Athena, and is easily one of the cheapest ways you can execute scans in the cloud.


    Features

    • Output results to your terminal, json, or to S3
    • Specify threads and parallel invocations in any desired number of batches
    • Specify any Nuclei arguments just like you would locally
    • Specify a single host or from a file
    • Run the http server to take scans from the API
    • Run the http server to get status of the scans
    • Query findings through Athena for searching S3
    • Specify a custom nuclei and reporting configurations

    Usage

    Think of Nuclear Pond as just a way for you to run Nuclei in the cloud. You can use it just as you would on your local machine but run them in parallel and with however many hosts you want to specify. All you need to think of is the nuclei command line flags you wish to pass to it.

    Setup & Installation

    To install Nuclear Pond, you need to configure the backend terraform module. You can do this by running terraform apply or by leveraging terragrunt.

    $ go install github.com/DevSecOpsDocs/nuclearpond@latest

    Environment Variables

    You can either pass in your backend with flags or through environment variables. You can use -f or --function-name to specify your Lambda function and -r or --region to the specified region. Below are environment variables you can use.

    • AWS_LAMBDA_FUNCTION_NAME is the name of your lambda function to execute the scans on
    • AWS_REGION is the region your resources are deployed
    • NUCLEARPOND_API_KEY is the API key for authenticating to the API
    • AWS_DYNAMODB_TABLE is the dynamodb table to store API scan states

    Command line flags

    Below are some of the flags you can specify when running nuclearpond. The primary flags you need are -t or -l for your target(s), -a for the nuclei args, and -o to specify your output. When specifying Nuclei args you must pass them in as base64 encoded strings by performing -a $(echo -ne "-t dns" | base64).

    Commands

    Below are the subcommands you can execute within nuclearpond.

    • run: Execute nuclei scans
    • service: Basic API to execute nuclei scans

    Run

    To run nuclearpond subcommand nuclearpond run -t devsecopsdocs.com -r us-east-1 -f jwalker-nuclei-runner-function -a $(echo -ne "-t dns" | base64) -o cmd -b 1 in which the target is devsecopsdocs.com, region is us-east-1, lambda function name is jwalker-nuclei-runner-function, nuclei arguments are -t dns, output is cmd, and executes one function through a batch of one host through -b 1.

    $ nuclearpond run -h
    Executes nuclei tasks in parallel by invoking lambda asynchronously

    Usage:
    nuclearpond run [flags]

    Flags:
    -a, --args string nuclei arguments as base64 encoded string
    -b, --batch-size int batch size for number of targets per execution (default 1)
    -f, --function-name string AWS Lambda function name
    -h, --help help for run
    -o, --output string output type to save nuclei results(s3, cmd, or json) (default "cmd")
    -r, --region string AWS region to run nuclei
    -s, --silent silent command line output
    -t, --target string individual target to specify
    -l, --targets string list of targets in a file
    -c, --threads int number of threads to run lambda funct ions, default is 1 which will be slow (default 1)

    Custom Templates

    The terraform module by default downloads the templates on execution as well as adds the templates as a layer. The variables to download templates use the terraform github provider to download the release zip. The folder name within the zip will be located within /opt. Since Nuclei downloads them on run we do not have to but to improve performance you can specify -t /opt/nuclei-templates-9.3.4/dns to execute templates from the downloaded zip. To specify your own templates you must reference a release. When doing so on your own repository you must specify these variables in the terraf orm module, github_token is not required if your repository is public.

    • github_repository
    • github_owner
    • release_tag
    • github_token

    Retrieving Findings

    If you have specified s3 as the output, your findings will be located in S3. The fastest way to get at them is to do so with Athena. Assuming you setup the terraform-module as your backend, all you need to do is query them directly through athena. You may have to configure query results if you have not done so already.

    select
    *
    from
    nuclei_db.findings_db
    limit 10;

    Advance Query

    In order to get down into queries a little deeper, I thought I would give you a quick example. In the select statement we drill down into info column, "matched-at" column must be in double quotes due to - character, and you are searching only for high and critical findings generated by Nuclei.

    SELECT
    info.name,
    host,
    type,
    info.severity,
    "matched-at",
    info.description,
    template,
    dt
    FROM
    "nuclei_db"."findings_db"
    where
    host like '%devsecopsdocs.com'
    and info.severity in ('high','critical')

    Infrastructure

    The backend infrastructure, all within terraform module. I would strongly recommend reading the readme associated to it as it will have some important notes.

    • Lambda function
    • S3 bucket
      • Stores nuclei binary
      • Stores configuration files
      • Stores findings
    • Glue Database and Table
      • Allows you to query the findings in S3
      • Partitioned by the hour
      • Partition projection
    • IAM Role for Lambda Function


    EAST - Extensible Azure Security Tool - Documentation


    Extensible Azure Security Tool (Later referred as E.A.S.T) is tool for assessing Azure and to some extent Azure AD security controls. Primary use case of EAST is Security data collection for evaluation in Azure Assessments. This information (JSON content) can then be used in various reporting tools, which we use to further correlate and investigate the data.


    This tool is licensed under MIT license.




    Collaborators

    Release notes

    • Preview branch introduced

      Changes:

      • Installation now accounts for use of Azure Cloud Shell's updated version in regards to depedencies (Cloud Shell has now Node.JS v 16 version installed)

      • Checking of Databricks cluster types as per advisory

        • Audits Databricks clusters for potential privilege elevation - This control requires typically permissions on the databricks cluster"
      • Content.json is has now key and content based sorting. This enables doing delta checks with git diff HEAD^1 ΒΉ as content.json has predetermined order of results

      ΒΉWord of caution, if want to check deltas of content.json, then content.json will need to be "unignored" from .gitignore exposing results to any upstream you might have configured.

      Use this feature with caution, and ensure you don't have public upstream set for the branch you are using this feature for

    • Change of programming patterns to avoid possible race conditions with larger datasets. This is mostly changes of using var to let in for await -style loops


    Important

    Current status of the tool is beta
    • Fixes, updates etc. are done on "Best effort" basis, with no guarantee of time, or quality of the possible fix applied
    • We do some additional tuning before using EAST in our daily work, such as apply various run and environment restrictions, besides formalizing ourselves with the environment in question. Thus we currently recommend, that EAST is run in only in test environments, and with read-only permissions.
      • All the calls in the service are largely to Azure Cloud IP's, so it should work well in hardened environments where outbound IP restrictions are applied. This reduces the risk of this tool containing malicious packages which could "phone home" without also having C2 in Azure.
        • Essentially running it in read-only mode, reduces a lot of the risk associated with possibly compromised NPM packages (Google compromised NPM)
        • Bugs etc: You can protect your environment against certain mistakes in this code by running the tool with reader-only permissions
    • Lot of the code is "AS IS": Meaning, it's been serving only the purpose of creating certain result; Lot of cleaning up and modularizing remains to be finished
    • There are no tests at the moment, apart from certain manual checks, that are run after changes to main.js and various more advanced controls.
    • The control descriptions at this stage are not the final product, so giving feedback on them, while appreciated, is not the focus of the tooling at this stage
    • As the name implies, we use it as tool to evaluate environments. It is not meant to be run as unmonitored for the time being, and should not be run in any internet exposed service that accepts incoming connections.
    • Documentation could be described as incomplete for the time being
    • EAST is mostly focused on PaaS resource, as most of our Azure assessments focus on this resource type
    • No Input sanitization is performed on launch params, as it is always assumed, that the input of these parameters are controlled. That being said, the tool uses extensively exec() - While I have not reviewed all paths, I believe that achieving shellcode execution is trivial. This tool does not assume hostile input, thus the recommendation is that you don't paste launch arguments into command line without reviewing them first.

    Tool operation

    Depedencies

    To reduce amount of code we use the following depedencies for operation and aesthetics are used (Kudos to the maintainers of these fantastic packages)

    package aesthetics operation license
    axios
    βœ…
    MIT
    yargs
    βœ…
    MIT
    jsonwebtoken
    βœ…
    MIT
    chalk
    βœ…
    MIT
    js-beautify
    βœ…
    MIT

    Other depedencies for running the tool: If you are planning to run this in Azure Cloud Shell you don't need to install Azure CLI:

    • This tool does not include or distribute Microsoft Azure CLI, but rather uses it when it has been installed on the source system (Such as Azure Cloud Shell, which is primary platform for running EAST)

    Azure Cloud Shell (BASH) or applicable Linux Distro / WSL

    Requirement description Install
    βœ…
    AZ CLI
    AZCLI USE curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
    βœ…
    Node.js runtime 14
    Node.js runtime for EAST install with NVM

    Controls

    EAST provides three categories of controls: Basic, Advanced, and Composite

    The machine readable control looks like this, regardless of the type (Basic/advanced/composite):

    {
    "name": "fn-sql-2079",
    "resource": "/subscriptions/6193053b-408b-44d0-b20f-4e29b9b67394/resourcegroups/rg-fn-2079/providers/microsoft.web/sites/fn-sql-2079",
    "controlId": "managedIdentity",
    "isHealthy": true,
    "id": "/subscriptions/6193053b-408b-44d0-b20f-4e29b9b67394/resourcegroups/rg-fn-2079/providers/microsoft.web/sites/fn-sql-2079",
    "Description": "\r\n Ensure The Service calls downstream resources with managed identity",
    "metadata": {
    "principalId": {
    "type": "SystemAssigned",
    "tenantId": "033794f5-7c9d-4e98-923d-7b49114b7ac3",
    "principalId": "cb073f1e-03bc-440e-874d-5ed3ce6df7f8"
    },
    "roles": [{
    "role": [{
    "properties": {
    "roleDefinitionId": "/subscriptions/6193053b-408b-44d0-b20f-4e29b9b67394/providers/Microsoft.Authorization/roleDefinitions/b24988ac-6180-42a0-ab88-20f7382dd24c",
    "principalId": "cb073f1e-03b c-440e-874d-5ed3ce6df7f8",
    "scope": "/subscriptions/6193053b-408b-44d0-b20f-4e29b9b67394/resourceGroups/RG-FN-2079",
    "createdOn": "2021-12-27T06:03:09.7052113Z",
    "updatedOn": "2021-12-27T06:03:09.7052113Z",
    "createdBy": "4257db31-3f22-4c0f-bd57-26cbbd4f5851",
    "updatedBy": "4257db31-3f22-4c0f-bd57-26cbbd4f5851"
    },
    "id": "/subscriptions/6193053b-408b-44d0-b20f-4e29b9b67394/resourceGroups/RG-FN-2079/providers/Microsoft.Authorization/roleAssignments/ada69f21-790e-4386-9f47-c9b8a8c15674",
    "type": "Microsoft.Authorization/roleAssignments",
    "name": "ada69f21-790e-4386-9f47-c9b8a8c15674",
    "RoleName": "Contributor"
    }]
    }]
    },
    "category": "Access"
    },

    Basic

    Basic controls include checks on the initial ARM object for simple "toggle on/off"- boolean settings of said service.

    Example: Azure Container Registry adminUser

    acr_adminUser


    Portal EAST

    if (item.properties?.adminUserEnabled == false ){returnObject.isHealthy = true }

    Advanced

    Advanced controls include checks beyond the initial ARM object. Often invoking new requests to get further information about the resource in scope and it's relation to other services.

    Example: Role Assignments

    Besides checking the role assignments of subscription, additional check is performed via Azure AD Conditional Access Reporting for MFA, and that privileged accounts are not only protected by passwords (SPN's with client secrets)

    Example: Azure Data Factory

    ADF_pipeLineRuns

    Azure Data Factory pipeline mapping combines pipelines -> activities -> and data targets together and then checks for secrets leaked on the logs via run history of the said activities.



    Composite

    Composite controls combines two or more control results from pipeline, in order to form one, or more new controls. Using composites solves two use cases for EAST

    1. You cant guarantee an order of control results being returned in the pipeline
    2. You need to return more than one control result from single check

    Example: composite_resolve_alerts

    1. Get alerts from Microsoft Cloud Defender on subscription check
    2. Form new controls per resourceProvider for alerts

    Reporting

    EAST is not focused to provide automated report generation, as it provides mostly JSON files with control and evaluation status. The idea is to use separate tooling to create reports, which are fairly trivial to automate via markdown creation scripts and tools such as Pandoc

    • While focus is not on the reporting, this repo includes example automation for report creation with pandoc to ease reading of the results in single document format.

    While this tool does not distribute pandoc, it can be used when creation of the reports, thus the following citation is added: https://github.com/jgm/pandoc/blob/master/CITATION.cff

    cff-version: 1.2.0
    title: Pandoc
    message: "If you use this software, please cite it as below."
    type: software
    url: "https://github.com/jgm/pandoc"
    authors:
    - given-names: John
    family-names: MacFarlane
    email: jgm@berkeley.edu
    orcid: 'https://orcid.org/0000-0003-2557-9090'
    - given-names: Albert
    family-names: Krewinkel
    email: tarleb+github@moltkeplatz.de
    orcid: '0000-0002-9455-0796'
    - given-names: Jesse
    family-names: Rosenthal
    email: jrosenthal@jhu.edu

    Running EAST scan

    This part has guide how to run this either on BASH@linux, or BASH on Azure Cloud Shell (obviously Cloud Shell is Linux too, but does not require that you have your own linux box to use this)

    ⚠️If you are running the tool in Cloud Shell, you might need to reapply some of the installations again as Cloud Shell does not persist various session settings.

    Fire and forget prerequisites on cloud shell

    curl -o- https://raw.githubusercontent.com/jsa2/EAST/preview/sh/initForuse.sh | bash;

    jump to next step

    Detailed Prerequisites (This is if you opted no to do the "fire and forget version")

    Prerequisites

    git clone https://github.com/jsa2/EAST --branch preview
    cd EAST;
    npm install

    Pandoc installation on cloud shell

    # Get pandoc for reporting (first time only)
    wget "https://github.com/jgm/pandoc/releases/download/2.17.1.1/pandoc-2.17.1.1-linux-amd64.tar.gz";
    tar xvzf "pandoc-2.17.1.1-linux-amd64.tar.gz" --strip-components 1 -C ~

    Installing pandoc on distros that support APT

    # Get pandoc for reporting (first time only)
    sudo apt install pandoc

    Login Az CLI and run the scan

    # Relogin is required to ensure token cache is placed on session on cloud shell

    az account clear
    az login

    #
    cd EAST
    # replace the subid below with your subscription ID!
    subId=6193053b-408b-44d0-b20f-4e29b9b67394
    #
    node ./plugins/main.js --batch=10 --nativescope=true --roleAssignments=true --helperTexts=true --checkAad=true --scanAuditLogs --composites --subInclude=$subId


    Generate report

    cd EAST; node templatehelpers/eastReports.js --doc

    • If you want to include all Azure Security Benchmark results in the report

    cd EAST; node templatehelpers/eastReports.js --doc --asb

    Export report from cloud shell

    pandoc -s fullReport2.md -f markdown -t docx --reference-doc=pandoc-template.docx -o fullReport2.docx


    Azure Devops (Experimental) There is Azure Devops control for dumping pipeline logs. You can specify the control run by following example:

    node ./plugins/main.js --batch=10 --nativescope=true --roleAssignments=true --helperTexts=true --checkAad=true --scanAuditLogs --composites --subInclude=$subId --azdevops "organizationName"

    Licensing

    Community use

    • Share relevant controls across multiple environments as community effort

    Company use

    • Companies have possibility to develop company specific controls which apply to company specific work. Companies can then control these implementations by decision to share, or not share them based on the operating principle of that company.

    Non IPR components

    • Code logic and functions are under MIT license. since code logic and functions are alredy based on open-source components & vendor API's, it does not make sense to restrict something that is already based on open source

    If you use this tool as part of your commercial effort we only require, that you follow the very relaxed terms of MIT license

    Read license

    Tool operation documentation

    Principles

    AZCLI USE

    Existing tooling enhanced with Node.js runtime

    Use rich and maintained context of Microsoft Azure CLI login & commands with Node.js control flow which supplies enhanced rest-requests and maps results to schema.

    • This tool does not include or distribute Microsoft Azure CLI, but rather uses it when it has been installed on the source system (Such as Azure Cloud Shell, which is primary platform for running EAST)

    Speedup

    View more details

    βœ…Using Node.js runtime as orchestrator utilises Nodes asynchronous nature allowing batching of requests. Batching of requests utilizes the full extent of Azure Resource Managers incredible speed.

    βœ…Compared to running requests one-by-one, the speedup can be up to 10x, when Node executes the batch of requests instead of single request at time

    Parameters reference

    Example:

    node ./plugins/main.js --batch=10 --nativescope --roleAssignments --helperTexts=true --checkAad --scanAuditLogs --composites --shuffle --clearTokens
    Param Description Default if undefined
    --nativescope Currently mandatory parameter no values
    --shuffle Can help with throttling. Shuffles the resource list to reduce the possibility of resource provider throttling threshold being met no values
    --roleAssignments Checks controls as per microsoft.authorization no values
    --includeRG Checks controls with ResourceGroups as per microsoft.authorization no values
    --checkAad Checks controls as per microsoft.azureactivedirectory no values
    --subInclude Defines subscription scope no default, requires subscriptionID/s, if not defined will enumerate all subscriptions the user have access to
    --namespace text filter which matches full, or part of the resource ID
    example /microsoft.storage/storageaccounts all storage accounts in the scope
    optional parameter
    --notIncludes text filter which matches full, or part of the resource ID
    example /microsoft.storage/storageaccounts all storage accounts in the scope are excluded
    optional parameter
    --batch size of batch interval between throttles 5
    --wait size of batch interval between throttles 1500
    --scanAuditLogs optional parameter. When defined in hours will toggle Azure Activity Log scanning for weak authentication events
    defined in: scanAuditLogs
    24h
    --composites read composite no values
    --clearTokens clears tokens in session folder, use this if you get authorization errors, or have just changed to other az login account
    use az account clear if you want to clear AZ CLI cache too
    no values
    --tag Filter all results in the end based on single tag--tag=svc=aksdev no values
    --ignorePreCheck use this option when used with browser delegated tokens no values
    --helperTexts Will append text descriptions from general to manual controls no values
    --reprocess Will update results to existing content.json. Useful for incremental runs no values

    Parameters reference for example report:

    node templatehelpers/eastReports.js --asb 
    Param Description Default if undefined
    --asb gets all ASB results available to users no values
    --policy gets all Policy results available to users no values
    --doc prints pandoc string for export to console no values

    (Highly experimental) Running in restricted environments where only browser use is available

    Read here Running in restricted environments

    Developing controls

    Developer guide including control flow description is here dev-guide.md

    Updates and examples

    Auditing Microsoft.Web provider (Functions and web apps)

    βœ…Check roles that are assigned to function managed identity in Azure AD and all Azure Subscriptions the audit account has access to
    βœ…Relation mapping, check which keyVaults the function uses across all subs the audit account has access to
    βœ…Check if Azure AD authentication is enabled
    βœ…Check that generation of access tokens to the api requires assigment .appRoleAssignmentRequired
    βœ…Audit bindings
    • Function or Azure AD Authentication enabled
    • Count and type of triggers

    βœ…Check if SCM and FTP endpoints are secured


    Azure RBAC baseline authorization

    ⚠️Detect principals in privileged subscriptions roles protected only by password-based single factor authentication.
    • Checks for users without MFA policies applied for set of conditions
    • Checks for ServicePrincipals protected only by password (as opposed to using Certificate Credential, workload federation and or workload identity CA policy)

    Maps to App Registration Best Practices

    • An unused credential on an application can result in security breach. While it's convenient to use password. secrets as a credential, we strongly recommend that you use x509 certificates as the only credential type for getting tokens for your application

    βœ…State healthy - User result example

    { 
    "subscriptionName": "EAST -msdn",
    "friendlyName": "joosua@thx138.onmicrosoft.com",
    "mfaResults": {
    "oid": "138ac68f-d8a7-4000-8d41-c10ff26a9097",
    "appliedPol": [{
    "GrantConditions": "challengeWithMfa",
    "policy": "baseline",
    "oid": "138ac68f-d8a7-4000-8d41-c10ff26a9097"
    }],
    "checkType": "mfa"
    },
    "basicAuthResults": {
    "oid": "138ac68f-d8a7-4000-8d41-c10aa26a9097",
    "appliedPol": [{
    "GrantConditions": "challengeWithMfa",
    "policy": "baseline",
    "oid": "138ac68f-d8a7-4000-8d41-c10aa26a9097"
    }],
    "checkType": "basicAuth"
    },
    }

    ⚠️State unHealthy - Application principal example

    { 
    "subscriptionName": "EAST - HoneyPot",
    "friendlyName": "thx138-kvref-6193053b-408b-44d0-b20f-4e29b9b67394",
    "creds": {
    "@odata.context": "https://graph.microsoft.com/beta/$metadata#servicePrincipals(id,displayName,appId,keyCredentials,passwordCredentials,servicePrincipalType)/$entity",
    "id": "babec804-037d-4caf-946e-7a2b6de3a45f",
    "displayName": "thx138-kvref-6193053b-408b-44d0-b20f-4e29b9b67394",
    "appId": "5af1760e-89ff-46e4-a968-0ac36a7b7b69",
    "servicePrincipalType": "Application",
    "keyCredentials": [],
    "passwordCredentials": [],
    "OnlySingleFactor": [{
    "customKeyIdentifier": null,
    "endDateTime": "2023-10-20T06:54:59.2014093Z",
    "keyId": "7df44f81-a52c-4fd6-b704-4b046771f85a",
    "startDateTime": "2021-10-20T06:54:59.2014093Z",
    "secretText": null,
    "hint": nu ll,
    "displayName": null
    }],
    "StrongSingleFactor": []
    }
    }

    Contributing

    Following methods work for contributing for the time being:

    1. Submit a pull request with code / documentation change
    2. Submit a issue
      • issue can be a:
      • ⚠️Problem (issue)
      • Feature request
      • ❔Question

    Other

    1. By default EAST tries to work with the current depedencies - Introducing new (direct) depedencies is not directly encouraged with EAST. If such vital depedency is introduced, then review licensing of such depedency, and update readme.md - depedencies
      • There is nothing to prevent you from creating your own fork of EAST with your own depedencies


    ❌