CyberChef - The Cyber Swiss Army Knife - A Web App For Encryption, Encoding, Compression And Data Analysis

CyberChef is a simple, intuitive web app for carrying out all manner of "cyber" operations within a web browser. These operations include simple encoding like XOR and Base64, more complex encryption like AES, DES and Blowfish, creating binary and hexdumps, compression and decompression of data, calculating hashes and checksums, IPv6 and X.509 parsing, changing character encodings, and much more.

The tool is designed to enable both technical and non-technical analysts to manipulate data in complex ways without having to deal with complex tools or algorithms. It was conceived, designed, built and incrementally improved by an analyst in their 10% innovation time over several years.

Live demo

CyberChef is still under active development. As a result, it shouldn't be considered a finished product. There is still testing and bug fixing to do, new features to be added and additional documentation to write. Please contribute!

Cryptographic operations in CyberChef should not be relied upon to provide security in any situation. No guarantee is offered for their correctness.

A live demo can be found here - have fun!

Containers

If you would like to try out CyberChef locally you can either build it yourself:

docker build --tag cyberchef --ulimit nofile=10000 .
docker run -it -p 8080:80 cyberchef

Or you can use our image directly:

docker run -it -p 8080:80 ghcr.io/gchq/cyberchef:latest

This image is built and published through our GitHub Workflows

How it works

There are four main areas in CyberChef:

1. The input box in the top right, where you can paste, type or drag the text or file you want to operate on.
2. The output box in the bottom right, where the outcome of your processing will be displayed.
3. The operations list on the far left, where you can find all the operations that CyberChef is capable of in categorised lists, or by searching.
4. The recipe area in the middle, where you can drag the operations that you want to use and specify arguments and options.

You can use as many operations as you like in simple or complex ways. Some examples are as follows:

• Decode a Base64-encoded string
• Convert a date and time to a different time zone
• Parse a Teredo IPv6 address
• Convert data from a hexdump, then decompress
• Decrypt and disassemble shellcode
• Display multiple timestamps as full dates
• Carry out different operations on data of different types
• Use parts of the input as arguments to operations
• Perform AES decryption, extracting the IV from the beginning of the cipher stream
• Automagically detect several layers of nested encoding

Features

• Drag and drop
  • Operations can be dragged in and out of the recipe list, or reorganised.
  • Files up to 2GB can be dragged over the input box to load them directly into the browser.
• Auto Bake
  • Whenever you modify the input or the recipe, CyberChef will automatically "bake" for you and produce the output immediately.
  • This can be turned off and operated manually if it is affecting performance (if the input is very large, for instance).
• Automated encoding detection
  • CyberChef uses a number of techniques to attempt to automatically detect which encodings your data is under. If it finds a suitable operation that make sense of your data, it displays the 'magic' icon in the Output field which you can click to decode your data.
• Breakpoints
  • You can set breakpoints on any operation in your recipe to pause execution before running it.
  • You can also step through the recipe one operation at a time to see what the data looks like at each stage.
• Save and load recipes
  • If you come up with an awesome recipe that you know you'll want to use again, just click "Save recipe" and add it to your local storage. It'll be waiting for you next time you visit CyberChef.
  • You can also copy the URL, which includes your recipe and input, to easily share it with others.
• Search
  • If you know the name of the operation you want or a word associated with it, start typing it into the search field and any matching operations will immediately be shown.
• Highlighting
  • When you highlight text in the input or output, the offset and length values will be displayed and, if possible, the corresponding data will be highlighted in the output or input respectively (example: highlight the word 'question' in the input to see where it appears in the output).
• Save to file and load from file
  • You can save the output to a file at any time or load a file by dragging and dropping it into the input field. Files up to around 2GB are supported (depending on your browser), however, some operations may take a very long time to run over this much data.
• CyberChef is entirely client-side
  • It should be noted that none of your recipe configuration or input (either text or files) is ever sent to the CyberChef web server - all processing is carried out within your browser, on your own computer.
  • Due to this feature, CyberChef can be downloaded and run locally. You can use the link in the top left corner of the app to download a full copy of CyberChef and drop it into a virtual machine, share it with other people, or host it in a closed network.

Deep linking

By manipulating CyberChef's URL hash, you can change the initial settings with which the page opens. The format is https://gchq.github.io/CyberChef/#recipe=Operation()&input=...

Supported arguments are recipe, input (encoded in Base64), and theme.

Browser support

CyberChef is built to support

• Google Chrome 50+
• Mozilla Firefox 38+

Node.js support

CyberChef is built to fully support Node.js v16. For more information, see the "Node API" wiki page

Contributing

Contributing a new operation to CyberChef is super easy! The quickstart script will walk you through the process. If you can write basic JavaScript, you can write a CyberChef operation.

An installation walkthrough, how-to guides for adding new operations and themes, descriptions of the repository structure, available data types and coding conventions can all be found in the "Contributing" wiki page.

• Push your changes to your fork.
• Submit a pull request. If you are doing this for the first time, you will be prompted to sign the GCHQ Contributor Licence Agreement via the CLA assistant on the pull request. This will also ask whether you are happy for GCHQ to contact you about a token of thanks for your contribution, or about job opportunities at GCHQ.

Pyxamstore - Python Utility For Parsing Xamarin AssemblyStore Blob Files

This is an alpha release of an assemblies.blob AssemblyStore parser written in Python. The tool is capable of unpack and repackaging assemblies.blob and assemblies.manifest Xamarin files from an APK.

Installing

Run the installer script:

python setup.py install

You can then use the tool by calling pyxamstore

Usage

Unpacking

I recommend using the tool in conjunction with apktool. The following commands can be used to unpack an APK and unpack the Xamarin DLLs:

apktool d yourapp.apk
pyxamstore unpack -d yourapp/unknown/assemblies/

Assemblies that are detected as compressed with LZ4 will be automatically decompressed in the extraction process.

Repacking

If you want to make changes to the DLLs within the AssemblyStore, you can use pyxamstore along with the assemblies.json generated during the unpack to create a new assemblies.blob file(s). The following command from the directory where your assemblies.json file exists:

pyxamstore pack

From here you'll need to copy the new manifest and blobs as well as repackage/sign the APK.

Additional Details

Additional file format details can be found on my personal website.

Known Limitations

• Python3 support (working on it!)
• DLLs that have debug/config data associated with them

AiCEF - An AI-assisted cyber exercise content generation framework using named entity recognition

AiCEF is a tool implementing the accompanying framework [1] in order to harness the intelligence that is available from online resources, as well as threat groups' activities, arsenal (eg. MITRE), to create relevant and timely cybersecurity exercise content. This way, we abstract the events from the reports in a machine-readable form. The produced graphs can be infused with additional intelligence, e.g. the threat actor profile from MITRE, also mapped in our ontology. While this may fill gaps that would be missing from a report, one can also manipulate the graph to create custom and unique models. Finally, we exploit transformer-based language models like GPT to convert the graph into text that can serve as the scenario of a cybersecurity exercise. We have tested and validated AiCEF with a group of experts in cybersecurity exercises, and the results clearly show that AiCEF significantly augments the capabilities in creating timely and relevant cybersecurity exercises in terms of both quality and time.

We used Python to create a machine-learning-powered Exercise Generation Framework and developed a set of tools to perform a set of individual tasks which would help an exercise planner (EP) to create a timely and targeted Cybersecurity Exercise Scenario, regardless of her experience.

Problems an Exercise Planner faces:

• Constant table-top research to have fresh content
• Realistic CSE scenario creation can be difficult and time-consuming
• Meeting objectives but also keeping it appealing for the target audience
• Is the relevance and timeliness aspects considered?
• Can all the above be automated?

Our Main Objective: Build an AI powered tool that can generate relevant and up-to-date Cyber Exercise Content in a few steps with little technical expertise from the user.

Release Roadmap

The updated project, AiCEF v.2.0 is planned to be publicly released by the end of 2023, pending heavy code review and functionality updates. Submodules with reduced functinality will start being release by early June 2023. Thank you for your patience.

Installation

The most convenient way to install AiCEF is by using the docker-compose command. For production deployment, we advise you deploy MySQL manually in a dedicated environment and then to start the other components using Docker.

First, make sure you have docker-compose installed in your environment:



Linux:

$ sudo apt-get install docker-compose

Then, clone the repository:

$ git clone https://github.com/grazvan/AiCEF/docker.git /<choose-a-path>/AiCEF-docker
$ cd /<choose-a-path>/AiCEF-docker

Configure the environment settings

Import the MySQL file in your

$ mysql -u <your_username> –-password=<your_password> AiCEF_db < AiCEF_db.sql 

Before running the docker-compose command, settings must be configured. Copy the sample settings file and change it accordingly to your needs.

$ cp .env.sample .env

Run AiCEF

Note: Make sure you have an OpenAI API key available. Load the environment setttings (including your MySQL connection details):

set -a ; source .env

Finally, run docker-compose in detached (-d) mode:

$ sudo docker-compose up -d

Usage

A common usage flow consists of generating a Trend Report to analyze patterns over time, parsing relevant articles and converting them into Incident Breadcrumbs using MLTP module and storing them in a knowledge database called KDb. Incidents are then generated using IncGen component and can be enhanced using the Graph Enhancer module to simulate known APT activity. The incidents come with injects that can be edited on the fly. The CSE scenario is then created using CEGen, which defines various attributes like CSE name, number of Events, and Incidents. MLCESO is a crucial step in the methodology where dedicated ML models are trained to extract information from the collected articles with over 80% accuracy. The Incident Generation & Enhancer (IncGen) workflow can be automated, generating a variety of incidents based on filtering parameters and the existing database. The knowledge database (KDB) consists of almost 3000 articles classified into six categories that can be augmented using APT Enhancer by using the activity of known APT groups from MITRE or manually.

Find below some sample usage screenshots:

Features

• An AI-powered Cyber Exercise Generation Framework
• Developed in Python & EEL
• Open source library Stixview
• Stores data in MYSQL
• API to Text Synthesis Models (ex. GPT-3.5)
• Can create incidents based on TTPs of 125 known APT actors
• Models Cyber Exercise Content in machine readable STIX2.1 [2] (.json) and human readable format (.pdf)

Authors

AiCEF is a product designed and developed by Alex Zacharis, Razvan Gavrila and Constantinos Patsakis.

References

[1] https://link.springer.com/article/10.1007/s10207-023-00693-z

[2] https://oasis-open.github.io/cti-documentation/stix/intro.html

Contributing

Contributions are welcome! If you'd like to contribute to AiCEF v2.0, please follow these steps:

1. Fork this repository
2. Create a new branch (git checkout -b feature/your-branch-name)
3. Make your changes and commit them (git commit -m 'Add some feature')
4. Push to the branch (git push origin feature/your-branch-name)
5. Open a new pull request

License

AiCEF is licensed under Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license. See for more information.

Under the following terms:

Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. NonCommercial — You may not use the material for commercial purposes. No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.