Password Advice for the Rest of Us

Cookie-Monster - BOF To Steal Browser Cookies & Credentials

Steal browser cookies for edge, chrome and firefox through a BOF or exe! Cookie-Monster will extract the WebKit master key, locate a browser process with a handle to the Cookies and Login Data files, copy the handle(s) and then filelessly download the target. Once the Cookies/Login Data file(s) are downloaded, the python decryption script can help extract those secrets! Firefox module will parse the profiles.ini and locate where the logins.json and key4.db files are located and download them. A seperate github repo is referenced for offline decryption.

BOF Usage

Usage: cookie-monster [ --chrome || --edge || --firefox || --chromeCookiePID <pid> || --chromeLoginDataPID <PID> || --edgeCookiePID <pid> || --edgeLoginDataPID <pid>] 
cookie-monster Example: 
   cookie-monster --chrome 
   cookie-monster --edge 
   cookie-moster --firefox 
   cookie-monster --chromeCookiePID 1337
   cookie-monster --chromeLoginDataPID 1337
   cookie-monster --edgeCookiePID 4444
   cookie-monster --edgeLoginDataPID 4444
cookie-monster Options: 
    --chrome, looks at all running processes and handles, if one matches chrome.exe it copies the handle to Cookies/Login Data and then copies the file to the CWD 
    --edge, looks at all running processes and handles, if one matches msedge.exe it copies the handle to Cookies/Login Data and then copies the file to the CWD 
    --firefox, looks for profiles.ini and locates the key4.db and logins.json file 
    --chromeCookiePID, if chrome PI   D is provided look for the specified process with a handle to cookies is known, specifiy the pid to duplicate its handle and file
    --chromeLoginDataPID, if chrome PID is provided look for the specified process with a handle to Login Data is known, specifiy the pid to duplicate its handle and file  
    --edgeCookiePID, if edge PID is provided look for the specified process with a handle to cookies is known, specifiy the pid to duplicate its handle and file
    --edgeLoginDataPID, if edge PID is provided look for the specified process with a handle to Login Data is known, specifiy the pid to duplicate its handle and file  

EXE usage

Cookie Monster Example:
  cookie-monster.exe --all 
Cookie Monster Options:
  -h, --help                     Show this help message and exit
  --all                          Run chrome, edge, and firefox methods
  --edge                         Extract edge keys and download Cookies/Login Data file to PWD
  --chrome                       Extract chrome keys and download Cookies/Login Data file to PWD
  --firefox                      Locate firefox key and Cookies, does not make a copy of either file

Decryption Steps

Install requirements

pip3 install -r requirements.txt

Base64 encode the webkit masterkey

python3 base64-encode.py "\xec\xfc...."

Decrypt Chrome/Edge Cookies File

python .\decrypt.py "XHh..." --cookies ChromeCookie.db

Results Example:
-----------------------------------
Host: .github.com
Path: /
Name: dotcom_user
Cookie: KingOfTheNOPs
Expires: Oct 28 2024 21:25:22

Host: github.com
Path: /
Name: user_session
Cookie: x123.....
Expires: Nov 11 2023 21:25:22

Decrypt Chome/Edge Passwords File

python .\decrypt.py "XHh..." --passwords ChromePasswords.db

Results Example:
-----------------------------------
URL: https://test.com/
Username: tester
Password: McTesty

Decrypt Firefox Cookies and Stored Credentials:
https://github.com/lclevy/firepwd

Installation

Ensure Mingw-w64 and make is installed on the linux prior to compiling.

make

to compile exe on windows

gcc .\cookie-monster.c -o cookie-monster.exe -lshlwapi -lcrypt32

TO-DO

• update decrypt.py to support firefox based on firepwd and add bruteforce module based on DonPAPI

References

This project could not have been done without the help of Mr-Un1k0d3r and his amazing seasonal videos! Highly recommend checking out his lessons!!!
Cookie Webkit Master Key Extractor: https://github.com/Mr-Un1k0d3r/Cookie-Graber-BOF
Fileless download: https://github.com/fortra/nanodump
Decrypt Cookies and Login Data: https://github.com/login-securite/DonPAPI

WiFi-password-stealer - Simple Windows And Linux Keystroke Injection Tool That Exfiltrates Stored WiFi Data (SSID And Password)

Have you ever watched a film where a hacker would plug-in, seemingly ordinary, USB drive into a victim's computer and steal data from it? - A proper wet dream for some.

Disclaimer: All content in this project is intended for security research purpose only.

Introduction

During the summer of 2022, I decided to do exactly that, to build a device that will allow me to steal data from a victim's computer. So, how does one deploy malware and exfiltrate data? In the following text I will explain all of the necessary steps, theory and nuances when it comes to building your own keystroke injection tool. While this project/tutorial focuses on WiFi passwords, payload code could easily be altered to do something more nefarious. You are only limited by your imagination (and your technical skills).

Setup

After creating pico-ducky, you only need to copy the modified payload (adjusted for your SMTP details for Windows exploit and/or adjusted for the Linux password and a USB drive name) to the RPi Pico.

Prerequisites

• Physical access to victim's computer.

• Unlocked victim's computer.

• Victim's computer has to have an internet access in order to send the stolen data using SMTP for the exfiltration over a network medium.

• Knowledge of victim's computer password for the Linux exploit.

Requirements - What you'll need

• Raspberry Pi Pico (RPi Pico)
• Micro USB to USB Cable
• Jumper Wire (optional)
• pico-ducky - Transformed RPi Pico into a USB Rubber Ducky
• USB flash drive (for the exploit over physical medium only)

Note:

• It is possible to build this tool using Rubber Ducky, but keep in mind that RPi Pico costs about $4.00 and the Rubber Ducky costs $80.00.

• However, while pico-ducky is a good and budget-friedly solution, Rubber Ducky does offer things like stealthiness and usage of the lastest DuckyScript version.

• In order to use Ducky Script to write the payload on your RPi Pico you first need to convert it to a pico-ducky. Follow these simple steps in order to create pico-ducky.

Keystroke injection tool

Keystroke injection tool, once connected to a host machine, executes malicious commands by running code that mimics keystrokes entered by a user. While it looks like a USB drive, it acts like a keyboard that types in a preprogrammed payload. Tools like Rubber Ducky can type over 1,000 words per minute. Once created, anyone with physical access can deploy this payload with ease.

Keystroke injection

The payload uses STRING command processes keystroke for injection. It accepts one or more alphanumeric/punctuation characters and will type the remainder of the line exactly as-is into the target machine. The ENTER/SPACE will simulate a press of keyboard keys.

Delays

We use DELAY command to temporarily pause execution of the payload. This is useful when a payload needs to wait for an element such as a Command Line to load. Delay is useful when used at the very beginning when a new USB device is connected to a targeted computer. Initially, the computer must complete a set of actions before it can begin accepting input commands. In the case of HIDs setup time is very short. In most cases, it takes a fraction of a second, because the drivers are built-in. However, in some instances, a slower PC may take longer to recognize the pico-ducky. The general advice is to adjust the delay time according to your target.

Exfiltration

Data exfiltration is an unauthorized transfer of data from a computer/device. Once the data is collected, adversary can package it to avoid detection while sending data over the network, using encryption or compression. Two most common way of exfiltration are:

• Exfiltration over the network medium.

• This approach was used for the Windows exploit. The whole payload can be seen here.

• Exfiltration over a physical medium.

• This approach was used for the Linux exploit. The whole payload can be seen here.

Windows exploit

In order to use the Windows payload (payload1.dd), you don't need to connect any jumper wire between pins.

Sending stolen data over email

Once passwords have been exported to the .txt file, payload will send the data to the appointed email using Yahoo SMTP. For more detailed instructions visit a following link. Also, the payload template needs to be updated with your SMTP information, meaning that you need to update RECEIVER_EMAIL, SENDER_EMAIL and yours email PASSWORD. In addition, you could also update the body and the subject of the email.

 Note:

• After sending data over the email, the .txt file is deleted.

• You can also use some an SMTP from another email provider, but you should be mindful of SMTP server and port number you will write in the payload.

• Keep in mind that some networks could be blocking usage of an unknown SMTP at the firewall.

Linux exploit

In order to use the Linux payload (payload2.dd) you need to connect a jumper wire between GND and GPIO5 in order to comply with the code in code.py on your RPi Pico. For more information about how to setup multiple payloads on your RPi Pico visit this link.

Storing stolen data to USB flash drive

Once passwords have been exported from the computer, data will be saved to the appointed USB flash drive. In order for this payload to function properly, it needs to be updated with the correct name of your USB drive, meaning you will need to replace USBSTICK with the name of your USB drive in two places.

In addition, you will also need to update the Linux PASSWORD in the payload in three places. As stated above, in order for this exploit to be successful, you will need to know the victim's Linux machine password, which makes this attack less plausible.

Bash script

In order to run the wifi_passwords_print.sh script you will need to update the script with the correct name of your USB stick after which you can type in the following command in your terminal:

echo PASSWORD | sudo -S sh wifi_passwords_print.sh USBSTICK

where PASSWORD is your account's password and USBSTICK is the name for your USB device.

Quick overview of the payload

NetworkManager is based on the concept of connection profiles, and it uses plugins for reading/writing data. It uses .ini-style keyfile format and stores network configuration profiles. The keyfile is a plugin that supports all the connection types and capabilities that NetworkManager has. The files are located in /etc/NetworkManager/system-connections/. Based on the keyfile format, the payload uses the grep command with regex in order to extract data of interest. For file filtering, a modified positive lookbehind assertion was used ((?<=keyword)). While the positive lookbehind assertion will match at a certain position in the string, sc. at a position right after the keyword without making that text itself part of the match, the regex (?<=keyword).* will match any text after the keyword. This allows the payload to match the values after SSID and psk (pre-shared key) keywords.

For more information about NetworkManager here is some useful links:

• Manually creating NetworkManager profiles in keyfile format
• Description of keyfile settings plugin

Exfiltrated data formatting

Below is an example of the exfiltrated and formatted data from a victim's machine in a .txt file.

USB Mass Storage Device Problem

One of the advantages of Rubber Ducky over RPi Pico is that it doesn't show up as a USB mass storage device once plugged in. Once plugged into the computer, all the machine sees it as a USB keyboard. This isn't a default behavior for the RPi Pico. If you want to prevent your RPi Pico from showing up as a USB mass storage device when plugged in, you need to connect a jumper wire between pin 18 (GND) and pin 20 (GPIO15). For more details visit this link.

 Tip:

• Upload your payload to RPi Pico before you connect the pins.
• Don't solder the pins because you will probably want to change/update the payload at some point.

Payload Writer

When creating a functioning payload file, you can use the writer.py script, or you can manually change the template file. In order to run the script successfully you will need to pass, in addition to the script file name, a name of the OS (windows or linux) and the name of the payload file (e.q. payload1.dd). Below you can find an example how to run the writer script when creating a Windows payload.

python3 writer.py windows payload1.dd

Limitations/Drawbacks

• This pico-ducky currently works only on Windows OS.

• This attack requires physical access to an unlocked device in order to be successfully deployed.

• The Linux exploit is far less likely to be successful, because in order to succeed, you not only need physical access to an unlocked device, you also need to know the admins password for the Linux machine.

• Machine's firewall or network's firewall may prevent stolen data from being sent over the network medium.

• Payload delays could be inadequate due to varying speeds of different computers used to deploy an attack.

• The pico-ducky device isn't really stealthy, actually it's quite the opposite, it's really bulky especially if you solder the pins.

• Also, the pico-ducky device is noticeably slower compared to the Rubber Ducky running the same script.

• If the Caps Lock is ON, some of the payload code will not be executed and the exploit will fail.

• If the computer has a non-English Environment set, this exploit won't be successful.

• Currently, pico-ducky doesn't support DuckyScript 3.0, only DuckyScript 1.0 can be used. If you need the 3.0 version you will have to use the Rubber Ducky.

To-Do List

• Fix Caps Lock bug.
• Fix non-English Environment bug.
• Obfuscate the command prompt.
• Implement exfiltration over a physical medium.
• Create a payload for Linux.
• Encode/Encrypt exfiltrated data before sending it over email.
• Implement indicator of successfully completed exploit.
• Implement command history clean-up for Linux exploit.
• Enhance the Linux exploit in order to avoid usage of sudo.

KnowsMore - A Swiss Army Knife Tool For Pentesting Microsoft Active Directory (NTLM Hashes, BloodHound, NTDS And DCSync)

KnowsMore officially supports Python 3.8+.

Main features

• Import NTLM Hashes from .ntds output txt file (generated by CrackMapExec or secretsdump.py)
• Import NTLM Hashes from NTDS.dit and SYSTEM
• Import Cracked NTLM hashes from hashcat output file
• Import BloodHound ZIP or JSON file
• BloodHound importer (import JSON to Neo4J without BloodHound UI)
• Analyse the quality of password (length , lower case, upper case, digit, special and latin)
• Analyse similarity of password with company and user name
• Search for users, passwords and hashes
• Export all cracked credentials direct to BloodHound Neo4j Database as 'owned object'
• Other amazing features...

Getting stats

knowsmore --stats

This command will produce several statistics about the passwords like the output bellow

KnowsMore v0.1.4 by Helvio Junior
Active Directory, BloodHound, NTDS hashes and Password Cracks correlation tool
https://github.com/helviojunior/knowsmore

 [+] Startup parameters
     command line: knowsmore --stats 
     module: stats
     database file: knowsmore.db

 [+] start time 2023-01-11 03:59:20
[?] General Statistics
+-------+----------------+-------+
|   top | description    |   qty |
|-------+----------------+-------|
|     1 | Total Users    | 95369 |
|     2 | Unique Hashes  | 74299 |
|     3 | Cracked Hashes | 23177 |
|     4 | Cracked Users  | 35078 |
+-------+----------------+-------+

 [?] General Top 10 passwords
+-------+-------------+-------+
|   top | password    |   qty |
|-------+-------------+-------|
|     1 | password    |  1111 |
|     2 | 123456      |   824 |
|     3 | 123456789   |   815 |
|     4 | guest       |      553 |
|     5 | qwerty      |   329 |
|     6 | 12345678    |   277 |
|     7 | 111111      |   268 |
|     8 | 12345       |   202 |
|     9 | secret      |   170 |
|    10 | sec4us      |   165 |
+-------+-------------+-------+

 [?] Top 10 weak passwords by company name similarity
+-------+--------------+---------+----------------------+-------+
|   top | password     |   score |   company_similarity |   qty |
|-------+--------------+---------+----------------------+-------|
|     1 | company123   |    7024 |                   80 |  1111 |
|     2 | Company123   |    5209 |                   80 |   824 |
|     3 | company      |    3674 |                  100 |   553 |
|     4 | Company@10   |    2080 |                   80 |   329 |
|     5 | company10    |    1722 |                   86 |   268 |
|     6 | Company@2022 |    1242 |                   71 |   202 |
|     7 | Company@2024 |    1015 |                      71 |   165 |
|     8 | Company2022  |     978 |                   75 |   157 |
|     9 | Company10    |     745 |                   86 |   116 |
|    10 | Company21    |     707 |                   86 |   110 |
+-------+--------------+---------+----------------------+-------+

Installation

Simple

pip3 install --upgrade knowsmore

Note: If you face problem with dependency version Check the Virtual ENV file

Execution Flow

There is no an obligation order to import data, but to get better correlation data we suggest the following execution flow:

1. Create database file
2. Import BloodHound files
   1. Domains
   2. GPOs
   3. OUs
   4. Groups
   5. Computers
   6. Users
3. Import NTDS file
4. Import cracked hashes

Create database file

All data are stored in a SQLite Database

knowsmore --create-db

Importing BloodHound files

We can import all full BloodHound files into KnowsMore, correlate data, and sync it to Neo4J BloodHound Database. So you can use only KnowsMore to import JSON files directly into Neo4j database instead of use extremely slow BloodHound User Interface

# Bloodhound ZIP File
knowsmore --bloodhound --import-data ~/Desktop/client.zip

# Bloodhound JSON File
knowsmore --bloodhound --import-data ~/Desktop/20220912105336_users.json

Note: The KnowsMore is capable to import BloodHound ZIP File and JSON files, but we recommend to use ZIP file, because the KnowsMore will automatically order the files to better data correlation.

Sync data to Neo4j BloodHound database

# Bloodhound ZIP File
knowsmore --bloodhound --sync 10.10.10.10:7687 -d neo4j -u neo4j -p 12345678

Note: The KnowsMore implementation of bloodhount-importer was inpired from Fox-It BloodHound Import implementation. We implemented several changes to save all data in KnowsMore SQLite database and after that do an incremental sync to Neo4J database. With this strategy we have several benefits such as at least 10x faster them original BloodHound User interface.

Importing NTDS file

Option 1

Note: Import hashes and clear-text passwords directly from NTDS.dit and SYSTEM registry

knowsmore --secrets-dump -target LOCAL -ntds ~/Desktop/ntds.dit -system ~/Desktop/SYSTEM

Option 2

Note: First use the secretsdump to extract ntds hashes with the command bellow

secretsdump.py -ntds ntds.dit -system system.reg -hashes lmhash:ntlmhash LOCAL -outputfile ~/Desktop/client_name

After that import

knowsmore --ntlm-hash --import-ntds ~/Desktop/client_name.ntds

Generating a custom wordlist

knowsmore --word-list -o "~/Desktop/Wordlist/my_custom_wordlist.txt" --batch --name company_name

Importing cracked hashes

Cracking hashes

First extract all hashes to a txt file

# Extract NTLM hashes to file
nowsmore --ntlm-hash --export-hashes "~/Desktop/ntlm_hash.txt"

# Or, extract NTLM hashes from NTDS file
cat ~/Desktop/client_name.ntds | cut -d ':' -f4 > ntlm_hashes.txt

In order to crack the hashes, I usually use hashcat with the command bellow

# Wordlist attack
hashcat -m 1000 -a 0 -O -o "~/Desktop/cracked.txt" --remove "~/Desktop/ntlm_hash.txt" "~/Desktop/Wordlist/*"

# Mask attack
hashcat -m 1000 -a 3 -O --increment --increment-min 4 -o "~/Desktop/cracked.txt" --remove "~/Desktop/ntlm_hash.txt" ?a?a?a?a?a?a?a?a

importing hashcat output file

knowsmore --ntlm-hash --company clientCompanyName --import-cracked ~/Desktop/cracked.txt

Note: Change clientCompanyName to name of your company

Wipe sensitive data

As the passwords and his hashes are extremely sensitive data, there is a module to replace the clear text passwords and respective hashes.

Note: This command will keep all generated statistics and imported user data.

knowsmore --wipe

BloodHound Mark as owned

One User

During the assessment you can find (in a several ways) users password, so you can add this to the Knowsmore database

knowsmore --user-pass --username administrator --password Sec4US@2023

# or adding the company name

knowsmore --user-pass --username administrator --password Sec4US@2023 --company sec4us

Integrate all credentials cracked to Neo4j Bloodhound database

knowsmore --bloodhound --mark-owned 10.10.10.10 -d neo4j -u neo4j -p 123456

To remote connection make sure that Neo4j database server is accepting remote connection. Change the line bellow at the config file /etc/neo4j/neo4j.conf and restart the service.

server.bolt.listen_address=0.0.0.0:7687

Mass-Bruter - Mass Bruteforce Network Protocols

Mass bruteforce network protocols

Info

Simple personal script to quickly mass bruteforce common services in a large scale of network.
It will check for default credentials on ftp, ssh, mysql, mssql...etc.
This was made for authorized red team penetration testing purpose only.

How it works

1. Use masscan(faster than nmap) to find alive hosts with common ports from network segment.
2. Parse ips and ports from masscan result.
3. Craft and run hydra commands to automatically bruteforce supported network services on devices.

Requirements

• Kali linux or any preferred linux distribution
• Python 3.10+

# Clone the repo
git clone https://github.com/opabravo/mass-bruter
cd mass-bruter

# Install required tools for the script
apt update && apt install seclists masscan hydra

How To Use

Private ip range : 10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12

Save masscan results under ./result/masscan/, with the format masscan_<name>.<ext>

Ex: masscan_192.168.0.0-16.txt

Example command:

masscan -p 3306,1433,21,22,23,445,3389,5900,6379,27017,5432,5984,11211,9200,1521 172.16.0.0/12 | tee ./result/masscan/masscan_test.txt

Example Resume Command:

masscan --resume paused.conf | tee -a ./result/masscan/masscan_test.txt

Command Options

┌──(root㉿root)-[~/mass-bruter]
└─# python3 mass_bruteforce.py
Usage:  [OPTIONS]

  Mass Bruteforce Script

Options:
  -q, --quick           Quick mode (Only brute telnet, ssh, ftp , mysql,
                        mssql, postgres, oracle)
  -a, --all             Brute all services(Very Slow)
  -s, --show            Show result with successful login
  -f, --file-path PATH  The directory or file that contains masscan result
                        [default: ./result/masscan/]
  --help                Show this message and exit.

Quick Bruteforce Example:

python3 mass_bruteforce.py -q -f ~/masscan_script.txt

Fetch cracked credentials:

python3 mass_bruteforce.py -s

Todo

• Migrate with dpl4hydra
• Optimize the code and functions
• MultiProcessing

Any contributions are welcomed!

Thoughts on scheduled password changes (don’t call them rotations!)

PassMute - PassMute - A Multi Featured Password Transmutation/Mutator Tool

This is a command-line tool written in Python that applies one or more transmutation rules to a given password or a list of passwords read from one or more files. The tool can be used to generate transformed passwords for security testing or research purposes. Also, while you doing pentesting it will be very useful tool for you to brute force the passwords!!

How Passmute can also help to secure our passwords more?

PassMute can help to generate strong and complex passwords by applying different transformation rules to the input password. However, password security also depends on other factors such as the length of the password, randomness, and avoiding common phrases or patterns.

The transformation rules include:

reverse: reverses the password string

uppercase: converts the password to uppercase letters

lowercase: converts the password to lowercase letters

swapcase: swaps the case of each letter in the password

capitalize: capitalizes the first letter of the password

leet: replaces some letters in the password with their leet equivalents

strip: removes all whitespace characters from the password

The tool can also write the transformed passwords to an output file and run the transformation process in parallel using multiple threads.

Installation

git clone https://HITH-Hackerinthehouse/PassMute.git

cd PassMute

chmod +x PassMute.py

Usage To use the tool, you need to have Python 3 installed on your system. Then, you can run the tool from the command line using the following options:

python PassMute.py [-h] [-f FILE [FILE ...]] -r RULES [RULES ...] [-v] [-p PASSWORD] [-o OUTPUT] [-t THREAD_TIMEOUT] [--max-threads MAX_THREADS]

Here's a brief explanation of the available options:

-h or --help: shows the help message and exits

-f (FILE) [FILE ...], --file (FILE) [FILE ...]: one or more files to read passwords from

-r (RULES) [RULES ...] or --rules (RULES) [RULES ...]: one or more transformation rules to apply

-v or --verbose: prints verbose output for each password transformation

-p (PASSWORD) or --password (PASSWORD): transforms a single password

-o (OUTPUT) or --output (OUTPUT): output file to save the transformed passwords

-t (THREAD_TIMEOUT) or --thread-timeout (THREAD_TIMEOUT): timeout for threads to complete (in seconds)

--max-threads (MAX_THREADS): maximum number of threads to run simultaneously (default: 10)

NOTE: If you are getting any error regarding argparse module then simply install the module by following command: pip install argparse

Examples

Here are some example commands those read passwords from a file, applies two transformation rules, and saves the transformed passwords to an output file:

Single Password transmutation: python PassMute.py -p HITHHack3r -r leet reverse swapcase -v -t 50

Multiple Password transmutation: python PassMute.py -f testwordlists.txt -r leet reverse -v -t 100 -o testupdatelists.txt

Here Verbose and Thread are recommended to use in case you're transmutating big files and also it depends upon your microprocessor as well, it's not required every time to use threads and verbose mode.

Legal Disclaimer:

You might be super excited to use this tool, we too. But here we need to confirm! Hackerinthehouse, any contributor of this project and Github won't be responsible for any actions made by you. This tool is made for security research and educational purposes only. It is the end user's responsibility to obey all applicable local, state and federal laws.

S3 Ep133: Apple takes “tight-lipped” to a whole new level

World Password Day: 2 + 2 = 4

To Infinity and Beyond, with Cloudflare Cache Reserve

What if I told you... that you could run a website from behind Cloudflare and only have 385 daily requests miss their cache and go through to the origin service?

No biggy, unless... that was out of a total of more than 166M requests in the same period:

Yep, we just hit "five nines" of cache hit ratio on Pwned Passwords being 99.999%. Actually, it was 99.9998% but we're at the point now where that's just splitting hairs, let's talk about how we've managed to only have two requests in a million hit the origin, beginning with a bit of history:

Optimising Caching on Pwned Passwords (with Workers)- @troyhunt - https://t.co/KjBtCwmhmT pic.twitter.com/BSfJbWyxMy

— Cloudflare (@Cloudflare) August 9, 2018

Ah, memories 😊 Back then, Pwned Passwords was serving way fewer requests in a month than what we do in a day now and the cache hit ratio was somewhere around 92%. Put another way, instead of 2 in every million requests hitting the origin it was 85k. And we were happy with that! As the years progressed, the traffic grew and the caching model was optimised so our stats improved:

There it is - Pwned Passwords is now doing north of 2 *billion* requests a month, peaking at 91.59M in a day with a cache-hit ratio of 99.52%. All free, open source and out there for the community to do good with 😊 pic.twitter.com/DSJOjb2CxZ

— Troy Hunt (@troyhunt) May 24, 2022

And that's pretty much where we levelled out, at about the 99-and-a-bit percent mark. We were really happy with that as it was now only 5k requests per million hitting the origin. There was bound to be a number somewhere around that mark due to the transient nature of cache and eviction criteria inevitably meaning a Cloudflare edge node somewhere would need to reach back to the origin website and pull a new copy of the data. But what if Cloudflare never had to do that unless explicitly instructed to do so? I mean, what if it just stayed in their cache unless we actually changed the source file and told them to update their version? Welcome to Cloudflare Cache Reserve:

Ok, so I may have annotated the important bit but that's what it feels like - magic - because you just turn it on and... that's it. You still serve your content the same way, you still need the appropriate cache headers and you still have the same tiered caching as before, but now there's a "cache reserve" sitting between that and your origin. It's backed by R2 which is their persistent data store and you can keep your cached things there for as long as you want. However, per the earlier link, it's not free:

You pay based on how much you store for how long, how much you write and how much you read. Let's put that in real terms and just as a brief refresher (longer version here), remember that Pwned Passwords is essentially just 16^5 (just over 1 million) text files of about 30kb each for the SHA-1 hashes and a similar number for the NTLM ones (albeit slight smaller file sizes). Here are the Cache Reserve usage stats for the last 9 days:

We can now do some pretty simple maths with that and working on the assumption of 9 days, here's what we get:

2 bucks a day 😲 But this has taken nearly 16M requests off my origin service over this period of time so I haven't paid for the Azure Function execution (which is cheap) nor the egress bandwidth (which is not cheap). But why are there only 16M read operations over 9 days when earlier we saw 167M requests to the API in a single day? Because if you scroll back up to the "insert magic here" diagram, Cache Reserve is only a fallback position and most requests (i.e. 99.52% of them) are still served from the edge caches.

Note also that there are nearly 1M write operations and there are 2 reasons for this:

1. Cache Reserve is being seeded with source data as requests come in and miss the edge cache. This means that our cache hit ratio is going to get much, much better yet as not even half all the potentially cacheable API queries are in Cache Reserve. It also means that the 48c per day cost is going to come way down 🙂
2. Every time the FBI feeds new passwords into the service, the impacted file is purged from cache. This means that there will always be write operations and, of course, read operations as the data flows to the edge cache and makes corresponding hits to the origin service. The prevalence of all this depends on how much data the feds feed in, but it'll never get to zero whilst they're seeding new passwords.

An untold number of businesses rely on Pwned Passwords as an integral part of their registration, login and password reset flows. Seriously, the number is "untold" because we have no idea who's actually using it, we just know the service got hit three and a quarter billion times in the last 30 days:

Giving consumers of the service confidence that not only is it highly resilient, but also massively fast is essential to adoption. In turn, more adoption helps drive better password practices, less account takeovers and more smiles all round 😊

As those remaining hash prefixes populate Cache Reserve, keep an eye on the "cf-cache-status" response header. If you ever see a value of "MISS" then congratulations, you're literally one in a million!

Full disclosure: Cloudflare provides services to HIBP for free and they helped in getting Cache Reserve up and running. However, they had no idea I was writing this blog post and reading it live in its entirety is the first anyone there has seen it. Surprise! 👋

Serious Security: Unravelling the LifeLock “hacked passwords” story

World Password Day – the 1960s just called and gave you your passwords back

IoT devices must “protect consumers from cyberharm”, says UK government

S3 Ep60: Exchange exploit, GoDaddy breach and cookies made public [Podcast]

GoDaddy admits to password breach: check your Managed WordPress site!