Login
A Model Context Protocol (MCP) server implementation that integrates with Firecrawl for web scraping capabilities.
Big thanks to @vrknetha, @cawstudios for the initial implementation!
You can also play around with our MCP Server on MCP.so's playground. Thanks to MCP.so for hosting and @gstarwd for integrating our server.
FreshRSS env FIRECRAWL_API_KEY=fc-YOUR_API_KEY npx -y firecrawl-mcp
npm install -g firecrawl-mcp
Configuring Cursor 🖥️ Note: Requires Cursor version 0.45.6+ For the most up-to-date configuration instructions, please refer to the official Cursor documentation on configuring MCP servers: Cursor MCP Server Configuration Guide
To configure Firecrawl MCP in Cursor v0.45.6
env FIRECRAWL_API_KEY=your-api-key npx -y firecrawl-mcp
To configure Firecrawl MCP in Cursor v0.48.6
json { "mcpServers": { "firecrawl-mcp": { "command": "npx", "args": ["-y", "firecrawl-mcp"], "env": { "FIRECRAWL_API_KEY": "YOUR-API-KEY" } } } }
If you are using Windows and are running into issues, try
cmd /c "set FIRECRAWL_API_KEY=your-api-key && npx -y firecrawl-mcp"
Replace your-api-key with your Firecrawl API key. If you don't have one yet, you can create an account and get it from https://www.firecrawl.dev/app/api-keys
After adding, refresh the MCP server list to see the new tools. The Composer Agent will automatically use Firecrawl MCP when appropriate, but you can explicitly request it by describing your web scraping needs. Access the Composer via Command+L (Mac), select "Agent" next to the submit button, and enter your query.
Add this to your ./codeium/windsurf/model_config.json:
{
"mcpServers": {
"mcp-server-firecrawl": {
"command": "npx",
"args": ["-y", "firecrawl-mcp"],
"env": {
"FIRECRAWL_API_KEY": "YOUR_API_KEY"
}
}
}
}
To install Firecrawl for Claude Desktop automatically via Smithery:
npx -y @smithery/cli install @mendableai/mcp-server-firecrawl --client claude
FIRECRAWL_API_KEY: Your Firecrawl API keyFIRECRAWL_API_URL
FIRECRAWL_API_URL (Optional): Custom API endpoint for self-hosted instanceshttps://firecrawl.your-domain.com
FIRECRAWL_RETRY_MAX_ATTEMPTS: Maximum number of retry attempts (default: 3)FIRECRAWL_RETRY_INITIAL_DELAY: Initial delay in milliseconds before first retry (default: 1000)FIRECRAWL_RETRY_MAX_DELAY: Maximum delay in milliseconds between retries (default: 10000)FIRECRAWL_RETRY_BACKOFF_FACTOR: Exponential backoff multiplier (default: 2)FIRECRAWL_CREDIT_WARNING_THRESHOLD: Credit usage warning threshold (default: 1000)FIRECRAWL_CREDIT_CRITICAL_THRESHOLD: Credit usage critical threshold (default: 100)For cloud API usage with custom retry and credit monitoring:
# Required for cloud API
export FIRECRAWL_API_KEY=your-api-key
# Optional retry configuration
export FIRECRAWL_RETRY_MAX_ATTEMPTS=5 # Increase max retry attempts
export FIRECRAWL_RETRY_INITIAL_DELAY=2000 # Start with 2s delay
export FIRECRAWL_RETRY_MAX_DELAY=30000 # Maximum 30s delay
export FIRECRAWL_RETRY_BACKOFF_FACTOR=3 # More aggressive backoff
# Optional credit monitoring
export FIRECRAWL_CREDIT_WARNING_THRESHOLD=2000 # Warning at 2000 credits
export FIRECRAWL_CREDIT_CRITICAL_THRESHOLD=500 # Critical at 500 credits
For self-hosted instance:
# Required for self-hosted
export FIRECRAWL_API_URL=https://firecrawl.your-domain.com
# Optional authentication for self-hosted
export FIRECRAWL_API_KEY=your-api-key # If your instance requires auth
# Custom retry configuration
export FIRECRAWL_RETRY_MAX_ATTEMPTS=10
export FIRECRAWL_RETRY_INITIAL_DELAY=500 # Start with faster retries
Add this to your claude_desktop_config.json:
{
"mcpServers": {
"mcp-server-firecrawl": {
"command": "npx",
"args": ["-y", "firecrawl-mcp"],
"env": {
"FIRECRAWL_API_KEY": "YOUR_API_KEY_HERE",
"FIRECRAWL_RETRY_MAX_ATTEMPTS": "5",
"FIRECRAWL_RETRY_INITIAL_DELAY": "2000",
"FIRECRAWL_RETRY_MAX_DELAY": "30000",
"FIRECRAWL_RETRY_BACKOFF_FACTOR": "3",
"FIRECRAWL_CREDIT_WARNING_THRESHOLD": "2000",
"FIRECRAWL_CREDIT_CRITICAL_THRESHOLD": "500"
}
}
}
}
The server includes several configurable parameters that can be set via environment variables. Here are the default values if not configured:
const CONFIG = {
retry: {
maxAttempts: 3, // Number of retry attempts for rate-limited requests
initialDelay: 1000, // Initial delay before first retry (in milliseconds)
maxDelay: 10000, // Maximum delay between retries (in milliseconds)
backoffFactor: 2, // Multiplier for exponential backoff
},
credit: {
warningThreshold: 1000, // Warn when credit usage reaches this level
criticalThreshold: 100, // Critical alert when credit usage reaches this level
},
};
These configurations control:
Retry Behavior
Automatically retries failed requests due to rate limits
Example: With default settings, retries will be attempted at:
Credit Usage Monitoring
The server utilizes Firecrawl's built-in rate limiting and batch processing capabilities:
firecrawl_scrape)Scrape content from a single URL with advanced options.
{
"name": "firecrawl_scrape",
"arguments": {
"url": "https://example.com",
"formats": ["markdown"],
"onlyMainContent": true,
"waitFor": 1000,
"timeout": 30000,
"mobile": false,
"includeTags": ["article", "main"],
"excludeTags": ["nav", "footer"],
"skipTlsVerification": false
}
}
firecrawl_batch_scrape)Scrape multiple URLs efficiently with built-in rate limiting and parallel processing.
{
"name": "firecrawl_batch_scrape",
"arguments": {
"urls": ["https://example1.com", "https://example2.com"],
"options": {
"formats": ["markdown"],
"onlyMainContent": true
}
}
}
Response includes operation ID for status checking:
{
"content": [
{
"type": "text",
"text": "Batch operation queued with ID: batch_1. Use firecrawl_check_batch_status to check progress."
}
],
"isError": false
}
firecrawl_check_batch_status)Check the status of a batch operation.
{
"name": "firecrawl_check_batch_status",
"arguments": {
"id": "batch_1"
}
}
firecrawl_search)Search the web and optionally extract content from search results.
{
"name": "firecrawl_search",
"arguments": {
"query": "your search query",
"limit": 5,
"lang": "en",
"country": "us",
"scrapeOptions": {
"formats": ["markdown"],
"onlyMainContent": true
}
}
}
firecrawl_crawl)Start an asynchronous crawl with advanced options.
{
"name": "firecrawl_crawl",
"arguments": {
"url": "https://example.com",
"maxDepth": 2,
"limit": 100,
"allowExternalLinks": false,
"deduplicateSimilarURLs": true
}
}
firecrawl_extract)Extract structured information from web pages using LLM capabilities. Supports both cloud AI and self-hosted LLM extraction.
{
"name": "firecrawl_extract",
"arguments": {
"urls": ["https://example.com/page1", "https://example.com/page2"],
"prompt": "Extract product information including name, price, and description",
"systemPrompt": "You are a helpful assistant that extracts product information",
"schema": {
"type": "object",
"properties": {
"name": { "type": "string" },
"price": { "type": "number" },
"description": { "type": "string" }
},
"required": ["name", "price"]
},
"allowExternalLinks": false,
"enableWebSearch": false,
"includeSubdomains": false
}
}
Example response:
{
"content": [
{
"type": "text",
"text": {
"name": "Example Product",
"price": 99.99,
"description": "This is an example product description"
}
}
],
"isError": false
}
urls: Array of URLs to extract information fromprompt: Custom prompt for the LLM extractionsystemPrompt: System prompt to guide the LLMschema: JSON schema for structured data extractionallowExternalLinks: Allow extraction from external linksenableWebSearch: Enable web search for additional contextincludeSubdomains: Include subdomains in extractionWhen using a self-hosted instance, the extraction will use your configured LLM. For cloud API, it uses Firecrawl's managed LLM service.
Conduct deep web research on a query using intelligent crawling, search, and LLM analysis.
{
"name": "firecrawl_deep_research",
"arguments": {
"query": "how does carbon capture technology work?",
"maxDepth": 3,
"timeLimit": 120,
"maxUrls": 50
}
}
Arguments:
Returns:
Generate a standardized llms.txt (and optionally llms-full.txt) file for a given domain. This file defines how large language models should interact with the site.
{
"name": "firecrawl_generate_llmstxt",
"arguments": {
"url": "https://example.com",
"maxUrls": 20,
"showFullText": true
}
}
Arguments:
Returns:
The server includes comprehensive logging:
Example log messages:
[INFO] Firecrawl MCP Server initialized successfully
[INFO] Starting scrape for URL: https://example.com
[INFO] Batch operation queued with ID: batch_1
[WARNING] Credit usage has reached warning threshold
[ERROR] Rate limit exceeded, retrying in 2s...
The server provides robust error handling:
Example error response:
{
"content": [
{
"type": "text",
"text": "Error: Rate limit exceeded. Retrying in 2 seconds..."
}
],
"isError": true
}
# Install dependencies
npm install
# Build
npm run build
# Run tests
npm test
npm test
MIT License - see LICENSE file for details
JSpector is a Burp Suite extension that passively crawls JavaScript files and automatically creates issues with URLs, endpoints and dangerous methods found on the JS files.
Before installing JSpector, you need to have Jython installed on Burp Suite.
Extensions tab.Add button in the Installed tab.Extension Details dialog box, select Python as the Extension Type.Select file button and navigate to the JSpector.py.Next button.Close button.Dashboard tab.
Focused on protecting highly sensitive data, temcrypt is an advanced multi-layer data evolutionary encryption mechanism that offers scalable complexity over time, and is resistant to common brute force attacks.
You can create your own applications, scripts and automations when deploying it.
Find out what temcrypt stands for, the features and inspiration that led me to create it and much more. READ THE KNOWLEDGE DOCUMENT. This is very important to you.
temcrypt is compatible with both Node.js v18 or major, and modern web browsers, allowing you to use it in various environments.
The only dependencies that temcrypt uses are crypto-js for handling encryption algorithms like AES-256, SHA-256 and some encoders and fs is used for file handling with Node.js
To use temcrypt, you need to have Node.js installed. Then, you can install temcrypt using npm:
npm install temcryptafter that, import it in your code as follows:
const temcrypt = require("temcrypt");Includes an auto-install feature for its dependencies, so you don't have to worry about installing them manually. Just run the temcrypt.js library and the dependencies will be installed automatically and then call it in your code, this was done to be portable:
node temcrypt.jsAlternatively, you can use temcrypt directly in the browser by including the following script tag:
<script src="temcrypt.js"></script>or minified:
<script src="temcrypt.min.js"></script>You can also call the library on your website or web application from a CDN:
<script src="https://cdn.jsdelivr.net/gh/jofpin/temcrypt/temcrypt.min.js"></script>temcrypt provides functions like encrypt and decrypt to securely protect and disclose your information.
Parameters
dataString (string): The string data to encrypt.dataFiles (string): The file path to encrypt. Provide either dataString or dataFiles.mainKey (string): The main key (private) for encryption.extraBytes (number, optional): Additional bytes to add to the encryption. Is an optional parameter used in the temcrypt encryption process. It allows you to add extra bytes to the encrypted data, increasing the complexity of the encryption, which requires more processing power to decrypt. It also serves to make patterns lose by changing the weight of the encryption.
Returns
status (boolean): true to indicate successful decryption.hash (string): The unique hash generated for the legitimacy verify of the encrypted data.dataString (string) or dataFiles: The decrypted string or the file path of the decrypted file, depending on the input.updatedEncryptedData (string): The updated encrypted data after decryption. The updated encrypted data after decryption. Every time the encryption is decrypted, the output is updated, because the mainKey changes its order and the new date of last decryption is saved.creationDate (string): The creation date of the encrypted data.lastDecryptionDate (string): The date of the last successful decryption of the data.dataString is provided: hash (string): The unique hash generated for the legitimacy verify of the encrypted data.mainKey (string): The main key (private) used for encryption.timeKey (string): The time key (private) of the encryption.dataString (string): The encrypted string.extraBytes (number, optional): The extra bytes used for encryption.dataFiles is provided: hash (string): The unique hash generated for the legitimacy verify of the encrypted data.mainKey (string): The main key used for encryption.timeKey (string): The time key of the encryption.dataFiles (string): The file path of the encrypted file.extraBytes (number, optional): The extra bytes used for encryption.status (boolean): false to indicate decryption failure.error_code (number): An error code indicating the reason for decryption failure.message (string): A descriptive error message explaining the decryption failure.Here are some examples of how to use temcrypt. Please note that when encrypting, you must enter a key and save the hour and minute that you encrypted the information. To decrypt the information, you must use the same main key at the same hour and minute on subsequent days:
const dataToEncrypt = "Sensitive data";
const mainKey = "your_secret_key"; // Insert your custom key
const encryptedData = temcrypt.encrypt({
dataString: dataToEncrypt,
mainKey: mainKey
});
console.log(encryptedData);const encryptedData = "..."; // Encrypted data obtained from the encryption process
const mainKey = "your_secret_key";
const decryptedData = temcrypt.decrypt({
dataString: encryptedData,
mainKey: mainKey
});
console.log(decryptedData);Encrypt a File:
To encrypt a file using temcrypt, you can use the encrypt function with the dataFiles parameter. Here's an example of how to encrypt a file and obtain the encryption result:
const temcrypt = require("temcrypt");
const filePath = "path/test.txt";
const mainKey = "your_secret_key";
const result = temcrypt.encrypt({
dataFiles: filePath,
mainKey: mainKey,
extraBytes: 128 // Optional: Add 128 extra bytes
});
console.log(result);In this example, replace 'test.txt' with the actual path to the file you want to encrypt and set 'your_secret_key' as the main key for the encryption. The result object will contain the encryption details, including the unique hash, main key, time key, and the file path of the encrypted file.
Decrypt a File:
To decrypt a file that was previously encrypted with temcrypt, you can use the decrypt function with the dataFiles parameter. Here's an example of how to decrypt a file and obtain the decryption result:
const temcrypt = require("temcrypt");
const filePath = "path/test.txt.trypt";
const mainKey = "your_secret_key";
const result = temcrypt.decrypt({
dataFiles: filePath,
mainKey: mainKey
});
console.log(result);In this example, replace 'path/test.txt.trypt' with the actual path to the encrypted file, and set 'your_secret_key' as the main key for decryption. The result object will contain the decryption status and the decrypted data, if successful.
Remember to provide the correct main key used during encryption to successfully decrypt the file, at the exact same hour and minute that it was encrypted. If the main key is wrong or the file was tampered with or the time is wrong, the decryption status will be false and the decrypted data will not be available.
temcrypt provides utils functions to perform additional operations beyond encryption and decryption. These utility functions are designed to enhance the functionality and usability.
Function List:
Below, you can see the details and how to implement its uses.
Update MainKey:
The changeKey utility function allows you to change the mainKey used to encrypt the data while keeping the encrypted data intact. This is useful when you want to enhance the security of your encrypted data or update the mainKey periodically.
dataFiles (optional): The path to the file that was encrypted using temcrypt.dataString (optional): The encrypted string that was generated using temcrypt.mainKey (string): The current mainKey used to encrypt the data.newKey(string): The new mainKey that will replace the current mainKey.const temcrypt = require("temcrypt");
const filePath = "test.txt.trypt";
const currentMainKey = "my_recent_secret_key";
const newMainKey = "new_recent_secret_key";
// Update mainKey for the encrypted file
const result = temcrypt.utils({
changeKey: {
dataFiles: filePath,
mainKey: currentMainKey,
newKey: newMainKey
}
});
console.log(result.message);Check Data Integrity:
The check utility function allows you to verify the integrity of the data encrypted using temcrypt. It checks whether a file or a string is a valid temcrypt encrypted data.
dataFiles (optional): The path to the file that you want to check.dataString (optional): The encrypted string that you want to check.const temcrypt = require("temcrypt");
const filePath = "test.txt.trypt";
const encryptedString = "..."; // Encrypted string generated by temcrypt
// Check the integrity of the encrypted File
const result = temcrypt.utils({
check: {
dataFiles: filePath
}
});
console.log(result.message);
// Check the integrity of the encrypted String
const result2 = temcrypt.utils({
check: {
dataString: encryptedString
}
});
console.log(result2.message);Verify Hash:
The verify utility function allows you to verify the integrity of encrypted data using its hash value. Checks if the encrypted data output matches the provided hash value.
hash (string): The hash value to verify against.dataFiles (optional): The path to the file whose hash you want to verify.dataString (optional): The encrypted string whose hash you want to verify.const temcrypt = require("temcrypt");
const filePath = "test.txt.trypt";
const hashToVerify = "..."; // The hash value to verify
// Verify the hash of the encrypted File
const result = temcrypt.utils({
verify: {
hash: hashToVerify,
dataFiles: filePath
}
});
console.log(result.message);
// Verify the hash of the encrypted String
const result2 = temcrypt.utils({
verify: {
hash: hashToVerify,
dataString: encryptedString
}
});
console.log(result2.message);The following table presents the important error codes and their corresponding error messages used by temcrypt to indicate various error scenarios.
| Code | Error Message | Description |
|---|---|---|
| 420 | Decryption time limit exceeded | The decryption process took longer than the allowed time limit. |
| 444 | Decryption failed | The decryption process encountered an error. |
| 777 | No data provided | No data was provided for the operation. |
| 859 | Invalid temcrypt encrypted string | The provided string is not a valid temcrypt encrypted string. |
Check out the examples directory for more detailed usage examples.
WARNINGThe encryption size of a string or file should be less than 16 KB (kilobytes). If it's larger, you must have enough computational power to decrypt it. Otherwise, your personal computer will exceed the time required to find the correct main key combination and proper encryption formation, and it won't be able to decrypt the information.
TIPS
- With temcrypt you can only decrypt your information in later days with the key that you entered at the same hour and minute that you encrypted.
- Focus on time, it is recommended to start the decryption between the first 2 to 10 seconds, so you have an advantage to generate the correct key formation.
The content of this project itself is licensed under the Creative Commons Attribution 3.0 license, and the underlying source code used to format and display that content is licensed under the MIT license.
Copyright (c) 2023 by Jose Pino
