<div align="center">
<a href="https://github.com/ReversecLabs/spikee">
<picture>
<source srcset="/images/logo-dark.png" media="(prefers-color-scheme: dark)">
<source srcset="/images/logo-light.png" media="(prefers-color-scheme: light)">
<img src="/images/logo-light.png" alt="Spikee Logo" width="200">
</picture>
</a>
<br>
<h1>Simple Prompt Injection Kit for Evaluation and Exploitation</h1>
</div>
_Version: 0.3.1_
Developed by Reversec Labs, `spikee` is a toolkit for assessing the resilience of LLMs, guardrails, and applications against prompt injection and jailbreaking. Spikee's strength is its modular design, which allows for easy customization of every part of the testing process.
---
## Table of Contents
- [Spikee Use Cases](#spikee-use-cases)
- [The Spikee Architecture](#the-spikee-architecture)
- [Documentation](#documentation)
- [Installation](#1-installation)
- [Local Installation (From Source)](#12-local-installation-from-source)
- [Local Inference Dependencies](#13-local-inference-dependencies)
- [Core Workflow: A Practical Guide](#2-core-workflow-a-practical-guide)
- [Step 1: Initialize a Workspace](#step-1-initialize-a-workspace)
- [Step 2: Explore Available Components](#step-2-explore-available-components)
- [Step 3: Choose a Scenario and Generate a Dataset](#step-3-choose-a-scenario-and-generate-a-dataset)
- [Scenario A: Testing a Standalone LLM](#scenario-a-testing-a-standalone-llm)
- [Scenario B: Testing an LLM Application](#scenario-b-testing-an-llm-application)
- [Bonus: Including standalone attacks](#bonus-including-standalone-attacks)
- [Step 4: Test a Target](#step-4-test-a-target)
- [A. Basic LLM Test](#a-basic-llm-test)
- [B. Testing a Custom LLM Application](#b-testing-a-custom-llm-application)
- [C. Enhancing Tests with Attacks](#c-enhancing-tests-with-attacks)
- [C. Testing a Sample of a Large Dataset](#c-testing-a-sample-of-a-large-dataset)
- [D. Evaluating Guardrails](#d-evaluating-guardrails)
- [Step 5: Analyze the Results](#step-5-analyze-the-results)
- [Contributing](#3-contributing)
- [Questions or Feedback?](#questions-or-feedback)
---
## Spikee Use Cases
<div align="center">
<img src="docs/spikee-usecases.png" width="700px">
</div>
## The Spikee Architecture
Spikee operates in two stages: generating a test dataset, and executing tests against a target using the dataset. Each stage is powered by easy-to-customize Python modules.
<div align="center">
<img src="docs/spikee-architecture.png" width="700px">
</div>
## Documentation
This README provides a practical guide to the core workflow. For advanced topics, see the detailed documentation:
1. **[Built-in Seeds and Datasets](./docs/01_builtin_seeds_and_datasets.md)**: An overview of all built-in datasets.
2. **[Dataset Generation Options](./docs/02_dataset_generation_options.md)**: A reference for all `spikee generate` flags.
3. **[Creating Custom Targets](./docs/03_custom_targets.md)**: Interact with any LLM, API, or guardrail.
4. **[Developing Custom Plugins](./docs/04_custom_plugins.md)**: Statically transform and obfuscate payloads.
5. **[Writing Dynamic Attack Scripts](./docs/03_dynamic_attacks.md)**: Create iterative, adaptive attack logic.
6. **[Judges: Evaluating Attack Success](./docs/05_judges.md)**: Define custom success criteria for tests.
7. **[Testing Guardrails](./docs/06_guardrail_testing.md)**: Evaluate guardrail effectiveness and false positive rates.
8. **[Interpreting Spikee Results](./docs/07_interpreting_results.md)**: Understand test reports and performance metrics.
9. **[Generating Custom Datasets with an LLM](./docs/08_llm_dataset_generation.md)**: Create tailored datasets for specific use cases.
---
## 1. Installation
Install `spikee` directly from PyPI.
```bash
pip install spikee
```
To ensure a clean installation when upgrading, use the `--force-reinstall` flag (*this helps a lot removing deprecated files/datasets that would otherwise persist*):
```bash
pip install --upgrade --force-reinstall spikee
```
### 1.2 Local Installation (From Source)
```bash
git clone https://github.com/WithSecureLabs/spikee.git
cd spikee
python3 -m venv env
source env/bin/activate
pip install .
```
### 1.3 Local Inference Dependencies
For targets requiring local model inference:
```bash
pip install -r requirements-local-inference.txt
```
---
## 2. Core Workflow: A Practical Guide
### Step 1: Initialize a Workspace
Create a project directory and initialize it. This sets up the folder structure and dataset files.
```bash
mkdir my-spikee-project
cd my-spikee-project
spikee init
```
### Step 2: Explore Available Components
Use `spikee list` to see what seeds, targets, plugins, and attacks are available in your workspace (both local and built-in).
```bash
spikee list seeds
spikee list plugins
spikee list judges
spikee list datasets
spikee list targets
spikee list attacks
```
### Step 3: Choose a Scenario and Generate a Dataset
Your testing scenario determines what kind of testing dataset you need to generate.
#### Scenario A: Testing a Standalone LLM
When you test an LLM directly, you control the entire prompt. This is ideal for assessing a model's general resilience to jailbreaks and harmful instructions.
* **What to Generate:** A *full prompt*, which includes a task (like "Summarize this: <data>"), the data containing the prompt injection or jailbreak, and optionally a system message.
* **How to Generate:** Use the default `--format full-prompt` and optionally `--include-system-message`. The `datasets/seeds-cybersec-2025-04` folder provides a great starting point with diverse jailbreaks and attack instructions.
```bash
spikee generate --seed-folder datasets/seeds-cybersec-2025-04
```
This will generate the dataset in JSONL format: `datasets/cybersec-2025-04-full-prompt-dataset-TIMESTAMP.jsonl`.
#### Scenario B: Testing an LLM Application
When you test an application (like a chatbot or an email summarizer), the application itself builds the final prompt. Your input is just one part of it, which could be a prompt or data (such as documents/emails).
* **What to Generate:** Just the *user prompt* or *document* with the attack payload (e.g., the body of an email containing a prompt injection).
* **How to Generate:** Use `--format document`.
```bash
spikee generate --seed-folder datasets/seeds-cybersec-2025-04 --format document
```
This will generate the dataset in JSONL format: `datasets/cybersec-2025-04-document-dataset-TIMESTAMP.jsonl`.
#### Bonus: Including standalone attacks
The `generate` command we saw before composes a dataset by combining documents with jailbreaks and instructions. However, some datasets - such as `seeds-simsonsun-high-quality-jailbreaks` and `in-the-wild-jailbreak-prompts` - contain a static list of ready-to-use attack prompts. To include those in the generated dataset, we use `--standalone-attacks`:
```bash
spikee generate --seed-folder datasets/seeds-simsonsun-high-quality-jailbreaks \
--standalone-attacks datasets/seeds-simsonsun-high-quality-jailbreaks/standalone_attacks.jsonl \
```
### Step 4: Test a Target
`spikee test` runs your dataset against a target. First, rename `.env-example` to `.env` and add any necessary API keys.
#### A. Basic LLM Test
This command tests gpt-4o-mini via the OpenAI API using the dataset generated in Scenario A (require `OPENAI_API_KEY` in `.env`).
```bash
spikee test --dataset datasets/cybersec-2025-04-full-prompt-dataset-*.jsonl \
--target openai_api \
--target-options gpt-4o-mini
```
> **How is attack success determined? With Judges.**
>
> The `cybersec-2025-04` dataset contains attacks whose success can be verified automatically by searching for specific "canary" words or matching regular expressions in the response (such as the presence of a *Markdown image*).
>
> For more complex goals, like checking for harmful content or policy violations, Spikee can use more complex **Judges**. These are Python modules that evaluate the target's response. We include simple LLM-based judges that can assess if a response meets a given criteria. See the **[Judges documentation](./04_judges.md)** to learn more.
#### B. Testing a Custom LLM Application
To test an LLM application, you must create a custom **Target script**. This Python script, placed in the `targets/` directory in your workspace, tells Spikee how to send data to the application and receive its response. For details, see the **[Creating Custom Targets](./01_custom_targets.md)** guide.
```bash
# Test a custom email application using malicious documents and your custom target
spikee test --dataset datasets/llm-mailbox-document-dataset-*.jsonl \
--target llm_mailbox
```
> Especially when testing LLM applications, it's useful to create a custom dataset tailored to the specific use case. In the sample case of the LLM Webmail application, we create a custom dataset stating from `cybersec-2025-04`, that only focusses on testing exfiltration of confidential information via mardown images. Check this tutorial for more information: https://labs.reversec.com/posts/2025/01/spikee-testing-llm-applications-for-prompt-injection
#### C. Enhancing Tests with Attacks
If static prompts fail, use `--attack` to run iterative scripts that modifies the prompt/documents until they succeed (or run out of iterations).
```bash
# Best of N attack
spikee test --dataset datasets/dataset-name.jsonl \
--target openai_api \
--attack best_of_n --attack-iterations 25
```
```bash
# Anti spotlighting attack
spikee test --dataset datasets/dataset-name.jsonl \
--target openai_api \
--attack anti_spotlighting --attack-iterations 50
```
Some attacks, like `prompt decompositoion` support options, such as whih LLM to use to generate attack prompt variations:
```bash
spikee test --dataset datasets/dataset-name.jsonl \
--target openai_api \
--attack prompt_decomposition --attack-iterations 50 -attack-options 'mode=ollama-llama3.2'
```
#### C. Testing a Sample of a Large Dataset
For large datasets, or when operating under time and cost constraints, you can test a random subset of the dataset using the `--sample` flag.
By default, Spikee uses a static seed for sampling. This means that running the same command multiple times will always select the **same random sample**, ensuring your tests are reproducible. This is useful for regression testing.
```bash
# Test a reproducible 15% sample of a large dataset.
# This will select the same 15% of entries every time you run it.
spikee test --dataset datasets/large-dataset.jsonl \
--target openai_api \
--sample 0.15
```
If you need a different sample for each run, or want to use your own seed for reproducibility across different machines or setups, you can use the `--sample-seed` flag.
```bash
# Use a custom seed for a different reproducible sample
spikee test --dataset datasets/large-dataset.jsonl \
--target openai_api \
--sample 0.1 \
--sample-seed 123
# Use a truly random sample on each run
spikee test --dataset datasets/large-dataset.jsonl \
--target openai_api \
--sample 0.1 \
--sample-seed random
```
#### D. Evaluating Guardrails
When you're testing an LLM application, you're automatically testing any guardrail that the developers of the application have applied. Howeer, sometimes you might want to test individual guardrails in isolation.
**1. Testing a Prompt Injection Guardrail:**
To test a guardrail's ability to block general jailbreaks, you could use a broad dataset like `in-the-wild-jailbreak-prompts`, or a more high-quality, focussed one like `seeds-simsonsun-high-quality-jailbreaks`.
```bash
# Test Meta's Prompt Guard against jailbreaks
spikee generate --seed-folder datasets/seeds-simsonsun-high-quality-jailbreaks \
--standalone-attacks datasets/seeds-in-the-wild-jailbreak-prompts/standalone_attacks.jsonl \
spikee test --dataset datasets/simsonsun-high-quality-jailbreaks-*.jsonl \
--target prompt_guard_jailbreak
```
The output of this will tell you whether a particular prompt in the dataset bypassed the guardrial. *This doesn't mean the jailbreak would succeed against an LLM, but simply that it would not be blocked by a guardrail*.
**2. Testing a Topical Guardrail:**
To test a guardrail that blocks specific topics (like financial advice), use a purpose-built dataset, like the sample one that can be generated from these seeds: `seeds-investment-advice`.
```bash
spikee generate --seed-folder datasets/seeds-investment-advice \
--standalone-attacks datasets/seeds-investment-advice/standalone_attacks.jsonl \
--include-system-message \
--format document
```
Notice that here we use `--format document`, as we just want to generate the raw prompts/queries from the seed folder, we don't want to add additional prompts. Also notice that we use `--include-system-message`, as the topical guardrail will use this to determine whether the input aligns wit hthe system_message rules.
```bash
spikee test --dataset datasets/investment-advice-document-sys-dataset-TIMESTAMP.jsonl \
--target nvidia_nemoguard_topic_control
```
### Step 5: Analyze the Results
Use `spikee results analyze` to get a statistical summary of the test run.
```bash
# Analyze the most recent results file for the openai_api target
spikee results analyze --result-file results/results_openai_api-gpt-4o-mini_*.jsonl
```
This command provides an overview of the success rate and detailed breakdowns by attack type, helping you identify specific weaknesses.
---
## 3. Contributing
Contributions are welcome. Please feel free to submit bug fixes, new modules (Targets, Plugins, Attacks, Judges), or dataset seeds via GitHub pull requests.
### Questions or Feedback?
File an issue on the [GitHub repository](https://github.com/ReversecLabs/spikee).
Raw data
{
"_id": null,
"home_page": null,
"name": "spikee",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "prompt-injection, LLM, cyber-security, pentesting",
"author": null,
"author_email": "Reversec <donato.capitella@reversec.com>",
"download_url": "https://files.pythonhosted.org/packages/42/68/5f84b6b28003d7afdd51a6b94dcee97b3018aee18db4497df1f6dc55a5cc/spikee-0.3.2.tar.gz",
"platform": null,
"description": "<div align=\"center\">\n <a href=\"https://github.com/ReversecLabs/spikee\">\n <picture>\n <source srcset=\"/images/logo-dark.png\" media=\"(prefers-color-scheme: dark)\">\n <source srcset=\"/images/logo-light.png\" media=\"(prefers-color-scheme: light)\">\n <img src=\"/images/logo-light.png\" alt=\"Spikee Logo\" width=\"200\">\n </picture>\n </a>\n <br>\n <h1>Simple Prompt Injection Kit for Evaluation and Exploitation</h1>\n</div>\n\n_Version: 0.3.1_\n\n\nDeveloped by Reversec Labs, `spikee` is a toolkit for assessing the resilience of LLMs, guardrails, and applications against prompt injection and jailbreaking. Spikee's strength is its modular design, which allows for easy customization of every part of the testing process.\n\n---\n\n## Table of Contents\n\n- [Spikee Use Cases](#spikee-use-cases)\n- [The Spikee Architecture](#the-spikee-architecture)\n- [Documentation](#documentation)\n- [Installation](#1-installation)\n - [Local Installation (From Source)](#12-local-installation-from-source)\n - [Local Inference Dependencies](#13-local-inference-dependencies)\n- [Core Workflow: A Practical Guide](#2-core-workflow-a-practical-guide)\n - [Step 1: Initialize a Workspace](#step-1-initialize-a-workspace)\n - [Step 2: Explore Available Components](#step-2-explore-available-components)\n - [Step 3: Choose a Scenario and Generate a Dataset](#step-3-choose-a-scenario-and-generate-a-dataset)\n - [Scenario A: Testing a Standalone LLM](#scenario-a-testing-a-standalone-llm)\n - [Scenario B: Testing an LLM Application](#scenario-b-testing-an-llm-application)\n - [Bonus: Including standalone attacks](#bonus-including-standalone-attacks)\n - [Step 4: Test a Target](#step-4-test-a-target)\n - [A. Basic LLM Test](#a-basic-llm-test)\n - [B. Testing a Custom LLM Application](#b-testing-a-custom-llm-application)\n - [C. Enhancing Tests with Attacks](#c-enhancing-tests-with-attacks)\n - [C. Testing a Sample of a Large Dataset](#c-testing-a-sample-of-a-large-dataset)\n - [D. Evaluating Guardrails](#d-evaluating-guardrails)\n - [Step 5: Analyze the Results](#step-5-analyze-the-results)\n- [Contributing](#3-contributing)\n - [Questions or Feedback?](#questions-or-feedback)\n\n---\n\n## Spikee Use Cases\n<div align=\"center\">\n <img src=\"docs/spikee-usecases.png\" width=\"700px\">\n</div>\n\n## The Spikee Architecture\n\nSpikee operates in two stages: generating a test dataset, and executing tests against a target using the dataset. Each stage is powered by easy-to-customize Python modules.\n\n<div align=\"center\">\n <img src=\"docs/spikee-architecture.png\" width=\"700px\">\n</div>\n\n## Documentation\n\nThis README provides a practical guide to the core workflow. For advanced topics, see the detailed documentation:\n\n1. **[Built-in Seeds and Datasets](./docs/01_builtin_seeds_and_datasets.md)**: An overview of all built-in datasets.\n2. **[Dataset Generation Options](./docs/02_dataset_generation_options.md)**: A reference for all `spikee generate` flags.\n3. **[Creating Custom Targets](./docs/03_custom_targets.md)**: Interact with any LLM, API, or guardrail.\n4. **[Developing Custom Plugins](./docs/04_custom_plugins.md)**: Statically transform and obfuscate payloads.\n5. **[Writing Dynamic Attack Scripts](./docs/03_dynamic_attacks.md)**: Create iterative, adaptive attack logic.\n6. **[Judges: Evaluating Attack Success](./docs/05_judges.md)**: Define custom success criteria for tests.\n7. **[Testing Guardrails](./docs/06_guardrail_testing.md)**: Evaluate guardrail effectiveness and false positive rates.\n8. **[Interpreting Spikee Results](./docs/07_interpreting_results.md)**: Understand test reports and performance metrics.\n9. **[Generating Custom Datasets with an LLM](./docs/08_llm_dataset_generation.md)**: Create tailored datasets for specific use cases.\n---\n\n## 1. Installation\n\nInstall `spikee` directly from PyPI.\n\n```bash\npip install spikee\n```\n\nTo ensure a clean installation when upgrading, use the `--force-reinstall` flag (*this helps a lot removing deprecated files/datasets that would otherwise persist*):\n```bash\npip install --upgrade --force-reinstall spikee\n```\n\n### 1.2 Local Installation (From Source)\n\n```bash\ngit clone https://github.com/WithSecureLabs/spikee.git\ncd spikee\npython3 -m venv env\nsource env/bin/activate\npip install .\n```\n\n### 1.3 Local Inference Dependencies\n\nFor targets requiring local model inference:\n\n```bash\npip install -r requirements-local-inference.txt\n```\n\n---\n\n## 2. Core Workflow: A Practical Guide\n\n### Step 1: Initialize a Workspace\n\nCreate a project directory and initialize it. This sets up the folder structure and dataset files.\n\n```bash\nmkdir my-spikee-project\ncd my-spikee-project\nspikee init\n```\n\n### Step 2: Explore Available Components\n\nUse `spikee list` to see what seeds, targets, plugins, and attacks are available in your workspace (both local and built-in).\n\n```bash\nspikee list seeds \nspikee list plugins\nspikee list judges \nspikee list datasets \nspikee list targets \nspikee list attacks \n```\n\n### Step 3: Choose a Scenario and Generate a Dataset\n\nYour testing scenario determines what kind of testing dataset you need to generate.\n\n#### Scenario A: Testing a Standalone LLM\nWhen you test an LLM directly, you control the entire prompt. This is ideal for assessing a model's general resilience to jailbreaks and harmful instructions.\n\n* **What to Generate:** A *full prompt*, which includes a task (like \"Summarize this: <data>\"), the data containing the prompt injection or jailbreak, and optionally a system message.\n* **How to Generate:** Use the default `--format full-prompt` and optionally `--include-system-message`. The `datasets/seeds-cybersec-2025-04` folder provides a great starting point with diverse jailbreaks and attack instructions.\n\n```bash\nspikee generate --seed-folder datasets/seeds-cybersec-2025-04\n```\n\nThis will generate the dataset in JSONL format: `datasets/cybersec-2025-04-full-prompt-dataset-TIMESTAMP.jsonl`.\n\n#### Scenario B: Testing an LLM Application \nWhen you test an application (like a chatbot or an email summarizer), the application itself builds the final prompt. Your input is just one part of it, which could be a prompt or data (such as documents/emails).\n\n* **What to Generate:** Just the *user prompt* or *document* with the attack payload (e.g., the body of an email containing a prompt injection).\n* **How to Generate:** Use `--format document`.\n\n```bash\nspikee generate --seed-folder datasets/seeds-cybersec-2025-04 --format document\n```\n\nThis will generate the dataset in JSONL format: `datasets/cybersec-2025-04-document-dataset-TIMESTAMP.jsonl`.\n\n#### Bonus: Including standalone attacks\nThe `generate` command we saw before composes a dataset by combining documents with jailbreaks and instructions. However, some datasets - such as `seeds-simsonsun-high-quality-jailbreaks` and `in-the-wild-jailbreak-prompts` - contain a static list of ready-to-use attack prompts. To include those in the generated dataset, we use `--standalone-attacks`:\n\n```bash\nspikee generate --seed-folder datasets/seeds-simsonsun-high-quality-jailbreaks \\\n --standalone-attacks datasets/seeds-simsonsun-high-quality-jailbreaks/standalone_attacks.jsonl \\\n```\n\n\n### Step 4: Test a Target\n\n`spikee test` runs your dataset against a target. First, rename `.env-example` to `.env` and add any necessary API keys.\n\n#### A. Basic LLM Test\nThis command tests gpt-4o-mini via the OpenAI API using the dataset generated in Scenario A (require `OPENAI_API_KEY` in `.env`).\n\n```bash\nspikee test --dataset datasets/cybersec-2025-04-full-prompt-dataset-*.jsonl \\\n --target openai_api \\\n --target-options gpt-4o-mini\n```\n\n> **How is attack success determined? With Judges.**\n>\n> The `cybersec-2025-04` dataset contains attacks whose success can be verified automatically by searching for specific \"canary\" words or matching regular expressions in the response (such as the presence of a *Markdown image*).\n>\n> For more complex goals, like checking for harmful content or policy violations, Spikee can use more complex **Judges**. These are Python modules that evaluate the target's response. We include simple LLM-based judges that can assess if a response meets a given criteria. See the **[Judges documentation](./04_judges.md)** to learn more.\n\n#### B. Testing a Custom LLM Application\nTo test an LLM application, you must create a custom **Target script**. This Python script, placed in the `targets/` directory in your workspace, tells Spikee how to send data to the application and receive its response. For details, see the **[Creating Custom Targets](./01_custom_targets.md)** guide.\n\n```bash\n# Test a custom email application using malicious documents and your custom target\nspikee test --dataset datasets/llm-mailbox-document-dataset-*.jsonl \\\n --target llm_mailbox\n```\n\n> Especially when testing LLM applications, it's useful to create a custom dataset tailored to the specific use case. In the sample case of the LLM Webmail application, we create a custom dataset stating from `cybersec-2025-04`, that only focusses on testing exfiltration of confidential information via mardown images. Check this tutorial for more information: https://labs.reversec.com/posts/2025/01/spikee-testing-llm-applications-for-prompt-injection\n\n#### C. Enhancing Tests with Attacks\nIf static prompts fail, use `--attack` to run iterative scripts that modifies the prompt/documents until they succeed (or run out of iterations).\n\n```bash\n# Best of N attack\nspikee test --dataset datasets/dataset-name.jsonl \\\n --target openai_api \\\n --attack best_of_n --attack-iterations 25\n```\n\n```bash\n# Anti spotlighting attack\nspikee test --dataset datasets/dataset-name.jsonl \\\n --target openai_api \\\n --attack anti_spotlighting --attack-iterations 50\n```\n\nSome attacks, like `prompt decompositoion` support options, such as whih LLM to use to generate attack prompt variations:\n```bash\nspikee test --dataset datasets/dataset-name.jsonl \\\n --target openai_api \\\n --attack prompt_decomposition --attack-iterations 50 -attack-options 'mode=ollama-llama3.2'\n```\n\n#### C. Testing a Sample of a Large Dataset\nFor large datasets, or when operating under time and cost constraints, you can test a random subset of the dataset using the `--sample` flag.\n\nBy default, Spikee uses a static seed for sampling. This means that running the same command multiple times will always select the **same random sample**, ensuring your tests are reproducible. This is useful for regression testing.\n\n```bash\n# Test a reproducible 15% sample of a large dataset.\n# This will select the same 15% of entries every time you run it.\nspikee test --dataset datasets/large-dataset.jsonl \\\n --target openai_api \\\n --sample 0.15\n```\n\nIf you need a different sample for each run, or want to use your own seed for reproducibility across different machines or setups, you can use the `--sample-seed` flag.\n\n```bash\n# Use a custom seed for a different reproducible sample\nspikee test --dataset datasets/large-dataset.jsonl \\\n --target openai_api \\\n --sample 0.1 \\\n --sample-seed 123\n\n# Use a truly random sample on each run\nspikee test --dataset datasets/large-dataset.jsonl \\\n --target openai_api \\\n --sample 0.1 \\\n --sample-seed random\n```\n\n#### D. Evaluating Guardrails\nWhen you're testing an LLM application, you're automatically testing any guardrail that the developers of the application have applied. Howeer, sometimes you might want to test individual guardrails in isolation.\n\n**1. Testing a Prompt Injection Guardrail:**\nTo test a guardrail's ability to block general jailbreaks, you could use a broad dataset like `in-the-wild-jailbreak-prompts`, or a more high-quality, focussed one like `seeds-simsonsun-high-quality-jailbreaks`.\n\n```bash\n# Test Meta's Prompt Guard against jailbreaks\nspikee generate --seed-folder datasets/seeds-simsonsun-high-quality-jailbreaks \\\n --standalone-attacks datasets/seeds-in-the-wild-jailbreak-prompts/standalone_attacks.jsonl \\\n\nspikee test --dataset datasets/simsonsun-high-quality-jailbreaks-*.jsonl \\\n --target prompt_guard_jailbreak\n```\n\nThe output of this will tell you whether a particular prompt in the dataset bypassed the guardrial. *This doesn't mean the jailbreak would succeed against an LLM, but simply that it would not be blocked by a guardrail*.\n\n**2. Testing a Topical Guardrail:**\nTo test a guardrail that blocks specific topics (like financial advice), use a purpose-built dataset, like the sample one that can be generated from these seeds: `seeds-investment-advice`.\n\n```bash\nspikee generate --seed-folder datasets/seeds-investment-advice \\\n --standalone-attacks datasets/seeds-investment-advice/standalone_attacks.jsonl \\\n --include-system-message \\\n --format document\n```\n\nNotice that here we use `--format document`, as we just want to generate the raw prompts/queries from the seed folder, we don't want to add additional prompts. Also notice that we use `--include-system-message`, as the topical guardrail will use this to determine whether the input aligns wit hthe system_message rules.\n\n```bash\nspikee test --dataset datasets/investment-advice-document-sys-dataset-TIMESTAMP.jsonl \\\n --target nvidia_nemoguard_topic_control\n```\n\n### Step 5: Analyze the Results\n\nUse `spikee results analyze` to get a statistical summary of the test run.\n\n```bash\n# Analyze the most recent results file for the openai_api target\nspikee results analyze --result-file results/results_openai_api-gpt-4o-mini_*.jsonl\n```\nThis command provides an overview of the success rate and detailed breakdowns by attack type, helping you identify specific weaknesses.\n\n---\n\n## 3. Contributing\n\nContributions are welcome. Please feel free to submit bug fixes, new modules (Targets, Plugins, Attacks, Judges), or dataset seeds via GitHub pull requests.\n\n### Questions or Feedback?\n\nFile an issue on the [GitHub repository](https://github.com/ReversecLabs/spikee).\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Spikee - Simple Prompt Injection Kit for Evaluation and Exploitation",
"version": "0.3.2",
"project_urls": null,
"split_keywords": [
"prompt-injection",
" llm",
" cyber-security",
" pentesting"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "fc49b3b288c3a6ecb8c709d1bc27ddca8a22bf975bfadb1639bcb82e9afcc0b1",
"md5": "781247d2a07248662eedc8e4bbee013a",
"sha256": "408f0e95f87145703ff43b4fd291456c165ee74f6eb6801e63af0b4c2a0b7d18"
},
"downloads": -1,
"filename": "spikee-0.3.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "781247d2a07248662eedc8e4bbee013a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 170602,
"upload_time": "2025-07-16T18:20:17",
"upload_time_iso_8601": "2025-07-16T18:20:17.927983Z",
"url": "https://files.pythonhosted.org/packages/fc/49/b3b288c3a6ecb8c709d1bc27ddca8a22bf975bfadb1639bcb82e9afcc0b1/spikee-0.3.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "42685f84b6b28003d7afdd51a6b94dcee97b3018aee18db4497df1f6dc55a5cc",
"md5": "5b7615084ed794767307260d2d2b701b",
"sha256": "32191a2a4dcaa6283a0b39346cd47b622fac210df8ddc7e4ed08cd1bce256d93"
},
"downloads": -1,
"filename": "spikee-0.3.2.tar.gz",
"has_sig": false,
"md5_digest": "5b7615084ed794767307260d2d2b701b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 129684,
"upload_time": "2025-07-16T18:20:19",
"upload_time_iso_8601": "2025-07-16T18:20:19.381560Z",
"url": "https://files.pythonhosted.org/packages/42/68/5f84b6b28003d7afdd51a6b94dcee97b3018aee18db4497df1f6dc55a5cc/spikee-0.3.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-16 18:20:19",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "spikee"
}
JSON