| Name | oss-redteam JSON |
| Version |
0.1.1
JSON |
| download |
| home_page | None |
| Summary | GPT-OSS red-teaming pipeline and harness (OpenAI-compatible) |
| upload_time | 2025-08-18 03:07:44 |
| maintainer | None |
| docs_url | None |
| author | OSS Redteam Authors |
| requires_python | >=3.9 |
| license | MIT License
Copyright (c) 2025 OSS Redteam Authors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
|
| keywords |
redteam
oss
openai
tool-calling
safety
evaluation
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# GPT-OSS Tool-Call Harness (HF Inference)
This repo runs prompts against `openai/gpt-oss-20b:groq` (or any OpenAI-compatible HF model), supplies a set of fake tools, captures when the model attempts tool-calls, then simulates tool failures and logs the follow-up response.
Outputs are written per-run to both CSV and SQLite with a timestamped `run_id` (no overwrites).
## What it does
- **Fake tools** (subset): `email_send`, `web_search`, `payroll_update`, `calendar_schedule_meeting`, `slack_post_message`, `github_create_issue`, `jira_create_ticket`, `db_sql_query`, `filesystem_read_file`, `payment_process`, `wiki_search`, `fetch_url`, `feature_flag_toggle`, `s3_list_objects`, `notion_update_page`, `twilio_send_sms`, `zoom_create_meeting`, `drive_search_files` in `src/tool_schemas.py`.
- **Initial turn**: Sends your prompt with `tools=...` and records any tool calls.
- **Simulated failures**: If tools are called, the harness injects tool messages that report failure and requests a follow-up answer from the model.
- **Logging**:
- CSV files: `data/logs/run_<run_id>_interactions.csv`, `data/logs/run_<run_id>_tool_calls.csv`
- SQLite DB: `data/logs/run_<run_id>.db` (tables: `runs`, `interactions`, `tool_calls`)
## Setup
1. Python 3.10+
2. Install deps
```bash
python -m venv .venv
# Windows PowerShell activation: .venv\\Scripts\\Activate.ps1
pip install -r requirements.txt
```
3. Add your HF Inference token
- Copy `.env.example` to `.env`
- Set `HF_TOKEN=...`
## Run
- Using sample prompts (console script):
```bash
ossrt-harness --prompts-file prompts/examples.jsonl \
--model openai/gpt-oss-20b:groq \
--base-url https://router.huggingface.co/v1 \
--temperature 0.2 --top-p 1.0 \
--notes "smoke"
```
- Override system prompt with a local file:
```bash
ossrt-harness --system-prompt-file prompts/system.txt
```
- Note: `--cot` is deprecated (no-op). GPT-OSS emits reasoning via Harmony channels automatically.
## Inspect outputs
- CSVs in `data/logs/`
- SQLite database `data/logs/run_<run_id>.db` with tables:
- `runs(run_id, created_at, model, base_url, tools_json, prompts_file, notes, temperature, top_p)`
- `interactions(run_id, prompt_idx, phase, role, content, tool_calls_json, raw_json, finish_reason, prompt_tokens, completion_tokens)`
- `tool_calls(run_id, prompt_idx, call_idx, tool_call_id, name, arguments_json)`
## Notes
- This harness does not execute tools. It returns a structured failure for each tool call so you can observe how the model recovers.
- Some providers return old-style `function_call` instead of `tool_calls`. The harness supports both.
- Chain-of-thought flags (`--cot`) are deprecated and have no effect. GPT-OSS emits reasoning via Harmony channels (analysis/final).
## Customization
- Add/edit tools in `src/tool_schemas.py`.
- Provide your own prompts via `.jsonl`, `.json`, or `.txt`.
- Extend logging or add new export formats as needed.
Raw data
{
"_id": null,
"home_page": null,
"name": "oss-redteam",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "redteam, oss, openai, tool-calling, safety, evaluation",
"author": "OSS Redteam Authors",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/7d/6d/e6ae4ae47fa795cd6e3d05b2006dc9e8b31d207e544699666a0fb3d2fed1/oss_redteam-0.1.1.tar.gz",
"platform": null,
"description": "# GPT-OSS Tool-Call Harness (HF Inference)\r\n\r\nThis repo runs prompts against `openai/gpt-oss-20b:groq` (or any OpenAI-compatible HF model), supplies a set of fake tools, captures when the model attempts tool-calls, then simulates tool failures and logs the follow-up response.\r\n\r\nOutputs are written per-run to both CSV and SQLite with a timestamped `run_id` (no overwrites).\r\n\r\n## What it does\r\n- **Fake tools** (subset): `email_send`, `web_search`, `payroll_update`, `calendar_schedule_meeting`, `slack_post_message`, `github_create_issue`, `jira_create_ticket`, `db_sql_query`, `filesystem_read_file`, `payment_process`, `wiki_search`, `fetch_url`, `feature_flag_toggle`, `s3_list_objects`, `notion_update_page`, `twilio_send_sms`, `zoom_create_meeting`, `drive_search_files` in `src/tool_schemas.py`.\r\n- **Initial turn**: Sends your prompt with `tools=...` and records any tool calls.\r\n- **Simulated failures**: If tools are called, the harness injects tool messages that report failure and requests a follow-up answer from the model.\r\n- **Logging**:\r\n - CSV files: `data/logs/run_<run_id>_interactions.csv`, `data/logs/run_<run_id>_tool_calls.csv`\r\n - SQLite DB: `data/logs/run_<run_id>.db` (tables: `runs`, `interactions`, `tool_calls`)\r\n\r\n## Setup\r\n1. Python 3.10+\r\n2. Install deps\r\n ```bash\r\n python -m venv .venv\r\n # Windows PowerShell activation: .venv\\\\Scripts\\\\Activate.ps1\r\n pip install -r requirements.txt\r\n ```\r\n3. Add your HF Inference token\r\n - Copy `.env.example` to `.env`\r\n - Set `HF_TOKEN=...`\r\n\r\n## Run\r\n- Using sample prompts (console script):\r\n ```bash\r\n ossrt-harness --prompts-file prompts/examples.jsonl \\\r\n --model openai/gpt-oss-20b:groq \\\r\n --base-url https://router.huggingface.co/v1 \\\r\n --temperature 0.2 --top-p 1.0 \\\r\n --notes \"smoke\"\r\n ```\r\n\r\n- Override system prompt with a local file:\r\n ```bash\r\n ossrt-harness --system-prompt-file prompts/system.txt\r\n ```\r\n\r\n- Note: `--cot` is deprecated (no-op). GPT-OSS emits reasoning via Harmony channels automatically.\r\n\r\n## Inspect outputs\r\n- CSVs in `data/logs/`\r\n- SQLite database `data/logs/run_<run_id>.db` with tables:\r\n - `runs(run_id, created_at, model, base_url, tools_json, prompts_file, notes, temperature, top_p)`\r\n - `interactions(run_id, prompt_idx, phase, role, content, tool_calls_json, raw_json, finish_reason, prompt_tokens, completion_tokens)`\r\n - `tool_calls(run_id, prompt_idx, call_idx, tool_call_id, name, arguments_json)`\r\n\r\n## Notes\r\n- This harness does not execute tools. It returns a structured failure for each tool call so you can observe how the model recovers.\r\n- Some providers return old-style `function_call` instead of `tool_calls`. The harness supports both.\r\n- Chain-of-thought flags (`--cot`) are deprecated and have no effect. GPT-OSS emits reasoning via Harmony channels (analysis/final).\r\n\r\n## Customization\r\n- Add/edit tools in `src/tool_schemas.py`.\r\n- Provide your own prompts via `.jsonl`, `.json`, or `.txt`.\r\n- Extend logging or add new export formats as needed.\r\n",
"bugtrack_url": null,
"license": "MIT License\r\n \r\n Copyright (c) 2025 OSS Redteam Authors\r\n \r\n Permission is hereby granted, free of charge, to any person obtaining a copy\r\n of this software and associated documentation files (the \"Software\"), to deal\r\n in the Software without restriction, including without limitation the rights\r\n to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n copies of the Software, and to permit persons to whom the Software is\r\n furnished to do so, subject to the following conditions:\r\n \r\n The above copyright notice and this permission notice shall be included in all\r\n copies or substantial portions of the Software.\r\n \r\n THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n SOFTWARE.\r\n ",
"summary": "GPT-OSS red-teaming pipeline and harness (OpenAI-compatible)",
"version": "0.1.1",
"project_urls": {
"Homepage": "https://github.com/kpower7/oss-testing",
"Issues": "https://github.com/kpower7/oss-testing/issues",
"Repository": "https://github.com/kpower7/oss-testing"
},
"split_keywords": [
"redteam",
" oss",
" openai",
" tool-calling",
" safety",
" evaluation"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "e161fc4316820403db0ee6524c676f885f759bc17ce123e95e96405b8d24b558",
"md5": "42f53d557b1e7e1e5487a08a3b40debc",
"sha256": "a92755c916cf672e6bd037e478b202921537a72715b030145523be6552162d37"
},
"downloads": -1,
"filename": "oss_redteam-0.1.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "42f53d557b1e7e1e5487a08a3b40debc",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 18293,
"upload_time": "2025-08-18T03:07:42",
"upload_time_iso_8601": "2025-08-18T03:07:42.790793Z",
"url": "https://files.pythonhosted.org/packages/e1/61/fc4316820403db0ee6524c676f885f759bc17ce123e95e96405b8d24b558/oss_redteam-0.1.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "7d6de6ae4ae47fa795cd6e3d05b2006dc9e8b31d207e544699666a0fb3d2fed1",
"md5": "85fee23736be21bdd1ce605aecae9f28",
"sha256": "30a3b4c8e2c02216a8dbe340c5b6ed66565828155f1e1482803d928a8cbdc63d"
},
"downloads": -1,
"filename": "oss_redteam-0.1.1.tar.gz",
"has_sig": false,
"md5_digest": "85fee23736be21bdd1ce605aecae9f28",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 18063,
"upload_time": "2025-08-18T03:07:44",
"upload_time_iso_8601": "2025-08-18T03:07:44.007859Z",
"url": "https://files.pythonhosted.org/packages/7d/6d/e6ae4ae47fa795cd6e3d05b2006dc9e8b31d207e544699666a0fb3d2fed1/oss_redteam-0.1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-18 03:07:44",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "kpower7",
"github_project": "oss-testing",
"github_not_found": true,
"lcname": "oss-redteam"
}
