# Security Lake Tools
[](https://github.com/tenzir/security-lake-tools/actions/workflows/test.yml)
[](https://badge.fury.io/py/security-lake-tools)
[](https://pypi.org/project/security-lake-tools/)
[](https://github.com/tenzir/security-lake-tools/blob/main/LICENSE)
Tools for managing AWS Security Lake custom sources with OCSF (Open Cybersecurity Schema Framework) support.
## Features
- ✨ Create Security Lake custom sources for all OCSF event classes
- 🔧 Automatic IAM role creation for AWS Glue crawlers
- 📋 Built-in OCSF event class mapping
- 🔍 Detailed error messages and troubleshooting guidance
- 🚀 Simple command-line interface
## Installation
### Using uvx (Recommended)
The easiest way to use this tool is with [uvx](https://github.com/astral-sh/uv), which runs the tool in an isolated environment:
```bash
# Run directly without installation
uvx --from security-lake-tools security-lake-create-source --help
# Or with a shorter alias
alias sl-create='uvx --from security-lake-tools security-lake-create-source'
sl-create 1001 --external-id your-external-id
```
### Traditional Installation
Using pip:
```bash
pip install security-lake-tools
```
Using [uv](https://github.com/astral-sh/uv):
```bash
uv pip install security-lake-tools
```
## Quick Start
### Create a custom source
```bash
# Using uvx (no installation needed)
uvx --from security-lake-tools security-lake-create-source 1001 \
--external-id your-external-id \
--region us-east-1
# Or if installed traditionally
security-lake-create-source 1001 \
--external-id your-external-id \
--region us-east-1
# With explicit configuration
uvx --from security-lake-tools security-lake-create-source 1001 \
--external-id your-external-id \
--region us-east-1 \
--account-id 123456789012 \
--profile production
```
### List available OCSF event classes
```bash
# Using uvx
uvx --from security-lake-tools security-lake-create-source --list
# Or if installed
security-lake-create-source --list
```
## Detailed Usage
### Prerequisites
1. **AWS Credentials**: Configure AWS credentials using one of:
- `aws configure`
- Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)
- IAM role (if running on EC2)
2. **Security Lake**: Ensure Security Lake is enabled in your target region
3. **IAM Permissions**: You need permissions to:
- Create IAM roles and policies
- Create Security Lake custom sources
- Create and manage Glue crawlers
### Command-Line Options
```bash
security-lake-create-source [OPTIONS] CLASS_UID
Arguments:
CLASS_UID OCSF class UID (e.g., 1001 for File System Activity)
Options:
--region AWS region (default: us-east-1)
--account-id AWS account ID (default: auto-detected)
--external-id External ID for trust relationship (required)
--glue-role-arn ARN of existing Glue service role
--profile AWS profile to use
--no-create-role Don't auto-create Glue role if missing
--skip-role-check Skip Glue role verification
--list List all available OCSF class UIDs
--help Show help message
```
### OCSF Event Classes
The tool supports all standard OCSF event classes:
#### System Activity (1xxx)
- 1001: File System Activity
- 1002: Kernel Extension Activity
- 1003: Kernel Activity
- 1004: Memory Activity
- 1005: Module Activity
- 1006: Scheduled Job Activity
- 1007: Process Activity
- 1008: Event Log Activity
- 1009: Script Activity
#### Findings (2xxx)
- 2001: Security Finding
- 2002: Vulnerability Finding
- 2003: Compliance Finding
- 2004: Detection Finding
- 2005: Incident Finding
- 2006: Data Security Finding
- 2007: Application Security Posture Finding
[See full list with `--list` option]
### IAM Role Management
By default, the tool automatically creates a Glue service role with:
- Trust relationship with `glue.amazonaws.com`
- AWS managed policy `AWSGlueServiceRole`
- Custom S3 policy for Security Lake buckets
- Lake Formation permissions
To use an existing role:
```bash
security-lake-create-source 1001 \
--external-id your-external-id \
--glue-role-arn arn:aws:iam::123456789012:role/MyExistingGlueRole
```
To prevent automatic role creation:
```bash
security-lake-create-source 1001 \
--external-id your-external-id \
--no-create-role
```
## What Gets Created
For each custom source, Security Lake creates:
1. **Provider Role**: `AmazonSecurityLake-Provider-{source-name}-{region}`
- Allows the specified account to write logs to Security Lake
2. **S3 Location**: `s3://aws-security-data-lake-{region}-{id}/ext/{source-name}/`
- Where your OCSF-formatted logs should be written
3. **Glue Resources**:
- Crawler: Discovers and catalogs your data
- Database: Stores metadata
- Table: Defines the schema
## Troubleshooting
### Common Issues
1. **"The Glue role does not exist"**
- Let the tool create it automatically (default behavior)
- Or create manually with proper permissions
- Or specify existing role with `--glue-role-arn`
2. **"Source already exists"**
- Delete the existing source first
- Or use a different class UID
3. **"Security Lake not enabled"**
- Enable Security Lake in the AWS Console
- Ensure you're using the correct region
4. **"Invalid principal" error**
- Ensure the account ID is correct
- Check that the external ID matches your configuration
### Debug Mode
For more detailed output, set the `AWS_DEBUG` environment variable:
```bash
AWS_DEBUG=1 security-lake-create-source 1001 --external-id test
```
## Development
### Setup
```bash
# Clone the repository
git clone https://github.com/yourusername/security-lake-tools
cd security-lake-tools
# Install with development dependencies using uv
uv pip install -e ".[dev]"
```
### Running Tests
```bash
# Run tests
uv run pytest
# With coverage
uv run pytest --cov=security_lake_tools
```
### Code Quality
```bash
# Format code
uv run black src tests
# Lint
uv run ruff check src tests
# Type checking
uv run mypy src
```
## Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
1. Fork the repository
2. Create your feature branch (`git checkout -b feature/AmazingFeature`)
3. Commit your changes (`git commit -m 'Add some AmazingFeature'`)
4. Push to the branch (`git push origin feature/AmazingFeature`)
5. Open a Pull Request
## License
This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.
## Acknowledgments
- AWS Security Lake team for the service
- OCSF community for the schema framework
- Contributors and users of this tool
Raw data
{
"_id": null,
"home_page": null,
"name": "security-lake-tools",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "aws, logging, ocsf, security, security-lake",
"author": null,
"author_email": "Tenzir <engineering@tenzir.com>",
"download_url": "https://files.pythonhosted.org/packages/44/8a/ca225e4d209363fa0f201c61a5daee5400acef02afcd4bae7b1eb7ef3518/security_lake_tools-0.1.2.tar.gz",
"platform": null,
"description": "# Security Lake Tools\n\n[](https://github.com/tenzir/security-lake-tools/actions/workflows/test.yml)\n[](https://badge.fury.io/py/security-lake-tools)\n[](https://pypi.org/project/security-lake-tools/)\n[](https://github.com/tenzir/security-lake-tools/blob/main/LICENSE)\n\nTools for managing AWS Security Lake custom sources with OCSF (Open Cybersecurity Schema Framework) support.\n\n## Features\n\n- \u2728 Create Security Lake custom sources for all OCSF event classes\n- \ud83d\udd27 Automatic IAM role creation for AWS Glue crawlers\n- \ud83d\udccb Built-in OCSF event class mapping\n- \ud83d\udd0d Detailed error messages and troubleshooting guidance\n- \ud83d\ude80 Simple command-line interface\n\n## Installation\n\n### Using uvx (Recommended)\n\nThe easiest way to use this tool is with [uvx](https://github.com/astral-sh/uv), which runs the tool in an isolated environment:\n\n```bash\n# Run directly without installation\nuvx --from security-lake-tools security-lake-create-source --help\n\n# Or with a shorter alias\nalias sl-create='uvx --from security-lake-tools security-lake-create-source'\nsl-create 1001 --external-id your-external-id\n```\n\n### Traditional Installation\n\nUsing pip:\n```bash\npip install security-lake-tools\n```\n\nUsing [uv](https://github.com/astral-sh/uv):\n```bash\nuv pip install security-lake-tools\n```\n\n## Quick Start\n\n### Create a custom source\n\n```bash\n# Using uvx (no installation needed)\nuvx --from security-lake-tools security-lake-create-source 1001 \\\n --external-id your-external-id \\\n --region us-east-1\n\n# Or if installed traditionally\nsecurity-lake-create-source 1001 \\\n --external-id your-external-id \\\n --region us-east-1\n\n# With explicit configuration\nuvx --from security-lake-tools security-lake-create-source 1001 \\\n --external-id your-external-id \\\n --region us-east-1 \\\n --account-id 123456789012 \\\n --profile production\n```\n\n### List available OCSF event classes\n\n```bash\n# Using uvx\nuvx --from security-lake-tools security-lake-create-source --list\n\n# Or if installed\nsecurity-lake-create-source --list\n```\n\n## Detailed Usage\n\n### Prerequisites\n\n1. **AWS Credentials**: Configure AWS credentials using one of:\n - `aws configure`\n - Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)\n - IAM role (if running on EC2)\n\n2. **Security Lake**: Ensure Security Lake is enabled in your target region\n\n3. **IAM Permissions**: You need permissions to:\n - Create IAM roles and policies\n - Create Security Lake custom sources\n - Create and manage Glue crawlers\n\n### Command-Line Options\n\n```bash\nsecurity-lake-create-source [OPTIONS] CLASS_UID\n\nArguments:\n CLASS_UID OCSF class UID (e.g., 1001 for File System Activity)\n\nOptions:\n --region AWS region (default: us-east-1)\n --account-id AWS account ID (default: auto-detected)\n --external-id External ID for trust relationship (required)\n --glue-role-arn ARN of existing Glue service role\n --profile AWS profile to use\n --no-create-role Don't auto-create Glue role if missing\n --skip-role-check Skip Glue role verification\n --list List all available OCSF class UIDs\n --help Show help message\n```\n\n### OCSF Event Classes\n\nThe tool supports all standard OCSF event classes:\n\n#### System Activity (1xxx)\n- 1001: File System Activity\n- 1002: Kernel Extension Activity\n- 1003: Kernel Activity\n- 1004: Memory Activity\n- 1005: Module Activity\n- 1006: Scheduled Job Activity\n- 1007: Process Activity\n- 1008: Event Log Activity\n- 1009: Script Activity\n\n#### Findings (2xxx)\n- 2001: Security Finding\n- 2002: Vulnerability Finding\n- 2003: Compliance Finding\n- 2004: Detection Finding\n- 2005: Incident Finding\n- 2006: Data Security Finding\n- 2007: Application Security Posture Finding\n\n[See full list with `--list` option]\n\n### IAM Role Management\n\nBy default, the tool automatically creates a Glue service role with:\n- Trust relationship with `glue.amazonaws.com`\n- AWS managed policy `AWSGlueServiceRole`\n- Custom S3 policy for Security Lake buckets\n- Lake Formation permissions\n\nTo use an existing role:\n```bash\nsecurity-lake-create-source 1001 \\\n --external-id your-external-id \\\n --glue-role-arn arn:aws:iam::123456789012:role/MyExistingGlueRole\n```\n\nTo prevent automatic role creation:\n```bash\nsecurity-lake-create-source 1001 \\\n --external-id your-external-id \\\n --no-create-role\n```\n\n## What Gets Created\n\nFor each custom source, Security Lake creates:\n\n1. **Provider Role**: `AmazonSecurityLake-Provider-{source-name}-{region}`\n - Allows the specified account to write logs to Security Lake\n\n2. **S3 Location**: `s3://aws-security-data-lake-{region}-{id}/ext/{source-name}/`\n - Where your OCSF-formatted logs should be written\n\n3. **Glue Resources**:\n - Crawler: Discovers and catalogs your data\n - Database: Stores metadata\n - Table: Defines the schema\n\n## Troubleshooting\n\n### Common Issues\n\n1. **\"The Glue role does not exist\"**\n - Let the tool create it automatically (default behavior)\n - Or create manually with proper permissions\n - Or specify existing role with `--glue-role-arn`\n\n2. **\"Source already exists\"**\n - Delete the existing source first\n - Or use a different class UID\n\n3. **\"Security Lake not enabled\"**\n - Enable Security Lake in the AWS Console\n - Ensure you're using the correct region\n\n4. **\"Invalid principal\" error**\n - Ensure the account ID is correct\n - Check that the external ID matches your configuration\n\n### Debug Mode\n\nFor more detailed output, set the `AWS_DEBUG` environment variable:\n```bash\nAWS_DEBUG=1 security-lake-create-source 1001 --external-id test\n```\n\n## Development\n\n### Setup\n\n```bash\n# Clone the repository\ngit clone https://github.com/yourusername/security-lake-tools\ncd security-lake-tools\n\n# Install with development dependencies using uv\nuv pip install -e \".[dev]\"\n```\n\n### Running Tests\n\n```bash\n# Run tests\nuv run pytest\n\n# With coverage\nuv run pytest --cov=security_lake_tools\n```\n\n### Code Quality\n\n```bash\n# Format code\nuv run black src tests\n\n# Lint\nuv run ruff check src tests\n\n# Type checking\nuv run mypy src\n```\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n1. Fork the repository\n2. Create your feature branch (`git checkout -b feature/AmazingFeature`)\n3. Commit your changes (`git commit -m 'Add some AmazingFeature'`)\n4. Push to the branch (`git push origin feature/AmazingFeature`)\n5. Open a Pull Request\n\n## License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.\n\n## Acknowledgments\n\n- AWS Security Lake team for the service\n- OCSF community for the schema framework\n- Contributors and users of this tool",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Tools for managing AWS Security Lake custom sources",
"version": "0.1.2",
"project_urls": {
"Documentation": "https://github.com/tenzir/security-lake-tools#readme",
"Homepage": "https://github.com/tenzir/security-lake-tools",
"Issues": "https://github.com/tenzir/security-lake-tools/issues",
"Repository": "https://github.com/tenzir/security-lake-tools"
},
"split_keywords": [
"aws",
" logging",
" ocsf",
" security",
" security-lake"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "0a461f91fd16a33d864d49ae81e87999f3c63259d778a6d27a06df39276e4e66",
"md5": "0f42de699197856999f31ed624b4b0a1",
"sha256": "e350bb666858a008f0f7da5e6d9a4be9b922940e64c45d370e7a1f696a1b3093"
},
"downloads": -1,
"filename": "security_lake_tools-0.1.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0f42de699197856999f31ed624b4b0a1",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 13595,
"upload_time": "2025-08-06T00:36:04",
"upload_time_iso_8601": "2025-08-06T00:36:04.502728Z",
"url": "https://files.pythonhosted.org/packages/0a/46/1f91fd16a33d864d49ae81e87999f3c63259d778a6d27a06df39276e4e66/security_lake_tools-0.1.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "448aca225e4d209363fa0f201c61a5daee5400acef02afcd4bae7b1eb7ef3518",
"md5": "661b9e668059d1b50fee3ec7b5722c83",
"sha256": "baa5dd47cf23f29f22204301dc2c6c8a2e1d1be65a2a32439b3ca2ab0420c35b"
},
"downloads": -1,
"filename": "security_lake_tools-0.1.2.tar.gz",
"has_sig": false,
"md5_digest": "661b9e668059d1b50fee3ec7b5722c83",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 42226,
"upload_time": "2025-08-06T00:36:05",
"upload_time_iso_8601": "2025-08-06T00:36:05.697898Z",
"url": "https://files.pythonhosted.org/packages/44/8a/ca225e4d209363fa0f201c61a5daee5400acef02afcd4bae7b1eb7ef3518/security_lake_tools-0.1.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-06 00:36:05",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "tenzir",
"github_project": "security-lake-tools#readme",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "security-lake-tools"
}
JSON
