# LogSentinelAI — Declarative LLM-Based Log Analyzer for Security Events, System Errors, and Anomalies
> **Benefits**: Transform unstructured logs into structured security intelligence by simply defining a Pydantic model—the LLM automatically extracts and validates data without manual parsing or regex rules.
**Keywords**: `AI log analysis` • `cybersecurity automation` • `SIEM integration` • `threat detection` • `DevSecOps` • `log monitoring` • `security intelligence` • `anomaly detection`
[](https://opensource.org/licenses/MIT)
[](https://github.com/call518/LogSentinelAI/actions/workflows/pypi-publish.yml)
[](https://deepwiki.com/call518/LogSentinelAI)
[](https://www.buymeacoffee.com/call518)
LogSentinelAI is an **AI-powered cybersecurity tool** that leverages LLM with **Declarative Extraction** to analyze security events, anomalies, and errors from various logs including Apache, Linux, and converts them into structured data for **SIEM integration** with Elasticsearch/Kibana. This **DevSecOps automation solution** enables **real-time threat detection** and **security monitoring** by simply declaring your desired result structure as a Pydantic class, and the AI automatically analyzes logs to return JSON matching that schema. No complex parsing or regex rules required.
---
## Dashboard Example
## JSON Output Example
## Telegram Alert Example
When critical security events are detected, LogSentinelAI can automatically send real-time alerts to Telegram:
```text
🚨 [CRITICAL+ EVENTS] 🚨
• Highest Severity: CRITICAL
• Immediate Attention: Not Required
📊 Alert Events Summary (1 total):
• CRITICAL: 1
📋 Summary
➤ The analysis indicates several potential security events in the system logs.
🔥 Event-1
• Severity: CRITICAL
• Event Type: AUTH_FAILURE
• Description: Multiple authentication failures attempted against the SSH daemon.
• Confidence: 0.9
• Human Review: Required
• Related Logs:
1. Jun 14 15:16:01 combo sshd(pam_unix)[19939]: authentication failure; logname= uid=0 euid=0 tty=NODEV...
2. Jun 14 15:16:02 combo sshd(pam_unix)[19937]: check pass; user unknown
3. Jun 15 02:04:59 combo sshd(pam_unix)[20882]: authentication failure; logname= uid=0 euid=0 tty=NODEV...
... and 5 more log entries
• Recommended Actions:
➤ Review login history and account activity for suspicious patterns.
➤ Implement multi-factor authentication to enhance security.
➤ Monitor network traffic for unauthorized connections.
📊 Statistics:
• total_events: 8
• auth_failures: 8
• unique_ips: 0
• unique_users: 0
🔍 ES/Kibana Metadata:
• Index: logsentinelai-analysis
• @chunk_analysis_start_utc: 2025-08-17T22:42:32Z
• @chunk_analysis_end_utc: 2025-08-17T22:43:02Z
• @chunk_analysis_elapsed_time: 30
• @processing_result: success
• @log_count: 10
• @processing_mode: batch
• @access_mode: local
• @llm_provider: vllm
• @llm_model: Qwen/Qwen2.5-1.5B-Instruct
• @log_path: /var/log/messages
• @token_size_input: 1834
• @token_size_output: 618
• @timestamp: 2025-08-17T22:43:02.261161
• @log_type: linux_system
• @document_id: linux_system_20250817_224302_261129_chunk_1
• @host: {"hostname":"linux.foo.com","ip_addresses":["123.123.123.123/24"]}
```
> Configure Telegram alerts by setting `TELEGRAM_ENABLED=true`, `TELEGRAM_TOKEN`, and `TELEGRAM_CHAT_ID` in your config file. Alerts are automatically sent for CRITICAL+ events (configurable via `TELEGRAM_ALERT_LEVEL`).
## Key Features
> ⚡️ **Declarative Extraction**
>
> In each analyzer script, simply declare the desired result structure as a Pydantic class, and the LLM will automatically analyze logs and return results as JSON matching that schema. No complex parsing or post-processing—just declare what you want, and the AI handles the rest. This approach enables developers to focus on "what to extract" declaratively, while the LLM takes care of "how to extract"—a modern paradigm for information extraction.
```python
# Example: Just declare the result structure you want in your HTTP Access log analyzer
from pydantic import BaseModel
class MyAccessLogResult(BaseModel):
ip: str
url: str
is_attack: bool
# By defining only the result structure (Pydantic class) like above,
# the LLM automatically analyzes each log and returns JSON like this:
# {
# "ip": "192.168.0.1",
# "url": "/admin.php",
# "is_attack": true
# }
```
## System Architecture
- **Log Sources**: Logs are collected from various sources, including local files, remote SSH connections, HTTP endpoints, Apache error logs, system logs, and TCPDump outputs.
- **LogSentinelAI Core**: Handles parsing and extraction using a declarative approach. Log structures are defined using Pydantic models, and the actual extraction is performed by LLMs. The system validates and structures the extracted data.
- **LLM Provider**: Integrates with external or local LLMs (e.g., OpenAI, vLLM, Ollama) to interpret and transform raw logs into structured JSON, based on user-defined schemas.
- **Elasticsearch**: Structured outputs, raw logs, and metadata are indexed into Elasticsearch for searchability and event correlation.
- **Kibana**: Provides visualization and dashboards for immediate insight into security events and operational data.
- **Telegram Alerts**: Automatically sends real-time notifications to Telegram groups/channels when CRITICAL security events are detected or processing failures occur, enabling immediate incident response.
### AI-powered Analysis
- **Declarative Extraction**: Just declare your desired result structure (Pydantic class) and the LLM analyzes logs automatically
- **LLM Providers**: OpenAI API, Ollama, vLLM
- **Supported Log Types**: HTTP Access, Apache Error, Linux System
- **Threat Detection**: SQL Injection, XSS, Brute Force, Network Anomaly Detection
- **Output**: Structured JSON validated by Pydantic
- **Just define a Pydantic class and the LLM generates results in that structure automatically**
- **Adaptive Sensitivity**: Detection sensitivity auto-adjusted by LLM model and log type prompt
### Processing Modes
- **Batch**: Bulk analysis of historical logs
- **Real-time**: Sampling-based live monitoring
- **Access Methods**: Local files, SSH remote
### Data Enrichment
- **GeoIP**: MaxMind GeoLite2 City lookup (including coordinates, Kibana geo_point support)
- **Statistics**: IP counts, response codes, various metrics
- **Multi-language Support**: Configurable result language (default: Korean)
### Integration & Output
- **Storage**: Elasticsearch (ILM policy support)
- **Visualization**: Kibana dashboard
- **Deployment**: Docker containers
- **Real-time Alerts**: Telegram notifications for CRITICAL security events and system failures
### CLI Command Mapping
```bash
# CLI commands mapped to analyzer scripts:
logsentinelai-httpd-access → analyzers/httpd_access.py
logsentinelai-httpd-server → analyzers/httpd_server.py
logsentinelai-linux-system → analyzers/linux_system.py
logsentinelai-geoip-download → utils/geoip_downloader.py
```
### Sample Log Preview
#### HTTP Access Log
```log
54.36.149.41 - - [22/Jan/2019:03:56:14 +0330] "GET /filter/27|13%20%D9%85%DA%AF%D8%A7%D9%BE%DB%8C%DA%A9%D8%B3%D9%84,27|%DA%A9%D9%85%D8%AA%D8%B1%20%D8%A7%D8%B2%205%20%D9%85%DA%AF%D8%A7%D9%BE%DB%8C%DA%A9%D8%B3%D9%84,p53 HTTP/1.1" 200 30577 "-" "Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)" "-"
31.56.96.51 - - [22/Jan/2019:03:56:16 +0330] "GET /image/60844/productModel/200x200 HTTP/1.1" 200 5667 "https://www.zanbil.ir/m/filter/b113" "Mozilla/5.0 (Linux; Android 6.0; ALE-L21 Build/HuaweiALE-L21) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.158 Mobile Safari/537.36" "-"
31.56.96.51 - - [22/Jan/2019:03:56:16 +0330] "GET /image/61474/productModel/200x200 HTTP/1.1" 200 5379 "https://www.zanbil.ir/m/filter/b113" "Mozilla/5.0 (Linux; Android 6.0; ALE-L21 Build/HuaweiALE-L21) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.158 Mobile Safari/537.36" "-"
40.77.167.129 - - [22/Jan/2019:03:56:17 +0330] "GET /image/14925/productModel/100x100 HTTP/1.1" 200 1696 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "-"
91.99.72.15 - - [22/Jan/2019:03:56:17 +0330] "GET /product/31893/62100/%D8%B3%D8%B4%D9%88%D8%A7%D8%B1-%D8%AE%D8%A7%D9%86%DA%AF%DB%8C-%D9%BE%D8%B1%D9%86%D8%B3%D9%84%DB%8C-%D9%85%D8%AF%D9%84-PR257AT HTTP/1.1" 200 41483 "-" "Mozilla/5.0 (Windows NT 6.2; Win64; x64; rv:16.0)Gecko/16.0 Firefox/16.0" "-"
40.77.167.129 - - [22/Jan/2019:03:56:17 +0330] "GET /image/23488/productModel/150x150 HTTP/1.1" 200 2654 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "-"
40.77.167.129 - - [22/Jan/2019:03:56:18 +0330] "GET /image/45437/productModel/150x150 HTTP/1.1" 200 3688 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "-"
40.77.167.129 - - [22/Jan/2019:03:56:18 +0330] "GET /image/576/article/100x100 HTTP/1.1" 200 14776 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "-"
66.249.66.194 - - [22/Jan/2019:03:56:18 +0330] "GET /filter/b41,b665,c150%7C%D8%A8%D8%AE%D8%A7%D8%B1%D9%BE%D8%B2,p56 HTTP/1.1" 200 34277 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"
40.77.167.129 - - [22/Jan/2019:03:56:18 +0330] "GET /image/57710/productModel/100x100 HTTP/1.1" 200 1695 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "-"
```
#### Apache Server Log
```log
[Thu Jun 09 06:07:04 2005] [notice] LDAP: Built with OpenLDAP LDAP SDK
[Thu Jun 09 06:07:04 2005] [notice] LDAP: SSL support unavailable
[Thu Jun 09 06:07:04 2005] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Jun 09 06:07:05 2005] [notice] Digest: generating secret for digest authentication ...
[Thu Jun 09 06:07:05 2005] [notice] Digest: done
[Thu Jun 09 06:07:05 2005] [notice] LDAP: Built with OpenLDAP LDAP SDK
[Thu Jun 09 06:07:05 2005] [notice] LDAP: SSL support unavailable
[Thu Jun 09 06:07:05 2005] [error] env.createBean2(): Factory error creating channel.jni:jni ( channel.jni, jni)
[Thu Jun 09 06:07:05 2005] [error] config.update(): Can't create channel.jni:jni
[Thu Jun 09 06:07:05 2005] [error] env.createBean2(): Factory error creating vm: ( vm, )
```
#### Linux System Log
```log
Jun 14 15:16:01 combo sshd(pam_unix)[19939]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=218.188.2.4
Jun 14 15:16:02 combo sshd(pam_unix)[19937]: check pass; user unknown
Jun 14 15:16:02 combo sshd(pam_unix)[19937]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=218.188.2.4
Jun 15 02:04:59 combo sshd(pam_unix)[20882]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net user=root
Jun 15 02:04:59 combo sshd(pam_unix)[20884]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net user=root
Jun 15 02:04:59 combo sshd(pam_unix)[20883]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net user=root
Jun 15 02:04:59 combo sshd(pam_unix)[20885]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net user=root
Jun 15 02:04:59 combo sshd(pam_unix)[20886]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net user=root
Jun 15 02:04:59 combo sshd(pam_unix)[20892]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net user=root
Jun 15 02:04:59 combo sshd(pam_unix)[20893]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net user=root
```
### More Public Sample Logs
To test more log types and formats, you can leverage this public sample logs repository:
- GitHub: [Sample Log Files Repository](https://github.com/SoftManiaTech/sample_log_files)
How to use with LogSentinelAI:
- Clone and pick appropriate files for your analyzer
- Use `--log-path` to point the analyzer CLI at the chosen file
## Frequently Asked Questions (FAQ)
### Q: How does LogSentinelAI differ from traditional log analysis tools?
**A**: Unlike traditional tools that require complex regex patterns and manual parsing rules, LogSentinelAI uses **declarative extraction** - you simply define a Pydantic model structure, and the LLM automatically extracts and validates security data. No programming required for new log formats.
### Q: Can I use LogSentinelAI for compliance and audit reporting?
**A**: Yes! LogSentinelAI provides structured JSON output with full audit trails, timestamps, and confidence scores - perfect for **SOX compliance**, **PCI DSS audits**, and **security incident reporting**. All analysis results are indexed in Elasticsearch for long-term retention.
### Q: Does it work with custom or proprietary log formats?
**A**: Absolutely! The AI can adapt to any log format. Simply create a new analyzer with your desired output schema, and the LLM will learn to parse your custom logs automatically. No need to write parsing logic.
### Q: Is it suitable for high-volume enterprise environments?
**A**: Yes, LogSentinelAI supports **real-time processing**, **batch analysis**, and **sampling-based monitoring** for high-volume scenarios. It integrates with enterprise SIEM solutions via Elasticsearch and provides **horizontal scaling** capabilities.
### Q: What about data privacy and on-premises deployment?
**A**: LogSentinelAI supports **local LLM deployment** using Ollama or vLLM - your logs never leave your infrastructure. Perfect for organizations with strict **data residency** and **privacy compliance** requirements.
- Some formats may require adapting analyzer prompts/schemas
## Installation Guide
**Requirements**: Python 3.11 or 3.12 (Python 3.13+ not supported due to dependency compatibility)
For installation, environment setup, CLI usage, Elasticsearch/Kibana integration, and all practical guides for LogSentinelAI, please refer to the installation documentation below.
**[Go to Installation and Usage Guide: INSTALL.md](./INSTALL.md)**
> ⚡️ For additional inquiries, please use GitHub Issues/Discussions!
## Acknowledgments
We would like to express our sincere gratitude to the following projects and communities that provided inspiration, guidance, and foundational technologies for LogSentinelAI:
### Core Technologies & Frameworks
- **[Outlines](https://dottxt-ai.github.io/outlines/latest/)** - Structured LLM output generation framework that powers our reliable AI analysis
- **[dottxt-ai Demos](https://github.com/dottxt-ai/demos/tree/main/logs)** - Excellent log analysis examples and implementation patterns
- **[STRESSED - YouTube](https://www.youtube.com/watch?v=csw6TVfzBcw)** - Creating a Structured AI Log Analysis System with Python & LLMs
- **[Docker ELK Stack](https://github.com/deviantony/docker-elk)** - Comprehensive Elasticsearch, Logstash, and Kibana Docker setup
### LLM Infrastructure & Deployment
- **[vLLM](https://github.com/vllm-project/vllm)** - High-performance LLM inference engine for GPU-accelerated local deployment
- **[Ollama](https://ollama.com/)** - Simplified local LLM deployment and management platform
### Open Source Community
We are deeply grateful to the broader open source community and the countless projects that have contributed to making AI-powered log analysis accessible and practical. This project stands on the shoulders of many innovative open source initiatives that continue to push the boundaries of what's possible.
## Contributing
🤝 **Got ideas? Found bugs? Want to add cool features?**
We're always excited to welcome new contributors! Whether you're fixing a typo, adding a new monitoring tool, or improving documentation - every contribution makes this project better.
**Ways to contribute:**
- 🐛 Report issues or bugs
- 💡 Suggest new PostgreSQL monitoring features
- 📝 Improve documentation
- 🚀 Submit pull requests
- ⭐ Star the repo if you find it useful!
## 📄 License
This project is licensed under the MIT License.
Raw data
{
"_id": null,
"home_page": null,
"name": "logsentinelai",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.13,>=3.11",
"maintainer_email": null,
"keywords": "security, log-analysis, ai, llm, cybersecurity, elasticsearch, threat-detection",
"author": null,
"author_email": "JungJungIn <call518@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/60/81/381fc72aef97b5b45e78f4cf621ec44202f5cd4d6a9e5daa5a8ffbd1016c/logsentinelai-1.1.7.tar.gz",
"platform": null,
"description": "# LogSentinelAI \u2014 Declarative LLM-Based Log Analyzer for Security Events, System Errors, and Anomalies\n\n> **Benefits**: Transform unstructured logs into structured security intelligence by simply defining a Pydantic model\u2014the LLM automatically extracts and validates data without manual parsing or regex rules.\n\n**Keywords**: `AI log analysis` \u2022 `cybersecurity automation` \u2022 `SIEM integration` \u2022 `threat detection` \u2022 `DevSecOps` \u2022 `log monitoring` \u2022 `security intelligence` \u2022 `anomaly detection`\n\n[](https://opensource.org/licenses/MIT)\n[](https://github.com/call518/LogSentinelAI/actions/workflows/pypi-publish.yml)\n[](https://deepwiki.com/call518/LogSentinelAI)\n[](https://www.buymeacoffee.com/call518)\n\nLogSentinelAI is an **AI-powered cybersecurity tool** that leverages LLM with **Declarative Extraction** to analyze security events, anomalies, and errors from various logs including Apache, Linux, and converts them into structured data for **SIEM integration** with Elasticsearch/Kibana. This **DevSecOps automation solution** enables **real-time threat detection** and **security monitoring** by simply declaring your desired result structure as a Pydantic class, and the AI automatically analyzes logs to return JSON matching that schema. No complex parsing or regex rules required.\n\n---\n\n## Dashboard Example\n\n\n\n## JSON Output Example\n\n\n\n## Telegram Alert Example\n\nWhen critical security events are detected, LogSentinelAI can automatically send real-time alerts to Telegram:\n\n```text\n\ud83d\udea8 [CRITICAL+ EVENTS] \ud83d\udea8\n \u2022 Highest Severity: CRITICAL\n \u2022 Immediate Attention: Not Required\n\n\ud83d\udcca Alert Events Summary (1 total):\n \u2022 CRITICAL: 1\n\n\ud83d\udccb Summary\n \u27a4 The analysis indicates several potential security events in the system logs.\n\n\ud83d\udd25 Event-1\n \u2022 Severity: CRITICAL\n \u2022 Event Type: AUTH_FAILURE\n \u2022 Description: Multiple authentication failures attempted against the SSH daemon.\n \u2022 Confidence: 0.9\n \u2022 Human Review: Required\n \u2022 Related Logs:\n 1. Jun 14 15:16:01 combo sshd(pam_unix)[19939]: authentication failure; logname= uid=0 euid=0 tty=NODEV...\n 2. Jun 14 15:16:02 combo sshd(pam_unix)[19937]: check pass; user unknown\n 3. Jun 15 02:04:59 combo sshd(pam_unix)[20882]: authentication failure; logname= uid=0 euid=0 tty=NODEV...\n ... and 5 more log entries\n \u2022 Recommended Actions:\n \u27a4 Review login history and account activity for suspicious patterns.\n \u27a4 Implement multi-factor authentication to enhance security.\n \u27a4 Monitor network traffic for unauthorized connections.\n\n\ud83d\udcca Statistics:\n \u2022 total_events: 8\n \u2022 auth_failures: 8\n \u2022 unique_ips: 0\n \u2022 unique_users: 0\n\n\ud83d\udd0d ES/Kibana Metadata:\n \u2022 Index: logsentinelai-analysis\n \u2022 @chunk_analysis_start_utc: 2025-08-17T22:42:32Z\n \u2022 @chunk_analysis_end_utc: 2025-08-17T22:43:02Z\n \u2022 @chunk_analysis_elapsed_time: 30\n \u2022 @processing_result: success\n \u2022 @log_count: 10\n \u2022 @processing_mode: batch\n \u2022 @access_mode: local\n \u2022 @llm_provider: vllm\n \u2022 @llm_model: Qwen/Qwen2.5-1.5B-Instruct\n \u2022 @log_path: /var/log/messages\n \u2022 @token_size_input: 1834\n \u2022 @token_size_output: 618\n \u2022 @timestamp: 2025-08-17T22:43:02.261161\n \u2022 @log_type: linux_system\n \u2022 @document_id: linux_system_20250817_224302_261129_chunk_1\n \u2022 @host: {\"hostname\":\"linux.foo.com\",\"ip_addresses\":[\"123.123.123.123/24\"]}\n```\n\n> Configure Telegram alerts by setting `TELEGRAM_ENABLED=true`, `TELEGRAM_TOKEN`, and `TELEGRAM_CHAT_ID` in your config file. Alerts are automatically sent for CRITICAL+ events (configurable via `TELEGRAM_ALERT_LEVEL`).\n\n## Key Features\n\n> \u26a1\ufe0f **Declarative Extraction**\n>\n> In each analyzer script, simply declare the desired result structure as a Pydantic class, and the LLM will automatically analyze logs and return results as JSON matching that schema. No complex parsing or post-processing\u2014just declare what you want, and the AI handles the rest. This approach enables developers to focus on \"what to extract\" declaratively, while the LLM takes care of \"how to extract\"\u2014a modern paradigm for information extraction.\n\n```python\n# Example: Just declare the result structure you want in your HTTP Access log analyzer\nfrom pydantic import BaseModel\n\nclass MyAccessLogResult(BaseModel):\n ip: str\n url: str\n is_attack: bool\n\n# By defining only the result structure (Pydantic class) like above,\n# the LLM automatically analyzes each log and returns JSON like this:\n# {\n# \"ip\": \"192.168.0.1\",\n# \"url\": \"/admin.php\",\n# \"is_attack\": true\n# }\n```\n\n## System Architecture\n\n\n\n- **Log Sources**: Logs are collected from various sources, including local files, remote SSH connections, HTTP endpoints, Apache error logs, system logs, and TCPDump outputs.\n- **LogSentinelAI Core**: Handles parsing and extraction using a declarative approach. Log structures are defined using Pydantic models, and the actual extraction is performed by LLMs. The system validates and structures the extracted data.\n- **LLM Provider**: Integrates with external or local LLMs (e.g., OpenAI, vLLM, Ollama) to interpret and transform raw logs into structured JSON, based on user-defined schemas.\n- **Elasticsearch**: Structured outputs, raw logs, and metadata are indexed into Elasticsearch for searchability and event correlation.\n- **Kibana**: Provides visualization and dashboards for immediate insight into security events and operational data.\n- **Telegram Alerts**: Automatically sends real-time notifications to Telegram groups/channels when CRITICAL security events are detected or processing failures occur, enabling immediate incident response.\n\n### AI-powered Analysis\n\n- **Declarative Extraction**: Just declare your desired result structure (Pydantic class) and the LLM analyzes logs automatically\n- **LLM Providers**: OpenAI API, Ollama, vLLM\n- **Supported Log Types**: HTTP Access, Apache Error, Linux System\n- **Threat Detection**: SQL Injection, XSS, Brute Force, Network Anomaly Detection\n- **Output**: Structured JSON validated by Pydantic\n- **Just define a Pydantic class and the LLM generates results in that structure automatically**\n- **Adaptive Sensitivity**: Detection sensitivity auto-adjusted by LLM model and log type prompt\n\n### Processing Modes\n\n- **Batch**: Bulk analysis of historical logs\n- **Real-time**: Sampling-based live monitoring\n- **Access Methods**: Local files, SSH remote\n\n### Data Enrichment\n\n- **GeoIP**: MaxMind GeoLite2 City lookup (including coordinates, Kibana geo_point support)\n- **Statistics**: IP counts, response codes, various metrics\n- **Multi-language Support**: Configurable result language (default: Korean)\n\n### Integration & Output\n\n- **Storage**: Elasticsearch (ILM policy support)\n- **Visualization**: Kibana dashboard\n- **Deployment**: Docker containers\n- **Real-time Alerts**: Telegram notifications for CRITICAL security events and system failures\n\n### CLI Command Mapping\n\n```bash\n# CLI commands mapped to analyzer scripts:\nlogsentinelai-httpd-access \u2192 analyzers/httpd_access.py\nlogsentinelai-httpd-server \u2192 analyzers/httpd_server.py \nlogsentinelai-linux-system \u2192 analyzers/linux_system.py\nlogsentinelai-geoip-download \u2192 utils/geoip_downloader.py\n```\n\n### Sample Log Preview\n\n#### HTTP Access Log\n\n```log\n54.36.149.41 - - [22/Jan/2019:03:56:14 +0330] \"GET /filter/27|13%20%D9%85%DA%AF%D8%A7%D9%BE%DB%8C%DA%A9%D8%B3%D9%84,27|%DA%A9%D9%85%D8%AA%D8%B1%20%D8%A7%D8%B2%205%20%D9%85%DA%AF%D8%A7%D9%BE%DB%8C%DA%A9%D8%B3%D9%84,p53 HTTP/1.1\" 200 30577 \"-\" \"Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)\" \"-\"\n31.56.96.51 - - [22/Jan/2019:03:56:16 +0330] \"GET /image/60844/productModel/200x200 HTTP/1.1\" 200 5667 \"https://www.zanbil.ir/m/filter/b113\" \"Mozilla/5.0 (Linux; Android 6.0; ALE-L21 Build/HuaweiALE-L21) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.158 Mobile Safari/537.36\" \"-\"\n31.56.96.51 - - [22/Jan/2019:03:56:16 +0330] \"GET /image/61474/productModel/200x200 HTTP/1.1\" 200 5379 \"https://www.zanbil.ir/m/filter/b113\" \"Mozilla/5.0 (Linux; Android 6.0; ALE-L21 Build/HuaweiALE-L21) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.158 Mobile Safari/537.36\" \"-\"\n40.77.167.129 - - [22/Jan/2019:03:56:17 +0330] \"GET /image/14925/productModel/100x100 HTTP/1.1\" 200 1696 \"-\" \"Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)\" \"-\"\n91.99.72.15 - - [22/Jan/2019:03:56:17 +0330] \"GET /product/31893/62100/%D8%B3%D8%B4%D9%88%D8%A7%D8%B1-%D8%AE%D8%A7%D9%86%DA%AF%DB%8C-%D9%BE%D8%B1%D9%86%D8%B3%D9%84%DB%8C-%D9%85%D8%AF%D9%84-PR257AT HTTP/1.1\" 200 41483 \"-\" \"Mozilla/5.0 (Windows NT 6.2; Win64; x64; rv:16.0)Gecko/16.0 Firefox/16.0\" \"-\"\n40.77.167.129 - - [22/Jan/2019:03:56:17 +0330] \"GET /image/23488/productModel/150x150 HTTP/1.1\" 200 2654 \"-\" \"Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)\" \"-\"\n40.77.167.129 - - [22/Jan/2019:03:56:18 +0330] \"GET /image/45437/productModel/150x150 HTTP/1.1\" 200 3688 \"-\" \"Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)\" \"-\"\n40.77.167.129 - - [22/Jan/2019:03:56:18 +0330] \"GET /image/576/article/100x100 HTTP/1.1\" 200 14776 \"-\" \"Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)\" \"-\"\n66.249.66.194 - - [22/Jan/2019:03:56:18 +0330] \"GET /filter/b41,b665,c150%7C%D8%A8%D8%AE%D8%A7%D8%B1%D9%BE%D8%B2,p56 HTTP/1.1\" 200 34277 \"-\" \"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\" \"-\"\n40.77.167.129 - - [22/Jan/2019:03:56:18 +0330] \"GET /image/57710/productModel/100x100 HTTP/1.1\" 200 1695 \"-\" \"Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)\" \"-\"\n```\n\n#### Apache Server Log\n\n```log\n[Thu Jun 09 06:07:04 2005] [notice] LDAP: Built with OpenLDAP LDAP SDK\n[Thu Jun 09 06:07:04 2005] [notice] LDAP: SSL support unavailable\n[Thu Jun 09 06:07:04 2005] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)\n[Thu Jun 09 06:07:05 2005] [notice] Digest: generating secret for digest authentication ...\n[Thu Jun 09 06:07:05 2005] [notice] Digest: done\n[Thu Jun 09 06:07:05 2005] [notice] LDAP: Built with OpenLDAP LDAP SDK\n[Thu Jun 09 06:07:05 2005] [notice] LDAP: SSL support unavailable\n[Thu Jun 09 06:07:05 2005] [error] env.createBean2(): Factory error creating channel.jni:jni ( channel.jni, jni)\n[Thu Jun 09 06:07:05 2005] [error] config.update(): Can't create channel.jni:jni\n[Thu Jun 09 06:07:05 2005] [error] env.createBean2(): Factory error creating vm: ( vm, )\n```\n\n#### Linux System Log\n\n```log\nJun 14 15:16:01 combo sshd(pam_unix)[19939]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=218.188.2.4 \nJun 14 15:16:02 combo sshd(pam_unix)[19937]: check pass; user unknown\nJun 14 15:16:02 combo sshd(pam_unix)[19937]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=218.188.2.4 \nJun 15 02:04:59 combo sshd(pam_unix)[20882]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net user=root\nJun 15 02:04:59 combo sshd(pam_unix)[20884]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net user=root\nJun 15 02:04:59 combo sshd(pam_unix)[20883]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net user=root\nJun 15 02:04:59 combo sshd(pam_unix)[20885]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net user=root\nJun 15 02:04:59 combo sshd(pam_unix)[20886]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net user=root\nJun 15 02:04:59 combo sshd(pam_unix)[20892]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net user=root\nJun 15 02:04:59 combo sshd(pam_unix)[20893]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net user=root\n```\n\n### More Public Sample Logs\n\nTo test more log types and formats, you can leverage this public sample logs repository:\n\n- GitHub: [Sample Log Files Repository](https://github.com/SoftManiaTech/sample_log_files)\n\nHow to use with LogSentinelAI:\n\n- Clone and pick appropriate files for your analyzer\n- Use `--log-path` to point the analyzer CLI at the chosen file\n\n## Frequently Asked Questions (FAQ)\n\n### Q: How does LogSentinelAI differ from traditional log analysis tools?\n\n**A**: Unlike traditional tools that require complex regex patterns and manual parsing rules, LogSentinelAI uses **declarative extraction** - you simply define a Pydantic model structure, and the LLM automatically extracts and validates security data. No programming required for new log formats.\n\n### Q: Can I use LogSentinelAI for compliance and audit reporting?\n\n**A**: Yes! LogSentinelAI provides structured JSON output with full audit trails, timestamps, and confidence scores - perfect for **SOX compliance**, **PCI DSS audits**, and **security incident reporting**. All analysis results are indexed in Elasticsearch for long-term retention.\n\n### Q: Does it work with custom or proprietary log formats?\n\n**A**: Absolutely! The AI can adapt to any log format. Simply create a new analyzer with your desired output schema, and the LLM will learn to parse your custom logs automatically. No need to write parsing logic.\n\n### Q: Is it suitable for high-volume enterprise environments?\n\n**A**: Yes, LogSentinelAI supports **real-time processing**, **batch analysis**, and **sampling-based monitoring** for high-volume scenarios. It integrates with enterprise SIEM solutions via Elasticsearch and provides **horizontal scaling** capabilities.\n\n### Q: What about data privacy and on-premises deployment?\n\n**A**: LogSentinelAI supports **local LLM deployment** using Ollama or vLLM - your logs never leave your infrastructure. Perfect for organizations with strict **data residency** and **privacy compliance** requirements.\n\n- Some formats may require adapting analyzer prompts/schemas\n\n## Installation Guide\n\n**Requirements**: Python 3.11 or 3.12 (Python 3.13+ not supported due to dependency compatibility)\n\nFor installation, environment setup, CLI usage, Elasticsearch/Kibana integration, and all practical guides for LogSentinelAI, please refer to the installation documentation below.\n\n**[Go to Installation and Usage Guide: INSTALL.md](./INSTALL.md)**\n\n> \u26a1\ufe0f For additional inquiries, please use GitHub Issues/Discussions!\n\n## Acknowledgments\n\nWe would like to express our sincere gratitude to the following projects and communities that provided inspiration, guidance, and foundational technologies for LogSentinelAI:\n\n### Core Technologies & Frameworks\n\n- **[Outlines](https://dottxt-ai.github.io/outlines/latest/)** - Structured LLM output generation framework that powers our reliable AI analysis\n- **[dottxt-ai Demos](https://github.com/dottxt-ai/demos/tree/main/logs)** - Excellent log analysis examples and implementation patterns\n- **[STRESSED - YouTube](https://www.youtube.com/watch?v=csw6TVfzBcw)** - Creating a Structured AI Log Analysis System with Python & LLMs\n- **[Docker ELK Stack](https://github.com/deviantony/docker-elk)** - Comprehensive Elasticsearch, Logstash, and Kibana Docker setup\n\n### LLM Infrastructure & Deployment\n\n- **[vLLM](https://github.com/vllm-project/vllm)** - High-performance LLM inference engine for GPU-accelerated local deployment\n- **[Ollama](https://ollama.com/)** - Simplified local LLM deployment and management platform\n\n### Open Source Community\n\nWe are deeply grateful to the broader open source community and the countless projects that have contributed to making AI-powered log analysis accessible and practical. This project stands on the shoulders of many innovative open source initiatives that continue to push the boundaries of what's possible.\n\n## Contributing\n\n\ud83e\udd1d **Got ideas? Found bugs? Want to add cool features?**\n\nWe're always excited to welcome new contributors! Whether you're fixing a typo, adding a new monitoring tool, or improving documentation - every contribution makes this project better.\n\n**Ways to contribute:**\n- \ud83d\udc1b Report issues or bugs\n- \ud83d\udca1 Suggest new PostgreSQL monitoring features\n- \ud83d\udcdd Improve documentation \n- \ud83d\ude80 Submit pull requests\n- \u2b50 Star the repo if you find it useful!\n\n## \ud83d\udcc4 License\n\nThis project is licensed under the MIT License.\n",
"bugtrack_url": null,
"license": null,
"summary": "AI-Powered Log Analyzer - Leverages LLM to analyze log files and detect security events",
"version": "1.1.7",
"project_urls": {
"Documentation": "https://github.com/call518/LogSentinelAI#readme",
"Homepage": "https://github.com/call518/LogSentinelAI",
"Issues": "https://github.com/call518/LogSentinelAI/issues",
"Repository": "https://github.com/call518/LogSentinelAI.git"
},
"split_keywords": [
"security",
" log-analysis",
" ai",
" llm",
" cybersecurity",
" elasticsearch",
" threat-detection"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "4bc237736f5fc9f44912214ba3c482747cae292cd2c13f90437f6a55cdc98a5f",
"md5": "e91941ccb98ad3e372a89d2eb84a0507",
"sha256": "9ae181750808dd56604a5c091ba289b21765508806db402df58639f2b06a82eb"
},
"downloads": -1,
"filename": "logsentinelai-1.1.7-py3-none-any.whl",
"has_sig": false,
"md5_digest": "e91941ccb98ad3e372a89d2eb84a0507",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.13,>=3.11",
"size": 68048,
"upload_time": "2025-09-05T18:23:07",
"upload_time_iso_8601": "2025-09-05T18:23:07.663287Z",
"url": "https://files.pythonhosted.org/packages/4b/c2/37736f5fc9f44912214ba3c482747cae292cd2c13f90437f6a55cdc98a5f/logsentinelai-1.1.7-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "6081381fc72aef97b5b45e78f4cf621ec44202f5cd4d6a9e5daa5a8ffbd1016c",
"md5": "78906848e645fabeef3b548374233d9f",
"sha256": "71633b88cf296025470c0a62c08ab6084cc67427f1a51e9ea5a2fd3b19b0546b"
},
"downloads": -1,
"filename": "logsentinelai-1.1.7.tar.gz",
"has_sig": false,
"md5_digest": "78906848e645fabeef3b548374233d9f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.13,>=3.11",
"size": 67215,
"upload_time": "2025-09-05T18:23:09",
"upload_time_iso_8601": "2025-09-05T18:23:09.457485Z",
"url": "https://files.pythonhosted.org/packages/60/81/381fc72aef97b5b45e78f4cf621ec44202f5cd4d6a9e5daa5a8ffbd1016c/logsentinelai-1.1.7.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-09-05 18:23:09",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "call518",
"github_project": "LogSentinelAI#readme",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "logsentinelai"
}
JSON
