# langchain-noctis
[](https://github.com/NoctisNet/langchain-noctis-retriever/actions/workflows/publish.yml)
[](https://github.com/NoctisNet/langchain-noctis-retriever/actions/workflows/test.yml)
[](https://badge.fury.io/py/langchain-noctis)
[](https://pepy.tech/project/langchain-noctis)
<p align="center">
<a href="https://www.noctisnet.com/">
<img src="https://cdn.prod.website-files.com/676bea044cb361883d143c96/67856b0790789b726a0ebc9d_d95bdca26bbd06ee46513e0f4c579b7e_Logo%20-%20White-p-500.png" alt="Noctis Logo" width="400"/>
</a>
</p>
This package contains the LangChain integration with [Noctis API](https://www.noctisnet.com/), providing domain information and mail server retrieval functionality, along with other domain-related tools.
## Table of Contents
- [Installation](#installation)
- [Retrievers](#retrievers)
- [NoctisRetriever](#noctisretriever)
- [Features](#features)
- [Configuration Options](#configuration-options)
- [Tools](#tools)
- [NoctisTool](#noctistool)
- [Tool Features](#tool-features)
- [Tool Configuration](#tool-configuration)
- [Examples](#examples)
- [Practical Use Cases](#practical-use-cases)
- [Security Analysis and Threat Intelligence](#security-analysis-and-threat-intelligence)
- [Local Development](#local-development)
- [Prerequisites](#prerequisites)
- [Installation for Development](#installation-for-development)
- [Running Tests](#running-tests)
- [Running Examples](#running-examples)
- [Troubleshooting](#troubleshooting)
- [Common Issues](#common-issues)
- [Adding Verbose Logging](#adding-verbose-logging)
- [Developer Documentation](#developer-documentation)
## Installation
```bash
pip install -U langchain-noctis
```
And you should configure credentials by setting the following environment variables:
```bash
export NOCTIS_API_KEY=your-api-key
```
## Retrievers
### NoctisRetriever
`NoctisRetriever` provides a way to retrieve domain information and mail server details from the Noctis API. It automatically extracts domains from natural language queries and returns structured information about each domain.
```python
from langchain_noctis import NoctisRetriever
# Create a retriever
retriever = NoctisRetriever(
k=5, # Number of documents to return
include_mail_servers=True, # Include mail server information
include_organization_domains=True, # Include domains from the same organization
organization_domains_limit=20, # Limit number of organization domains
include_ip_sharing_domains=True, # Include domains sharing the same IP address
ip_sharing_domains_limit=20, # Limit number of IP sharing domains
parallel_requests=True # Process multiple domains in parallel
)
# Get domain information from a query with one or more domains
documents = retriever.invoke("Tell me about example.com and github.com")
# Get information about related domains from the same organization
documents = retriever.invoke("Find domains related to icann.org")
# Get information about domains sharing the same IP address
documents = retriever.invoke("Find domains that share the same IP as google.com")
# Use in a chain
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_openai import ChatOpenAI
def format_docs(docs):
return "\n\n".join(doc.page_content for doc in docs)
prompt = ChatPromptTemplate.from_template(
"""Answer the question based on the context.
Context: {context}
Question: {question}
"""
)
llm = ChatOpenAI(model="gpt-3.5-turbo")
chain = (
{"context": retriever | format_docs, "question": RunnablePassthrough()}
| prompt
| llm
| StrOutputParser()
)
result = chain.invoke("What mail servers does example.com use?")
result = chain.invoke("What other domains are managed by the same organization as github.com?")
result = chain.invoke("What domains share the same IP address as google.com?")
```
### Features
1. **Domain Extraction**: Automatically extracts domain names from natural language queries.
2. **Domain Information**: Retrieves comprehensive domain information including IP addresses, ASN details, and more.
3. **Mail Server Information**: Retrieves mail server (MX record) information for domains.
4. **Organization Domains**: Discovers other domains managed by the same organization.
5. **IP Sharing Domains**: Identifies domains that share the same IP address.
6. **Network Prefix Domains**: Finds domains on the same network prefix (subnet).
7. **Parallel Processing**: Efficiently processes multiple domains in parallel.
8. **Configurable Output**: Control how many documents are returned and what information to include.
### Configuration Options
- `k` (int, default=3): Number of documents to return.
- `include_mail_servers` (bool, default=True): Whether to include mail server information.
- `include_organization_domains` (bool, default=False): Whether to include other domains managed by the same organization.
- `organization_domains_limit` (int, default=25): Maximum number of organization domains to return.
- `include_ip_sharing_domains` (bool, default=False): Whether to include domains sharing the same IP address.
- `ip_sharing_domains_limit` (int, default=25): Maximum number of IP sharing domains to return.
- `include_network_prefix_domains` (bool, default=False): Whether to include domains on the same network prefix.
- `network_prefix_domains_limit` (int, default=25): Maximum number of network prefix domains to return.
- `parallel_requests` (bool, default=True): Process multiple domains in parallel.
- `max_workers` (int, default=5): Maximum number of parallel workers when `parallel_requests` is True.
## Tools
### NoctisTool
`NoctisTool` is a LangChain tool for retrieving domain information through the Noctis API. It's designed to be used in agent workflows, allowing LLMs to query specific domain information directly.
```python
from langchain_noctis import NoctisTool
from langchain_core.agents import AgentExecutor, create_openai_tools_agent
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
# Create the tool
tool = NoctisTool()
# Create an agent with the tool
llm = ChatOpenAI(model="gpt-4-turbo", temperature=0)
prompt = ChatPromptTemplate.from_messages([
("system", "You are a helpful assistant that analyzes domain information."),
("user", "{input}")
])
agent = create_openai_tools_agent(llm, [tool], prompt)
agent_executor = AgentExecutor(agent=agent, tools=[tool])
# Use the agent to analyze domains
result = agent_executor.invoke({
"input": "What mail servers does example.com use?"
})
print(result["output"])
# Use directly in tool calling
result = tool.invoke({
"domain": "example.com",
"info_type": "domain_info"
})
print(result)
```
### Tool Features
1. **Specific Domain Queries**: Allows direct queries about a specific domain without natural language extraction.
2. **Multiple Information Types**: Supports querying different types of domain information:
- `domain_info`: Basic domain information
- `mail_servers`: Mail server records
- `organization_domains`: Other domains managed by the same organization
- `ip_sharing_domains`: Domains sharing the same IP address
- `network_prefix_domains`: Domains on the same network prefix
3. **Formatted Output**: Provides well-formatted, human-readable responses suitable for LLM consumption.
4. **Error Handling**: Robust error handling for API exceptions and invalid queries.
5. **Tool Calling Integration**: Designed for integration with LLM tool calling capabilities.
### Tool Configuration
The `NoctisTool` accepts the following parameters:
- **During initialization**:
- `api_client` (Optional): A custom Noctis API client instance
- `domains_api` (Optional): A custom Noctis DomainsApi instance
- **During invocation**:
- `domain` (str, required): The domain name to retrieve information for
- `info_type` (str, default="domain_info"): The type of information to retrieve
Example invocation:
```python
# Direct tool invocation
tool = NoctisTool()
result = tool.invoke({
"domain": "github.com",
"info_type": "organization_domains"
})
# As a tool call via JSON format
result = tool.invoke({
"args": {
"domain": "google.com",
"info_type": "ip_sharing_domains"
},
"id": "1",
"name": tool.name,
"type": "tool_call"
})
```
## Examples
The `examples/` directory contains several examples demonstrating different features of the package:
### Retriever Examples
- **mail_server_example.py**: Shows how to retrieve mail server information for domains
- **organization_domains_example.py**: Demonstrates finding domains belonging to the same organization
- **ip_sharing_domains_example.py**: Shows how to find domains sharing the same IP address
- **security_analysis_example.py**: Advanced example of using the retriever for security analysis and threat intelligence
### Tool Examples
- **noctis_tool_example.py**: Comprehensive example showing various ways to use the NoctisTool:
- Direct invocation for different information types (domain_info, mail_servers, etc.)
- Tool calling format for agent integration
- Using the tool with an LLM agent
- Creating a custom enhanced tool with extended functionality
- **noctis_tool_security_analysis.py**: Advanced example implementing a domain security analyzer:
- Domain analysis with detailed security assessment
- Domain comparison for finding relationships
- Phishing domain detection and analysis
- LLM-powered risk assessment
To run the examples:
```bash
# Run retriever examples
python examples/mail_server_example.py
python examples/organization_domains_example.py
python examples/ip_sharing_domains_example.py
python examples/security_analysis_example.py
# Run tool examples
python examples/noctis_tool_example.py
python examples/noctis_tool_security_analysis.py
```
Each example requires a valid `NOCTIS_API_KEY` environment variable. The agent and security analysis examples also require an `OPENAI_API_KEY` environment variable.
## Practical Use Cases
### Security Analysis and Threat Intelligence
The NoctisRetriever can be used for security analysis and threat intelligence by identifying potentially suspicious domain relationships and patterns:
```python
from langchain_noctis import NoctisRetriever
# Create a retriever focused on security analysis
retriever = NoctisRetriever(
include_ip_sharing_domains=True,
ip_sharing_domains_limit=30,
include_organization_domains=True
)
# Analyze potential phishing domains
legitimate_domain = "paypal.com"
suspect_domain = "paypa1-secure.com" # Note the "1" instead of "l"
# Get information about both domains
legitimate_docs = retriever.invoke(f"Find information about {legitimate_domain}")
suspect_docs = retriever.invoke(f"Find information about {suspect_domain}")
# Check if domains share IP addresses
ip_sharing_docs = retriever.invoke(f"Find domains that share the same IP as {suspect_domain}")
# Pass to an LLM for analysis
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
llm = ChatOpenAI(model="gpt-3.5-turbo")
prompt = ChatPromptTemplate.from_template(
"""You are a cybersecurity analyst. Analyze these domain relationships:
{context}
Is {suspect_domain} likely to be a phishing attempt targeting {legitimate_domain}?
Provide a security assessment and recommended actions.
"""
)
# ... continue with chain creation and invocation
```
For a complete implementation, see [security_analysis_example.py](examples/security_analysis_example.py) in the examples directory. This example demonstrates:
1. Analyzing a set of domains for suspicious IP sharing patterns
2. Identifying domains that share IPs with multiple different domains (potential infrastructure overlap)
3. Generating security assessments using LangChain and LLMs
4. Detecting potential phishing or typosquatting domains
By combining domain intelligence from Noctis with LLMs, security teams can automate the initial analysis of domain relationships and identify potential threats more efficiently.
## Local Development
### Prerequisites
To set up the local development environment, you'll need:
- Python 3.8 or higher
- Poetry (for dependency management)
- A Noctis API key (sign up at [noctis.io](https://noctis.io))
- Optional: OpenAI API key (for LLM examples)
### Installation for Development
1. Clone the repository:
```bash
git clone https://github.com/yourusername/langchain-noctis-retriever.git
cd langchain-noctis-retriever
```
2. Install dependencies using Poetry:
```bash
poetry install --with test,lint,test_integration,codespell,typing
```
3. Set up your API keys as environment variables:
```bash
export NOCTIS_API_KEY=your-noctis-api-key
export OPENAI_API_KEY=your-openai-api-key # Optional, for LLM examples
```
### Running Tests
You can run the test suite using pytest:
```bash
# Run all tests
poetry run pytest
# Run unit tests only
poetry run pytest tests/unit_tests/
# Run integration tests only
poetry run pytest tests/integration_tests/
# Run with verbose output
poetry run pytest -v
```
The integration tests with the live API require a valid `NOCTIS_API_KEY` environment variable. Tests without this key will be skipped automatically.
### Running Examples
The package includes several example scripts to demonstrate different features:
```bash
# Run mail server example
poetry run python examples/mail_server_example.py
# Run organization domains example
poetry run python examples/organization_domains_example.py
# Run IP sharing domains example
poetry run python examples/ip_sharing_domains_example.py
# Run security analysis example
poetry run python examples/security_analysis_example.py
```
Each example requires a valid `NOCTIS_API_KEY` environment variable. Some examples (like the security analysis) also utilize OpenAI's LLM capabilities and require an `OPENAI_API_KEY` environment variable.
Note that running examples with real API calls may incur costs depending on your Noctis API subscription tier.
### Troubleshooting
#### Common Issues
1. **API Key Issues**:
- Ensure your `NOCTIS_API_KEY` is correctly set in your environment variables
- Verify that your API key is active and has sufficient permissions/credits
2. **Rate Limiting**:
- If you're making many requests, you might encounter rate limiting
- Use the `parallel_requests=False` option to serialize requests
#### Adding Verbose Logging
To enable debug logging for troubleshooting:
```python
import logging
logging.basicConfig(level=logging.DEBUG)
retriever = NoctisRetriever(
# your configuration here
)
```
## Developer Documentation
Comprehensive technical documentation is available in the `docs/` directory:
- **[Architecture Overview](architecture/architecture.md)** - High-level architecture and component explanations
- **[Developer Guide](architecture/developer_guide.md)** - Guide for using and extending the retriever
- **[Creating New Sources](architecture/creating_new_sources.md)** - How to implement custom information sources
- **[API Reference](architecture/api_reference.md)** - Complete reference of all classes and methods
The documentation covers both usage patterns and extension mechanisms.
This will output detailed logs about API calls, request processing, and any errors encountered.
Raw data
{
"_id": null,
"home_page": "https://github.com/NoctisNet/langchain-noctis",
"name": "langchain-noctis",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.9",
"maintainer_email": null,
"keywords": "langchain, noctis, domain, dns, security, ai, retriever, llm, tool",
"author": "Noctis Team",
"author_email": "support@noctis.io",
"download_url": "https://files.pythonhosted.org/packages/fe/f1/b1e64bf62e3995490801e0d32e288f2fa55ec73f5914a191df58fff2361e/langchain_noctis-0.0.1b1.tar.gz",
"platform": null,
"description": "# langchain-noctis\n\n[](https://github.com/NoctisNet/langchain-noctis-retriever/actions/workflows/publish.yml)\n[](https://github.com/NoctisNet/langchain-noctis-retriever/actions/workflows/test.yml)\n[](https://badge.fury.io/py/langchain-noctis)\n[](https://pepy.tech/project/langchain-noctis)\n\n\n<p align=\"center\">\n <a href=\"https://www.noctisnet.com/\">\n <img src=\"https://cdn.prod.website-files.com/676bea044cb361883d143c96/67856b0790789b726a0ebc9d_d95bdca26bbd06ee46513e0f4c579b7e_Logo%20-%20White-p-500.png\" alt=\"Noctis Logo\" width=\"400\"/>\n </a>\n</p>\n\nThis package contains the LangChain integration with [Noctis API](https://www.noctisnet.com/), providing domain information and mail server retrieval functionality, along with other domain-related tools.\n\n## Table of Contents\n- [Installation](#installation)\n- [Retrievers](#retrievers)\n - [NoctisRetriever](#noctisretriever)\n - [Features](#features)\n - [Configuration Options](#configuration-options)\n- [Tools](#tools)\n - [NoctisTool](#noctistool)\n - [Tool Features](#tool-features)\n - [Tool Configuration](#tool-configuration)\n- [Examples](#examples)\n- [Practical Use Cases](#practical-use-cases)\n - [Security Analysis and Threat Intelligence](#security-analysis-and-threat-intelligence)\n- [Local Development](#local-development)\n - [Prerequisites](#prerequisites)\n - [Installation for Development](#installation-for-development)\n - [Running Tests](#running-tests)\n - [Running Examples](#running-examples)\n - [Troubleshooting](#troubleshooting)\n - [Common Issues](#common-issues)\n - [Adding Verbose Logging](#adding-verbose-logging)\n- [Developer Documentation](#developer-documentation)\n\n## Installation\n\n```bash\npip install -U langchain-noctis\n```\n\nAnd you should configure credentials by setting the following environment variables:\n\n```bash\nexport NOCTIS_API_KEY=your-api-key\n```\n\n## Retrievers\n\n### NoctisRetriever\n\n`NoctisRetriever` provides a way to retrieve domain information and mail server details from the Noctis API. It automatically extracts domains from natural language queries and returns structured information about each domain.\n\n```python\nfrom langchain_noctis import NoctisRetriever\n\n# Create a retriever\nretriever = NoctisRetriever(\n k=5, # Number of documents to return\n include_mail_servers=True, # Include mail server information\n include_organization_domains=True, # Include domains from the same organization\n organization_domains_limit=20, # Limit number of organization domains\n include_ip_sharing_domains=True, # Include domains sharing the same IP address\n ip_sharing_domains_limit=20, # Limit number of IP sharing domains\n parallel_requests=True # Process multiple domains in parallel\n)\n\n# Get domain information from a query with one or more domains\ndocuments = retriever.invoke(\"Tell me about example.com and github.com\")\n\n# Get information about related domains from the same organization\ndocuments = retriever.invoke(\"Find domains related to icann.org\")\n\n# Get information about domains sharing the same IP address\ndocuments = retriever.invoke(\"Find domains that share the same IP as google.com\")\n\n# Use in a chain\nfrom langchain_core.output_parsers import StrOutputParser\nfrom langchain_core.prompts import ChatPromptTemplate\nfrom langchain_core.runnables import RunnablePassthrough\nfrom langchain_openai import ChatOpenAI\n\ndef format_docs(docs):\n return \"\\n\\n\".join(doc.page_content for doc in docs)\n\nprompt = ChatPromptTemplate.from_template(\n \"\"\"Answer the question based on the context.\n \n Context: {context}\n \n Question: {question}\n \"\"\"\n)\n\nllm = ChatOpenAI(model=\"gpt-3.5-turbo\")\n\nchain = (\n {\"context\": retriever | format_docs, \"question\": RunnablePassthrough()}\n | prompt\n | llm\n | StrOutputParser()\n)\n\nresult = chain.invoke(\"What mail servers does example.com use?\")\nresult = chain.invoke(\"What other domains are managed by the same organization as github.com?\")\nresult = chain.invoke(\"What domains share the same IP address as google.com?\")\n```\n\n### Features\n\n1. **Domain Extraction**: Automatically extracts domain names from natural language queries.\n2. **Domain Information**: Retrieves comprehensive domain information including IP addresses, ASN details, and more.\n3. **Mail Server Information**: Retrieves mail server (MX record) information for domains.\n4. **Organization Domains**: Discovers other domains managed by the same organization.\n5. **IP Sharing Domains**: Identifies domains that share the same IP address.\n6. **Network Prefix Domains**: Finds domains on the same network prefix (subnet).\n7. **Parallel Processing**: Efficiently processes multiple domains in parallel.\n8. **Configurable Output**: Control how many documents are returned and what information to include.\n\n### Configuration Options\n\n- `k` (int, default=3): Number of documents to return.\n- `include_mail_servers` (bool, default=True): Whether to include mail server information.\n- `include_organization_domains` (bool, default=False): Whether to include other domains managed by the same organization.\n- `organization_domains_limit` (int, default=25): Maximum number of organization domains to return.\n- `include_ip_sharing_domains` (bool, default=False): Whether to include domains sharing the same IP address.\n- `ip_sharing_domains_limit` (int, default=25): Maximum number of IP sharing domains to return.\n- `include_network_prefix_domains` (bool, default=False): Whether to include domains on the same network prefix.\n- `network_prefix_domains_limit` (int, default=25): Maximum number of network prefix domains to return.\n- `parallel_requests` (bool, default=True): Process multiple domains in parallel.\n- `max_workers` (int, default=5): Maximum number of parallel workers when `parallel_requests` is True.\n\n## Tools\n\n### NoctisTool\n\n`NoctisTool` is a LangChain tool for retrieving domain information through the Noctis API. It's designed to be used in agent workflows, allowing LLMs to query specific domain information directly.\n\n```python\nfrom langchain_noctis import NoctisTool\nfrom langchain_core.agents import AgentExecutor, create_openai_tools_agent\nfrom langchain_openai import ChatOpenAI\nfrom langchain_core.prompts import ChatPromptTemplate\n\n# Create the tool\ntool = NoctisTool()\n\n# Create an agent with the tool\nllm = ChatOpenAI(model=\"gpt-4-turbo\", temperature=0)\nprompt = ChatPromptTemplate.from_messages([\n (\"system\", \"You are a helpful assistant that analyzes domain information.\"),\n (\"user\", \"{input}\")\n])\n\nagent = create_openai_tools_agent(llm, [tool], prompt)\nagent_executor = AgentExecutor(agent=agent, tools=[tool])\n\n# Use the agent to analyze domains\nresult = agent_executor.invoke({\n \"input\": \"What mail servers does example.com use?\"\n})\nprint(result[\"output\"])\n\n# Use directly in tool calling\nresult = tool.invoke({\n \"domain\": \"example.com\", \n \"info_type\": \"domain_info\"\n})\nprint(result)\n```\n\n### Tool Features\n\n1. **Specific Domain Queries**: Allows direct queries about a specific domain without natural language extraction.\n2. **Multiple Information Types**: Supports querying different types of domain information:\n - `domain_info`: Basic domain information\n - `mail_servers`: Mail server records\n - `organization_domains`: Other domains managed by the same organization\n - `ip_sharing_domains`: Domains sharing the same IP address\n - `network_prefix_domains`: Domains on the same network prefix\n3. **Formatted Output**: Provides well-formatted, human-readable responses suitable for LLM consumption.\n4. **Error Handling**: Robust error handling for API exceptions and invalid queries.\n5. **Tool Calling Integration**: Designed for integration with LLM tool calling capabilities.\n\n### Tool Configuration\n\nThe `NoctisTool` accepts the following parameters:\n\n- **During initialization**:\n - `api_client` (Optional): A custom Noctis API client instance\n - `domains_api` (Optional): A custom Noctis DomainsApi instance\n\n- **During invocation**:\n - `domain` (str, required): The domain name to retrieve information for\n - `info_type` (str, default=\"domain_info\"): The type of information to retrieve\n\nExample invocation:\n\n```python\n# Direct tool invocation\ntool = NoctisTool()\nresult = tool.invoke({\n \"domain\": \"github.com\",\n \"info_type\": \"organization_domains\"\n})\n\n# As a tool call via JSON format\nresult = tool.invoke({\n \"args\": {\n \"domain\": \"google.com\",\n \"info_type\": \"ip_sharing_domains\"\n },\n \"id\": \"1\",\n \"name\": tool.name,\n \"type\": \"tool_call\"\n})\n```\n\n## Examples\n\nThe `examples/` directory contains several examples demonstrating different features of the package:\n\n### Retriever Examples\n\n- **mail_server_example.py**: Shows how to retrieve mail server information for domains\n- **organization_domains_example.py**: Demonstrates finding domains belonging to the same organization\n- **ip_sharing_domains_example.py**: Shows how to find domains sharing the same IP address\n- **security_analysis_example.py**: Advanced example of using the retriever for security analysis and threat intelligence\n\n### Tool Examples\n\n- **noctis_tool_example.py**: Comprehensive example showing various ways to use the NoctisTool:\n - Direct invocation for different information types (domain_info, mail_servers, etc.)\n - Tool calling format for agent integration\n - Using the tool with an LLM agent\n - Creating a custom enhanced tool with extended functionality\n\n- **noctis_tool_security_analysis.py**: Advanced example implementing a domain security analyzer:\n - Domain analysis with detailed security assessment\n - Domain comparison for finding relationships\n - Phishing domain detection and analysis\n - LLM-powered risk assessment\n\nTo run the examples:\n\n```bash\n# Run retriever examples\npython examples/mail_server_example.py\npython examples/organization_domains_example.py\npython examples/ip_sharing_domains_example.py\npython examples/security_analysis_example.py\n\n# Run tool examples\npython examples/noctis_tool_example.py\npython examples/noctis_tool_security_analysis.py\n```\n\nEach example requires a valid `NOCTIS_API_KEY` environment variable. The agent and security analysis examples also require an `OPENAI_API_KEY` environment variable.\n\n## Practical Use Cases\n\n### Security Analysis and Threat Intelligence\n\nThe NoctisRetriever can be used for security analysis and threat intelligence by identifying potentially suspicious domain relationships and patterns:\n\n```python\nfrom langchain_noctis import NoctisRetriever\n\n# Create a retriever focused on security analysis\nretriever = NoctisRetriever(\n include_ip_sharing_domains=True,\n ip_sharing_domains_limit=30,\n include_organization_domains=True\n)\n\n# Analyze potential phishing domains\nlegitimate_domain = \"paypal.com\"\nsuspect_domain = \"paypa1-secure.com\" # Note the \"1\" instead of \"l\"\n\n# Get information about both domains\nlegitimate_docs = retriever.invoke(f\"Find information about {legitimate_domain}\")\nsuspect_docs = retriever.invoke(f\"Find information about {suspect_domain}\")\n\n# Check if domains share IP addresses\nip_sharing_docs = retriever.invoke(f\"Find domains that share the same IP as {suspect_domain}\")\n\n# Pass to an LLM for analysis\nfrom langchain_openai import ChatOpenAI\nfrom langchain_core.prompts import ChatPromptTemplate\n\nllm = ChatOpenAI(model=\"gpt-3.5-turbo\")\nprompt = ChatPromptTemplate.from_template(\n \"\"\"You are a cybersecurity analyst. Analyze these domain relationships:\n \n {context}\n \n Is {suspect_domain} likely to be a phishing attempt targeting {legitimate_domain}?\n Provide a security assessment and recommended actions.\n \"\"\"\n)\n\n# ... continue with chain creation and invocation\n```\n\nFor a complete implementation, see [security_analysis_example.py](examples/security_analysis_example.py) in the examples directory. This example demonstrates:\n\n1. Analyzing a set of domains for suspicious IP sharing patterns\n2. Identifying domains that share IPs with multiple different domains (potential infrastructure overlap)\n3. Generating security assessments using LangChain and LLMs\n4. Detecting potential phishing or typosquatting domains\n\nBy combining domain intelligence from Noctis with LLMs, security teams can automate the initial analysis of domain relationships and identify potential threats more efficiently.\n\n## Local Development\n\n### Prerequisites\n\nTo set up the local development environment, you'll need:\n\n- Python 3.8 or higher\n- Poetry (for dependency management)\n- A Noctis API key (sign up at [noctis.io](https://noctis.io))\n- Optional: OpenAI API key (for LLM examples)\n\n### Installation for Development\n\n1. Clone the repository:\n```bash\ngit clone https://github.com/yourusername/langchain-noctis-retriever.git\ncd langchain-noctis-retriever\n```\n\n2. Install dependencies using Poetry:\n```bash\npoetry install --with test,lint,test_integration,codespell,typing\n```\n\n3. Set up your API keys as environment variables:\n```bash\nexport NOCTIS_API_KEY=your-noctis-api-key\nexport OPENAI_API_KEY=your-openai-api-key # Optional, for LLM examples\n```\n\n### Running Tests\n\nYou can run the test suite using pytest:\n\n```bash\n# Run all tests\npoetry run pytest\n\n# Run unit tests only\npoetry run pytest tests/unit_tests/\n\n# Run integration tests only\npoetry run pytest tests/integration_tests/\n\n# Run with verbose output\npoetry run pytest -v\n```\n\nThe integration tests with the live API require a valid `NOCTIS_API_KEY` environment variable. Tests without this key will be skipped automatically.\n\n### Running Examples\n\nThe package includes several example scripts to demonstrate different features:\n\n```bash\n# Run mail server example\npoetry run python examples/mail_server_example.py\n\n# Run organization domains example\npoetry run python examples/organization_domains_example.py\n\n# Run IP sharing domains example\npoetry run python examples/ip_sharing_domains_example.py\n\n# Run security analysis example\npoetry run python examples/security_analysis_example.py\n```\n\nEach example requires a valid `NOCTIS_API_KEY` environment variable. Some examples (like the security analysis) also utilize OpenAI's LLM capabilities and require an `OPENAI_API_KEY` environment variable.\n\nNote that running examples with real API calls may incur costs depending on your Noctis API subscription tier.\n\n### Troubleshooting\n\n#### Common Issues\n\n1. **API Key Issues**:\n - Ensure your `NOCTIS_API_KEY` is correctly set in your environment variables\n - Verify that your API key is active and has sufficient permissions/credits\n\n2. **Rate Limiting**:\n - If you're making many requests, you might encounter rate limiting\n - Use the `parallel_requests=False` option to serialize requests\n\n#### Adding Verbose Logging\n\nTo enable debug logging for troubleshooting:\n\n```python\nimport logging\nlogging.basicConfig(level=logging.DEBUG)\n\nretriever = NoctisRetriever(\n # your configuration here\n)\n```\n\n## Developer Documentation\n\nComprehensive technical documentation is available in the `docs/` directory:\n\n- **[Architecture Overview](architecture/architecture.md)** - High-level architecture and component explanations\n- **[Developer Guide](architecture/developer_guide.md)** - Guide for using and extending the retriever\n- **[Creating New Sources](architecture/creating_new_sources.md)** - How to implement custom information sources\n- **[API Reference](architecture/api_reference.md)** - Complete reference of all classes and methods\n\nThe documentation covers both usage patterns and extension mechanisms.\n\nThis will output detailed logs about API calls, request processing, and any errors encountered.",
"bugtrack_url": null,
"license": "MIT",
"summary": "An integration package connecting Noctis API and LangChain",
"version": "0.0.1b1",
"project_urls": {
"Bug Tracker": "https://github.com/NoctisNet/langchain-noctis/issues",
"Changelog": "https://github.com/NoctisNet/langchain-noctis/CHANGELOG.md",
"Documentation": "https://github.com/NoctisNet/langchain-noctis/blob/main/architecture/README.md",
"HomePage": "https://www.noctisnet.com/",
"Homepage": "https://github.com/NoctisNet/langchain-noctis",
"Release Notes": "https://github.com/NoctisNet/langchain-noctis/releases",
"Repository": "https://github.com/NoctisNet/langchain-noctis",
"Source Code": "https://github.com/NoctisNet/langchain-noctis"
},
"split_keywords": [
"langchain",
" noctis",
" domain",
" dns",
" security",
" ai",
" retriever",
" llm",
" tool"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "cd6ba55a1e46fcaa110c9b63ff90a2ddbad86085e5a599d25c7e0892c0de6e52",
"md5": "fc0e3c080adbfd6cdcb531f02821207d",
"sha256": "894fcfa819400284f562bc775b591cda2ed5ecc2ed5e9538b6dff1679d356bc3"
},
"downloads": -1,
"filename": "langchain_noctis-0.0.1b1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "fc0e3c080adbfd6cdcb531f02821207d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.9",
"size": 22718,
"upload_time": "2025-04-07T13:32:05",
"upload_time_iso_8601": "2025-04-07T13:32:05.084305Z",
"url": "https://files.pythonhosted.org/packages/cd/6b/a55a1e46fcaa110c9b63ff90a2ddbad86085e5a599d25c7e0892c0de6e52/langchain_noctis-0.0.1b1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "fef1b1e64bf62e3995490801e0d32e288f2fa55ec73f5914a191df58fff2361e",
"md5": "71c1d01ba36fcc7da8b5ce6e969b41ec",
"sha256": "5477265b46c519896d4a2b241ac5432ed5cd660a76548a23d403035c4b1dd064"
},
"downloads": -1,
"filename": "langchain_noctis-0.0.1b1.tar.gz",
"has_sig": false,
"md5_digest": "71c1d01ba36fcc7da8b5ce6e969b41ec",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.9",
"size": 20851,
"upload_time": "2025-04-07T13:32:06",
"upload_time_iso_8601": "2025-04-07T13:32:06.352546Z",
"url": "https://files.pythonhosted.org/packages/fe/f1/b1e64bf62e3995490801e0d32e288f2fa55ec73f5914a191df58fff2361e/langchain_noctis-0.0.1b1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-04-07 13:32:06",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "NoctisNet",
"github_project": "langchain-noctis",
"github_not_found": true,
"lcname": "langchain-noctis"
}
JSON
