# python-proto-importer
[](https://crates.io/crates/python-proto-importer)
[](https://pypi.org/project/python-proto-importer/)
[](https://github.com/K-dash/python-proto-importer/actions)
[](https://codecov.io/gh/K-dash/python-proto-importer)
Rust-based CLI to streamline Python gRPC/Protobuf workflows: generate code, stabilize imports, and run type checks in a single command. Ships as a PyPI package (via maturin) and as a Rust crate.
### Why this project (Motivation)
- **Fragile imports from stock protoc output**: vanilla `grpcio-tools`/`protoc` emit absolute imports (e.g. `import foo.bar_pb2`) that break when you move the generated tree, split packages, or embed code under a different root. This tool rewrites them into stable **relative imports** inside the generated package.
- **Package structure friction**: projects often forget to add `__init__.py` or need namespace packages. We can auto-create `__init__.py` (opt-in/out) to make the tree importable and CI-friendly.
- **Type-checking pain**: mixing generated `.py` and `.pyi` frequently leads to noisy type warnings. We optionally integrate with `mypy-protobuf` / `mypy-grpc`, and recommend `.pyi`-first verification via Pyright.
- **Non-reproducible, multi-step scripts**: teams maintain ad‑hoc scripts for generation, postprocessing, and verification. This CLI runs the full pipeline in one command and stores configuration in `pyproject.toml`.
- **Silent breakages**: generated trees “import” locally but fail in CI or different PYTHONPATHs. A built-in **import dry-run** validates the entire package layout deterministically.
### How it differs from existing tools
- **Postprocess with awareness of your output tree**: relative-import rewriting targets only `_pb2[_grpc]` modules that actually exist beneath your configured `out`, leaving third‑party modules (e.g. `google.protobuf`) untouched by default.
- **Package hygiene by default**: opt-in `__init__.py` generation and path‑robust computation (uses canonical paths) reduce environment‑dependent surprises.
- **Verification built-in**: import dry‑run for all generated modules, plus easy hooks to run `mypy`/`pyright` as part of the same command.
- **Single source of truth in `pyproject.toml`**: keeps your team’s proto generation policy declarative and reviewable.
- **Fast, portable binary**: implemented in Rust with a small runtime footprint; distributed both on PyPI (wheel) and crates.io.
- **Backends**: `protoc` (v0.1), `buf generate` (planned v0.2)
- **Postprocess**: convert internal imports to relative; generate `__init__.py`
- **Typing**: optional `mypy-protobuf` / `mypy-grpc` emission
- **Verification**: import dry-run, optional mypy/pyright
For Japanese documentation, see: [docs/日本語 README](doc/README.ja.md)
## Table of Contents
- [Quick Start](#quick-start)
- [Commands](#commands)
- [Configuration](#configuration)
- [Core Configuration](#core-configuration)
- [Postprocess Configuration](#postprocess-configuration)
- [Verification Configuration](#verification-configuration)
- [Configuration Examples](#configuration-examples)
- [Advanced Usage](#advanced-usage)
- [Limitations](#limitations)
- [Contributing](#contributing)
- [License](#license)
## Quick Start
```bash
pip install python-proto-importer
# or
cargo install python-proto-importer
```
Create a `pyproject.toml` with your configuration:
```toml
[tool.python_proto_importer]
backend = "protoc"
python_exe = "python3"
include = ["proto"]
inputs = ["proto/**/*.proto"]
out = "generated/python"
```
Run the build:
```bash
proto-importer build
```
## Commands
### `proto-importer doctor`
Environment diagnostics with versions and helpful hints:
- Detects Python runner (uv/python) and shows versions
- Checks for `grpcio-tools` (required), `mypy-protobuf` / `mypy-grpc` (optional per config)
- Shows `protoc` / `buf` versions (informational in v0.1)
- Checks `mypy` / `pyright` CLIs and prints hints if configured but missing
### `proto-importer build [--pyproject PATH]`
Generate Python code from proto files, apply postprocessing, and run verification.
Options:
- `--pyproject PATH`: Path to pyproject.toml (default: `./pyproject.toml`)
- `--no-verify`: Skip verification after generation
- `--postprocess-only`: Skip generation, only run postprocessing (experimental)
### `proto-importer check [--pyproject PATH]`
Run verification only (import dry-run and type checks) without generation.
### `proto-importer clean [--pyproject PATH] --yes`
Remove generated output directory. Requires `--yes` confirmation.
## Configuration
All configuration is done through `pyproject.toml` under the `[tool.python_proto_importer]` section.
### Core Configuration
| Option | Type | Default | Description |
| ------------ | ------- | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `backend` | string | `"protoc"` | Code generation backend. Currently only `"protoc"` is supported. `"buf"` planned for v0.2. |
| `python_exe` | string | `"python3"` | Python executable to use for generation and verification. Can be `"python3"`, `"python"`, `"uv"` (fully tested), or a path like `".venv/bin/python"`. |
| `include` | array | `["."]` | Proto import paths (passed as `--proto_path` to protoc). Empty array defaults to `["."]`. See [Include Path Behavior](#include-path-behavior) for details. |
| `inputs` | array | `[]` | Glob patterns for proto files to generate. Example: `["proto/**/*.proto"]`. Files are filtered by `include` paths. |
| `out` | string | `"generated/python"` | Output directory for generated Python files. |
| `mypy` | boolean | `false` | Generate mypy type stubs (`.pyi` files) using `mypy-protobuf`. |
| `mypy_grpc` | boolean | `false` | Generate gRPC mypy stubs (`_grpc.pyi` files) using `mypy-grpc`. |
#### Include Path Behavior
The `include` option controls proto import paths and has important interactions with `inputs`:
1. **Default Behavior**: If `include` is empty or not specified, it defaults to `["."]` (current directory).
2. **Path Resolution**: Each path in `include` is passed to protoc as `--proto_path`. Proto files can only import other protos within these paths.
3. **Input Filtering**: Files matched by `inputs` globs are automatically filtered to only include those under `include` paths. This prevents protoc errors when globs match files outside the include paths.
4. **Output Structure**: Generated files maintain the directory structure relative to the `include` path. For example:
- With `include = ["proto"]` and a file at `proto/service/api.proto`
- Output will be at `{out}/service/api_pb2.py`
5. **Multiple Include Paths**: When specifying multiple paths like `include = ["proto/common", "proto/services"]`, be aware that files with the same relative path may cause conflicts.
**Examples:**
```toml
# Simple case - all protos under proto/ directory
include = ["proto"]
inputs = ["proto/**/*.proto"]
# Multiple include paths - useful for separate proto roots
include = ["common/proto", "services/proto"]
inputs = ["**/*.proto"]
# Selective generation - only specific services
include = ["."] # Use current directory to avoid path conflicts
inputs = ["proto/payment/**/*.proto", "proto/user/**/*.proto"]
# Alternative proto structure
include = ["api/definitions"]
inputs = ["api/definitions/**/*.proto"]
```
### Postprocess Configuration
The `postprocess` table controls post-generation transformations:
| Option | Type | Default | Description |
| ------------------ | ------- | --------- | ----------------------------------------------------------------------------------------------- |
| `relative_imports` | boolean | `true` | Convert absolute imports to relative imports within generated files. |
| `fix_pyi` | boolean | `true` | Fix type annotations in `.pyi` files (currently reserved for future use). |
| `create_package` | boolean | `true` | Create `__init__.py` files in all directories. Set to `false` for namespace packages (PEP 420). |
| `exclude_google` | boolean | `true` | Exclude `google.protobuf` imports from relative import conversion. |
| `pyright_header` | boolean | `false` | Add Pyright suppression header to generated `_pb2.py` and `_pb2_grpc.py` files. |
| `module_suffixes` | array | See below | File suffixes to process during postprocessing. |
Default `module_suffixes`:
```toml
module_suffixes = ["_pb2.py", "_pb2.pyi", "_pb2_grpc.py", "_pb2_grpc.pyi"]
```
#### Import Rewrite Coverage and Limitations
- Covered patterns:
- `import pkg.module_pb2` / `import pkg.module_pb2 as alias`
- `import pkg.mod1_pb2, pkg.sub.mod2_pb2 as alias` (split into multiple `from` lines)
- `from pkg import module_pb2` / `from pkg import module_pb2 as alias`
- `from pkg import mod1_pb2, mod2_pb2 as alias`
- `from pkg import (\n mod1_pb2,\n mod2_pb2 as alias,\n )`
- Exclusions/known behaviors:
- `google.protobuf.*` is excluded when `exclude_google = true` (default).
- Parentheses-based line continuation is supported for `from ... import (...)`; backslash continuations (e.g. `\\`) are not currently handled.
- Only modules matching `_pb2` / `_pb2_grpc` are candidates; other imports are left unchanged.
- Mixed lists are split: rewritten items go to a relative `from` line; non-target items remain as their original import.
- Rewrites only apply if the target module file exists under the configured `out` tree.
#### Path Resolution Robustness
- The tool computes relative import prefixes using canonicalized paths (`realpath`),
which reduces inconsistencies from relative segments (e.g., `./`, `../`) and
symlinks. If canonicalization fails (non-existent paths, permission), it falls
back to a best-effort relative computation.
- Practical tip: ensure your generated tree exists before postprocessing so the
canonicalization can establish a stable common prefix.
### Verification Configuration
The `[tool.python_proto_importer.verify]` section configures optional verification commands:
| Option | Type | Default | Description |
| ------------- | ----- | ------- | ------------------------------------------------------------------------------- |
| `mypy_cmd` | array | `null` | Command to run mypy type checking. Example: `["mypy", "--strict", "generated"]` |
| `pyright_cmd` | array | `null` | Command to run pyright type checking. Example: `["pyright", "generated"]` |
**Important Notes:**
1. **Import Dry-run**: Always performed automatically. The tool attempts to import all generated Python modules to ensure they're valid.
2. **Type Checking**: Only runs if configured. The tools (mypy/pyright) must be available in your environment.
3. **Command Arrays**: Commands are specified as arrays where the first element is the executable and remaining elements are arguments.
**Examples:**
```toml
[tool.python_proto_importer.verify]
# Using uv to run type checkers
mypy_cmd = ["uv", "run", "mypy", "--strict", "generated/python"]
pyright_cmd = ["uv", "run", "pyright", "generated/python"]
# Direct execution
mypy_cmd = ["mypy", "--config-file", "mypy.ini", "generated"]
# Check only .pyi files with pyright
pyright_cmd = ["pyright", "generated/**/*.pyi"]
# Exclude generated gRPC files from mypy strict checking
mypy_cmd = ["mypy", "--strict", "--exclude", ".*_grpc\\.py$", "generated"]
```
## Configuration Examples
### Minimal Configuration
```toml
[tool.python_proto_importer]
backend = "protoc"
inputs = ["proto/**/*.proto"]
out = "generated"
```
### Full-Featured Configuration
```toml
[tool.python_proto_importer]
backend = "protoc"
python_exe = ".venv/bin/python"
include = ["proto"]
inputs = ["proto/**/*.proto"]
out = "src/generated"
mypy = true
mypy_grpc = true
[tool.python_proto_importer.postprocess]
relative_imports = true
fix_pyi = true
create_package = true
exclude_google = true
pyright_header = true
[tool.python_proto_importer.verify]
mypy_cmd = ["uv", "run", "mypy", "--strict", "--exclude", ".*_grpc\\.py$", "src/generated"]
pyright_cmd = ["uv", "run", "pyright", "src/generated/**/*.pyi"]
```
Note: For pyright, we recommend focusing on `.pyi` stubs (as shown) to avoid warnings from generated `.py` that intentionally reference experimental or dynamically provided attributes.
### Pyi-only Verification Example
```toml
[tool.python_proto_importer]
backend = "protoc"
include = ["proto"]
inputs = ["proto/**/*.proto"]
out = "generated/python"
mypy = true
[tool.python_proto_importer.verify]
# Validate only the generated stubs with pyright
pyright_cmd = ["uv", "run", "pyright", "generated/python/**/*.pyi"]
```
### Namespace Package Configuration (PEP 420)
```toml
[tool.python_proto_importer]
backend = "protoc"
include = ["proto"]
inputs = ["proto/**/*.proto"]
out = "generated"
[tool.python_proto_importer.postprocess]
create_package = false # Don't create __init__.py files
```
### Selective Service Generation
```toml
[tool.python_proto_importer]
backend = "protoc"
include = ["."]
# Only generate specific services
inputs = [
"proto/authentication/**/*.proto",
"proto/user_management/**/*.proto"
]
out = "services/generated"
```
### Custom Directory Structure
```toml
[tool.python_proto_importer]
backend = "protoc"
# For non-standard proto locations
include = ["api/v1/definitions"]
inputs = ["api/v1/definitions/**/*.proto"]
out = "build/python/api"
```
## Advanced Usage
### Using with uv
[uv](https://github.com/astral-sh/uv) is a fast Python package manager that can replace pip and virtualenv:
```toml
[tool.python_proto_importer]
python_exe = "uv" # or ".venv/bin/python" if using uv venv
# ... rest of config
[tool.python_proto_importer.verify]
mypy_cmd = ["uv", "run", "mypy", "--strict", "generated"]
```
### CI/CD Integration
```yaml
# GitHub Actions example
- name: Install dependencies
run: |
pip install python-proto-importer
pip install grpcio-tools mypy-protobuf
- name: Generate Python code from protos
run: proto-importer build
- name: Run tests
run: pytest tests/
```
### Understanding `include` vs `inputs`
One of the most important concepts to understand when configuring python-proto-importer is the difference between `include` and `inputs`:
#### 🗂️ `include` - "Where to Look" (Search Paths)
Specifies **where** the protobuf compiler (protoc) should **search** for `.proto` files.
#### 📄 `inputs` - "What to Compile" (Target Files)
Specifies **which** `.proto` files you want to **compile** using glob patterns.
#### 🏗️ Example Project Structure
```
my-project/
├── api/
│ ├── user/
│ │ └── user.proto # Want to compile this
│ └── order/
│ └── order.proto # Want to compile this
├── third_party/
│ └── google/
│ └── protobuf/
│ └── timestamp.proto # Referenced as dependency
└── generated/ # Output directory
```
#### ⚙️ Configuration Example
```toml
[tool.python_proto_importer]
include = ["api", "third_party"] # Search paths
inputs = ["api/**/*.proto"] # Files to compile
out = "generated"
```
#### 🔍 How It Works
1. **`inputs`**: `api/**/*.proto` → finds `user.proto` and `order.proto`
2. **`include`**: Sets `api` and `third_party` as search paths
3. **Compilation**:
- When compiling `user.proto`, if it contains `import "google/protobuf/timestamp.proto"`
- The compiler can automatically find `third_party/google/protobuf/timestamp.proto`
#### 🚫 Common Mistakes
**❌ Wrong Pattern:**
```toml
# Wrong: Including dependencies in inputs causes them to be generated
inputs = ["api/**/*.proto", "third_party/**/*.proto"] # Generates unwanted files
include = ["api"] # Missing search paths
```
**✅ Correct Pattern:**
```toml
# Correct: Only compile what you need, but include all search paths
inputs = ["api/**/*.proto"] # Only compile your API files
include = ["api", "third_party"] # Include all paths for dependencies
```
#### 🎯 Key Takeaway
- **`include`** = Compiler's "eyes" (what it can see)
- **`inputs`** = Compiler's "hands" (what it grabs and compiles)
Dependencies are **not compiled** (excluded from `inputs`) but **must be searchable** (included in `include`).
This approach ensures you **generate only the files you need** while **properly resolving all dependencies**.
### Handling Complex Proto Dependencies
When dealing with complex proto dependencies across multiple directories:
```toml
[tool.python_proto_importer]
# Include all necessary proto roots
include = [
".",
"third_party/proto",
"vendor/proto"
]
# Use specific patterns to avoid conflicts
inputs = [
"src/proto/**/*.proto",
"third_party/proto/specific_service/**/*.proto"
]
out = "generated"
```
## Limitations
- **v0.1 limitations**:
- Only `protoc` backend is supported. `buf generate` support is planned for v0.2.
- Import rewriting targets common `_pb2(_grpc)?.py[i]` patterns; broader coverage is added incrementally with tests.
- Import dry-run verifies only generated `.py` modules (excluding `__init__.py`). `.pyi` files are not imported and should be validated via type checkers (e.g., configure `pyright_cmd` to point at `**/*.pyi`).
- The `fix_pyi` flag is reserved for future use in v0.1 and currently has no effect.
- **Known behaviors**:
- When using multiple `include` paths with files of the same name, protoc may report "shadowing" errors. Use selective `inputs` patterns to avoid this.
- Generated file structure follows protoc conventions: files are placed relative to their `--proto_path`.
- Type checkers (mypy/pyright) must be installed separately and available in PATH or the Python environment.
## Contributing
See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup and guidelines.
## License
Apache-2.0. See LICENSE for details.
Raw data
{
"_id": null,
"home_page": "https://github.com/K-dash/python-proto-importer",
"name": "python-proto-importer",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.11",
"maintainer_email": null,
"keywords": "protobuf, grpc, python, generator, cli",
"author": null,
"author_email": "K-dash <maintainers+python-proto-importer@kdash.dev>",
"download_url": "https://files.pythonhosted.org/packages/20/7e/9a7d3e0aec3136ba8b0c34be6dd3a908d16756ea6b3dd4934828aca8513b/python_proto_importer-0.1.3.tar.gz",
"platform": null,
"description": "# python-proto-importer\n\n[](https://crates.io/crates/python-proto-importer)\n[](https://pypi.org/project/python-proto-importer/)\n[](https://github.com/K-dash/python-proto-importer/actions)\n[](https://codecov.io/gh/K-dash/python-proto-importer)\n\nRust-based CLI to streamline Python gRPC/Protobuf workflows: generate code, stabilize imports, and run type checks in a single command. Ships as a PyPI package (via maturin) and as a Rust crate.\n\n### Why this project (Motivation)\n\n- **Fragile imports from stock protoc output**: vanilla `grpcio-tools`/`protoc` emit absolute imports (e.g. `import foo.bar_pb2`) that break when you move the generated tree, split packages, or embed code under a different root. This tool rewrites them into stable **relative imports** inside the generated package.\n- **Package structure friction**: projects often forget to add `__init__.py` or need namespace packages. We can auto-create `__init__.py` (opt-in/out) to make the tree importable and CI-friendly.\n- **Type-checking pain**: mixing generated `.py` and `.pyi` frequently leads to noisy type warnings. We optionally integrate with `mypy-protobuf` / `mypy-grpc`, and recommend `.pyi`-first verification via Pyright.\n- **Non-reproducible, multi-step scripts**: teams maintain ad\u2011hoc scripts for generation, postprocessing, and verification. This CLI runs the full pipeline in one command and stores configuration in `pyproject.toml`.\n- **Silent breakages**: generated trees \u201cimport\u201d locally but fail in CI or different PYTHONPATHs. A built-in **import dry-run** validates the entire package layout deterministically.\n\n### How it differs from existing tools\n\n- **Postprocess with awareness of your output tree**: relative-import rewriting targets only `_pb2[_grpc]` modules that actually exist beneath your configured `out`, leaving third\u2011party modules (e.g. `google.protobuf`) untouched by default.\n- **Package hygiene by default**: opt-in `__init__.py` generation and path\u2011robust computation (uses canonical paths) reduce environment\u2011dependent surprises.\n- **Verification built-in**: import dry\u2011run for all generated modules, plus easy hooks to run `mypy`/`pyright` as part of the same command.\n- **Single source of truth in `pyproject.toml`**: keeps your team\u2019s proto generation policy declarative and reviewable.\n- **Fast, portable binary**: implemented in Rust with a small runtime footprint; distributed both on PyPI (wheel) and crates.io.\n\n- **Backends**: `protoc` (v0.1), `buf generate` (planned v0.2)\n- **Postprocess**: convert internal imports to relative; generate `__init__.py`\n- **Typing**: optional `mypy-protobuf` / `mypy-grpc` emission\n- **Verification**: import dry-run, optional mypy/pyright\n\nFor Japanese documentation, see: [docs/\u65e5\u672c\u8a9e README](doc/README.ja.md)\n\n## Table of Contents\n\n- [Quick Start](#quick-start)\n- [Commands](#commands)\n- [Configuration](#configuration)\n - [Core Configuration](#core-configuration)\n - [Postprocess Configuration](#postprocess-configuration)\n - [Verification Configuration](#verification-configuration)\n- [Configuration Examples](#configuration-examples)\n- [Advanced Usage](#advanced-usage)\n- [Limitations](#limitations)\n- [Contributing](#contributing)\n- [License](#license)\n\n## Quick Start\n\n```bash\npip install python-proto-importer\n# or\ncargo install python-proto-importer\n```\n\nCreate a `pyproject.toml` with your configuration:\n\n```toml\n[tool.python_proto_importer]\nbackend = \"protoc\"\npython_exe = \"python3\"\ninclude = [\"proto\"]\ninputs = [\"proto/**/*.proto\"]\nout = \"generated/python\"\n```\n\nRun the build:\n\n```bash\nproto-importer build\n```\n\n## Commands\n\n### `proto-importer doctor`\n\nEnvironment diagnostics with versions and helpful hints:\n\n- Detects Python runner (uv/python) and shows versions\n- Checks for `grpcio-tools` (required), `mypy-protobuf` / `mypy-grpc` (optional per config)\n- Shows `protoc` / `buf` versions (informational in v0.1)\n- Checks `mypy` / `pyright` CLIs and prints hints if configured but missing\n\n### `proto-importer build [--pyproject PATH]`\n\nGenerate Python code from proto files, apply postprocessing, and run verification.\n\nOptions:\n\n- `--pyproject PATH`: Path to pyproject.toml (default: `./pyproject.toml`)\n- `--no-verify`: Skip verification after generation\n- `--postprocess-only`: Skip generation, only run postprocessing (experimental)\n\n### `proto-importer check [--pyproject PATH]`\n\nRun verification only (import dry-run and type checks) without generation.\n\n### `proto-importer clean [--pyproject PATH] --yes`\n\nRemove generated output directory. Requires `--yes` confirmation.\n\n## Configuration\n\nAll configuration is done through `pyproject.toml` under the `[tool.python_proto_importer]` section.\n\n### Core Configuration\n\n| Option | Type | Default | Description |\n| ------------ | ------- | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `backend` | string | `\"protoc\"` | Code generation backend. Currently only `\"protoc\"` is supported. `\"buf\"` planned for v0.2. |\n| `python_exe` | string | `\"python3\"` | Python executable to use for generation and verification. Can be `\"python3\"`, `\"python\"`, `\"uv\"` (fully tested), or a path like `\".venv/bin/python\"`. |\n| `include` | array | `[\".\"]` | Proto import paths (passed as `--proto_path` to protoc). Empty array defaults to `[\".\"]`. See [Include Path Behavior](#include-path-behavior) for details. |\n| `inputs` | array | `[]` | Glob patterns for proto files to generate. Example: `[\"proto/**/*.proto\"]`. Files are filtered by `include` paths. |\n| `out` | string | `\"generated/python\"` | Output directory for generated Python files. |\n| `mypy` | boolean | `false` | Generate mypy type stubs (`.pyi` files) using `mypy-protobuf`. |\n| `mypy_grpc` | boolean | `false` | Generate gRPC mypy stubs (`_grpc.pyi` files) using `mypy-grpc`. |\n\n#### Include Path Behavior\n\nThe `include` option controls proto import paths and has important interactions with `inputs`:\n\n1. **Default Behavior**: If `include` is empty or not specified, it defaults to `[\".\"]` (current directory).\n\n2. **Path Resolution**: Each path in `include` is passed to protoc as `--proto_path`. Proto files can only import other protos within these paths.\n\n3. **Input Filtering**: Files matched by `inputs` globs are automatically filtered to only include those under `include` paths. This prevents protoc errors when globs match files outside the include paths.\n\n4. **Output Structure**: Generated files maintain the directory structure relative to the `include` path. For example:\n\n - With `include = [\"proto\"]` and a file at `proto/service/api.proto`\n - Output will be at `{out}/service/api_pb2.py`\n\n5. **Multiple Include Paths**: When specifying multiple paths like `include = [\"proto/common\", \"proto/services\"]`, be aware that files with the same relative path may cause conflicts.\n\n**Examples:**\n\n```toml\n# Simple case - all protos under proto/ directory\ninclude = [\"proto\"]\ninputs = [\"proto/**/*.proto\"]\n\n# Multiple include paths - useful for separate proto roots\ninclude = [\"common/proto\", \"services/proto\"]\ninputs = [\"**/*.proto\"]\n\n# Selective generation - only specific services\ninclude = [\".\"] # Use current directory to avoid path conflicts\ninputs = [\"proto/payment/**/*.proto\", \"proto/user/**/*.proto\"]\n\n# Alternative proto structure\ninclude = [\"api/definitions\"]\ninputs = [\"api/definitions/**/*.proto\"]\n```\n\n### Postprocess Configuration\n\nThe `postprocess` table controls post-generation transformations:\n\n| Option | Type | Default | Description |\n| ------------------ | ------- | --------- | ----------------------------------------------------------------------------------------------- |\n| `relative_imports` | boolean | `true` | Convert absolute imports to relative imports within generated files. |\n| `fix_pyi` | boolean | `true` | Fix type annotations in `.pyi` files (currently reserved for future use). |\n| `create_package` | boolean | `true` | Create `__init__.py` files in all directories. Set to `false` for namespace packages (PEP 420). |\n| `exclude_google` | boolean | `true` | Exclude `google.protobuf` imports from relative import conversion. |\n| `pyright_header` | boolean | `false` | Add Pyright suppression header to generated `_pb2.py` and `_pb2_grpc.py` files. |\n| `module_suffixes` | array | See below | File suffixes to process during postprocessing. |\n\nDefault `module_suffixes`:\n\n```toml\nmodule_suffixes = [\"_pb2.py\", \"_pb2.pyi\", \"_pb2_grpc.py\", \"_pb2_grpc.pyi\"]\n```\n\n#### Import Rewrite Coverage and Limitations\n\n- Covered patterns:\n - `import pkg.module_pb2` / `import pkg.module_pb2 as alias`\n - `import pkg.mod1_pb2, pkg.sub.mod2_pb2 as alias` (split into multiple `from` lines)\n - `from pkg import module_pb2` / `from pkg import module_pb2 as alias`\n - `from pkg import mod1_pb2, mod2_pb2 as alias`\n - `from pkg import (\\n mod1_pb2,\\n mod2_pb2 as alias,\\n )`\n- Exclusions/known behaviors:\n - `google.protobuf.*` is excluded when `exclude_google = true` (default).\n - Parentheses-based line continuation is supported for `from ... import (...)`; backslash continuations (e.g. `\\\\`) are not currently handled.\n - Only modules matching `_pb2` / `_pb2_grpc` are candidates; other imports are left unchanged.\n - Mixed lists are split: rewritten items go to a relative `from` line; non-target items remain as their original import.\n - Rewrites only apply if the target module file exists under the configured `out` tree.\n\n#### Path Resolution Robustness\n\n- The tool computes relative import prefixes using canonicalized paths (`realpath`),\n which reduces inconsistencies from relative segments (e.g., `./`, `../`) and\n symlinks. If canonicalization fails (non-existent paths, permission), it falls\n back to a best-effort relative computation.\n- Practical tip: ensure your generated tree exists before postprocessing so the\n canonicalization can establish a stable common prefix.\n\n### Verification Configuration\n\nThe `[tool.python_proto_importer.verify]` section configures optional verification commands:\n\n| Option | Type | Default | Description |\n| ------------- | ----- | ------- | ------------------------------------------------------------------------------- |\n| `mypy_cmd` | array | `null` | Command to run mypy type checking. Example: `[\"mypy\", \"--strict\", \"generated\"]` |\n| `pyright_cmd` | array | `null` | Command to run pyright type checking. Example: `[\"pyright\", \"generated\"]` |\n\n**Important Notes:**\n\n1. **Import Dry-run**: Always performed automatically. The tool attempts to import all generated Python modules to ensure they're valid.\n\n2. **Type Checking**: Only runs if configured. The tools (mypy/pyright) must be available in your environment.\n\n3. **Command Arrays**: Commands are specified as arrays where the first element is the executable and remaining elements are arguments.\n\n**Examples:**\n\n```toml\n[tool.python_proto_importer.verify]\n# Using uv to run type checkers\nmypy_cmd = [\"uv\", \"run\", \"mypy\", \"--strict\", \"generated/python\"]\npyright_cmd = [\"uv\", \"run\", \"pyright\", \"generated/python\"]\n\n# Direct execution\nmypy_cmd = [\"mypy\", \"--config-file\", \"mypy.ini\", \"generated\"]\n\n# Check only .pyi files with pyright\npyright_cmd = [\"pyright\", \"generated/**/*.pyi\"]\n\n# Exclude generated gRPC files from mypy strict checking\nmypy_cmd = [\"mypy\", \"--strict\", \"--exclude\", \".*_grpc\\\\.py$\", \"generated\"]\n```\n\n## Configuration Examples\n\n### Minimal Configuration\n\n```toml\n[tool.python_proto_importer]\nbackend = \"protoc\"\ninputs = [\"proto/**/*.proto\"]\nout = \"generated\"\n```\n\n### Full-Featured Configuration\n\n```toml\n[tool.python_proto_importer]\nbackend = \"protoc\"\npython_exe = \".venv/bin/python\"\ninclude = [\"proto\"]\ninputs = [\"proto/**/*.proto\"]\nout = \"src/generated\"\nmypy = true\nmypy_grpc = true\n\n[tool.python_proto_importer.postprocess]\nrelative_imports = true\nfix_pyi = true\ncreate_package = true\nexclude_google = true\npyright_header = true\n\n[tool.python_proto_importer.verify]\nmypy_cmd = [\"uv\", \"run\", \"mypy\", \"--strict\", \"--exclude\", \".*_grpc\\\\.py$\", \"src/generated\"]\npyright_cmd = [\"uv\", \"run\", \"pyright\", \"src/generated/**/*.pyi\"]\n```\n\nNote: For pyright, we recommend focusing on `.pyi` stubs (as shown) to avoid warnings from generated `.py` that intentionally reference experimental or dynamically provided attributes.\n\n### Pyi-only Verification Example\n\n```toml\n[tool.python_proto_importer]\nbackend = \"protoc\"\ninclude = [\"proto\"]\ninputs = [\"proto/**/*.proto\"]\nout = \"generated/python\"\nmypy = true\n\n[tool.python_proto_importer.verify]\n# Validate only the generated stubs with pyright\npyright_cmd = [\"uv\", \"run\", \"pyright\", \"generated/python/**/*.pyi\"]\n```\n\n### Namespace Package Configuration (PEP 420)\n\n```toml\n[tool.python_proto_importer]\nbackend = \"protoc\"\ninclude = [\"proto\"]\ninputs = [\"proto/**/*.proto\"]\nout = \"generated\"\n\n[tool.python_proto_importer.postprocess]\ncreate_package = false # Don't create __init__.py files\n```\n\n### Selective Service Generation\n\n```toml\n[tool.python_proto_importer]\nbackend = \"protoc\"\ninclude = [\".\"]\n# Only generate specific services\ninputs = [\n \"proto/authentication/**/*.proto\",\n \"proto/user_management/**/*.proto\"\n]\nout = \"services/generated\"\n```\n\n### Custom Directory Structure\n\n```toml\n[tool.python_proto_importer]\nbackend = \"protoc\"\n# For non-standard proto locations\ninclude = [\"api/v1/definitions\"]\ninputs = [\"api/v1/definitions/**/*.proto\"]\nout = \"build/python/api\"\n```\n\n## Advanced Usage\n\n### Using with uv\n\n[uv](https://github.com/astral-sh/uv) is a fast Python package manager that can replace pip and virtualenv:\n\n```toml\n[tool.python_proto_importer]\npython_exe = \"uv\" # or \".venv/bin/python\" if using uv venv\n# ... rest of config\n\n[tool.python_proto_importer.verify]\nmypy_cmd = [\"uv\", \"run\", \"mypy\", \"--strict\", \"generated\"]\n```\n\n### CI/CD Integration\n\n```yaml\n# GitHub Actions example\n- name: Install dependencies\n run: |\n pip install python-proto-importer\n pip install grpcio-tools mypy-protobuf\n\n- name: Generate Python code from protos\n run: proto-importer build\n\n- name: Run tests\n run: pytest tests/\n```\n\n### Understanding `include` vs `inputs`\n\nOne of the most important concepts to understand when configuring python-proto-importer is the difference between `include` and `inputs`:\n\n#### \ud83d\uddc2\ufe0f `include` - \"Where to Look\" (Search Paths)\n\nSpecifies **where** the protobuf compiler (protoc) should **search** for `.proto` files.\n\n#### \ud83d\udcc4 `inputs` - \"What to Compile\" (Target Files)\n\nSpecifies **which** `.proto` files you want to **compile** using glob patterns.\n\n#### \ud83c\udfd7\ufe0f Example Project Structure\n\n```\nmy-project/\n\u251c\u2500\u2500 api/\n\u2502 \u251c\u2500\u2500 user/\n\u2502 \u2502 \u2514\u2500\u2500 user.proto # Want to compile this\n\u2502 \u2514\u2500\u2500 order/\n\u2502 \u2514\u2500\u2500 order.proto # Want to compile this\n\u251c\u2500\u2500 third_party/\n\u2502 \u2514\u2500\u2500 google/\n\u2502 \u2514\u2500\u2500 protobuf/\n\u2502 \u2514\u2500\u2500 timestamp.proto # Referenced as dependency\n\u2514\u2500\u2500 generated/ # Output directory\n```\n\n#### \u2699\ufe0f Configuration Example\n\n```toml\n[tool.python_proto_importer]\ninclude = [\"api\", \"third_party\"] # Search paths\ninputs = [\"api/**/*.proto\"] # Files to compile\nout = \"generated\"\n```\n\n#### \ud83d\udd0d How It Works\n\n1. **`inputs`**: `api/**/*.proto` \u2192 finds `user.proto` and `order.proto`\n2. **`include`**: Sets `api` and `third_party` as search paths\n3. **Compilation**:\n - When compiling `user.proto`, if it contains `import \"google/protobuf/timestamp.proto\"`\n - The compiler can automatically find `third_party/google/protobuf/timestamp.proto`\n\n#### \ud83d\udeab Common Mistakes\n\n**\u274c Wrong Pattern:**\n\n```toml\n# Wrong: Including dependencies in inputs causes them to be generated\ninputs = [\"api/**/*.proto\", \"third_party/**/*.proto\"] # Generates unwanted files\ninclude = [\"api\"] # Missing search paths\n```\n\n**\u2705 Correct Pattern:**\n\n```toml\n# Correct: Only compile what you need, but include all search paths\ninputs = [\"api/**/*.proto\"] # Only compile your API files\ninclude = [\"api\", \"third_party\"] # Include all paths for dependencies\n```\n\n#### \ud83c\udfaf Key Takeaway\n\n- **`include`** = Compiler's \"eyes\" (what it can see)\n- **`inputs`** = Compiler's \"hands\" (what it grabs and compiles)\n\nDependencies are **not compiled** (excluded from `inputs`) but **must be searchable** (included in `include`).\n\nThis approach ensures you **generate only the files you need** while **properly resolving all dependencies**.\n\n### Handling Complex Proto Dependencies\n\nWhen dealing with complex proto dependencies across multiple directories:\n\n```toml\n[tool.python_proto_importer]\n# Include all necessary proto roots\ninclude = [\n \".\",\n \"third_party/proto\",\n \"vendor/proto\"\n]\n# Use specific patterns to avoid conflicts\ninputs = [\n \"src/proto/**/*.proto\",\n \"third_party/proto/specific_service/**/*.proto\"\n]\nout = \"generated\"\n```\n\n## Limitations\n\n- **v0.1 limitations**:\n - Only `protoc` backend is supported. `buf generate` support is planned for v0.2.\n - Import rewriting targets common `_pb2(_grpc)?.py[i]` patterns; broader coverage is added incrementally with tests.\n - Import dry-run verifies only generated `.py` modules (excluding `__init__.py`). `.pyi` files are not imported and should be validated via type checkers (e.g., configure `pyright_cmd` to point at `**/*.pyi`).\n - The `fix_pyi` flag is reserved for future use in v0.1 and currently has no effect.\n- **Known behaviors**:\n - When using multiple `include` paths with files of the same name, protoc may report \"shadowing\" errors. Use selective `inputs` patterns to avoid this.\n - Generated file structure follows protoc conventions: files are placed relative to their `--proto_path`.\n - Type checkers (mypy/pyright) must be installed separately and available in PATH or the Python environment.\n\n## Contributing\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md) for development setup and guidelines.\n\n## License\n\nApache-2.0. See LICENSE for details.\n\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Rust-based CLI to streamline Python gRPC/Protobuf workflows",
"version": "0.1.3",
"project_urls": {
"Homepage": "https://github.com/K-dash/python-proto-importer",
"Repository": "https://github.com/K-dash/python-proto-importer"
},
"split_keywords": [
"protobuf",
" grpc",
" python",
" generator",
" cli"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "85be7b74e93b8e84c596684b990f05276516755dc02f1dac109fc367720e7466",
"md5": "b4225da81f69a63367d497a2b3a9d529",
"sha256": "b7aeaac822faa467edba1f7992ef295b6dff817ec006fe6259d048d6fb95bb5f"
},
"downloads": -1,
"filename": "python_proto_importer-0.1.3-cp38-abi3-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "b4225da81f69a63367d497a2b3a9d529",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.11",
"size": 2074558,
"upload_time": "2025-08-11T02:59:05",
"upload_time_iso_8601": "2025-08-11T02:59:05.540247Z",
"url": "https://files.pythonhosted.org/packages/85/be/7b74e93b8e84c596684b990f05276516755dc02f1dac109fc367720e7466/python_proto_importer-0.1.3-cp38-abi3-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "3dc28f64649a871c018bb3ae011617c8c8a0e19897dd97cacbed4d1635e5c617",
"md5": "aa3084d7745c665fb83f4ab3391de8ee",
"sha256": "1a2724361637d44bea31c4e17c79b282b57feeb8290ac97eab46870f15fd272f"
},
"downloads": -1,
"filename": "python_proto_importer-0.1.3-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "aa3084d7745c665fb83f4ab3391de8ee",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.11",
"size": 2418417,
"upload_time": "2025-08-11T03:00:11",
"upload_time_iso_8601": "2025-08-11T03:00:11.517162Z",
"url": "https://files.pythonhosted.org/packages/3d/c2/8f64649a871c018bb3ae011617c8c8a0e19897dd97cacbed4d1635e5c617/python_proto_importer-0.1.3-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "72e82cbd2e09f62fc274b31079db7a0715adad7801366c41b952313b2b37d68c",
"md5": "9a801a6f08383dcfb2b73eeb10ab0924",
"sha256": "33d750994135b7b2c20f0b585a4ef21357ffa4902952507a500558c366b2d964"
},
"downloads": -1,
"filename": "python_proto_importer-0.1.3-cp38-abi3-win_amd64.whl",
"has_sig": false,
"md5_digest": "9a801a6f08383dcfb2b73eeb10ab0924",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.11",
"size": 1938828,
"upload_time": "2025-08-11T03:00:39",
"upload_time_iso_8601": "2025-08-11T03:00:39.377306Z",
"url": "https://files.pythonhosted.org/packages/72/e8/2cbd2e09f62fc274b31079db7a0715adad7801366c41b952313b2b37d68c/python_proto_importer-0.1.3-cp38-abi3-win_amd64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "207e9a7d3e0aec3136ba8b0c34be6dd3a908d16756ea6b3dd4934828aca8513b",
"md5": "f1456b5a541cbad1a09455aa90d46889",
"sha256": "1b49fe2a071a4955058c0a7f0d9b55b630523ae3109236772f8e888248eb6ff1"
},
"downloads": -1,
"filename": "python_proto_importer-0.1.3.tar.gz",
"has_sig": false,
"md5_digest": "f1456b5a541cbad1a09455aa90d46889",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.11",
"size": 54627,
"upload_time": "2025-08-11T03:00:13",
"upload_time_iso_8601": "2025-08-11T03:00:13.019376Z",
"url": "https://files.pythonhosted.org/packages/20/7e/9a7d3e0aec3136ba8b0c34be6dd3a908d16756ea6b3dd4934828aca8513b/python_proto_importer-0.1.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-11 03:00:13",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "K-dash",
"github_project": "python-proto-importer",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "python-proto-importer"
}
JSON
