## Why package your Python code?
1. Distributing your code
2. Non-painful import statements
3. Reproduciblity
* **
## `PYTHONPATH` Variable
- [**Refer to this README for indepth details**](https://github.com/avr2002/python-packaging/tree/main/packaging_demo/my_folder)
* **
## Packaging terms:
- module
- package
- sub-package
- distribution package
> [**Official Python Packaging Guide**](https://packaging.python.org/en/latest/)
### Module
A **module** in Python is a single file (with a `.py` extension) that contains Python code. It typically consists of classes, functions, and variables that can be used by other Python code. Modules are used to organize code into logical units and facilitate code reusability.
For example, consider a module named `math_operations.py` that contains functions to perform mathematical operations like addition, subtraction, multiplication, etc. You can import and use these functions in other Python scripts.
```python
# math_operations.py
def add(x, y):
return x + y
def subtract(x, y):
return x - y
```
### Package
A **package** in Python is a collection of modules grouped together in a directory. A package is typically represented by a directory containing an `__init__.py` file (which can be empty) and one or more Python modules. The `__init__.py` file indicates to Python that the directory should be treated as a package.
For example, consider a package named `my_package`:
```
my_package/
├── __init__.py
├── module1.py
└── module2.py
```
Here, `my_package` is a package containing `module1.py` and `module2.py`, which can be imported using dot notation (`my_package.module1`, `my_package.module2`).
### Sub-package
A **sub-package** in Python is a package nested within another package. This means that a package can contain other packages as well as modules. Sub-packages are created by organizing directories and adding `__init__.py` files appropriately to define the package structure.
For example:
```
my_parent_package/
├── __init__.py
└── my_sub_package/
├── __init__.py
├── module3.py
└── module4.py
```
In this structure, `my_sub_package` is a sub-package of `my_parent_package`, and it can contain its own modules (`module3.py`, `module4.py`). The sub-package can be imported using dot notation (`my_parent_package.my_sub_package.module3`).
### Distribution Package
A **distribution package** (or simply **distribution**) in Python refers to a packaged collection of Python code and resources that is made available for installation. It typically includes modules, packages, data files, configuration files, and other resources needed for a specific purpose (e.g., a library or application).
Distribution packages are often distributed and installed using Python package managers such as `pip` and can be uploaded to package repositories like PyPI (Python Package Index) for easy distribution and installation by other developers.
For example, popular distribution packages include `numpy`, `fast-api`, `pandas`, etc., which are installed using `pip` and provide functionalities that can be used in Python projects.
## Building a Distribution Package
### `sdist` format
1. Configure `setup.py`
2. Run `python setup.py build sdist` to build the source distribution
3. Run `pip install ./dist/<package_name>.tar.gz` to install the package
4. Use `pip list` to see if the package is installed.
* **
1. If changes are made in the package then use `pip install .` which will build and install the latest package on the fly.
2. Or simply use editable so that you don't always have to rebuild the package evrerytime new changes are made:
- `pip install --editable .`
* **
- `sdist` is short for source distribution, a `.tar` file containing our code called an "sdist". What that means is that the distribution package only contains a subset of our source code.
- A "source distribution" is essentially a zipped folder containing our *source* code.
* **
### `wheel` format
#### Understanding Source Distribution (sdist) and Wheels
- **Source Distribution (`sdist`)**:
- Represents a zipped folder containing the source code of a Python package.
- Primarily includes Python source files (`*.py`), configuration files, and other project-related assets.
- Provides a portable distribution format that can be used to install the package on any platform.
- Can be created using `python setup.py sdist`.
- **Wheels (`bdist_wheel`)**:
- A more advanced distribution format compared to `sdist`.
- Represents a zipped file with the `.whl` extension.
- Potentially includes more than just source code; can contain pre-compiled binaries (e.g., C extensions) for faster installation.
- Preferred for distribution when the package includes compiled, non-Python components.
- Created using `python setup.py bdist_wheel`.
#### Benefits of Wheels Over Source Distributions
- **Faster Installation**:
- Installing from wheels (`*.whl`) is typically faster than installing from source distributions (`*.tar.gz`).
- Wheels can include pre-compiled binaries, reducing the need for on-the-fly compilation during installation.
- Most packages are “pure Python” though, so unless you are working with Python bindings such as code written in Rust, C++, C, etc. building a wheel will be just as easy as building an sdist.
- **Ease of Use for Users**:
- Users benefit from quicker installations, especially when dealing with packages that have complex dependencies or require compilation.
#### Building Both `sdist` and `bdist_wheel`
- It's recommended to build and publish both `sdist` and `bdist_wheel` for Python packages to accommodate different use cases and platforms.
- `python setup.py sdist bdist_wheel`
- Building both types of distributions allows users to choose the most suitable distribution format based on their needs and environment.
#### Challenges and Considerations
- **Compilation Requirements**:
- Unfortunately, building wheels for *all* operating systems gets difficult if you have a compilation step required, so some OSS maintainers only build wheels for a single OS.
- Whenever a wheel is not available for your OS, `pip` actually executes the [`setup.py`](http://setup.py) (or equivalent files) on *your* machine, right after downloading the sdist.
- Building the wheel might require compiling code
- Compiling code can be *really* slow, 5-10-30 minutes or even more, especially if the machine is weak
- Compiling code requires dependencies, e.g. `gcc` if the source code is in C, but other languages require their own compilers. The user must install these on their on machine or the `pip install ...` will simply fail. This can happen when you install `numpy`, `pandas`, `scipy`, `pytorch`, `tensorflow`, etc.
- [`setup.py`](http://setup.py) may contain arbitrary code. This is highly insecure. A `setup.py` might contain malicious code.
- **Dependency Management**:
- Users may need to install additional development tools and dependencies (`gcc`, etc.) to successfully install packages that require compilation.
- **Security Concerns**:
- Using `setup.py` for package installation can be insecure as it executes arbitrary code.
- Ensure that packages obtained from untrusted sources are reviewed and validated.
#### Naming Scheme of a `wheel` file
- A wheel filename is broken down into parts separated by hyphens:
- `{dist}-{version}(-{build})?-{python}-{abi}-{platform}.whl`
- Each section in `{brackets}` is a tag, or a component of the wheel name that carries some meaning about what the wheel contains and where the wheel will or will not work.
- For Example: `dist/packaging-0.0.0-py3-none-any.whl`
- `packaging` is the package name
- `0.0.0` is the verison number
- `py3` denotes it's build for Python3
- `abi` tag. ABI stands for application binary interface.
- `any` stands for that this package is build to work on any platform.
- Other Examples:
- `cryptography-2.9.2-cp35-abi3-macosx_10_9_x86_64.whl`
- `chardet-3.0.4-py2.py3-none-any.whl`
- `PyYAML-5.3.1-cp38-cp38-win_amd64.whl`
- `numpy-1.18.4-cp38-cp38-win32.whl`
- `scipy-1.4.1-cp36-cp36m-macosx_10_6_intel.whl`
- The reason for this is, the wheel file contains the pre-complied binary code that helps to install the package fast.
#### Conclusion
- **Package Maintenance**:
- Maintainers should strive to provide both `sdist` and `bdist_wheel` distributions to maximize compatibility and ease of use for users.
- **Consider User Experience**:
- Consider the user's perspective when choosing the appropriate distribution format.
- Provide clear documentation and guidance for users installing packages with compilation requirements.
## `build` CLI tool and `pyproject.toml`
- "Build dependencies" are anything that must be installed on your system in order to build your distribution package into an sdist or wheel.
For example, we needed to `pip install wheel` in order to run `python setup.py bdist_wheel` so `wheel` is a build dependency for building wheels.
- [`setup.py`](http://setup.py) files can get complex.
You may need to `pip install ...` external libraries and import them into your `setup.py` file to accommodate complex build processes.
The lecture shows `pytorch` and `airflow` as examples of packages with complex [`setup.py`](http://setup.py) files.
- Somehow you need to be able to document build dependencies *outside* of [`setup.py`](http://setup.py).
If they were documented in the `setup.py` file… you would not be able to execute the `setup.py` file to read the documented dependencies (like if they were specified in an `list` somewhere in the file).
This is the original problem `pyproject.toml` was meant to solve.
```toml
# pyproject.toml
[build-system]
# Minimum requirements for the build system to execute.
requires = ["setuptools>=62.0.0", "wheel"]
```
`pyproject.toml` sits adjacent to [`setup.py`](http://setup.py) in the file tree
- The `build` CLI tool (`pip install build`) is a special project by the Python Packaging Authority (PyPA) which
1. reads the `[build-system]` table in the `pyproject.toml`,
2. installs those dependencies into an isolated virtual environment,
3. and then builds the sdist and wheel
```bash
pip install build
# both setup.py and pypproject.toml should be together, ideally in the root directory
# python -m build --sdist --wheel path/to/dir/with/setup.py/and/pyproject.toml
python -m build --sdist --wheel .
```
## Moving from `setup.py` to [`setup.cfg`](https://setuptools.pypa.io/en/latest/userguide/declarative_config.html) config file
>>Moving from
```python
# setup.py
from pathlib import Path
from setuptools import find_packages, setup
import wheel
# Function to read the contents of README.md
def read_file(filename: str) -> str:
filepath = Path(__file__).resolve().parent / filename
with open(filepath, encoding="utf-8") as file:
return file.read()
setup(
name="packaging-demo",
version="0.0.0",
packages=find_packages(),
# package meta-data
author="Amit Vikram Raj",
author_email="avr13405@gmail.com",
description="Demo for Python Packaging",
license="MIT",
# Set the long description from README.md
long_description=read_file("README.md"),
long_description_content_type="text/markdown",
# install requires: libraries that are needed for the package to work
install_requires=[
"numpy", # our package depends on numpy
],
# setup requires: the libraries that are needed to setup/build
# the package distribution
# setup_requires=[
# "wheel", # to build the binary distribution we need wheel package
# ],
)
```
>>TO
```python
# setup.py
from setuptools import setup
# Now setup.py takes it's configurations from setup.cfg file
setup()
```
```ini
# setup.cfg
[metadata]
name = packaging-demo
version = attr: packaging_demo.VERSION
author = Amit Vikram Raj
author_email = avr13405@gmail.com
description = Demo for Python Packaging
long_description = file: README.md
keywords = one, two
license = MIT
classifiers =
Framework :: Django
Programming Language :: Python :: 3
[options]
zip_safe = False
include_package_data = True
# same as find_packages() in setup()
packages = find:
python_requires = >=3.8
install_requires =
numpy
importlib-metadata; python_version<"3.10"
```
>>ALSO addtional setting is passed to `pyproject.toml` file.
>>Here we have specified `build-system` similar to `setup_requires` in `setup.py`
```toml
# pyproject.toml
[build-system]
# Minimum requirements for the build system to execute
requires = ["setuptools", "wheel", "numpy<1.24.3"]
# Adding ruff.toml to pyproject.toml
[tool.ruff]
line-length = 99
[tool.ruff.lint]
# 1. Enable flake8-bugbear (`B`) rules, in addition to the defaults.
select = ["E", "F", "B", "ERA"]
# 2. Avoid enforcing line-length violations (`E501`)
ignore = ["E501"]
# 3. Avoid trying to fix flake8-bugbear (`B`) violations.
unfixable = ["B"]
# 4. Ignore `E402` (import violations) in all `__init__.py` files, and in select subdirectories.
[tool.ruff.lint.per-file-ignores]
"__init__.py" = ["E402"]
"**/{tests,docs,tools}/*" = ["E402"]
# copying isort configurations from .isort.cfg to pyproject.toml
[tool.isort]
profile = "black"
multi_line_output = "VERTICAL_HANGING_INDENT"
force_grid_wrap = 2
line_length = 99
# copying balck config from .black.toml to pyproject.toml
[tool.black]
line-length = 99
exclude = ".venv"
# copying flake8 config from .flake8 to pyproject.toml
[tool.flake8]
docstring-convention = "all"
extend-ignore = ["D107", "D212", "E501", "W503", "W605", "D203", "D100",
"E305", "E701", "DAR101", "DAR201"]
exclude = [".venv"]
max-line-length = 99
# radon
radon-max-cc = 10
# copying pylint config from .pylintrc to pyproject.toml
[tool.pylint."messages control"]
disable = [
"line-too-long",
"trailing-whitespace",
"missing-function-docstring",
"consider-using-f-string",
"import-error",
"too-few-public-methods",
"redefined-outer-name",
]
```
- We have been treating [`setup.py`](http://setup.py) as a glorified config file, not really taking advantage of the fact that it is a Python file by adding logic to it.
This is more common than not. Also, there has been a general shift away from using Python for config files because doing so adds complexity to *using* the config files (like having to install libraries in order to execute the config file).
- `setup.cfg` is a companion file to [`setup.py`](http://setup.py) that allows us to define our package configuration in a static text file—specifically an [INI format](https://en.wikipedia.org/wiki/INI_file) file.
<aside>
💡 INI is a problematic, weak file format compared to more “modern” formats like JSON, YAML, and *TOML*. We will prefer TOML as we move forward.
</aside>
- Any values that we do not directly pass as arguments to setup() will be looked for by the setup() invocation in a setup.cfg file, which is meant to sit adjacent to setup.py in the file tree if used.
* **
- Now we are accumulating a lot of files!
- `setup.py`
- `setup.cfg`
- `pyproject.toml`
- `README.md`
- More files for linting and other code quality tools, e.g. `.pylintrc`, `.flake8`, `.blackrc`, `ruff.toml`, `.mypy`, `pre-commit-config.yaml`, etc.
- More files we have not talked about yet:
- `CHANGELOG` or `CHANGELOG.md`
- `VERSION` or `version.txt`
It turns out that nearly all of these files can be replaced with `pyproject.toml` . Nearly every linting / code quality tool supports parsing a section called `[tool.<name>]` e.g. `[tool.black]` section of `pyproject.toml` to read its configuration!
The docs of each individual tool should tell you how to accomplish this.
Above shown is a `pyproject.toml` with configurations for many of the linting tools we have used in the course.
>**Can `setup.cfg` and `setup.py` be replaced as well?**
## [Moving `setup.cfg` to `pyproject.toml`](https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html)
>FROM `setup.cfg`
```ini
# setup.cfg
[metadata]
name = packaging-demo
version = attr: packaging_demo.VERSION
author = Amit Vikram Raj
author_email = avr13405@gmail.com
description = Demo for Python Packaging
long_description = file: README.md
keywords = one, two
license = MIT
classifiers =
Programming Language :: Python :: 3
[options]
zip_safe = False
include_package_data = True
# same as find_packages() in setup()
packages = find:
python_requires = >=3.8
install_requires =
numpy
importlib-metadata; python_version<"3.10"
```
>TO
```toml
# pyproject.toml
[build-system]
# Minimum requirements for the build system to execute
requires = ["setuptools>=61.0.0", "wheel"]
# Adding these from setup.cfg in pyproject.toml file
[project]
name = "packaging-demo"
authors = [{ name = "Amit Vikram Raj", email = "avr13405@gmail.com" }]
description = "Demo for Python Packaging"
readme = "README.md"
requires-python = ">=3.8"
keywords = ["one", "two"]
license = { text = "MIT" }
classifiers = ["Programming Language :: Python :: 3"]
dependencies = ["numpy", 'importlib-metadata; python_version<"3.10"']
dynamic = ["version"]
# version = "0.0.3"
# https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html#dynamic-metadata
[tool.setuptools.dynamic]
# every while making changes in package, you can change the verison in one of these files
# version = {attr = "packaging_demo.VERSION"} # version read by 'packaging_demo/__init__.py' file
version = {file = ["version.txt"]} # version read by 'version.txt' file in root folder
```
>> `python -m build --sdist --wheel .` - Runs perfectly, we got rid of another config file (`setup.cfg`)
## Replacing `setup.py` with `build-backend`
- [PEP 517](https://peps.python.org/pep-0517/) added a `build-backend` argument to `pyproject.toml` like so:
```toml
[build-system]
# Defined by PEP 518:
requires = ["flit"]
# Defined by this PEP:
build-backend = "flit.api:main"
```
```python
# The above toml config is equivalent to
import flit.api
backend = flit.api.main
```
- The `build-backend` defines an entrypoint (executable Python module in this case) that the `build` CLI uses to actually do the work of parsing `pyproject.toml` and building the wheel and sdist.
- This means *you* could implement your own build backend today by writing a program that does that, and you could use it by adding your package to `requires = [...]` and specifying the entrypoint in `build-backend = ...`.
- If you do not specify a `build-backend` in `pyproject.toml`, setuptools is assumed and package will get bulit prefectly fine.
- If we remove `setup.py` and run `python -m build --sdist --wheel .` it runs perfectly without it because the default value of `build-system` is set as `build-backend = "setuptools.build_meta"` in `build` CLI which builds our package.
- But you can still explicitly declare `setuptools` as your build backend like this
```toml
# pyproject.toml
...
[build-system]
requires = ["setuptools>=61.0.0", "wheel"]
build-backend = "setuptools.build_meta"
...
```
Each build backend typically extends the `pyproject.toml` file with its own configuration options. For example,
```toml
# pyproject.toml
...
[tool.setuptools.package-data]
package_demo = ["*.json"]
[tool.setuptools.dynamic]
version = {file = "version.txt"}
long_description = {file = "README.md"}
...
```
- If you choose to use `setuptools` in your project, you can add these sections to `pyproject.toml`. You can read more about this in the `setuptools` documentation
## Adding Data Files in our Package
- It's often beneficial to include non-python files like data files or binary files inside of your package because oftentimes your Python code relies on these non-python files.
- And then we saw that if we're going to include those files, we need to get those files to end up inside of our package folder because it's our package folder that ends up inside of our users virtual environments upon installation.
- We also saw that by default, all non-python files don't make it into that final virtual environment folder. That is, don't actually make it into our package folder during build procedure.
So, how do we make sure that these files end up in our wheel/dist build of our package? For example here we demo with `cities.json` file that we want in our package as it's used by `states_info.py` file.
>[**Official `setuptools` Docs for Data Support**](https://setuptools.pypa.io/en/latest/userguide/datafiles.html)
### Using [MANIFEST.in](https://setuptools.pypa.io/en/latest/userguide/miscellaneous.html#using-manifest-in) File
```toml
# pyprject.toml
[tool.setuptools]
# ...
# By default, include-package-data is true in pyproject.toml, so you do
# NOT have to specify this line.
include-package-data = true
```
So, setuptools by default has this `include-package-data` value set to `true` as shown in the [official docs](https://setuptools.pypa.io/en/latest/userguide/datafiles.html) but we need to create an extra file `MANIFEST.in` and specify the data which we want to inculde in our package present at root dir.
>>**IMP:** It's import all the folders in the package directory should have `__init__.py` file inculing the data directory which we want to include because the `find_packages()` recusrive process that setuptools does will not go into fo;ders that have `__init__.py` file in it.
```in
# MANIFEST.in
include packaging_demo/*.json
include packaging_demo/my_folder/*.json
OR
Recursive include all json files in the package directory
recursive-include packaging_demo/ *.json
```
>[Docs on configuring MANIFEST.in file](https://setuptools.pypa.io/en/latest/userguide/miscellaneous.html#using-manifest-in)
### Without using `MANIFEST.in` file
From [setuptools docs](https://setuptools.pypa.io/en/latest/userguide/datafiles.html) we can add this in our `pyproject.toml` file:
```toml
# this is by default true so no need to explicitly add it
# but as mentioned in the docs, it is false for other methods like setup.py or setup.cfg
[tool.setuptools]
include-package-data = true
# add the data here, it's finding the files recursively
[tool.setuptools.package-data]
package_demo = ["*.json"]
```
## Other `build-backend` systems than `setuptools`
Other than `setuptools` we can use these build backend systems. The point to note is when using other systems the `pyproject.toml` cofiguration should follow their standerds.
1. [Hatch](https://hatch.pypa.io/1.9/config/build)
```toml
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
```
2. [Poetry](https://python-poetry.org/docs/pyproject/)
```toml
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
```
## Reproducibility, Dependency Graph, Locking Dependencies, Dependency Hell
- Documenting the exact versions of our dependencies and their dependencies and so on.
- It's advisable to have as little as possible number of dependencies associated with our package, as it can lead to dependency hell, or dependency conflict with other package as explained in the lectures.
- The more complex the dependency tree higher the chance of conflicts with future version of other libraries.
- [Dependency Graph Analysis By Eric](https://github.com/avr2002/dependency-graph-pip-analysis)
- Keeping the pinned versions of dependencies and python versions is advicable for troubleshooting purposes:
- `pip freeze > requirements.txt`
```bash
pip install pipdeptree graphviz
sudo apt-get install graphviz
# generate the dependency graph
pipdeptree -p packaging-demo --graph-output png > dependency-graph.png
```
<a href='https://raw.githubusercontent.com/avr2002/python-packaging/main/packaging_demo/assets/dependency-graph.png' target='_blank'>
<img src='https://raw.githubusercontent.com/avr2002/python-packaging/main/packaging_demo/assets/dependency-graph.png' alt='Dependency Graph of packaging-demo package' title='Dependency Graph of packaging-demo package'>
</a>
* **
## Adding Optional/Extra Dependencies to our Project
```toml
[project.optional-dependencies]
dev = ["ruff", "mypy", "black"]
```
```bash
# installing our package with optional dependencies
pip install '.[dev]'
```
* **
```toml
[project.optional-dependencies]
# for developement
dev = ["ruff", "mypy", "black"]
# plugin based architecture
colors = ["rich"]
```
```bash
# plugin based installation
pip install '.[colors]'
# here we demo with rich library, if user wants the output to be
# colorized then they can install our package like this.
# we can add multiple optional dependencies like:
pip install '.[colors, dev]'
```
* **
```toml
[project.optional-dependencies]
# for developement
dev = ["ruff", "mypy", "black"]
# plugin based architecture
colors = ["rich"]
# install all dependencies
all = ["packaging-demo[dev, colors]"]
```
```bash
# Installing all dependencies all at once
pip install '.[all]'
```
## Shopping for dependencies
We can use [Snyk](https://snyk.io/advisor/python/fastapi) to check how stable, well supported, if any security issues, etc. are present for the dependencies which we are going to use for our package and then take the decision on using it in our project.
* **
# Continuous Delivery: Publishing to PyPI
- Few key terms realted to Continuous Delivery
- DevOps, Waterfall, Agile, Scrum
### Publishing to PyPI
To pusblish our package to PyPI[Python Packaging Index], as stated in the [official guide](https://packaging.python.org/en/latest/), we use [`twine` CLI tool](https://twine.readthedocs.io/en/latest/).
```bash
pip install twine
twine upload --help
```
1. Generate API Token for [PyPI Test](https://test.pypi.org/) or [PyPI Prod](https://pypi.org/)
2. Build your python package: `python -m build --sdist --wheel "${PACKAGE_DIR}"`, here we're building both sdist and wheel, as recommended.
3. Run the twine tool: `twine upload --repository testpypi ./dist/*`, uplading to test-pypi
### Task Runner
- `CMake` and `Makefile`
- `sudo apt-get install make`
- [Taskfile](https://github.com/adriancooney/Taskfile)
- [`justfile`](https://github.com/casey/just)
- [pyinvoke](https://www.pyinvoke.org/)
* **
# Continuous Delivery using GitHub Actions
## Goals
1. Understand Continuous Delivery
2. Know the parts of a CI/CD pipeline for Python Packages
3. Have an advanced understanding of GitHub Actions
## Delivery "Environments"
>Dev $\rightarrow$ QA/Staging $\rightarrow$ Prod
- Pre-release version namings
- 0.0.0rc0 (rc = release candidate)
- 0.0.0.rc1
- 0.0.0a0 (alpha)
- 0.0.0b1 (beta)
## High-level CI/CD Workflow for Python Packages
## Detailed CI/CD Workflow for Python Packages
### GitHub CI/CD Workflow in worflows yaml file
```yaml
# .github/workflows/publish.yaml
name: Build, Test, and Publish
# triggers: whenever there is new changes pulled/pushed on this
# repo under given conditions, run the below jobs
on:
pull_request:
types: [opened, synchronize]
push:
branches:
- main
# Manually trigger a workflow
# https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_dispatch
workflow_dispatch:
jobs:
build-test-and-publish:
runs-on: ubuntu-latest
steps:
# github actions checksout, clones our repo, and checks out the branch we're working in
- uses: actions/checkout@v3
with:
# Number of commits to fetch. 0 indicates all history for all branches and tags
# fetching all tags so to aviod duplicate version tagging in 'Tag with the Release Version'
fetch-depth: 0
- name: Set up Python 3.8
uses: actions/setup-python@v3
with:
python-version: 3.8
# tagging the release version to avoid duplicate releases
- name: Tag with the Release Version
run: |
git tag $(cat version.txt)
- name: Install Python Dependencies
run: |
/bin/bash -x run.sh install
- name: Lint, Format, and Other Static Code Quality Check
run: |
/bin/bash -x run.sh lint:ci
- name: Build Python Package
run: |
/bin/bash -x run.sh build
- name: Publish to Test PyPI
# setting -x in below publish:test will not leak any secrets as they are masked in github
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
run: |
/bin/bash -x run.sh publish:test
env:
TEST_PYPI_TOKEN: ${{ secrets.TEST_PYPI_TOKEN }}
- name: Publish to Prod PyPI
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
run: |
/bin/bash -x run.sh publish:prod
env:
PROD_PYPI_TOKEN: ${{ secrets.PROD_PYPI_TOKEN }}
- name: Push Tags
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
run: |
git push origin --tags
```
### GitHub Actions Optimizations
1. Locking Requirements
- It's not really recommended to pin exact versions of dependencies to avoid future conflict
- But it's good practice to store them in the requirements file for future debugging.
- Tools:
2. Dependency Caching
- Whenever github actions gets executed in the github CI, everytime it's run on a fresh container.
Thus, everytime we'll have to download and re-install dependencies from pip again and again;
which is not a good as it's inefficeint and slows our workflow.
- Thus we would want to install all the dependencies when the workflow ran first and use it every
time a new worflow is run.
- GitHub Actions provide this functionality by caching the dependencies, it stores the installed
dependencies(`~/.cache/pip`) and downloads it everytime a new workflow is run.
[**Docs**](https://github.com/actions/cache/blob/main/examples.md#python---pip)
```yaml
- uses: actions/cache@v3
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
restore-keys: |
${{ runner.os }}-pip-
```
- Pip dependency caching using [`setup-python`](https://github.com/actions/setup-python?tab=readme-ov-file#caching-packages-dependencies) github action
```yaml
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.9'
cache: 'pip' # caching pip dependencies
- run: pip install -r requirements.txt
```
3. Parallelization
- We moved from above shown workflow to now a parallelized workflow as shown below.
- This helps in faster running of workflow, helping discover bugs in any steps
at the same time which was not possible in linear flow as earlier.
```yaml
# See .github/workflows/publish.yaml
jobs:
check-verison-txt:
...
lint-format-and-static-code-checks:
....
build-wheel-and-sdist:
...
publish:
needs:
- check-verison-txt
- lint-format-and-static-code-checks
- build-wheel-and-sdist
...
```
Raw data
{
"_id": null,
"home_page": null,
"name": "packaging-demo-avr",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "python, bash, makefile, pypi, ci-cd, setuptools, wheels, package-development, github-actions, pypi-package, pre-commit-hooks, pyproject-toml, gitactions-workflow, github-actions-enabled, pre-commit-ci, pre-commit-config",
"author": null,
"author_email": "Amit Vikram Raj <avr13405@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/d5/5f/8cc5906962cd25c9c49518aec336dd0b1d6a65cac0b25c3657c302a246f0/packaging_demo_avr-0.0.11.tar.gz",
"platform": null,
"description": "## Why package your Python code?\n\n1. Distributing your code\n2. Non-painful import statements\n3. Reproduciblity\n\n* **\n\n## `PYTHONPATH` Variable\n\n- [**Refer to this README for indepth details**](https://github.com/avr2002/python-packaging/tree/main/packaging_demo/my_folder) \n\n* **\n\n## Packaging terms:\n\n- module\n- package\n- sub-package\n- distribution package\n\n> [**Official Python Packaging Guide**](https://packaging.python.org/en/latest/)\n\n\n\n### Module\n\nA **module** in Python is a single file (with a `.py` extension) that contains Python code. It typically consists of classes, functions, and variables that can be used by other Python code. Modules are used to organize code into logical units and facilitate code reusability.\n\nFor example, consider a module named `math_operations.py` that contains functions to perform mathematical operations like addition, subtraction, multiplication, etc. You can import and use these functions in other Python scripts.\n\n```python\n# math_operations.py\n\ndef add(x, y):\n return x + y\n\ndef subtract(x, y):\n return x - y\n```\n\n### Package\n\nA **package** in Python is a collection of modules grouped together in a directory. A package is typically represented by a directory containing an `__init__.py` file (which can be empty) and one or more Python modules. The `__init__.py` file indicates to Python that the directory should be treated as a package.\n\nFor example, consider a package named `my_package`:\n\n```\nmy_package/\n\u251c\u2500\u2500 __init__.py\n\u251c\u2500\u2500 module1.py\n\u2514\u2500\u2500 module2.py\n```\n\nHere, `my_package` is a package containing `module1.py` and `module2.py`, which can be imported using dot notation (`my_package.module1`, `my_package.module2`).\n\n### Sub-package\n\nA **sub-package** in Python is a package nested within another package. This means that a package can contain other packages as well as modules. Sub-packages are created by organizing directories and adding `__init__.py` files appropriately to define the package structure.\n\nFor example:\n\n```\nmy_parent_package/\n\u251c\u2500\u2500 __init__.py\n\u2514\u2500\u2500 my_sub_package/\n \u251c\u2500\u2500 __init__.py\n \u251c\u2500\u2500 module3.py\n \u2514\u2500\u2500 module4.py\n```\n\nIn this structure, `my_sub_package` is a sub-package of `my_parent_package`, and it can contain its own modules (`module3.py`, `module4.py`). The sub-package can be imported using dot notation (`my_parent_package.my_sub_package.module3`).\n\n### Distribution Package\n\nA **distribution package** (or simply **distribution**) in Python refers to a packaged collection of Python code and resources that is made available for installation. It typically includes modules, packages, data files, configuration files, and other resources needed for a specific purpose (e.g., a library or application).\n\nDistribution packages are often distributed and installed using Python package managers such as `pip` and can be uploaded to package repositories like PyPI (Python Package Index) for easy distribution and installation by other developers.\n\nFor example, popular distribution packages include `numpy`, `fast-api`, `pandas`, etc., which are installed using `pip` and provide functionalities that can be used in Python projects.\n\n\n\n## Building a Distribution Package\n\n\n### `sdist` format\n\n\n1. Configure `setup.py`\n2. Run `python setup.py build sdist` to build the source distribution\n3. Run `pip install ./dist/<package_name>.tar.gz` to install the package\n4. Use `pip list` to see if the package is installed.\n\n* **\n\n1. If changes are made in the package then use `pip install .` which will build and install the latest package on the fly.\n\n2. Or simply use editable so that you don't always have to rebuild the package evrerytime new changes are made:\n - `pip install --editable .`\n\n\n* **\n\n- `sdist` is short for source distribution, a `.tar` file containing our code called an \"sdist\". What that means is that the distribution package only contains a subset of our source code.\n\n- A \"source distribution\" is essentially a zipped folder containing our *source* code.\n\n* **\n\n### `wheel` format\n\n\n#### Understanding Source Distribution (sdist) and Wheels\n\n- **Source Distribution (`sdist`)**:\n - Represents a zipped folder containing the source code of a Python package.\n - Primarily includes Python source files (`*.py`), configuration files, and other project-related assets.\n - Provides a portable distribution format that can be used to install the package on any platform.\n - Can be created using `python setup.py sdist`.\n\n- **Wheels (`bdist_wheel`)**:\n - A more advanced distribution format compared to `sdist`.\n - Represents a zipped file with the `.whl` extension.\n - Potentially includes more than just source code; can contain pre-compiled binaries (e.g., C extensions) for faster installation.\n - Preferred for distribution when the package includes compiled, non-Python components.\n - Created using `python setup.py bdist_wheel`.\n\n#### Benefits of Wheels Over Source Distributions\n\n- **Faster Installation**:\n - Installing from wheels (`*.whl`) is typically faster than installing from source distributions (`*.tar.gz`).\n - Wheels can include pre-compiled binaries, reducing the need for on-the-fly compilation during installation.\n - Most packages are \u201cpure Python\u201d though, so unless you are working with Python bindings such as code written in Rust, C++, C, etc. building a wheel will be just as easy as building an sdist.\n\n- **Ease of Use for Users**:\n - Users benefit from quicker installations, especially when dealing with packages that have complex dependencies or require compilation.\n\n#### Building Both `sdist` and `bdist_wheel`\n\n- It's recommended to build and publish both `sdist` and `bdist_wheel` for Python packages to accommodate different use cases and platforms.\n - `python setup.py sdist bdist_wheel`\n- Building both types of distributions allows users to choose the most suitable distribution format based on their needs and environment.\n\n#### Challenges and Considerations\n\n- **Compilation Requirements**:\n - Unfortunately, building wheels for *all* operating systems gets difficult if you have a compilation step required, so some OSS maintainers only build wheels for a single OS.\n\n - Whenever a wheel is not available for your OS, `pip` actually executes the [`setup.py`](http://setup.py) (or equivalent files) on *your* machine, right after downloading the sdist.\n\n - Building the wheel might require compiling code\n - Compiling code can be *really* slow, 5-10-30 minutes or even more, especially if the machine is weak\n - Compiling code requires dependencies, e.g. `gcc` if the source code is in C, but other languages require their own compilers. The user must install these on their on machine or the `pip install ...` will simply fail. This can happen when you install `numpy`, `pandas`, `scipy`, `pytorch`, `tensorflow`, etc.\n\n- [`setup.py`](http://setup.py) may contain arbitrary code. This is highly insecure. A `setup.py` might contain malicious code.\n\n\n- **Dependency Management**:\n - Users may need to install additional development tools and dependencies (`gcc`, etc.) to successfully install packages that require compilation.\n\n- **Security Concerns**:\n - Using `setup.py` for package installation can be insecure as it executes arbitrary code.\n - Ensure that packages obtained from untrusted sources are reviewed and validated.\n\n\n#### Naming Scheme of a `wheel` file\n\n- A wheel filename is broken down into parts separated by hyphens:\n - `{dist}-{version}(-{build})?-{python}-{abi}-{platform}.whl`\n\n- Each section in `{brackets}` is a tag, or a component of the wheel name that carries some meaning about what the wheel contains and where the wheel will or will not work.\n\n- For Example: `dist/packaging-0.0.0-py3-none-any.whl`\n\n - `packaging` is the package name\n - `0.0.0` is the verison number\n - `py3` denotes it's build for Python3\n - `abi` tag. ABI stands for application binary interface.\n - `any` stands for that this package is build to work on any platform.\n\n- Other Examples:\n - `cryptography-2.9.2-cp35-abi3-macosx_10_9_x86_64.whl`\n - `chardet-3.0.4-py2.py3-none-any.whl`\n - `PyYAML-5.3.1-cp38-cp38-win_amd64.whl`\n - `numpy-1.18.4-cp38-cp38-win32.whl`\n - `scipy-1.4.1-cp36-cp36m-macosx_10_6_intel.whl`\n\n- The reason for this is, the wheel file contains the pre-complied binary code that helps to install the package fast.\n\n\n#### Conclusion\n\n- **Package Maintenance**:\n - Maintainers should strive to provide both `sdist` and `bdist_wheel` distributions to maximize compatibility and ease of use for users.\n\n- **Consider User Experience**:\n - Consider the user's perspective when choosing the appropriate distribution format.\n - Provide clear documentation and guidance for users installing packages with compilation requirements.\n\n\n## `build` CLI tool and `pyproject.toml`\n\n- \"Build dependencies\" are anything that must be installed on your system in order to build your distribution package into an sdist or wheel.\n\n For example, we needed to `pip install wheel` in order to run `python setup.py bdist_wheel` so `wheel` is a build dependency for building wheels.\n\n- [`setup.py`](http://setup.py) files can get complex.\n\n You may need to `pip install ...` external libraries and import them into your `setup.py` file to accommodate complex build processes.\n\n The lecture shows `pytorch` and `airflow` as examples of packages with complex [`setup.py`](http://setup.py) files.\n\n- Somehow you need to be able to document build dependencies *outside* of [`setup.py`](http://setup.py).\n\n If they were documented in the `setup.py` file\u2026 you would not be able to execute the `setup.py` file to read the documented dependencies (like if they were specified in an `list` somewhere in the file).\n\n This is the original problem `pyproject.toml` was meant to solve.\n\n ```toml\n # pyproject.toml\n\n [build-system]\n # Minimum requirements for the build system to execute.\n requires = [\"setuptools>=62.0.0\", \"wheel\"]\n ```\n\n `pyproject.toml` sits adjacent to [`setup.py`](http://setup.py) in the file tree\n\n- The `build` CLI tool (`pip install build`) is a special project by the Python Packaging Authority (PyPA) which\n 1. reads the `[build-system]` table in the `pyproject.toml`,\n 2. installs those dependencies into an isolated virtual environment,\n 3. and then builds the sdist and wheel\n\n ```bash\n pip install build\n\n # both setup.py and pypproject.toml should be together, ideally in the root directory\n # python -m build --sdist --wheel path/to/dir/with/setup.py/and/pyproject.toml\n\n python -m build --sdist --wheel .\n ```\n\n\n## Moving from `setup.py` to [`setup.cfg`](https://setuptools.pypa.io/en/latest/userguide/declarative_config.html) config file\n\n>>Moving from\n\n```python\n# setup.py\nfrom pathlib import Path\n\nfrom setuptools import find_packages, setup\nimport wheel\n\n\n# Function to read the contents of README.md\ndef read_file(filename: str) -> str:\n filepath = Path(__file__).resolve().parent / filename\n with open(filepath, encoding=\"utf-8\") as file:\n return file.read()\n\n\nsetup(\n name=\"packaging-demo\",\n version=\"0.0.0\",\n packages=find_packages(),\n # package meta-data\n author=\"Amit Vikram Raj\",\n author_email=\"avr13405@gmail.com\",\n description=\"Demo for Python Packaging\",\n license=\"MIT\",\n # Set the long description from README.md\n long_description=read_file(\"README.md\"),\n long_description_content_type=\"text/markdown\",\n # install requires: libraries that are needed for the package to work\n install_requires=[\n \"numpy\", # our package depends on numpy\n ],\n # setup requires: the libraries that are needed to setup/build\n # the package distribution\n # setup_requires=[\n # \"wheel\", # to build the binary distribution we need wheel package\n # ],\n)\n```\n\n>>TO\n\n```python\n# setup.py\nfrom setuptools import setup\n\n# Now setup.py takes it's configurations from setup.cfg file\nsetup()\n```\n\n```ini\n# setup.cfg\n\n[metadata]\nname = packaging-demo\nversion = attr: packaging_demo.VERSION\nauthor = Amit Vikram Raj\nauthor_email = avr13405@gmail.com\ndescription = Demo for Python Packaging\nlong_description = file: README.md\nkeywords = one, two\nlicense = MIT\nclassifiers =\n Framework :: Django\n Programming Language :: Python :: 3\n\n[options]\nzip_safe = False\ninclude_package_data = True\n# same as find_packages() in setup()\npackages = find:\npython_requires = >=3.8\ninstall_requires =\n numpy\n importlib-metadata; python_version<\"3.10\"\n```\n\n>>ALSO addtional setting is passed to `pyproject.toml` file.\n>>Here we have specified `build-system` similar to `setup_requires` in `setup.py`\n\n```toml\n# pyproject.toml\n\n[build-system]\n# Minimum requirements for the build system to execute\nrequires = [\"setuptools\", \"wheel\", \"numpy<1.24.3\"]\n\n\n# Adding ruff.toml to pyproject.toml\n[tool.ruff]\nline-length = 99\n\n[tool.ruff.lint]\n# 1. Enable flake8-bugbear (`B`) rules, in addition to the defaults.\nselect = [\"E\", \"F\", \"B\", \"ERA\"]\n\n# 2. Avoid enforcing line-length violations (`E501`)\nignore = [\"E501\"]\n\n# 3. Avoid trying to fix flake8-bugbear (`B`) violations.\nunfixable = [\"B\"]\n\n# 4. Ignore `E402` (import violations) in all `__init__.py` files, and in select subdirectories.\n[tool.ruff.lint.per-file-ignores]\n\"__init__.py\" = [\"E402\"]\n\"**/{tests,docs,tools}/*\" = [\"E402\"]\n\n\n# copying isort configurations from .isort.cfg to pyproject.toml\n[tool.isort]\nprofile = \"black\"\nmulti_line_output = \"VERTICAL_HANGING_INDENT\"\nforce_grid_wrap = 2\nline_length = 99\n\n# copying balck config from .black.toml to pyproject.toml\n[tool.black]\nline-length = 99\nexclude = \".venv\"\n\n# copying flake8 config from .flake8 to pyproject.toml\n[tool.flake8]\ndocstring-convention = \"all\"\nextend-ignore = [\"D107\", \"D212\", \"E501\", \"W503\", \"W605\", \"D203\", \"D100\",\n \"E305\", \"E701\", \"DAR101\", \"DAR201\"]\nexclude = [\".venv\"]\nmax-line-length = 99\n\n# radon\nradon-max-cc = 10\n\n\n# copying pylint config from .pylintrc to pyproject.toml\n[tool.pylint.\"messages control\"]\ndisable = [\n \"line-too-long\",\n \"trailing-whitespace\",\n \"missing-function-docstring\",\n \"consider-using-f-string\",\n \"import-error\",\n \"too-few-public-methods\",\n \"redefined-outer-name\",\n]\n```\n\n- We have been treating [`setup.py`](http://setup.py) as a glorified config file, not really taking advantage of the fact that it is a Python file by adding logic to it.\n\n This is more common than not. Also, there has been a general shift away from using Python for config files because doing so adds complexity to *using* the config files (like having to install libraries in order to execute the config file).\n\n- `setup.cfg` is a companion file to [`setup.py`](http://setup.py) that allows us to define our package configuration in a static text file\u2014specifically an [INI format](https://en.wikipedia.org/wiki/INI_file) file.\n\n <aside>\n \ud83d\udca1 INI is a problematic, weak file format compared to more \u201cmodern\u201d formats like JSON, YAML, and *TOML*. We will prefer TOML as we move forward.\n\n </aside>\n\n- Any values that we do not directly pass as arguments to setup() will be looked for by the setup() invocation in a setup.cfg file, which is meant to sit adjacent to setup.py in the file tree if used.\n\n\n* **\n\n\n- Now we are accumulating a lot of files!\n - `setup.py`\n - `setup.cfg`\n - `pyproject.toml`\n - `README.md`\n - More files for linting and other code quality tools, e.g. `.pylintrc`, `.flake8`, `.blackrc`, `ruff.toml`, `.mypy`, `pre-commit-config.yaml`, etc.\n - More files we have not talked about yet:\n - `CHANGELOG` or `CHANGELOG.md`\n - `VERSION` or `version.txt`\n\n It turns out that nearly all of these files can be replaced with `pyproject.toml` . Nearly every linting / code quality tool supports parsing a section called `[tool.<name>]` e.g. `[tool.black]` section of `pyproject.toml` to read its configuration!\n\n The docs of each individual tool should tell you how to accomplish this.\n\n Above shown is a `pyproject.toml` with configurations for many of the linting tools we have used in the course.\n\n\n>**Can `setup.cfg` and `setup.py` be replaced as well?**\n\n\n## [Moving `setup.cfg` to `pyproject.toml`](https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html)\n\n>FROM `setup.cfg`\n```ini\n# setup.cfg\n[metadata]\nname = packaging-demo\nversion = attr: packaging_demo.VERSION\nauthor = Amit Vikram Raj\nauthor_email = avr13405@gmail.com\ndescription = Demo for Python Packaging\nlong_description = file: README.md\nkeywords = one, two\nlicense = MIT\nclassifiers =\n Programming Language :: Python :: 3\n\n[options]\nzip_safe = False\ninclude_package_data = True\n# same as find_packages() in setup()\npackages = find:\npython_requires = >=3.8\ninstall_requires =\n numpy\n importlib-metadata; python_version<\"3.10\"\n```\n\n>TO\n\n```toml\n# pyproject.toml\n\n[build-system]\n# Minimum requirements for the build system to execute\nrequires = [\"setuptools>=61.0.0\", \"wheel\"]\n\n# Adding these from setup.cfg in pyproject.toml file\n[project]\nname = \"packaging-demo\"\nauthors = [{ name = \"Amit Vikram Raj\", email = \"avr13405@gmail.com\" }]\ndescription = \"Demo for Python Packaging\"\nreadme = \"README.md\"\nrequires-python = \">=3.8\"\nkeywords = [\"one\", \"two\"]\nlicense = { text = \"MIT\" }\nclassifiers = [\"Programming Language :: Python :: 3\"]\ndependencies = [\"numpy\", 'importlib-metadata; python_version<\"3.10\"']\ndynamic = [\"version\"]\n# version = \"0.0.3\"\n# https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html#dynamic-metadata\n\n[tool.setuptools.dynamic]\n# every while making changes in package, you can change the verison in one of these files\n# version = {attr = \"packaging_demo.VERSION\"} # version read by 'packaging_demo/__init__.py' file\nversion = {file = [\"version.txt\"]} # version read by 'version.txt' file in root folder\n```\n\n>> `python -m build --sdist --wheel .` - Runs perfectly, we got rid of another config file (`setup.cfg`)\n\n\n## Replacing `setup.py` with `build-backend`\n\n\n- [PEP 517](https://peps.python.org/pep-0517/) added a `build-backend` argument to `pyproject.toml` like so:\n\n ```toml\n [build-system]\n # Defined by PEP 518:\n requires = [\"flit\"]\n # Defined by this PEP:\n build-backend = \"flit.api:main\"\n ```\n\n ```python\n # The above toml config is equivalent to\n import flit.api\n backend = flit.api.main\n ```\n\n- The `build-backend` defines an entrypoint (executable Python module in this case) that the `build` CLI uses to actually do the work of parsing `pyproject.toml` and building the wheel and sdist.\n\n- This means *you* could implement your own build backend today by writing a program that does that, and you could use it by adding your package to `requires = [...]` and specifying the entrypoint in `build-backend = ...`.\n\n\n- If you do not specify a `build-backend` in `pyproject.toml`, setuptools is assumed and package will get bulit prefectly fine.\n - If we remove `setup.py` and run `python -m build --sdist --wheel .` it runs perfectly without it because the default value of `build-system` is set as `build-backend = \"setuptools.build_meta\"` in `build` CLI which builds our package.\n\n- But you can still explicitly declare `setuptools` as your build backend like this\n\n ```toml\n # pyproject.toml\n\n ...\n\n [build-system]\n requires = [\"setuptools>=61.0.0\", \"wheel\"]\n build-backend = \"setuptools.build_meta\"\n\n ...\n ```\n\n Each build backend typically extends the `pyproject.toml` file with its own configuration options. For example,\n\n ```toml\n # pyproject.toml\n\n ...\n\n [tool.setuptools.package-data]\n package_demo = [\"*.json\"]\n\n [tool.setuptools.dynamic]\n version = {file = \"version.txt\"}\n long_description = {file = \"README.md\"}\n\n ...\n ```\n\n- If you choose to use `setuptools` in your project, you can add these sections to `pyproject.toml`. You can read more about this in the `setuptools` documentation\n\n\n## Adding Data Files in our Package\n\n- It's often beneficial to include non-python files like data files or binary files inside of your package because oftentimes your Python code relies on these non-python files.\n\n- And then we saw that if we're going to include those files, we need to get those files to end up inside of our package folder because it's our package folder that ends up inside of our users virtual environments upon installation.\n\n- We also saw that by default, all non-python files don't make it into that final virtual environment folder. That is, don't actually make it into our package folder during build procedure.\n\nSo, how do we make sure that these files end up in our wheel/dist build of our package? For example here we demo with `cities.json` file that we want in our package as it's used by `states_info.py` file.\n\n>[**Official `setuptools` Docs for Data Support**](https://setuptools.pypa.io/en/latest/userguide/datafiles.html)\n\n### Using [MANIFEST.in](https://setuptools.pypa.io/en/latest/userguide/miscellaneous.html#using-manifest-in) File\n\n\n```toml\n# pyprject.toml\n\n[tool.setuptools]\n# ...\n# By default, include-package-data is true in pyproject.toml, so you do\n# NOT have to specify this line.\ninclude-package-data = true\n```\n\nSo, setuptools by default has this `include-package-data` value set to `true` as shown in the [official docs](https://setuptools.pypa.io/en/latest/userguide/datafiles.html) but we need to create an extra file `MANIFEST.in` and specify the data which we want to inculde in our package present at root dir.\n\n>>**IMP:** It's import all the folders in the package directory should have `__init__.py` file inculing the data directory which we want to include because the `find_packages()` recusrive process that setuptools does will not go into fo;ders that have `__init__.py` file in it.\n\n```in\n# MANIFEST.in\n\ninclude packaging_demo/*.json\ninclude packaging_demo/my_folder/*.json\n\nOR\nRecursive include all json files in the package directory\n\nrecursive-include packaging_demo/ *.json\n```\n\n>[Docs on configuring MANIFEST.in file](https://setuptools.pypa.io/en/latest/userguide/miscellaneous.html#using-manifest-in)\n\n### Without using `MANIFEST.in` file\n\nFrom [setuptools docs](https://setuptools.pypa.io/en/latest/userguide/datafiles.html) we can add this in our `pyproject.toml` file:\n\n```toml\n# this is by default true so no need to explicitly add it\n# but as mentioned in the docs, it is false for other methods like setup.py or setup.cfg\n[tool.setuptools]\ninclude-package-data = true\n\n# add the data here, it's finding the files recursively\n[tool.setuptools.package-data]\npackage_demo = [\"*.json\"]\n```\n\n\n## Other `build-backend` systems than `setuptools`\n\nOther than `setuptools` we can use these build backend systems. The point to note is when using other systems the `pyproject.toml` cofiguration should follow their standerds.\n\n1. [Hatch](https://hatch.pypa.io/1.9/config/build)\n\n ```toml\n [build-system]\n requires = [\"hatchling\"]\n build-backend = \"hatchling.build\"\n ```\n2. [Poetry](https://python-poetry.org/docs/pyproject/)\n\n ```toml\n [build-system]\n requires = [\"poetry-core>=1.0.0\"]\n build-backend = \"poetry.core.masonry.api\"\n ```\n\n## Reproducibility, Dependency Graph, Locking Dependencies, Dependency Hell\n\n- Documenting the exact versions of our dependencies and their dependencies and so on.\n\n- It's advisable to have as little as possible number of dependencies associated with our package, as it can lead to dependency hell, or dependency conflict with other package as explained in the lectures.\n\n- The more complex the dependency tree higher the chance of conflicts with future version of other libraries.\n\n- [Dependency Graph Analysis By Eric](https://github.com/avr2002/dependency-graph-pip-analysis)\n\n- Keeping the pinned versions of dependencies and python versions is advicable for troubleshooting purposes:\n - `pip freeze > requirements.txt`\n\n\n```bash\npip install pipdeptree graphviz\n\nsudo apt-get install graphviz \n\n# generate the dependency graph\npipdeptree -p packaging-demo --graph-output png > dependency-graph.png\n```\n\n<a href='https://raw.githubusercontent.com/avr2002/python-packaging/main/packaging_demo/assets/dependency-graph.png' target='_blank'>\n <img src='https://raw.githubusercontent.com/avr2002/python-packaging/main/packaging_demo/assets/dependency-graph.png' alt='Dependency Graph of packaging-demo package' title='Dependency Graph of packaging-demo package'>\n</a>\n\n* **\n\n## Adding Optional/Extra Dependencies to our Project\n\n```toml\n[project.optional-dependencies]\ndev = [\"ruff\", \"mypy\", \"black\"]\n```\n\n```bash\n# installing our package with optional dependencies\npip install '.[dev]'\n```\n\n* **\n\n```toml\n[project.optional-dependencies]\n# for developement\ndev = [\"ruff\", \"mypy\", \"black\"]\n# plugin based architecture\ncolors = [\"rich\"]\n```\n\n```bash\n# plugin based installation\npip install '.[colors]'\n# here we demo with rich library, if user wants the output to be\n# colorized then they can install our package like this.\n\n\n# we can add multiple optional dependencies like:\npip install '.[colors, dev]'\n```\n\n* **\n\n```toml\n[project.optional-dependencies]\n# for developement\ndev = [\"ruff\", \"mypy\", \"black\"]\n# plugin based architecture\ncolors = [\"rich\"]\n# install all dependencies\nall = [\"packaging-demo[dev, colors]\"]\n```\n\n```bash\n# Installing all dependencies all at once\npip install '.[all]'\n```\n\n\n## Shopping for dependencies\n\nWe can use [Snyk](https://snyk.io/advisor/python/fastapi) to check how stable, well supported, if any security issues, etc. are present for the dependencies which we are going to use for our package and then take the decision on using it in our project.\n\n* **\n\n# Continuous Delivery: Publishing to PyPI\n\n- Few key terms realted to Continuous Delivery\n - DevOps, Waterfall, Agile, Scrum\n\n\n### Publishing to PyPI\n\nTo pusblish our package to PyPI[Python Packaging Index], as stated in the [official guide](https://packaging.python.org/en/latest/), we use [`twine` CLI tool](https://twine.readthedocs.io/en/latest/).\n\n```bash\npip install twine\n\ntwine upload --help\n```\n\n1. Generate API Token for [PyPI Test](https://test.pypi.org/) or [PyPI Prod](https://pypi.org/)\n\n2. Build your python package: `python -m build --sdist --wheel \"${PACKAGE_DIR}\"`, here we're building both sdist and wheel, as recommended.\n\n3. Run the twine tool: `twine upload --repository testpypi ./dist/*`, uplading to test-pypi\n\n\n### Task Runner\n\n- `CMake` and `Makefile`\n\n - `sudo apt-get install make`\n\n\n- [Taskfile](https://github.com/adriancooney/Taskfile)\n\n\n- [`justfile`](https://github.com/casey/just)\n- [pyinvoke](https://www.pyinvoke.org/)\n\n* **\n\n# Continuous Delivery using GitHub Actions\n\n## Goals\n1. Understand Continuous Delivery\n2. Know the parts of a CI/CD pipeline for Python Packages\n3. Have an advanced understanding of GitHub Actions\n\n\n\n## Delivery \"Environments\"\n\n>Dev $\\rightarrow$ QA/Staging $\\rightarrow$ Prod\n\n\n- Pre-release version namings\n - 0.0.0rc0 (rc = release candidate)\n - 0.0.0.rc1\n - 0.0.0a0 (alpha)\n - 0.0.0b1 (beta)\n\n\n\n\n## High-level CI/CD Workflow for Python Packages\n\n\n\n\n## Detailed CI/CD Workflow for Python Packages\n\n\n\n\n### GitHub CI/CD Workflow in worflows yaml file\n\n```yaml\n# .github/workflows/publish.yaml\n\nname: Build, Test, and Publish\n\n# triggers: whenever there is new changes pulled/pushed on this\n# repo under given conditions, run the below jobs\non:\n pull_request:\n types: [opened, synchronize]\n\n push:\n branches:\n - main\n\n # Manually trigger a workflow\n # https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_dispatch\n workflow_dispatch:\n\njobs:\n\n build-test-and-publish:\n\n runs-on: ubuntu-latest\n\n steps:\n # github actions checksout, clones our repo, and checks out the branch we're working in\n - uses: actions/checkout@v3\n with:\n # Number of commits to fetch. 0 indicates all history for all branches and tags\n # fetching all tags so to aviod duplicate version tagging in 'Tag with the Release Version'\n fetch-depth: 0\n\n - name: Set up Python 3.8\n uses: actions/setup-python@v3\n with:\n python-version: 3.8\n\n # tagging the release version to avoid duplicate releases\n - name: Tag with the Release Version\n run: |\n git tag $(cat version.txt)\n\n - name: Install Python Dependencies\n run: |\n /bin/bash -x run.sh install\n\n - name: Lint, Format, and Other Static Code Quality Check\n run: |\n /bin/bash -x run.sh lint:ci\n\n - name: Build Python Package\n run: |\n /bin/bash -x run.sh build\n\n - name: Publish to Test PyPI\n # setting -x in below publish:test will not leak any secrets as they are masked in github\n if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}\n run: |\n /bin/bash -x run.sh publish:test\n env:\n TEST_PYPI_TOKEN: ${{ secrets.TEST_PYPI_TOKEN }}\n\n - name: Publish to Prod PyPI\n if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}\n run: |\n /bin/bash -x run.sh publish:prod\n env:\n PROD_PYPI_TOKEN: ${{ secrets.PROD_PYPI_TOKEN }}\n\n - name: Push Tags\n if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}\n run: |\n git push origin --tags\n```\n\n### GitHub Actions Optimizations\n\n1. Locking Requirements\n - It's not really recommended to pin exact versions of dependencies to avoid future conflict\n - But it's good practice to store them in the requirements file for future debugging.\n - Tools:\n\n2. Dependency Caching\n - Whenever github actions gets executed in the github CI, everytime it's run on a fresh container.\n Thus, everytime we'll have to download and re-install dependencies from pip again and again;\n which is not a good as it's inefficeint and slows our workflow.\n\n - Thus we would want to install all the dependencies when the workflow ran first and use it every\n time a new worflow is run.\n\n - GitHub Actions provide this functionality by caching the dependencies, it stores the installed\n dependencies(`~/.cache/pip`) and downloads it everytime a new workflow is run.\n [**Docs**](https://github.com/actions/cache/blob/main/examples.md#python---pip)\n\n ```yaml\n - uses: actions/cache@v3\n with:\n path: ~/.cache/pip\n key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}\n restore-keys: |\n ${{ runner.os }}-pip-\n ```\n\n - Pip dependency caching using [`setup-python`](https://github.com/actions/setup-python?tab=readme-ov-file#caching-packages-dependencies) github action\n\n ```yaml\n steps:\n - uses: actions/checkout@v4\n - uses: actions/setup-python@v5\n with:\n python-version: '3.9'\n cache: 'pip' # caching pip dependencies\n - run: pip install -r requirements.txt\n ```\n\n3. Parallelization\n\n - We moved from above shown workflow to now a parallelized workflow as shown below.\n - This helps in faster running of workflow, helping discover bugs in any steps\n at the same time which was not possible in linear flow as earlier.\n\n```yaml\n# See .github/workflows/publish.yaml\n\njobs:\n\n check-verison-txt:\n ...\n\n lint-format-and-static-code-checks:\n ....\n\n build-wheel-and-sdist:\n ...\n\n publish:\n needs:\n - check-verison-txt\n - lint-format-and-static-code-checks\n - build-wheel-and-sdist\n ...\n```\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Demo for Python Packaging",
"version": "0.0.11",
"project_urls": {
"Documentation": "https://github.com/avr2002/python-packaging/blob/main/README.md",
"Repository": "https://github.com/avr2002/python-packaging"
},
"split_keywords": [
"python",
" bash",
" makefile",
" pypi",
" ci-cd",
" setuptools",
" wheels",
" package-development",
" github-actions",
" pypi-package",
" pre-commit-hooks",
" pyproject-toml",
" gitactions-workflow",
" github-actions-enabled",
" pre-commit-ci",
" pre-commit-config"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2a9e3967176f7f7b996615459c3f8ecda2f83dedb70e7e4f3173f9dd8dd9a033",
"md5": "086cfcceb03d1365f2707a3f17447082",
"sha256": "9385bc574e3e141d99e8bc10b395a27909926073f72d3c6f2bd6dde61608e40b"
},
"downloads": -1,
"filename": "packaging_demo_avr-0.0.11-py3-none-any.whl",
"has_sig": false,
"md5_digest": "086cfcceb03d1365f2707a3f17447082",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 16891,
"upload_time": "2024-06-01T09:35:44",
"upload_time_iso_8601": "2024-06-01T09:35:44.435341Z",
"url": "https://files.pythonhosted.org/packages/2a/9e/3967176f7f7b996615459c3f8ecda2f83dedb70e7e4f3173f9dd8dd9a033/packaging_demo_avr-0.0.11-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d55f8cc5906962cd25c9c49518aec336dd0b1d6a65cac0b25c3657c302a246f0",
"md5": "ab553b2343913562e3e5c76b6638c58c",
"sha256": "ea42043ff4aefe73db155037c21ebc8c01585b573f0d5bac673336ba75f5e3f1"
},
"downloads": -1,
"filename": "packaging_demo_avr-0.0.11.tar.gz",
"has_sig": false,
"md5_digest": "ab553b2343913562e3e5c76b6638c58c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 36956,
"upload_time": "2024-06-01T09:35:46",
"upload_time_iso_8601": "2024-06-01T09:35:46.722623Z",
"url": "https://files.pythonhosted.org/packages/d5/5f/8cc5906962cd25c9c49518aec336dd0b1d6a65cac0b25c3657c302a246f0/packaging_demo_avr-0.0.11.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-06-01 09:35:46",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "avr2002",
"github_project": "python-packaging",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "annotated-types",
"specs": [
[
"==",
"0.6.0"
]
]
},
{
"name": "anyio",
"specs": [
[
"==",
"4.3.0"
]
]
},
{
"name": "backports.tarfile",
"specs": [
[
"==",
"1.1.1"
]
]
},
{
"name": "black",
"specs": [
[
"==",
"24.4.2"
]
]
},
{
"name": "build",
"specs": [
[
"==",
"1.2.1"
]
]
},
{
"name": "certifi",
"specs": [
[
"==",
"2024.2.2"
]
]
},
{
"name": "cffi",
"specs": [
[
"==",
"1.16.0"
]
]
},
{
"name": "cfgv",
"specs": [
[
"==",
"3.4.0"
]
]
},
{
"name": "charset-normalizer",
"specs": [
[
"==",
"3.3.2"
]
]
},
{
"name": "click",
"specs": [
[
"==",
"8.1.7"
]
]
},
{
"name": "coverage",
"specs": [
[
"==",
"7.5.1"
]
]
},
{
"name": "cryptography",
"specs": [
[
"==",
"42.0.7"
]
]
},
{
"name": "distlib",
"specs": [
[
"==",
"0.3.8"
]
]
},
{
"name": "dnspython",
"specs": [
[
"==",
"2.6.1"
]
]
},
{
"name": "docutils",
"specs": [
[
"==",
"0.21.2"
]
]
},
{
"name": "email_validator",
"specs": [
[
"==",
"2.1.1"
]
]
},
{
"name": "fastapi",
"specs": [
[
"==",
"0.111.0"
]
]
},
{
"name": "fastapi-cli",
"specs": [
[
"==",
"0.0.3"
]
]
},
{
"name": "filelock",
"specs": [
[
"==",
"3.14.0"
]
]
},
{
"name": "h11",
"specs": [
[
"==",
"0.14.0"
]
]
},
{
"name": "httpcore",
"specs": [
[
"==",
"1.0.5"
]
]
},
{
"name": "httptools",
"specs": [
[
"==",
"0.6.1"
]
]
},
{
"name": "httpx",
"specs": [
[
"==",
"0.27.0"
]
]
},
{
"name": "identify",
"specs": [
[
"==",
"2.5.36"
]
]
},
{
"name": "idna",
"specs": [
[
"==",
"3.7"
]
]
},
{
"name": "importlib_metadata",
"specs": [
[
"==",
"7.1.0"
]
]
},
{
"name": "iniconfig",
"specs": [
[
"==",
"2.0.0"
]
]
},
{
"name": "jaraco.classes",
"specs": [
[
"==",
"3.4.0"
]
]
},
{
"name": "jaraco.context",
"specs": [
[
"==",
"5.3.0"
]
]
},
{
"name": "jaraco.functools",
"specs": [
[
"==",
"4.0.1"
]
]
},
{
"name": "jeepney",
"specs": [
[
"==",
"0.8.0"
]
]
},
{
"name": "Jinja2",
"specs": [
[
"==",
"3.1.4"
]
]
},
{
"name": "keyring",
"specs": [
[
"==",
"25.2.1"
]
]
},
{
"name": "markdown-it-py",
"specs": [
[
"==",
"3.0.0"
]
]
},
{
"name": "MarkupSafe",
"specs": [
[
"==",
"2.1.5"
]
]
},
{
"name": "mdurl",
"specs": [
[
"==",
"0.1.2"
]
]
},
{
"name": "more-itertools",
"specs": [
[
"==",
"10.2.0"
]
]
},
{
"name": "mypy",
"specs": [
[
"==",
"1.10.0"
]
]
},
{
"name": "mypy-extensions",
"specs": [
[
"==",
"1.0.0"
]
]
},
{
"name": "nh3",
"specs": [
[
"==",
"0.2.17"
]
]
},
{
"name": "nodeenv",
"specs": [
[
"==",
"1.8.0"
]
]
},
{
"name": "numpy",
"specs": [
[
"==",
"1.26.4"
]
]
},
{
"name": "orjson",
"specs": [
[
"==",
"3.10.3"
]
]
},
{
"name": "packaging",
"specs": [
[
"==",
"24.0"
]
]
},
{
"name": "pathspec",
"specs": [
[
"==",
"0.12.1"
]
]
},
{
"name": "pkginfo",
"specs": [
[
"==",
"1.10.0"
]
]
},
{
"name": "platformdirs",
"specs": [
[
"==",
"4.2.2"
]
]
},
{
"name": "pluggy",
"specs": [
[
"==",
"1.5.0"
]
]
},
{
"name": "pre-commit",
"specs": [
[
"==",
"3.7.1"
]
]
},
{
"name": "pycparser",
"specs": [
[
"==",
"2.22"
]
]
},
{
"name": "pydantic",
"specs": [
[
"==",
"2.7.1"
]
]
},
{
"name": "pydantic_core",
"specs": [
[
"==",
"2.18.2"
]
]
},
{
"name": "Pygments",
"specs": [
[
"==",
"2.18.0"
]
]
},
{
"name": "pyproject_hooks",
"specs": [
[
"==",
"1.1.0"
]
]
},
{
"name": "pytest",
"specs": [
[
"==",
"8.2.0"
]
]
},
{
"name": "pytest-cov",
"specs": [
[
"==",
"5.0.0"
]
]
},
{
"name": "python-dotenv",
"specs": [
[
"==",
"1.0.1"
]
]
},
{
"name": "python-multipart",
"specs": [
[
"==",
"0.0.9"
]
]
},
{
"name": "pytz",
"specs": [
[
"==",
"2024.1"
]
]
},
{
"name": "PyYAML",
"specs": [
[
"==",
"6.0.1"
]
]
},
{
"name": "readme_renderer",
"specs": [
[
"==",
"43.0"
]
]
},
{
"name": "requests",
"specs": [
[
"==",
"2.31.0"
]
]
},
{
"name": "requests-toolbelt",
"specs": [
[
"==",
"1.0.0"
]
]
},
{
"name": "rfc3986",
"specs": [
[
"==",
"2.0.0"
]
]
},
{
"name": "rich",
"specs": [
[
"==",
"13.7.1"
]
]
},
{
"name": "ruff",
"specs": [
[
"==",
"0.4.4"
]
]
},
{
"name": "SecretStorage",
"specs": [
[
"==",
"3.3.3"
]
]
},
{
"name": "shellingham",
"specs": [
[
"==",
"1.5.4"
]
]
},
{
"name": "sniffio",
"specs": [
[
"==",
"1.3.1"
]
]
},
{
"name": "starlette",
"specs": [
[
"==",
"0.37.2"
]
]
},
{
"name": "twine",
"specs": [
[
"==",
"5.1.0"
]
]
},
{
"name": "typer",
"specs": [
[
"==",
"0.12.3"
]
]
},
{
"name": "typing_extensions",
"specs": [
[
"==",
"4.11.0"
]
]
},
{
"name": "ujson",
"specs": [
[
"==",
"5.10.0"
]
]
},
{
"name": "urllib3",
"specs": [
[
"==",
"2.2.1"
]
]
},
{
"name": "uvicorn",
"specs": [
[
"==",
"0.29.0"
]
]
},
{
"name": "uvloop",
"specs": [
[
"==",
"0.19.0"
]
]
},
{
"name": "virtualenv",
"specs": [
[
"==",
"20.26.2"
]
]
},
{
"name": "watchfiles",
"specs": [
[
"==",
"0.21.0"
]
]
},
{
"name": "websockets",
"specs": [
[
"==",
"12.0"
]
]
},
{
"name": "zipp",
"specs": [
[
"==",
"3.18.2"
]
]
}
],
"lcname": "packaging-demo-avr"
}
