# Package auto assembler
Package auto assembler is a tool that meant to streamline creation of single module packages.
Its purpose is to automate as many aspects of python package creation as possible,
to shorten a development cycle of reusable components, maintain certain standard of quality
for reusable code. It provides tool to simplify the process of package creatrion
to a point that it can be triggered automatically within ci/cd pipelines,
with minimal preparations and requirements for new modules.
```python
import sys
sys.path.append('../')
from package_auto_assembler import (VersionHandler, \
ImportMappingHandler, RequirementsHandler, MetadataHandler, \
LocalDependaciesHandler, LongDocHandler, SetupDirHandler, \
ReleaseNotesHandler, MkDocsHandler, PackageAutoAssembler, \
DependenciesAnalyser)
```
### 1. Package versioning
Package versioning within paa is done based on semantic versioning.
`major.minor.patch`
By default, patch is updated, but the minor and major could also be update based on, for example, commit messages or manually from the log file.
Package auto assembler does try to pull latest version from package storage, but in case of failure uses version logs.
#### Initialize VersionHandler
```python
pv = VersionHandler(
# required
versions_filepath = '../tests/package_auto_assembler/other/lsts_package_versions.yml',
log_filepath = '../tests/package_auto_assembler/other/version_logs.csv',
# optional
default_version = "0.0.1")
```
#### Add new package
```python
pv.add_package(
package_name = "new_package",
# optional
version = "0.0.1"
)
```
#### Update package version
```python
pv.increment_patch(
package_name = "new_package"
)
## for not tracked package
pv.increment_patch(
package_name = "another_new_package",
# optional
default_version = "0.0.1"
)
```
There are no known versions of 'another_new_package', 0.0.1 will be used!
#### Display current versions and logs
```python
pv.get_versions(
# optional
versions_filepath = '../tests/package_auto_assembler/other/lsts_package_versions.yml'
)
```
{'another_new_package': '0.0.1', 'new_package': '0.0.2'}
```python
pv.get_version(
package_name='new_package'
)
```
'0.0.2'
```python
pv.get_logs(
# optional
log_filepath = '../tests/package_auto_assembler/other/version_logs.csv'
)
```
<div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>Timestamp</th>
<th>Package</th>
<th>Version</th>
</tr>
</thead>
<tbody>
<tr>
<th>0</th>
<td>2024-07-29 03:26:39</td>
<td>new_package</td>
<td>0.0.1</td>
</tr>
<tr>
<th>1</th>
<td>2024-07-29 03:26:40</td>
<td>new_package</td>
<td>0.0.2</td>
</tr>
<tr>
<th>2</th>
<td>2024-07-29 03:26:40</td>
<td>another_new_package</td>
<td>0.0.1</td>
</tr>
</tbody>
</table>
</div>
#### Flush versions and logs
```python
pv.flush_versions()
pv.flush_logs()
```
#### Get latest available version with pip
```python
pv.get_latest_pip_version(package_name = 'package-auto-assembler')
```
'0.3.1'
### 2. Import mapping
Install and import names of dependencies may vary. The mapping files maps import names to install names so that requirements extraction from `.py` files is possible.
#### Initialize ImportMappingHandler
```python
im = ImportMappingHandler(
# required
mapping_filepath = "../env_spec/package_mapping.json"
)
```
#### Load package mappings
```python
im.load_package_mappings(
# optional
mapping_filepath = "../env_spec/package_mapping.json"
)
```
{'PIL': 'Pillow',
'bs4': 'beautifulsoup4',
'fitz': 'PyMuPDF',
'attr': 'attrs',
'dotenv': 'python-dotenv',
'googleapiclient': 'google-api-python-client',
'google_auth_oauthlib': 'google-auth-oauthlib',
'sentence_transformers': 'sentence-transformers',
'flask': 'Flask',
'stdlib_list': 'stdlib-list',
'sklearn': 'scikit-learn',
'yaml': 'pyyaml',
'package_auto_assembler': 'package-auto-assembler',
'git': 'gitpython'}
### 3. Extracting and merging requirements
Maintaining requirements is much simpler, when done automatically based on the `.py` files.
The actual requirements files is still constructed. Standard libraries are not added, others are added with their versions, if specified. Local files are also used as dependencies, from which imports are extracted as well.
For example:
```python
import os
import pandas
import attr #>=22.2.0
from .components.local_dep import *
```
produces
``` txt
pandas
attrs >=22.2.0
yaml
```
as requirements file, where `yaml` is extracted from `local_dep.py` file.
Checking dependecies for vulnerabilities is usefull and it is done with `pip audit` which is integrated into the paa package and is used by default.
Optional requirements for `extras_require` could be probided the same way normal requirements are, but each like that contains an import like that should be commented out in a special way, starting with `#!`, for example:
```python
import os
import pandas
import attr #>=22.2.0
#! import hnswlib #==0.8.0
```
produces
``` txt
pandas
attrs >=22.2.0
hnswlib==0.8.0; extra == "hnswlib"
```
#### Initialize RequirementsHandler
```python
rh = RequirementsHandler(
# optional/required later
module_filepath = "../tests/package_auto_assembler/other/example_module.py",
package_mappings = {'PIL': 'Pillow',
'bs4': 'beautifulsoup4',
'fitz': 'PyMuPDF',
'attr': 'attrs',
'dotenv': 'python-dotenv',
'googleapiclient': 'google-api-python-client',
'sentence_transformers': 'sentence-transformers',
'flask': 'Flask',
'stdlib_list': 'stdlib-list',
'sklearn': 'scikit-learn',
'yaml': 'pyyaml'},
requirements_output_path = "../tests/package_auto_assembler/other/",
output_requirements_prefix = "requirements_",
custom_modules_filepath = "../tests/package_auto_assembler/dependancies",
python_version = '3.8',
add_header = True
)
```
#### List custom modules for a given directory
```python
rh.list_custom_modules(
# optional
custom_modules_filepath="../tests/package_auto_assembler/dependancies"
)
```
['example_local_dependacy_1', 'example_local_dependacy_2']
#### Check if module is a standard python library
```python
rh.is_standard_library(
# required
module_name = 'example_local_dependacy_1',
# optional
python_version = '3.8'
)
```
False
```python
rh.is_standard_library(
# required
module_name = 'logging',
# optional
python_version = '3.8'
)
```
True
#### Extract requirements from the module file
```python
rh.extract_requirements(
# optional
module_filepath = "../tests/package_auto_assembler/other/example_module.py",
custom_modules = ['example_local_dependacy_2', 'example_local_dependacy_1'],
package_mappings = {'PIL': 'Pillow',
'bs4': 'beautifulsoup4',
'fitz': 'PyMuPDF',
'attr': 'attrs',
'dotenv': 'python-dotenv',
'googleapiclient': 'google-api-python-client',
'sentence_transformers': 'sentence-transformers',
'flask': 'Flask',
'stdlib_list': 'stdlib-list',
'sklearn': 'scikit-learn',
'yaml': 'pyyaml'},
python_version = '3.8',
add_header=True
)
```
(['attrs>=22.2.0'],
['torch<=2.4.1', 'fastapi[all]', 'scikit-learn==1.5.1', 'numpy'])
```python
rh.requirements_list
```
['attrs>=22.2.0']
```python
rh.optional_requirements_list
```
['torch<=2.4.1', 'fastapi[all]', 'scikit-learn==1.5.1', 'numpy']
#### Audit dependencies
```python
rh.check_vulnerabilities(
# optional if ran extract_requirements() before
requirements_list = None,
raise_error = True
)
```
No known vulnerabilities found
```python
rh.vulnerabilities
```
[]
```python
try:
rh.check_vulnerabilities(
# optional if ran extract_requirements() before
requirements_list = ['attrs>=22.2.0', 'pandas', 'hnswlib==0.7.0'],
raise_error = True
)
except Exception as e:
print(f"Error: {e}")
```
Found 1 known vulnerability in 1 package
Name Version ID Fix Versions
------- ------- ------------------- ------------
hnswlib 0.7.0 GHSA-xwc8-rf6m-xr86
Error: Found vulnerabilities, resolve them or ignore check to move forwards!
```python
rh.vulnerabilities
```
[{'name': 'hnswlib',
'version': '0.7.0',
'id': 'GHSA-xwc8-rf6m-xr86',
'fix_versions': None}]
#### Save requirements to a file
```python
rh.write_requirements_file(
# optional/required later
module_name = 'example_module',
requirements = ['### example_module.py', 'attrs>=22.2.0'],
output_path = "../tests/package_auto_assembler/other/",
prefix = "requirements_"
)
```
#### Read requirements
```python
rh.read_requirements_file(
# required
requirements_filepath = "../tests/package_auto_assembler/other/requirements_example_module.txt"
)
```
['attrs>=22.2.0']
### 4. Preparing metadata
Since all of the necessary information for building a package needs to be contained within main component `.py` file, basic metadata is provided with the use of `__package_metadata__` dictionary object, defined within that `.py` file. It is also used as a trigger for package building within paa pipeline.
Even though some general information shared between packages could be provided through general config, but package specific info should be provided through `__package_metadata__`. It should support most text fields from setup file, but for others the following fields are available:
- `classifiers`: adds classifiers to the general ones from config
- `extras_require`: a dictionary of optional package that wouldn't be installed during normal installation. The key could be used during installation and the value would be a list of dependencies.
- `install_requires` : adds requirements to the list read from imports
\* Note that providing dependencies this way does not check them through pip-audit or translate them through package mapping
#### Initializing MetadataHandler
```python
mh = MetadataHandler(
# optional/required later
module_filepath = "../tests/package_auto_assembler/other/example_module.py"
)
```
#### Check if metadata is available
```python
mh.is_metadata_available(
# optional
module_filepath = "../tests/package_auto_assembler/other/example_module.py"
)
```
True
#### Extract metadata from module
```python
mh.get_package_metadata(
# optional
module_filepath = "../tests/package_auto_assembler/other/example_module.py"
)
```
{'author': 'Kyrylo Mordan',
'author_email': 'parachute.repo@gmail.com',
'version': '0.0.1',
'description': 'A mock handler for simulating a vector database.',
'keywords': ['python', 'vector database', 'similarity search']}
### 5. Merging local dependacies into single module
Package auto assembler creates `single module packages`, meaning that once package is built all of the object are imported from a single place. The packaging tool does allow for `local dependecies` which are `.py` files imported from specified dependencies directory and its subfolders. Packaging structure may look like the following:
```
packaging repo/
└src/
├ <package names>.py
└ components
├local_dependecy.py
└subdir_1
└local_dependency_2.py
```
During packaging process paa merges main module with its local dependies into a single file.
#### Initializing LocalDependaciesHandler
```python
ldh = LocalDependaciesHandler(
# required
main_module_filepath = "../tests/package_auto_assembler/other/example_module.py",
dependencies_dir = "../tests/package_auto_assembler/dependancies/",
# optional
save_filepath = "./combined_example_module.py"
)
```
#### Combine main module with dependacies
```python
print(ldh.combine_modules(
# optional
main_module_filepath = "../tests/package_auto_assembler/other/example_module.py",
dependencies_dir = "../tests/package_auto_assembler/dependancies/",
add_empty_design_choices = False
)[0:1000])
```
"""
Mock Vector Db Handler
This class is a mock handler for simulating a vector database, designed primarily for testing and development scenarios.
It offers functionalities such as text embedding, hierarchical navigable small world (HNSW) search,
and basic data management within a simulated environment resembling a vector database.
"""
import logging
import json
import time
import attr #>=22.2.0
import sklearn
__design_choices__ = {}
@attr.s
class Shouter:
"""
A class for managing and displaying formatted log messages.
This class uses the logging module to create and manage a logger
for displaying formatted messages. It provides a method to output
various types of lines and headers, with customizable message and
line lengths.
"""
# Formatting settings
dotline_length = attr.ib(default=50)
# Logger settings
logger = attr.ib(default=None)
logger_name = attr.ib(default='Shouter')
loggerLvl = attr.ib(default=logging.DEBUG)
log
```python
ldh.dependencies_names_list
```
['example_local_dependacy_2', 'example_local_dependacy_1', 'dep_from_bundle_1']
#### Save combined module
```python
ldh.save_combined_modules(
# optional
combined_module = ldh.combine_modules(),
save_filepath = "./combined_example_module.py"
)
```
### 6. Prepare README
Package description is based on `.ipynb` with same name as the `.py`. By default it is converted to markdown as is, but there is also an option to execute it.
```python
import logging
ldh = LongDocHandler(
# optional/required later
notebook_path = "../tests/package_auto_assembler/other/example_module.ipynb",
markdown_filepath = "../example_module.md",
timeout = 600,
kernel_name = 'python3',
# logger
loggerLvl = logging.DEBUG
)
```
#### Convert notebook to md without executing
```python
ldh.convert_notebook_to_md(
# optional
notebook_path = "../tests/package_auto_assembler/other/example_module.ipynb",
output_path = "../example_module.md"
)
```
Converted ../tests/package_auto_assembler/example_module.ipynb to ../example_module.md
#### Convert notebook to md with executing
```python
ldh.convert_and_execute_notebook_to_md(
# optional
notebook_path = "../tests/package_auto_assembler/other/example_module.ipynb",
output_path = "../example_module.md",
timeout = 600,
kernel_name = 'python3'
)
```
Converted and executed ../tests/package_auto_assembler/example_module.ipynb to ../example_module.md
#### Return long description
```python
long_description = ldh.return_long_description(
# optional
markdown_filepath = "../example_module.md"
)
```
### 7. Assembling setup directory
Package are created following rather simple sequence of steps. At some point of the process a temporary directory is created to store the following files:
- `__init__.py` is a simple import from a single module
- `cli.py` is optional packaged cli tool
- `<package name>.py` is a single module with all of the local dependecies
- `README.md` is a package description file based on `.ipynb` file
- `setup.py` is a setup file for making a package
#### Initializing SetupDirHandler
```python
sdh = SetupDirHandler(
# required
module_filepath = "../tests/package_auto_assembler/other/example_module.py",
# optional/ required
module_name = "example_module",
metadata = {'author': 'Kyrylo Mordan',
'version': '0.0.1',
'description': 'Example module.',
'long_description' : long_description,
'keywords': ['python']},
license_path = "../LICENSE",
requirements = ['attrs>=22.2.0'],
classifiers = ['Development Status :: 3 - Alpha',
'Intended Audience :: Developers',
'Intended Audience :: Science/Research',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.9',
'Programming Language :: Python :: 3.10',
'Programming Language :: Python :: 3.11',
'License :: OSI Approved :: MIT License',
'Topic :: Scientific/Engineering'],
setup_directory = "./example_setup_dir"
)
```
#### Create empty setup dir
```python
sdh.flush_n_make_setup_dir(
# optional
setup_directory = "./example_setup_dir"
)
```
#### Copy module to setup dir
```python
sdh.copy_module_to_setup_dir(
# optional
module_filepath = "./combined_example_module.py",
setup_directory = "./example_setup_dir"
)
```
#### Copy license to setup dir
```python
sdh.copy_module_to_setup_dir(
# optional
license_path = "../LICENSE",
setup_directory = "./example_setup_dir"
)
```
#### Create init file
```python
sdh.create_init_file(
# optional
module_name = "example_module",
setup_directory = "./example_setup_dir"
)
```
#### Create setup file
```python
sdh.write_setup_file(
# optional
module_name = "example_module",
metadata = {'author': 'Kyrylo Mordan',
'version': '0.0.1',
'description': 'Example Module',
'keywords': ['python']},
requirements = ['attrs>=22.2.0'],
classifiers = ['Development Status :: 3 - Alpha',
'Intended Audience :: Developers',
'Intended Audience :: Science/Research',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.9',
'Programming Language :: Python :: 3.10',
'Programming Language :: Python :: 3.11',
'License :: OSI Approved :: MIT License',
'Topic :: Scientific/Engineering'],
setup_directory = "./example_setup_dir"
)
```
### 8. Creating release notes from commit messages
Package versioning could be enhanced with release notes. Since the tool is mainly meant for ci/cd, it takes advantage of commit messages to construct a release note for every version.
Commit history is analysed from the last merge, if nothiong found then the next and the next, until at least one of `[<package name>]` labels are found within commit messages. They are bundled together to for a note, where each commit message or messages deliminated with `;` are turned in a list element. Previos notes are used to establish which part of commit history to use as a starting point.
Commit messages could also be used to increment version by something other then a default patch.
- `[<package name>][..+]` increments patch (default behavior)
- `[<package name>][.+.]` increments minor
- `[<package name>][+..]` increments major
- `[<package name>][0.1.2]` forces specific version `0.1.2`
\* First release within new packaging repo may struggle to extract release note since commit messages are only analysed from merges in the commit history.
```python
rnh = ReleaseNotesHandler(
# path to existing or new release notes file
filepath = '../tests/package_auto_assembler/other/release_notes.md',
# name of label in commit message [example_module] for filter
label_name = 'example_module',
# new version to be used in release notes
version = '0.0.1'
)
```
No relevant commit messages found!
..trying depth 2 !
No relevant commit messages found!
No messages to clean were provided
##### - overwritting commit messages from example
```python
# commit messages from last merge
rnh.commit_messages
```
['fixing paa tests',
'fixing paa tests',
'fixing paa tests',
'[package_auto_assembler] increasing default max search depth for commit history to 5',
'fixing mocker-db release notes',
'Update package version tracking files',
'Update README',
'Update requirements']
```python
example_commit_messages = [
'[example_module] usage example for initial release notes; bugfixes for RNH',
'[BUGFIX] missing parameterframe usage example and reduntant png file',
'[example_module][0.1.2] initial release notes handler',
'Update README',
'Update requirements'
]
rnh.commit_messages = example_commit_messages
```
##### - internal methods that run on intialiazation of ReleaseNotesHandler
```python
# get messages relevant only for label
rnh._filter_commit_messages_by_package()
print("Example filtered_messaged:")
print(rnh.filtered_messages)
# clean messages
rnh._clean_and_split_commit_messages()
print("Example processed_messages:")
print(rnh.processed_messages)
```
Example filtered_messaged:
['[example_module] usage example for initial release notes; bugfixes for RNH', '[example_module][0.1.2] initial release notes handler']
Example processed_messages:
['usage example for initial release notes', 'bugfixes for RNH', 'initial release notes handler']
##### - get version update from relevant messages
```python
version_update = rnh.extract_latest_version()
print(f"Example version_update: {version_update}")
```
Example version_update: 0.1.2
##### - get latest version from relevant release notes
```python
latest_version = rnh.extract_latest_version()
print(f"Example latest_version: {latest_version}")
```
Example latest_version: 0.1.2
##### - augment existing release note with new entries or create new
```python
# augment existing release note with new entries or create new
rnh.create_release_note_entry(
# optional
existing_contents=rnh.existing_contents,
version=rnh.version,
new_messages=rnh.processed_messages
)
print("Example processed_note_entries:")
print(rnh.processed_note_entries)
```
Example processed_note_entries:
['# Release notes\n', '\n', '### 0.1.2\n', '\n', ' - usage example for initial release notes\n', '\n', ' - bugfixes for RNH\n', '\n', ' - initial release notes handler\n', '\n', '### 0.0.1\n', '\n', ' - initial version of example_module\n']
##### - saving updated relese notes
```python
rnh.existing_contents
```
['# Release notes\n',
'\n',
'### 0.1.2\n',
'\n',
' - usage example for initial release notes\n',
' - bugfixes for RNH\n',
' - initial release notes handler\n',
'### 0.1.2\n',
'\n',
' - usage example for initial release notes\n',
'\n',
' - bugfixes for RNH\n',
'\n',
' - initial release notes handler\n',
'\n',
'### 0.0.1\n',
'\n',
' - initial version of example_module\n']
```python
rnh.save_release_notes()
```
```python
# updated content
rnh.get_release_notes_content()
```
['# Release notes\n',
'\n',
'### 0.1.2\n',
'\n',
' - usage example for initial release notes\n',
'\n',
' - bugfixes for RNH\n',
'\n',
' - initial release notes handler\n',
'\n',
'### 0.0.1\n',
'\n',
' - initial version of example_module\n']
### 9. Analysing package dependencies
Extracting info from installed dependencies can provide important insight into inner workings of a package and help avoid some of the licenses.
Licenses are extracted from package metadata and normalized for analysis. Missing labels are marked with `-` and not recognized licenses with `unknown`.
Information about unrecognized license labels could be provided through package_licenses json file that contains install package name and corresponding license label.
```python
da = DependenciesAnalyser(
# optional
package_name = 'mocker-db',
package_licenses_filepath = '../tests/package_auto_assembler/other/package_licenses.json',
allowed_licenses = ['mit', 'apache-2.0', 'lgpl-3.0', 'bsd-3-clause', 'bsd-2-clause', '-', 'mpl-2.0']
)
```
#### Finding installed packages with a list of tags
```python
da.filter_packages_by_tags(tags=['aa-paa-tool'])
```
[('comparisonframe', '0.0.0'),
('mocker-db', '0.0.1'),
('package-auto-assembler', '0.0.0'),
('proompter', '0.0.0')]
#### Extracting some metadata from the installed package
```python
package_metadata = da.get_package_metadata(
package_name = 'mocker-db'
)
package_metadata
```
{'keywords': ['aa-paa-tool'],
'version': '0.0.1',
'author': 'Kyrylo Mordan',
'author_email': 'parachute.repo@gmail.com',
'classifiers': ['Development Status :: 3 - Alpha',
'Intended Audience :: Developers',
'Intended Audience :: Science/Research',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.9',
'Programming Language :: Python :: 3.10',
'Programming Language :: Python :: 3.11',
'License :: OSI Approved :: MIT License',
'Topic :: Scientific/Engineering',
'PAA-Version :: 0.4.3',
'PAA-CLI :: False'],
'paa_version': '0.4.3',
'paa_cli': 'False',
'license_label': 'MIT'}
#### Extracting package requirements
```python
package_requirements = da.get_package_requirements(
package_name = 'mocker-db'
)
package_requirements
```
['requests',
'attrs >=22.2.0',
'httpx',
'hnswlib ==0.8.0',
'gridlooper ==0.0.1',
'dill ==0.3.7',
'numpy ==1.26.0',
"sentence-transformers ; extra == 'sentence_transformers'"]
#### Extracting tree of dependencies
```python
extracted_dependencies_tree = da.extract_dependencies_tree(
package_name = 'mocker-db'
)
extracted_dependencies_tree
```
{'requests': {'charset-normalizer': [],
'idna': [],
'urllib3': [],
'certifi': []},
'attrs': {'importlib-metadata': {'zipp': [], 'typing-extensions': []}},
'httpx': {'anyio': {'idna': [],
'sniffio': [],
'exceptiongroup': [],
'typing-extensions': []},
'certifi': [],
'httpcore': {'certifi': [], 'h11': {'typing-extensions': []}},
'idna': [],
'sniffio': []},
'hnswlib': {'numpy': []},
'gridlooper': {'dill': [],
'attrs': {'importlib-metadata': {'zipp': [], 'typing-extensions': []}},
'tqdm': {'colorama': []}},
'dill': [],
'numpy': []}
#### Addding license labels to tree of dependencies
```python
extracted_dependencies_tree_license = da.add_license_labels_to_dep_tree(
dependencies_tree = extracted_dependencies_tree
)
extracted_dependencies_tree_license
```
{'requests': 'apache-2.0',
'requests.charset-normalizer': '-',
'requests.idna': '-',
'requests.urllib3': '-',
'requests.certifi': 'mpl-2.0',
'attrs': '-',
'attrs.importlib-metadata': '-',
'attrs.importlib-metadata.zipp': '-',
'attrs.importlib-metadata.typing-extensions': '-',
'httpx': '-',
'httpx.anyio': 'mit',
'httpx.anyio.idna': '-',
'httpx.anyio.sniffio': '-',
'httpx.anyio.exceptiongroup': '-',
'httpx.anyio.typing-extensions': '-',
'httpx.certifi': 'mpl-2.0',
'httpx.httpcore': '-',
'httpx.httpcore.certifi': 'mpl-2.0',
'httpx.httpcore.h11': 'mit',
'httpx.httpcore.h11.typing-extensions': '-',
'httpx.idna': '-',
'httpx.sniffio': '-',
'hnswlib': '-',
'hnswlib.numpy': 'bsd-3-clause',
'gridlooper': '-',
'gridlooper.dill': 'bsd-3-clause',
'gridlooper.attrs': '-',
'gridlooper.attrs.importlib-metadata': '-',
'gridlooper.attrs.importlib-metadata.zipp': '-',
'gridlooper.attrs.importlib-metadata.typing-extensions': '-',
'gridlooper.tqdm': '-',
'gridlooper.tqdm.colorama': '-',
'dill': 'bsd-3-clause',
'numpy': 'bsd-3-clause'}
#### Printing extracted tree of dependencies
```python
da.print_flattened_tree(extracted_dependencies_tree_license)
```
└── requests : apache-2.0
├── charset-normalizer : -
├── idna : -
├── urllib3 : -
└── certifi : mpl-2.0
└── attrs : -
└── importlib-metadata : -
├── zipp : -
└── typing-extensions : -
└── httpx : -
├── anyio : mit
├── idna : -
├── sniffio : -
├── exceptiongroup : -
└── typing-extensions : -
├── certifi : mpl-2.0
├── httpcore : -
├── certifi : mpl-2.0
└── h11 : mit
└── typing-extensions : -
├── idna : -
└── sniffio : -
└── hnswlib : -
└── numpy : bsd-3-clause
└── gridlooper : -
├── dill : bsd-3-clause
├── attrs : -
└── importlib-metadata : -
├── zipp : -
└── typing-extensions : -
└── tqdm : -
└── colorama : -
└── dill : bsd-3-clause
└── numpy : bsd-3-clause
#### Filtering for unexpected licenses in tree of dependencies
```python
allowed_licenses = ['mit', 'apache-2.0', 'lgpl-3.0', 'mpl-2.0', '-']
da.find_unexpected_licenses_in_deps_tree(
tree_dep_license = extracted_dependencies_tree_license,
# optional
allowed_licenses = allowed_licenses,
raise_error = True
)
```
{'hnswlib': '', 'gridlooper': ''}
└── dill : bsd-3-clause
└── numpy : bsd-3-clause
└── hnswlib :
└── numpy : bsd-3-clause
└── gridlooper :
└── dill : bsd-3-clause
---------------------------------------------------------------------------
Exception Traceback (most recent call last)
Cell In[9], line 3
1 allowed_licenses = ['mit', 'apache-2.0', 'lgpl-3.0', 'mpl-2.0', '-']
----> 3 da.find_unexpected_licenses_in_deps_tree(
4 tree_dep_license = extracted_dependencies_tree_license,
5 # optional
6 allowed_licenses = allowed_licenses,
7 raise_error = True
8 )
File ~/miniforge3/envs/testenv/lib/python3.10/site-packages/package_auto_assembler/package_auto_assembler.py:2670, in DependenciesAnalyser.find_unexpected_licenses_in_deps_tree(self, tree_dep_license, allowed_licenses, raise_error)
2668 if raise_error and out != {}:
2669 self.print_flattened_tree(flattened_dict = out)
-> 2670 raise Exception("Found unexpected licenses!")
2671 else:
2672 self.logger.info("No unexpected licenses found")
Exception: Found unexpected licenses!
### 10. Adding cli interfaces
The tool allows to make a package with optional cli interfaces. These could be sometimes preferable when a package contains a standalone tool that would be called from script anyway.
All of the cli logic would need to be included within a `.py` file which should be stored within `cli_dir` provided in `.paa.config`.
Dependencies from these files are extracted in the similar manner to the main module.
Tools from main `.py` file could still be imported like the following:
```python
from package_name.package_name import ToBeImported
```
The code is wired in `setup.py` via the following automatically assuming that appropriate file with the same name as the package exists within `cli_dir` location.
```python
...,
entry_points = {'console_scripts': [
'<package_alias> = package_name.cli:cli']} ,
...
```
Alias for name could be provided via the following piece of code, defined after imports, otherwise package name would be used.
```python
__cli_metadata__ = {
"name" : <package_alias>
}
```
Package-auto-assembler tool itself uses [`click`](https://pypi.org/project/click/) dependency to build that file, use its [cli definition](https://github.com/Kiril-Mordan/reusables/blob/main/cli/package_auto_assembler.py) as example.
### 11. Adding routes and running FastAPI application
The tool allows to make a package with optional routes for FastAPI application and run them. Each packages can have one routes file where its logic should be defined. Package-auto-assembler itself can combine multiple routes from packages and filepaths into one application.
A `.py` file with the same name of the package should be stored within `api_routes_dir` provided in `.paa.config`.
Dependencies from these files are extracted in the similar manner to the main module.
Tools from main `.py` file could still be imported like the following:
```python
from package_name.package_name import ToBeImported
```
Api description, middleware and run parameters could be provided via optional `yml` file, which for example would look like:
```
DESCRIPTION : {
'version' : 0.0.0
}
MIDDLEWARE : {
origin : ['*']
}
RUN : {
host : 0.0.0.0
}
```
where DESCRIPTION contains parameters for `FastAPI`, MIDDLEWARE for `CORSMiddleware` and RUN for `uvicorn.run`
### 12. Making a package
Main wrapper for the package integrates described above components into a class that could be used to build package building pipelines within python scripts.
To simplify usage [cli interface](https://kiril-mordan.github.io/reusables/package_auto_assembler/cli/) is recomended instead.
#### Initializing PackageAutoAssembler
```python
paa = PackageAutoAssembler(
# required
module_name = "example_module",
module_filepath = "../tests/package_auto_assembler/other/example_module.py",
# optional
mapping_filepath = "../env_spec/package_mapping.json",
licenses_filepath = "../tests/package_auto_assembler/other/package_licenses.json",
allowed_licenses = ['mit', 'apache-2.0', 'lgpl-3.0', 'mpl-2.0', '-'],
dependencies_dir = "../tests/package_auto_assembler/dependancies/",
example_notebook_path = "./mock_vector_database.ipynb",
versions_filepath = '../tests/package_auto_assembler/other/lsts_package_versions.yml',
log_filepath = '../tests/package_auto_assembler/other/version_logs.csv',
setup_directory = "./example_module",
release_notes_filepath = "../tests/package_auto_assembler/other/release_notes.md",
license_path = "../LICENSE",
license_label = "mit",
classifiers = ['Development Status :: 3 - Alpha',
'Intended Audience :: Developers',
'Intended Audience :: Science/Research',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.9',
'Programming Language :: Python :: 3.10',
'Programming Language :: Python :: 3.11',
'License :: OSI Approved :: MIT License',
'Topic :: Scientific/Engineering'],
requirements_list = [],
execute_readme_notebook = True,
python_version = "3.8",
version_increment_type = "patch",
default_version = "0.0.1",
check_vulnerabilities = True,
check_dependencies_licenses = False,
add_requirements_header = True
)
```
#### Add metadata from module
```python
paa.add_metadata_from_module(
# optional
module_filepath = "../tests/package_auto_assembler/other/example_module.py"
)
```
Adding metadata ...
#### Add or update version
```python
paa.add_or_update_version(
# overwrites auto mode (not suggested)
version_increment_type = "patch",
version = "1.2.6",
# optional
module_name = "example_module",
versions_filepath = '../tests/package_auto_assembler/lsts_package_versions.yml',
log_filepath = '../tests/package_auto_assembler/version_logs.csv'
)
```
Incrementing version ...
No relevant commit messages found!
..trying depth 2 !
No relevant commit messages found!
..trying depth 3 !
No relevant commit messages found!
..trying depth 4 !
No relevant commit messages found!
..trying depth 5 !
No relevant commit messages found!
No messages to clean were provided
#### Add release notes from commit messages
```python
paa.add_or_update_release_notes(
# optional
filepath="../tests/package_auto_assembler/release_notes.md",
version=paa.metadata['version']
)
```
Updating release notes ...
#### Prepare setup directory
```python
paa.prep_setup_dir()
```
Preparing setup directory ...
#### Merge local dependacies
```python
paa.merge_local_dependacies(
# optional
main_module_filepath = "../tests/package_auto_assembler/other/example_module.py",
dependencies_dir= "../tests/package_auto_assembler/dependancies/",
save_filepath = "./example_module/example_module.py"
)
```
Merging ../tests/package_auto_assembler/other/example_module.py with dependecies from ../tests/package_auto_assembler/dependancies/ into ./example_module/example_module.py
#### Add requirements from module
```python
paa.add_requirements_from_module(
# optional
module_filepath = "../tests/package_auto_assembler/other/example_module.py",
import_mappings = {'PIL': 'Pillow',
'bs4': 'beautifulsoup4',
'fitz': 'PyMuPDF',
'attr': 'attrs',
'dotenv': 'python-dotenv',
'googleapiclient': 'google-api-python-client',
'sentence_transformers': 'sentence-transformers',
'flask': 'Flask',
'stdlib_list': 'stdlib-list',
'sklearn': 'scikit-learn',
'yaml': 'pyyaml',
'git' : 'gitpython'}
)
```
Adding requirements from ../tests/package_auto_assembler/other/example_module.py
No known vulnerabilities found
```python
paa.requirements_list
```
['### example_module.py', 'attrs>=22.2.0']
#### Make README out of example notebook
```python
paa.add_readme(
# optional
example_notebook_path = "../tests/package_auto_assembler/other/example_module.ipynb",
output_path = "./example_module/README.md",
execute_notebook=False,
)
```
Adding README from ../tests/package_auto_assembler/other/example_module.ipynb to ./example_module/README.md
#### Prepare setup file
```python
paa.prep_setup_file(
# optional
metadata = {'author': 'Kyrylo Mordan',
'version': '0.0.1',
'description': 'Example module',
'keywords': ['python'],
'license' : 'mit'},
requirements = ['### example_module.py',
'attrs>=22.2.0'],
classifiers = ['Development Status :: 3 - Alpha',
'Intended Audience :: Developers',
'Intended Audience :: Science/Research',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.9',
'Programming Language :: Python :: 3.10',
'Programming Language :: Python :: 3.11',
'License :: OSI Approved :: MIT License',
'Topic :: Scientific/Engineering'],
cli_module_filepath = "../tests/package_auto_assembler/other/cli.py"
)
```
Preparing setup file for example-module package ...
#### Make package
```python
paa.make_package(
# optional
setup_directory = "./example_module"
)
```
Making package from ./example_module ...
CompletedProcess(args=['python', './example_module/setup.py', 'sdist', 'bdist_wheel'], returncode=0, stdout="running sdist\nrunning egg_info\nwriting example_module.egg-info/PKG-INFO\nwriting dependency_links to example_module.egg-info/dependency_links.txt\nwriting entry points to example_module.egg-info/entry_points.txt\nwriting requirements to example_module.egg-info/requires.txt\nwriting top-level names to example_module.egg-info/top_level.txt\nreading manifest file 'example_module.egg-info/SOURCES.txt'\nwriting manifest file 'example_module.egg-info/SOURCES.txt'\nrunning check\ncreating example_module-0.0.0\ncreating example_module-0.0.0/example_module\ncreating example_module-0.0.0/example_module.egg-info\ncopying files to example_module-0.0.0...\ncopying example_module/__init__.py -> example_module-0.0.0/example_module\ncopying example_module/cli.py -> example_module-0.0.0/example_module\ncopying example_module/example_module.py -> example_module-0.0.0/example_module\ncopying example_module/setup.py -> example_module-0.0.0/example_module\ncopying example_module.egg-info/PKG-INFO -> example_module-0.0.0/example_module.egg-info\ncopying example_module.egg-info/SOURCES.txt -> example_module-0.0.0/example_module.egg-info\ncopying example_module.egg-info/dependency_links.txt -> example_module-0.0.0/example_module.egg-info\ncopying example_module.egg-info/entry_points.txt -> example_module-0.0.0/example_module.egg-info\ncopying example_module.egg-info/requires.txt -> example_module-0.0.0/example_module.egg-info\ncopying example_module.egg-info/top_level.txt -> example_module-0.0.0/example_module.egg-info\ncopying example_module.egg-info/SOURCES.txt -> example_module-0.0.0/example_module.egg-info\nWriting example_module-0.0.0/setup.cfg\nCreating tar archive\nremoving 'example_module-0.0.0' (and everything under it)\nrunning bdist_wheel\nrunning build\nrunning build_py\ncopying example_module/example_module.py -> build/lib/example_module\ncopying example_module/__init__.py -> build/lib/example_module\ncopying example_module/setup.py -> build/lib/example_module\ncopying example_module/cli.py -> build/lib/example_module\ninstalling to build/bdist.linux-x86_64/wheel\nrunning install\nrunning install_lib\ncreating build/bdist.linux-x86_64/wheel\ncreating build/bdist.linux-x86_64/wheel/example_module\ncopying build/lib/example_module/example_module.py -> build/bdist.linux-x86_64/wheel/example_module\ncopying build/lib/example_module/__init__.py -> build/bdist.linux-x86_64/wheel/example_module\ncopying build/lib/example_module/setup.py -> build/bdist.linux-x86_64/wheel/example_module\ncopying build/lib/example_module/cli.py -> build/bdist.linux-x86_64/wheel/example_module\nrunning install_egg_info\nCopying example_module.egg-info to build/bdist.linux-x86_64/wheel/example_module-0.0.0-py3.10.egg-info\nrunning install_scripts\ncreating build/bdist.linux-x86_64/wheel/example_module-0.0.0.dist-info/WHEEL\ncreating 'dist/example_module-0.0.0-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it\nadding 'example_module/__init__.py'\nadding 'example_module/cli.py'\nadding 'example_module/example_module.py'\nadding 'example_module/setup.py'\nadding 'example_module-0.0.0.dist-info/METADATA'\nadding 'example_module-0.0.0.dist-info/WHEEL'\nadding 'example_module-0.0.0.dist-info/entry_points.txt'\nadding 'example_module-0.0.0.dist-info/top_level.txt'\nadding 'example_module-0.0.0.dist-info/RECORD'\nremoving build/bdist.linux-x86_64/wheel\n", stderr='warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md\n\n/home/kyriosskia/miniconda3/envs/testenv/lib/python3.10/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.\n!!\n\n ********************************************************************************\n Please avoid running ``setup.py`` directly.\n Instead, use pypa/build, pypa/installer or other\n standards-based tools.\n\n See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.\n ********************************************************************************\n\n!!\n self.initialize_options()\n')
### 13. Making simple MkDocs site
Package documentation can be presented in a form of mkdocs static site, which could be either served or deployed to something like github packages.
Main module docstring is used as intro package that contains something like optional pypi and license badges. Package description and realease notes are turned into separate tabs. Png with diagrams for example could be provided and displayed as their own separate tabs as well.
The one for this package can be seen [here](https://kiril-mordan.github.io/reusables/package_auto_assembler/)
It can be packaged with the package and be displayed in webrowser like documentation for api via `{package_name}\docs` when using included api handling capabilities.
##### - preparing inputs
```python
package_name = "example_module"
module_content = LongDocHandler().read_module_content(filepath=f"../tests/package_auto_assembler/{package_name}.py")
docstring = LongDocHandler().extract_module_docstring(module_content=module_content)
pypi_link = LongDocHandler().get_pypi_badge(module_name=package_name)
docs_file_paths = {
"../example_module.md" : "usage-examples.md",
'../tests/package_auto_assembler/release_notes.md' : 'release_notes.md'
}
```
```python
mdh = MkDocsHandler(
# required
## name of the package to be displayed
package_name = package_name,
## dictionary of markdown files, with path as keys
docs_file_paths = docs_file_paths,
# optional
## module docstring to be displayed in the index
module_docstring = docstring,
## pypi badge to be displayed in the index
pypi_badge = pypi_link,
## license badge to be displayed in the index
license_badge="[](https://github.com/Kiril-Mordan/reusables/blob/main/LICENSE)",
## name of the project directory
project_name = "temp_project")
```
##### - preparing site
```python
mdh.create_mkdocs_dir()
mdh.move_files_to_docs()
mdh.generate_markdown_for_images()
mdh.create_index()
mdh.create_mkdocs_yml()
mdh.build_mkdocs_site()
```
Created new MkDocs dir: temp_project
Copied ../example_module.md to temp_project/docs/usage-examples.md
Copied ../tests/package_auto_assembler/release_notes.md to temp_project/docs/release_notes.md
index.md has been created with site_name: example-module
mkdocs.yml has been created with site_name: Example module
Custom CSS created at temp_project/docs/css/extra.css
INFO - Cleaning site directory
INFO - Building documentation to directory: /home/kyriosskia/Documents/nlp/reusables/example_notebooks/temp_project/site
INFO - Documentation built in 0.12 seconds
##### - test runing site
```python
mdh.serve_mkdocs_site()
```
Raw data
{
"_id": null,
"home_page": null,
"name": "package-auto-assembler",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "['python', 'packaging', 'aa-paa-tool']",
"author": "Kyrylo Mordan",
"author_email": "parachute.repo@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/a8/02/138e4ced42ec33224d331768acc6f4d82491287b69c0f98d8bd75abbb9aa/package_auto_assembler-0.5.2.tar.gz",
"platform": null,
"description": "# Package auto assembler\n\nPackage auto assembler is a tool that meant to streamline creation of single module packages.\nIts purpose is to automate as many aspects of python package creation as possible,\nto shorten a development cycle of reusable components, maintain certain standard of quality\nfor reusable code. It provides tool to simplify the process of package creatrion\nto a point that it can be triggered automatically within ci/cd pipelines,\nwith minimal preparations and requirements for new modules.\n\n```python\nimport sys\nsys.path.append('../')\nfrom package_auto_assembler import (VersionHandler, \\\n ImportMappingHandler, RequirementsHandler, MetadataHandler, \\\n LocalDependaciesHandler, LongDocHandler, SetupDirHandler, \\\n ReleaseNotesHandler, MkDocsHandler, PackageAutoAssembler, \\\n DependenciesAnalyser)\n```\n\n### 1. Package versioning\n\nPackage versioning within paa is done based on semantic versioning.\n\n`major.minor.patch`\n\nBy default, patch is updated, but the minor and major could also be update based on, for example, commit messages or manually from the log file. \n\nPackage auto assembler does try to pull latest version from package storage, but in case of failure uses version logs.\n\n#### Initialize VersionHandler\n\n\n```python\npv = VersionHandler(\n # required\n versions_filepath = '../tests/package_auto_assembler/other/lsts_package_versions.yml',\n log_filepath = '../tests/package_auto_assembler/other/version_logs.csv',\n # optional\n default_version = \"0.0.1\")\n```\n\n#### Add new package\n\n\n```python\npv.add_package(\n package_name = \"new_package\",\n # optional\n version = \"0.0.1\"\n)\n```\n\n#### Update package version\n\n\n```python\npv.increment_patch(\n package_name = \"new_package\"\n)\n## for not tracked package\npv.increment_patch(\n package_name = \"another_new_package\",\n # optional\n default_version = \"0.0.1\"\n)\n```\n\n There are no known versions of 'another_new_package', 0.0.1 will be used!\n\n\n#### Display current versions and logs\n\n\n```python\npv.get_versions(\n # optional\n versions_filepath = '../tests/package_auto_assembler/other/lsts_package_versions.yml'\n)\n```\n\n\n\n\n {'another_new_package': '0.0.1', 'new_package': '0.0.2'}\n\n\n\n\n```python\npv.get_version(\n package_name='new_package'\n)\n```\n\n\n\n\n '0.0.2'\n\n\n\n\n```python\npv.get_logs(\n # optional\n log_filepath = '../tests/package_auto_assembler/other/version_logs.csv'\n)\n```\n\n\n\n\n<div>\n<style scoped>\n .dataframe tbody tr th:only-of-type {\n vertical-align: middle;\n }\n\n .dataframe tbody tr th {\n vertical-align: top;\n }\n\n .dataframe thead th {\n text-align: right;\n }\n</style>\n<table border=\"1\" class=\"dataframe\">\n <thead>\n <tr style=\"text-align: right;\">\n <th></th>\n <th>Timestamp</th>\n <th>Package</th>\n <th>Version</th>\n </tr>\n </thead>\n <tbody>\n <tr>\n <th>0</th>\n <td>2024-07-29 03:26:39</td>\n <td>new_package</td>\n <td>0.0.1</td>\n </tr>\n <tr>\n <th>1</th>\n <td>2024-07-29 03:26:40</td>\n <td>new_package</td>\n <td>0.0.2</td>\n </tr>\n <tr>\n <th>2</th>\n <td>2024-07-29 03:26:40</td>\n <td>another_new_package</td>\n <td>0.0.1</td>\n </tr>\n </tbody>\n</table>\n</div>\n\n\n\n#### Flush versions and logs\n\n\n```python\npv.flush_versions()\npv.flush_logs()\n```\n\n#### Get latest available version with pip\n\n\n```python\npv.get_latest_pip_version(package_name = 'package-auto-assembler')\n```\n\n\n\n\n '0.3.1'\n\n\n\n### 2. Import mapping\n\nInstall and import names of dependencies may vary. The mapping files maps import names to install names so that requirements extraction from `.py` files is possible.\n\n#### Initialize ImportMappingHandler\n\n\n```python\nim = ImportMappingHandler(\n # required\n mapping_filepath = \"../env_spec/package_mapping.json\"\n)\n```\n\n#### Load package mappings\n\n\n```python\nim.load_package_mappings(\n # optional\n mapping_filepath = \"../env_spec/package_mapping.json\"\n)\n```\n\n\n\n\n {'PIL': 'Pillow',\n 'bs4': 'beautifulsoup4',\n 'fitz': 'PyMuPDF',\n 'attr': 'attrs',\n 'dotenv': 'python-dotenv',\n 'googleapiclient': 'google-api-python-client',\n 'google_auth_oauthlib': 'google-auth-oauthlib',\n 'sentence_transformers': 'sentence-transformers',\n 'flask': 'Flask',\n 'stdlib_list': 'stdlib-list',\n 'sklearn': 'scikit-learn',\n 'yaml': 'pyyaml',\n 'package_auto_assembler': 'package-auto-assembler',\n 'git': 'gitpython'}\n\n\n\n### 3. Extracting and merging requirements\n\nMaintaining requirements is much simpler, when done automatically based on the `.py` files. \n\nThe actual requirements files is still constructed. Standard libraries are not added, others are added with their versions, if specified. Local files are also used as dependencies, from which imports are extracted as well. \n\nFor example:\n\n```python\nimport os\nimport pandas\nimport attr #>=22.2.0\nfrom .components.local_dep import *\n```\n\nproduces \n\n``` txt\npandas\nattrs >=22.2.0\nyaml\n```\n\nas requirements file, where `yaml` is extracted from `local_dep.py` file.\n\nChecking dependecies for vulnerabilities is usefull and it is done with `pip audit` which is integrated into the paa package and is used by default.\n\nOptional requirements for `extras_require` could be probided the same way normal requirements are, but each like that contains an import like that should be commented out in a special way, starting with `#!`, for example:\n\n```python\nimport os\nimport pandas\nimport attr #>=22.2.0\n#! import hnswlib #==0.8.0\n```\n\nproduces\n\n``` txt\npandas\nattrs >=22.2.0\nhnswlib==0.8.0; extra == \"hnswlib\"\n```\n\n\n#### Initialize RequirementsHandler\n\n\n```python\nrh = RequirementsHandler(\n # optional/required later\n module_filepath = \"../tests/package_auto_assembler/other/example_module.py\",\n package_mappings = {'PIL': 'Pillow',\n 'bs4': 'beautifulsoup4',\n 'fitz': 'PyMuPDF',\n 'attr': 'attrs',\n 'dotenv': 'python-dotenv',\n 'googleapiclient': 'google-api-python-client',\n 'sentence_transformers': 'sentence-transformers',\n 'flask': 'Flask',\n 'stdlib_list': 'stdlib-list',\n 'sklearn': 'scikit-learn',\n 'yaml': 'pyyaml'},\n requirements_output_path = \"../tests/package_auto_assembler/other/\",\n output_requirements_prefix = \"requirements_\",\n custom_modules_filepath = \"../tests/package_auto_assembler/dependancies\",\n python_version = '3.8',\n add_header = True\n)\n```\n\n#### List custom modules for a given directory\n\n\n```python\nrh.list_custom_modules(\n # optional\n custom_modules_filepath=\"../tests/package_auto_assembler/dependancies\"\n)\n```\n\n\n\n\n ['example_local_dependacy_1', 'example_local_dependacy_2']\n\n\n\n#### Check if module is a standard python library\n\n\n```python\nrh.is_standard_library(\n # required\n module_name = 'example_local_dependacy_1',\n # optional\n python_version = '3.8'\n)\n```\n\n\n\n\n False\n\n\n\n\n```python\nrh.is_standard_library(\n # required\n module_name = 'logging',\n # optional\n python_version = '3.8'\n)\n```\n\n\n\n\n True\n\n\n\n#### Extract requirements from the module file\n\n\n```python\nrh.extract_requirements(\n # optional\n module_filepath = \"../tests/package_auto_assembler/other/example_module.py\",\n custom_modules = ['example_local_dependacy_2', 'example_local_dependacy_1'],\n package_mappings = {'PIL': 'Pillow',\n 'bs4': 'beautifulsoup4',\n 'fitz': 'PyMuPDF',\n 'attr': 'attrs',\n 'dotenv': 'python-dotenv',\n 'googleapiclient': 'google-api-python-client',\n 'sentence_transformers': 'sentence-transformers',\n 'flask': 'Flask',\n 'stdlib_list': 'stdlib-list',\n 'sklearn': 'scikit-learn',\n 'yaml': 'pyyaml'},\n python_version = '3.8',\n add_header=True\n)\n```\n\n\n\n\n (['attrs>=22.2.0'],\n ['torch<=2.4.1', 'fastapi[all]', 'scikit-learn==1.5.1', 'numpy'])\n\n\n\n\n```python\nrh.requirements_list\n```\n\n\n\n\n ['attrs>=22.2.0']\n\n\n\n\n```python\nrh.optional_requirements_list\n```\n\n\n\n\n ['torch<=2.4.1', 'fastapi[all]', 'scikit-learn==1.5.1', 'numpy']\n\n\n\n#### Audit dependencies\n\n\n```python\nrh.check_vulnerabilities(\n # optional if ran extract_requirements() before\n requirements_list = None,\n raise_error = True\n)\n```\n\n No known vulnerabilities found\n \n\n\n \n\n\n\n```python\nrh.vulnerabilities\n```\n\n\n\n\n []\n\n\n\n\n```python\ntry:\n rh.check_vulnerabilities(\n # optional if ran extract_requirements() before\n requirements_list = ['attrs>=22.2.0', 'pandas', 'hnswlib==0.7.0'],\n raise_error = True\n )\nexcept Exception as e:\n print(f\"Error: {e}\")\n```\n\n Found 1 known vulnerability in 1 package\n \n\n\n Name Version ID Fix Versions\n ------- ------- ------------------- ------------\n hnswlib 0.7.0 GHSA-xwc8-rf6m-xr86\n \n Error: Found vulnerabilities, resolve them or ignore check to move forwards!\n\n\n\n```python\nrh.vulnerabilities\n```\n\n\n\n\n [{'name': 'hnswlib',\n 'version': '0.7.0',\n 'id': 'GHSA-xwc8-rf6m-xr86',\n 'fix_versions': None}]\n\n\n\n#### Save requirements to a file\n\n\n```python\nrh.write_requirements_file(\n # optional/required later\n module_name = 'example_module',\n requirements = ['### example_module.py', 'attrs>=22.2.0'],\n output_path = \"../tests/package_auto_assembler/other/\",\n prefix = \"requirements_\"\n)\n```\n\n#### Read requirements\n\n\n```python\nrh.read_requirements_file(\n # required\n requirements_filepath = \"../tests/package_auto_assembler/other/requirements_example_module.txt\"\n)\n```\n\n\n\n\n ['attrs>=22.2.0']\n\n\n\n### 4. Preparing metadata\n\nSince all of the necessary information for building a package needs to be contained within main component `.py` file, basic metadata is provided with the use of `__package_metadata__` dictionary object, defined within that `.py` file. It is also used as a trigger for package building within paa pipeline. \n\nEven though some general information shared between packages could be provided through general config, but package specific info should be provided through `__package_metadata__`. It should support most text fields from setup file, but for others the following fields are available:\n\n- `classifiers`: adds classifiers to the general ones from config\n- `extras_require`: a dictionary of optional package that wouldn't be installed during normal installation. The key could be used during installation and the value would be a list of dependencies.\n- `install_requires` : adds requirements to the list read from imports\n\n\\* Note that providing dependencies this way does not check them through pip-audit or translate them through package mapping\n\n\n#### Initializing MetadataHandler\n\n\n```python\nmh = MetadataHandler(\n # optional/required later\n module_filepath = \"../tests/package_auto_assembler/other/example_module.py\"\n)\n```\n\n#### Check if metadata is available\n\n\n```python\nmh.is_metadata_available(\n # optional\n module_filepath = \"../tests/package_auto_assembler/other/example_module.py\"\n)\n```\n\n\n\n\n True\n\n\n\n#### Extract metadata from module\n\n\n```python\nmh.get_package_metadata(\n # optional\n module_filepath = \"../tests/package_auto_assembler/other/example_module.py\"\n)\n```\n\n\n\n\n {'author': 'Kyrylo Mordan',\n 'author_email': 'parachute.repo@gmail.com',\n 'version': '0.0.1',\n 'description': 'A mock handler for simulating a vector database.',\n 'keywords': ['python', 'vector database', 'similarity search']}\n\n\n\n### 5. Merging local dependacies into single module\n\nPackage auto assembler creates `single module packages`, meaning that once package is built all of the object are imported from a single place. The packaging tool does allow for `local dependecies` which are `.py` files imported from specified dependencies directory and its subfolders. Packaging structure may look like the following:\n\n```\npackaging repo/\n\u2514src/\n \u251c <package names>.py\n \u2514 components\n \u251clocal_dependecy.py\n \u2514subdir_1\n \u2514local_dependency_2.py \n```\n\nDuring packaging process paa merges main module with its local dependies into a single file.\n\n#### Initializing LocalDependaciesHandler\n\n\n```python\nldh = LocalDependaciesHandler(\n # required\n main_module_filepath = \"../tests/package_auto_assembler/other/example_module.py\",\n dependencies_dir = \"../tests/package_auto_assembler/dependancies/\",\n # optional\n save_filepath = \"./combined_example_module.py\"\n)\n```\n\n#### Combine main module with dependacies\n\n\n```python\nprint(ldh.combine_modules(\n # optional\n main_module_filepath = \"../tests/package_auto_assembler/other/example_module.py\",\n dependencies_dir = \"../tests/package_auto_assembler/dependancies/\",\n add_empty_design_choices = False\n)[0:1000])\n```\n\n \"\"\"\n Mock Vector Db Handler\n \n This class is a mock handler for simulating a vector database, designed primarily for testing and development scenarios.\n It offers functionalities such as text embedding, hierarchical navigable small world (HNSW) search,\n and basic data management within a simulated environment resembling a vector database.\n \"\"\"\n \n import logging\n import json\n import time\n import attr #>=22.2.0\n import sklearn\n \n __design_choices__ = {}\n \n @attr.s\n class Shouter:\n \n \"\"\"\n A class for managing and displaying formatted log messages.\n \n This class uses the logging module to create and manage a logger\n for displaying formatted messages. It provides a method to output\n various types of lines and headers, with customizable message and\n line lengths.\n \"\"\"\n \n # Formatting settings\n dotline_length = attr.ib(default=50)\n \n # Logger settings\n logger = attr.ib(default=None)\n logger_name = attr.ib(default='Shouter')\n loggerLvl = attr.ib(default=logging.DEBUG)\n log\n\n\n\n```python\nldh.dependencies_names_list\n```\n\n\n\n\n ['example_local_dependacy_2', 'example_local_dependacy_1', 'dep_from_bundle_1']\n\n\n\n#### Save combined module\n\n\n```python\nldh.save_combined_modules(\n # optional\n combined_module = ldh.combine_modules(),\n save_filepath = \"./combined_example_module.py\"\n)\n```\n\n### 6. Prepare README\n\nPackage description is based on `.ipynb` with same name as the `.py`. By default it is converted to markdown as is, but there is also an option to execute it.\n\n\n```python\nimport logging\nldh = LongDocHandler(\n # optional/required later\n notebook_path = \"../tests/package_auto_assembler/other/example_module.ipynb\",\n markdown_filepath = \"../example_module.md\",\n timeout = 600,\n kernel_name = 'python3',\n # logger\n loggerLvl = logging.DEBUG\n)\n```\n\n#### Convert notebook to md without executing\n\n\n```python\nldh.convert_notebook_to_md(\n # optional\n notebook_path = \"../tests/package_auto_assembler/other/example_module.ipynb\",\n output_path = \"../example_module.md\"\n)\n```\n\n Converted ../tests/package_auto_assembler/example_module.ipynb to ../example_module.md\n\n\n#### Convert notebook to md with executing\n\n\n```python\nldh.convert_and_execute_notebook_to_md(\n # optional\n notebook_path = \"../tests/package_auto_assembler/other/example_module.ipynb\",\n output_path = \"../example_module.md\",\n timeout = 600,\n kernel_name = 'python3'\n)\n```\n\n Converted and executed ../tests/package_auto_assembler/example_module.ipynb to ../example_module.md\n\n\n#### Return long description\n\n\n```python\nlong_description = ldh.return_long_description(\n # optional\n markdown_filepath = \"../example_module.md\"\n)\n```\n\n### 7. Assembling setup directory\n\nPackage are created following rather simple sequence of steps. At some point of the process a temporary directory is created to store the following files:\n\n- `__init__.py` is a simple import from a single module\n- `cli.py` is optional packaged cli tool\n- `<package name>.py` is a single module with all of the local dependecies\n- `README.md` is a package description file based on `.ipynb` file\n- `setup.py` is a setup file for making a package\n\n#### Initializing SetupDirHandler\n\n\n```python\nsdh = SetupDirHandler(\n # required\n module_filepath = \"../tests/package_auto_assembler/other/example_module.py\",\n # optional/ required\n module_name = \"example_module\",\n metadata = {'author': 'Kyrylo Mordan',\n 'version': '0.0.1',\n 'description': 'Example module.',\n 'long_description' : long_description,\n 'keywords': ['python']},\n license_path = \"../LICENSE\",\n requirements = ['attrs>=22.2.0'],\n classifiers = ['Development Status :: 3 - Alpha',\n 'Intended Audience :: Developers',\n 'Intended Audience :: Science/Research',\n 'Programming Language :: Python :: 3',\n 'Programming Language :: Python :: 3.9',\n 'Programming Language :: Python :: 3.10',\n 'Programming Language :: Python :: 3.11',\n 'License :: OSI Approved :: MIT License',\n 'Topic :: Scientific/Engineering'],\n setup_directory = \"./example_setup_dir\"\n)\n```\n\n#### Create empty setup dir\n\n\n```python\nsdh.flush_n_make_setup_dir(\n # optional\n setup_directory = \"./example_setup_dir\"\n)\n```\n\n#### Copy module to setup dir\n\n\n```python\nsdh.copy_module_to_setup_dir(\n # optional\n module_filepath = \"./combined_example_module.py\",\n setup_directory = \"./example_setup_dir\"\n)\n```\n\n#### Copy license to setup dir\n\n\n```python\nsdh.copy_module_to_setup_dir(\n # optional\n license_path = \"../LICENSE\",\n setup_directory = \"./example_setup_dir\"\n)\n```\n\n#### Create init file\n\n\n```python\nsdh.create_init_file(\n # optional\n module_name = \"example_module\",\n setup_directory = \"./example_setup_dir\"\n)\n```\n\n#### Create setup file\n\n\n```python\nsdh.write_setup_file(\n # optional\n module_name = \"example_module\",\n metadata = {'author': 'Kyrylo Mordan',\n 'version': '0.0.1',\n 'description': 'Example Module',\n 'keywords': ['python']},\n requirements = ['attrs>=22.2.0'],\n classifiers = ['Development Status :: 3 - Alpha',\n 'Intended Audience :: Developers',\n 'Intended Audience :: Science/Research',\n 'Programming Language :: Python :: 3',\n 'Programming Language :: Python :: 3.9',\n 'Programming Language :: Python :: 3.10',\n 'Programming Language :: Python :: 3.11',\n 'License :: OSI Approved :: MIT License',\n 'Topic :: Scientific/Engineering'],\n setup_directory = \"./example_setup_dir\"\n)\n```\n\n### 8. Creating release notes from commit messages\n\nPackage versioning could be enhanced with release notes. Since the tool is mainly meant for ci/cd, it takes advantage of commit messages to construct a release note for every version. \n\nCommit history is analysed from the last merge, if nothiong found then the next and the next, until at least one of `[<package name>]` labels are found within commit messages. They are bundled together to for a note, where each commit message or messages deliminated with `;` are turned in a list element. Previos notes are used to establish which part of commit history to use as a starting point.\n\nCommit messages could also be used to increment version by something other then a default patch. \n\n- `[<package name>][..+]` increments patch (default behavior)\n- `[<package name>][.+.]` increments minor\n- `[<package name>][+..]` increments major\n- `[<package name>][0.1.2]` forces specific version `0.1.2`\n\n\\* First release within new packaging repo may struggle to extract release note since commit messages are only analysed from merges in the commit history. \n\n\n\n```python\nrnh = ReleaseNotesHandler(\n # path to existing or new release notes file\n filepath = '../tests/package_auto_assembler/other/release_notes.md',\n # name of label in commit message [example_module] for filter\n label_name = 'example_module',\n # new version to be used in release notes\n version = '0.0.1'\n)\n```\n\n No relevant commit messages found!\n ..trying depth 2 !\n No relevant commit messages found!\n No messages to clean were provided\n\n\n##### - overwritting commit messages from example\n\n\n```python\n# commit messages from last merge\nrnh.commit_messages\n```\n\n\n\n\n ['fixing paa tests',\n 'fixing paa tests',\n 'fixing paa tests',\n '[package_auto_assembler] increasing default max search depth for commit history to 5',\n 'fixing mocker-db release notes',\n 'Update package version tracking files',\n 'Update README',\n 'Update requirements']\n\n\n\n\n```python\nexample_commit_messages = [\n '[example_module] usage example for initial release notes; bugfixes for RNH',\n '[BUGFIX] missing parameterframe usage example and reduntant png file',\n '[example_module][0.1.2] initial release notes handler',\n 'Update README',\n 'Update requirements'\n]\nrnh.commit_messages = example_commit_messages\n```\n\n##### - internal methods that run on intialiazation of ReleaseNotesHandler\n\n\n```python\n# get messages relevant only for label\nrnh._filter_commit_messages_by_package()\nprint(\"Example filtered_messaged:\")\nprint(rnh.filtered_messages)\n\n# clean messages\nrnh._clean_and_split_commit_messages()\nprint(\"Example processed_messages:\")\nprint(rnh.processed_messages)\n```\n\n Example filtered_messaged:\n ['[example_module] usage example for initial release notes; bugfixes for RNH', '[example_module][0.1.2] initial release notes handler']\n Example processed_messages:\n ['usage example for initial release notes', 'bugfixes for RNH', 'initial release notes handler']\n\n\n##### - get version update from relevant messages\n\n\n```python\nversion_update = rnh.extract_latest_version()\nprint(f\"Example version_update: {version_update}\")\n```\n\n Example version_update: 0.1.2\n\n\n##### - get latest version from relevant release notes\n\n\n```python\nlatest_version = rnh.extract_latest_version()\nprint(f\"Example latest_version: {latest_version}\")\n```\n\n Example latest_version: 0.1.2\n\n\n##### - augment existing release note with new entries or create new\n\n\n```python\n# augment existing release note with new entries or create new\nrnh.create_release_note_entry(\n # optional\n existing_contents=rnh.existing_contents,\n version=rnh.version,\n new_messages=rnh.processed_messages\n)\nprint(\"Example processed_note_entries:\")\nprint(rnh.processed_note_entries)\n```\n\n Example processed_note_entries:\n ['# Release notes\\n', '\\n', '### 0.1.2\\n', '\\n', ' - usage example for initial release notes\\n', '\\n', ' - bugfixes for RNH\\n', '\\n', ' - initial release notes handler\\n', '\\n', '### 0.0.1\\n', '\\n', ' - initial version of example_module\\n']\n\n\n##### - saving updated relese notes\n\n\n```python\nrnh.existing_contents\n```\n\n\n\n\n ['# Release notes\\n',\n '\\n',\n '### 0.1.2\\n',\n '\\n',\n ' - usage example for initial release notes\\n',\n ' - bugfixes for RNH\\n',\n ' - initial release notes handler\\n',\n '### 0.1.2\\n',\n '\\n',\n ' - usage example for initial release notes\\n',\n '\\n',\n ' - bugfixes for RNH\\n',\n '\\n',\n ' - initial release notes handler\\n',\n '\\n',\n '### 0.0.1\\n',\n '\\n',\n ' - initial version of example_module\\n']\n\n\n\n\n```python\nrnh.save_release_notes()\n```\n\n\n```python\n# updated content\nrnh.get_release_notes_content()\n```\n\n\n\n\n ['# Release notes\\n',\n '\\n',\n '### 0.1.2\\n',\n '\\n',\n ' - usage example for initial release notes\\n',\n '\\n',\n ' - bugfixes for RNH\\n',\n '\\n',\n ' - initial release notes handler\\n',\n '\\n',\n '### 0.0.1\\n',\n '\\n',\n ' - initial version of example_module\\n']\n\n\n\n### 9. Analysing package dependencies\n\nExtracting info from installed dependencies can provide important insight into inner workings of a package and help avoid some of the licenses. \n\nLicenses are extracted from package metadata and normalized for analysis. Missing labels are marked with `-` and not recognized licenses with `unknown`.\n\nInformation about unrecognized license labels could be provided through package_licenses json file that contains install package name and corresponding license label.\n\n\n```python\nda = DependenciesAnalyser(\n # optional\n package_name = 'mocker-db',\n package_licenses_filepath = '../tests/package_auto_assembler/other/package_licenses.json',\n allowed_licenses = ['mit', 'apache-2.0', 'lgpl-3.0', 'bsd-3-clause', 'bsd-2-clause', '-', 'mpl-2.0']\n)\n```\n\n#### Finding installed packages with a list of tags\n\n\n```python\nda.filter_packages_by_tags(tags=['aa-paa-tool'])\n```\n\n\n\n\n [('comparisonframe', '0.0.0'),\n ('mocker-db', '0.0.1'),\n ('package-auto-assembler', '0.0.0'),\n ('proompter', '0.0.0')]\n\n\n\n#### Extracting some metadata from the installed package\n\n\n```python\npackage_metadata = da.get_package_metadata(\n package_name = 'mocker-db'\n)\npackage_metadata\n```\n\n\n\n\n {'keywords': ['aa-paa-tool'],\n 'version': '0.0.1',\n 'author': 'Kyrylo Mordan',\n 'author_email': 'parachute.repo@gmail.com',\n 'classifiers': ['Development Status :: 3 - Alpha',\n 'Intended Audience :: Developers',\n 'Intended Audience :: Science/Research',\n 'Programming Language :: Python :: 3',\n 'Programming Language :: Python :: 3.9',\n 'Programming Language :: Python :: 3.10',\n 'Programming Language :: Python :: 3.11',\n 'License :: OSI Approved :: MIT License',\n 'Topic :: Scientific/Engineering',\n 'PAA-Version :: 0.4.3',\n 'PAA-CLI :: False'],\n 'paa_version': '0.4.3',\n 'paa_cli': 'False',\n 'license_label': 'MIT'}\n\n\n\n#### Extracting package requirements\n\n\n```python\npackage_requirements = da.get_package_requirements(\n package_name = 'mocker-db'\n)\npackage_requirements\n```\n\n\n\n\n ['requests',\n 'attrs >=22.2.0',\n 'httpx',\n 'hnswlib ==0.8.0',\n 'gridlooper ==0.0.1',\n 'dill ==0.3.7',\n 'numpy ==1.26.0',\n \"sentence-transformers ; extra == 'sentence_transformers'\"]\n\n\n\n#### Extracting tree of dependencies\n\n\n```python\nextracted_dependencies_tree = da.extract_dependencies_tree(\n package_name = 'mocker-db'\n)\nextracted_dependencies_tree\n```\n\n\n\n\n {'requests': {'charset-normalizer': [],\n 'idna': [],\n 'urllib3': [],\n 'certifi': []},\n 'attrs': {'importlib-metadata': {'zipp': [], 'typing-extensions': []}},\n 'httpx': {'anyio': {'idna': [],\n 'sniffio': [],\n 'exceptiongroup': [],\n 'typing-extensions': []},\n 'certifi': [],\n 'httpcore': {'certifi': [], 'h11': {'typing-extensions': []}},\n 'idna': [],\n 'sniffio': []},\n 'hnswlib': {'numpy': []},\n 'gridlooper': {'dill': [],\n 'attrs': {'importlib-metadata': {'zipp': [], 'typing-extensions': []}},\n 'tqdm': {'colorama': []}},\n 'dill': [],\n 'numpy': []}\n\n\n\n#### Addding license labels to tree of dependencies\n\n\n```python\nextracted_dependencies_tree_license = da.add_license_labels_to_dep_tree(\n dependencies_tree = extracted_dependencies_tree\n)\nextracted_dependencies_tree_license\n```\n\n\n\n\n {'requests': 'apache-2.0',\n 'requests.charset-normalizer': '-',\n 'requests.idna': '-',\n 'requests.urllib3': '-',\n 'requests.certifi': 'mpl-2.0',\n 'attrs': '-',\n 'attrs.importlib-metadata': '-',\n 'attrs.importlib-metadata.zipp': '-',\n 'attrs.importlib-metadata.typing-extensions': '-',\n 'httpx': '-',\n 'httpx.anyio': 'mit',\n 'httpx.anyio.idna': '-',\n 'httpx.anyio.sniffio': '-',\n 'httpx.anyio.exceptiongroup': '-',\n 'httpx.anyio.typing-extensions': '-',\n 'httpx.certifi': 'mpl-2.0',\n 'httpx.httpcore': '-',\n 'httpx.httpcore.certifi': 'mpl-2.0',\n 'httpx.httpcore.h11': 'mit',\n 'httpx.httpcore.h11.typing-extensions': '-',\n 'httpx.idna': '-',\n 'httpx.sniffio': '-',\n 'hnswlib': '-',\n 'hnswlib.numpy': 'bsd-3-clause',\n 'gridlooper': '-',\n 'gridlooper.dill': 'bsd-3-clause',\n 'gridlooper.attrs': '-',\n 'gridlooper.attrs.importlib-metadata': '-',\n 'gridlooper.attrs.importlib-metadata.zipp': '-',\n 'gridlooper.attrs.importlib-metadata.typing-extensions': '-',\n 'gridlooper.tqdm': '-',\n 'gridlooper.tqdm.colorama': '-',\n 'dill': 'bsd-3-clause',\n 'numpy': 'bsd-3-clause'}\n\n\n\n#### Printing extracted tree of dependencies\n\n\n```python\nda.print_flattened_tree(extracted_dependencies_tree_license)\n```\n\n \u2514\u2500\u2500 requests : apache-2.0\n \u251c\u2500\u2500 charset-normalizer : -\n \u251c\u2500\u2500 idna : -\n \u251c\u2500\u2500 urllib3 : -\n \u2514\u2500\u2500 certifi : mpl-2.0\n \u2514\u2500\u2500 attrs : -\n \u2514\u2500\u2500 importlib-metadata : -\n \u251c\u2500\u2500 zipp : -\n \u2514\u2500\u2500 typing-extensions : -\n \u2514\u2500\u2500 httpx : -\n \u251c\u2500\u2500 anyio : mit\n \u251c\u2500\u2500 idna : -\n \u251c\u2500\u2500 sniffio : -\n \u251c\u2500\u2500 exceptiongroup : -\n \u2514\u2500\u2500 typing-extensions : -\n \u251c\u2500\u2500 certifi : mpl-2.0\n \u251c\u2500\u2500 httpcore : -\n \u251c\u2500\u2500 certifi : mpl-2.0\n \u2514\u2500\u2500 h11 : mit\n \u2514\u2500\u2500 typing-extensions : -\n \u251c\u2500\u2500 idna : -\n \u2514\u2500\u2500 sniffio : -\n \u2514\u2500\u2500 hnswlib : -\n \u2514\u2500\u2500 numpy : bsd-3-clause\n \u2514\u2500\u2500 gridlooper : -\n \u251c\u2500\u2500 dill : bsd-3-clause\n \u251c\u2500\u2500 attrs : -\n \u2514\u2500\u2500 importlib-metadata : -\n \u251c\u2500\u2500 zipp : -\n \u2514\u2500\u2500 typing-extensions : -\n \u2514\u2500\u2500 tqdm : -\n \u2514\u2500\u2500 colorama : -\n \u2514\u2500\u2500 dill : bsd-3-clause\n \u2514\u2500\u2500 numpy : bsd-3-clause\n\n\n#### Filtering for unexpected licenses in tree of dependencies\n\n\n```python\nallowed_licenses = ['mit', 'apache-2.0', 'lgpl-3.0', 'mpl-2.0', '-']\n\nda.find_unexpected_licenses_in_deps_tree(\n tree_dep_license = extracted_dependencies_tree_license,\n # optional\n allowed_licenses = allowed_licenses,\n raise_error = True\n)\n```\n\n {'hnswlib': '', 'gridlooper': ''}\n \u2514\u2500\u2500 dill : bsd-3-clause\n \u2514\u2500\u2500 numpy : bsd-3-clause\n \u2514\u2500\u2500 hnswlib : \n \u2514\u2500\u2500 numpy : bsd-3-clause\n \u2514\u2500\u2500 gridlooper : \n \u2514\u2500\u2500 dill : bsd-3-clause\n\n\n\n ---------------------------------------------------------------------------\n\n Exception Traceback (most recent call last)\n\n Cell In[9], line 3\n 1 allowed_licenses = ['mit', 'apache-2.0', 'lgpl-3.0', 'mpl-2.0', '-']\n ----> 3 da.find_unexpected_licenses_in_deps_tree(\n 4 tree_dep_license = extracted_dependencies_tree_license,\n 5 # optional\n 6 allowed_licenses = allowed_licenses,\n 7 raise_error = True\n 8 )\n\n\n File ~/miniforge3/envs/testenv/lib/python3.10/site-packages/package_auto_assembler/package_auto_assembler.py:2670, in DependenciesAnalyser.find_unexpected_licenses_in_deps_tree(self, tree_dep_license, allowed_licenses, raise_error)\n 2668 if raise_error and out != {}:\n 2669 self.print_flattened_tree(flattened_dict = out)\n -> 2670 raise Exception(\"Found unexpected licenses!\")\n 2671 else:\n 2672 self.logger.info(\"No unexpected licenses found\")\n\n\n Exception: Found unexpected licenses!\n\n\n### 10. Adding cli interfaces\n\nThe tool allows to make a package with optional cli interfaces. These could be sometimes preferable when a package contains a standalone tool that would be called from script anyway.\n\nAll of the cli logic would need to be included within a `.py` file which should be stored within `cli_dir` provided in `.paa.config`. \nDependencies from these files are extracted in the similar manner to the main module.\n\nTools from main `.py` file could still be imported like the following:\n\n```python\nfrom package_name.package_name import ToBeImported\n```\n\nThe code is wired in `setup.py` via the following automatically assuming that appropriate file with the same name as the package exists within `cli_dir` location.\n\n```python\n...,\nentry_points = {'console_scripts': [\n '<package_alias> = package_name.cli:cli']} ,\n...\n```\n\nAlias for name could be provided via the following piece of code, defined after imports, otherwise package name would be used.\n\n```python \n__cli_metadata__ = {\n \"name\" : <package_alias>\n}\n```\n\nPackage-auto-assembler tool itself uses [`click`](https://pypi.org/project/click/) dependency to build that file, use its [cli definition](https://github.com/Kiril-Mordan/reusables/blob/main/cli/package_auto_assembler.py) as example.\n\n\n### 11. Adding routes and running FastAPI application\n\nThe tool allows to make a package with optional routes for FastAPI application and run them. Each packages can have one routes file where its logic should be defined. Package-auto-assembler itself can combine multiple routes from packages and filepaths into one application.\n\nA `.py` file with the same name of the package should be stored within `api_routes_dir` provided in `.paa.config`.\n\nDependencies from these files are extracted in the similar manner to the main module.\n\nTools from main `.py` file could still be imported like the following:\n\n```python\nfrom package_name.package_name import ToBeImported\n```\n\nApi description, middleware and run parameters could be provided via optional `yml` file, which for example would look like:\n\n```\nDESCRIPTION : {\n 'version' : 0.0.0\n}\nMIDDLEWARE : {\n origin : ['*']\n}\nRUN : {\n host : 0.0.0.0\n}\n```\n\nwhere DESCRIPTION contains parameters for `FastAPI`, MIDDLEWARE for `CORSMiddleware` and RUN for `uvicorn.run`\n\n\n### 12. Making a package\n\nMain wrapper for the package integrates described above components into a class that could be used to build package building pipelines within python scripts. \n\nTo simplify usage [cli interface](https://kiril-mordan.github.io/reusables/package_auto_assembler/cli/) is recomended instead. \n\n#### Initializing PackageAutoAssembler\n\n\n```python\npaa = PackageAutoAssembler(\n # required\n module_name = \"example_module\",\n module_filepath = \"../tests/package_auto_assembler/other/example_module.py\",\n # optional\n mapping_filepath = \"../env_spec/package_mapping.json\",\n licenses_filepath = \"../tests/package_auto_assembler/other/package_licenses.json\",\n allowed_licenses = ['mit', 'apache-2.0', 'lgpl-3.0', 'mpl-2.0', '-'],\n dependencies_dir = \"../tests/package_auto_assembler/dependancies/\",\n example_notebook_path = \"./mock_vector_database.ipynb\",\n versions_filepath = '../tests/package_auto_assembler/other/lsts_package_versions.yml',\n log_filepath = '../tests/package_auto_assembler/other/version_logs.csv',\n setup_directory = \"./example_module\",\n release_notes_filepath = \"../tests/package_auto_assembler/other/release_notes.md\",\n license_path = \"../LICENSE\",\n license_label = \"mit\",\n classifiers = ['Development Status :: 3 - Alpha',\n 'Intended Audience :: Developers',\n 'Intended Audience :: Science/Research',\n 'Programming Language :: Python :: 3',\n 'Programming Language :: Python :: 3.9',\n 'Programming Language :: Python :: 3.10',\n 'Programming Language :: Python :: 3.11',\n 'License :: OSI Approved :: MIT License',\n 'Topic :: Scientific/Engineering'],\n requirements_list = [],\n execute_readme_notebook = True,\n python_version = \"3.8\",\n version_increment_type = \"patch\",\n default_version = \"0.0.1\",\n check_vulnerabilities = True,\n check_dependencies_licenses = False,\n add_requirements_header = True\n)\n```\n\n#### Add metadata from module\n\n\n```python\npaa.add_metadata_from_module(\n # optional\n module_filepath = \"../tests/package_auto_assembler/other/example_module.py\"\n)\n```\n\n Adding metadata ...\n\n\n#### Add or update version\n\n\n```python\npaa.add_or_update_version(\n # overwrites auto mode (not suggested)\n version_increment_type = \"patch\",\n version = \"1.2.6\",\n # optional\n module_name = \"example_module\",\n versions_filepath = '../tests/package_auto_assembler/lsts_package_versions.yml',\n log_filepath = '../tests/package_auto_assembler/version_logs.csv'\n)\n```\n\n Incrementing version ...\n No relevant commit messages found!\n ..trying depth 2 !\n No relevant commit messages found!\n ..trying depth 3 !\n No relevant commit messages found!\n ..trying depth 4 !\n No relevant commit messages found!\n ..trying depth 5 !\n No relevant commit messages found!\n No messages to clean were provided\n\n\n#### Add release notes from commit messages\n\n\n```python\npaa.add_or_update_release_notes(\n # optional\n filepath=\"../tests/package_auto_assembler/release_notes.md\",\n version=paa.metadata['version']\n)\n```\n\n Updating release notes ...\n\n\n#### Prepare setup directory\n\n\n```python\npaa.prep_setup_dir()\n```\n\n Preparing setup directory ...\n\n\n#### Merge local dependacies\n\n\n```python\npaa.merge_local_dependacies(\n # optional\n main_module_filepath = \"../tests/package_auto_assembler/other/example_module.py\",\n dependencies_dir= \"../tests/package_auto_assembler/dependancies/\",\n save_filepath = \"./example_module/example_module.py\"\n)\n```\n\n Merging ../tests/package_auto_assembler/other/example_module.py with dependecies from ../tests/package_auto_assembler/dependancies/ into ./example_module/example_module.py\n\n\n#### Add requirements from module\n\n\n```python\npaa.add_requirements_from_module(\n # optional\n module_filepath = \"../tests/package_auto_assembler/other/example_module.py\",\n import_mappings = {'PIL': 'Pillow',\n 'bs4': 'beautifulsoup4',\n 'fitz': 'PyMuPDF',\n 'attr': 'attrs',\n 'dotenv': 'python-dotenv',\n 'googleapiclient': 'google-api-python-client',\n 'sentence_transformers': 'sentence-transformers',\n 'flask': 'Flask',\n 'stdlib_list': 'stdlib-list',\n 'sklearn': 'scikit-learn',\n 'yaml': 'pyyaml',\n 'git' : 'gitpython'}\n)\n```\n\n Adding requirements from ../tests/package_auto_assembler/other/example_module.py\n No known vulnerabilities found\n \n\n\n \n\n\n\n```python\npaa.requirements_list\n```\n\n\n\n\n ['### example_module.py', 'attrs>=22.2.0']\n\n\n\n#### Make README out of example notebook\n\n\n```python\npaa.add_readme(\n # optional\n example_notebook_path = \"../tests/package_auto_assembler/other/example_module.ipynb\",\n output_path = \"./example_module/README.md\",\n execute_notebook=False,\n)\n```\n\n Adding README from ../tests/package_auto_assembler/other/example_module.ipynb to ./example_module/README.md\n\n\n#### Prepare setup file\n\n\n```python\npaa.prep_setup_file(\n # optional\n metadata = {'author': 'Kyrylo Mordan',\n 'version': '0.0.1',\n 'description': 'Example module',\n 'keywords': ['python'],\n 'license' : 'mit'},\n requirements = ['### example_module.py',\n 'attrs>=22.2.0'],\n classifiers = ['Development Status :: 3 - Alpha',\n 'Intended Audience :: Developers',\n 'Intended Audience :: Science/Research',\n 'Programming Language :: Python :: 3',\n 'Programming Language :: Python :: 3.9',\n 'Programming Language :: Python :: 3.10',\n 'Programming Language :: Python :: 3.11',\n 'License :: OSI Approved :: MIT License',\n 'Topic :: Scientific/Engineering'],\n cli_module_filepath = \"../tests/package_auto_assembler/other/cli.py\"\n\n)\n```\n\n Preparing setup file for example-module package ...\n\n\n#### Make package\n\n\n```python\npaa.make_package(\n # optional\n setup_directory = \"./example_module\"\n)\n```\n\n Making package from ./example_module ...\n\n\n\n\n\n CompletedProcess(args=['python', './example_module/setup.py', 'sdist', 'bdist_wheel'], returncode=0, stdout=\"running sdist\\nrunning egg_info\\nwriting example_module.egg-info/PKG-INFO\\nwriting dependency_links to example_module.egg-info/dependency_links.txt\\nwriting entry points to example_module.egg-info/entry_points.txt\\nwriting requirements to example_module.egg-info/requires.txt\\nwriting top-level names to example_module.egg-info/top_level.txt\\nreading manifest file 'example_module.egg-info/SOURCES.txt'\\nwriting manifest file 'example_module.egg-info/SOURCES.txt'\\nrunning check\\ncreating example_module-0.0.0\\ncreating example_module-0.0.0/example_module\\ncreating example_module-0.0.0/example_module.egg-info\\ncopying files to example_module-0.0.0...\\ncopying example_module/__init__.py -> example_module-0.0.0/example_module\\ncopying example_module/cli.py -> example_module-0.0.0/example_module\\ncopying example_module/example_module.py -> example_module-0.0.0/example_module\\ncopying example_module/setup.py -> example_module-0.0.0/example_module\\ncopying example_module.egg-info/PKG-INFO -> example_module-0.0.0/example_module.egg-info\\ncopying example_module.egg-info/SOURCES.txt -> example_module-0.0.0/example_module.egg-info\\ncopying example_module.egg-info/dependency_links.txt -> example_module-0.0.0/example_module.egg-info\\ncopying example_module.egg-info/entry_points.txt -> example_module-0.0.0/example_module.egg-info\\ncopying example_module.egg-info/requires.txt -> example_module-0.0.0/example_module.egg-info\\ncopying example_module.egg-info/top_level.txt -> example_module-0.0.0/example_module.egg-info\\ncopying example_module.egg-info/SOURCES.txt -> example_module-0.0.0/example_module.egg-info\\nWriting example_module-0.0.0/setup.cfg\\nCreating tar archive\\nremoving 'example_module-0.0.0' (and everything under it)\\nrunning bdist_wheel\\nrunning build\\nrunning build_py\\ncopying example_module/example_module.py -> build/lib/example_module\\ncopying example_module/__init__.py -> build/lib/example_module\\ncopying example_module/setup.py -> build/lib/example_module\\ncopying example_module/cli.py -> build/lib/example_module\\ninstalling to build/bdist.linux-x86_64/wheel\\nrunning install\\nrunning install_lib\\ncreating build/bdist.linux-x86_64/wheel\\ncreating build/bdist.linux-x86_64/wheel/example_module\\ncopying build/lib/example_module/example_module.py -> build/bdist.linux-x86_64/wheel/example_module\\ncopying build/lib/example_module/__init__.py -> build/bdist.linux-x86_64/wheel/example_module\\ncopying build/lib/example_module/setup.py -> build/bdist.linux-x86_64/wheel/example_module\\ncopying build/lib/example_module/cli.py -> build/bdist.linux-x86_64/wheel/example_module\\nrunning install_egg_info\\nCopying example_module.egg-info to build/bdist.linux-x86_64/wheel/example_module-0.0.0-py3.10.egg-info\\nrunning install_scripts\\ncreating build/bdist.linux-x86_64/wheel/example_module-0.0.0.dist-info/WHEEL\\ncreating 'dist/example_module-0.0.0-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it\\nadding 'example_module/__init__.py'\\nadding 'example_module/cli.py'\\nadding 'example_module/example_module.py'\\nadding 'example_module/setup.py'\\nadding 'example_module-0.0.0.dist-info/METADATA'\\nadding 'example_module-0.0.0.dist-info/WHEEL'\\nadding 'example_module-0.0.0.dist-info/entry_points.txt'\\nadding 'example_module-0.0.0.dist-info/top_level.txt'\\nadding 'example_module-0.0.0.dist-info/RECORD'\\nremoving build/bdist.linux-x86_64/wheel\\n\", stderr='warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md\\n\\n/home/kyriosskia/miniconda3/envs/testenv/lib/python3.10/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.\\n!!\\n\\n ********************************************************************************\\n Please avoid running ``setup.py`` directly.\\n Instead, use pypa/build, pypa/installer or other\\n standards-based tools.\\n\\n See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.\\n ********************************************************************************\\n\\n!!\\n self.initialize_options()\\n')\n\n\n\n### 13. Making simple MkDocs site\n\nPackage documentation can be presented in a form of mkdocs static site, which could be either served or deployed to something like github packages. \n\nMain module docstring is used as intro package that contains something like optional pypi and license badges. Package description and realease notes are turned into separate tabs. Png with diagrams for example could be provided and displayed as their own separate tabs as well.\n \nThe one for this package can be seen [here](https://kiril-mordan.github.io/reusables/package_auto_assembler/)\n\nIt can be packaged with the package and be displayed in webrowser like documentation for api via `{package_name}\\docs` when using included api handling capabilities.\n\n##### - preparing inputs\n\n\n```python\npackage_name = \"example_module\"\n\nmodule_content = LongDocHandler().read_module_content(filepath=f\"../tests/package_auto_assembler/{package_name}.py\")\n\ndocstring = LongDocHandler().extract_module_docstring(module_content=module_content)\npypi_link = LongDocHandler().get_pypi_badge(module_name=package_name)\n\n\ndocs_file_paths = {\n \"../example_module.md\" : \"usage-examples.md\",\n '../tests/package_auto_assembler/release_notes.md' : 'release_notes.md'\n}\n```\n\n\n```python\nmdh = MkDocsHandler(\n # required\n ## name of the package to be displayed\n package_name = package_name,\n ## dictionary of markdown files, with path as keys\n docs_file_paths = docs_file_paths,\n # optional\n ## module docstring to be displayed in the index\n module_docstring = docstring,\n ## pypi badge to be displayed in the index\n pypi_badge = pypi_link,\n ## license badge to be displayed in the index\n license_badge=\"[](https://github.com/Kiril-Mordan/reusables/blob/main/LICENSE)\",\n ## name of the project directory\n project_name = \"temp_project\")\n```\n\n##### - preparing site\n\n\n```python\nmdh.create_mkdocs_dir()\nmdh.move_files_to_docs()\nmdh.generate_markdown_for_images()\nmdh.create_index()\nmdh.create_mkdocs_yml()\nmdh.build_mkdocs_site()\n```\n\n Created new MkDocs dir: temp_project\n Copied ../example_module.md to temp_project/docs/usage-examples.md\n Copied ../tests/package_auto_assembler/release_notes.md to temp_project/docs/release_notes.md\n index.md has been created with site_name: example-module\n mkdocs.yml has been created with site_name: Example module\n Custom CSS created at temp_project/docs/css/extra.css\n\n\n INFO - Cleaning site directory\n INFO - Building documentation to directory: /home/kyriosskia/Documents/nlp/reusables/example_notebooks/temp_project/site\n INFO - Documentation built in 0.12 seconds\n\n\n##### - test runing site\n\n\n```python\nmdh.serve_mkdocs_site()\n```\n",
"bugtrack_url": null,
"license": "mit",
"summary": "A tool to automate package creation within ci based on just .py and optionally .ipynb file.",
"version": "0.5.2",
"project_urls": null,
"split_keywords": [
"['python'",
" 'packaging'",
" 'aa-paa-tool']"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "06c78be5c4db8de407af3a6a02fa89aad9a1735ff4e9d9e05416ea0eef453ec0",
"md5": "37e3eb695232197928b6355ca2894fc6",
"sha256": "a6ae6e18e227076831064ca49ff0436777ddd0227d250f2d06920fceec718728"
},
"downloads": -1,
"filename": "package_auto_assembler-0.5.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "37e3eb695232197928b6355ca2894fc6",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 1353335,
"upload_time": "2024-09-22T23:49:50",
"upload_time_iso_8601": "2024-09-22T23:49:50.832547Z",
"url": "https://files.pythonhosted.org/packages/06/c7/8be5c4db8de407af3a6a02fa89aad9a1735ff4e9d9e05416ea0eef453ec0/package_auto_assembler-0.5.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a802138e4ced42ec33224d331768acc6f4d82491287b69c0f98d8bd75abbb9aa",
"md5": "ff1fee1744cc2bc0e67ee09205cb512c",
"sha256": "577131fe951b5a70f050ae4916a1b6aa7847a9bcbb58397059ebc15ef309a8ed"
},
"downloads": -1,
"filename": "package_auto_assembler-0.5.2.tar.gz",
"has_sig": false,
"md5_digest": "ff1fee1744cc2bc0e67ee09205cb512c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 1322151,
"upload_time": "2024-09-22T23:49:52",
"upload_time_iso_8601": "2024-09-22T23:49:52.862311Z",
"url": "https://files.pythonhosted.org/packages/a8/02/138e4ced42ec33224d331768acc6f4d82491287b69c0f98d8bd75abbb9aa/package_auto_assembler-0.5.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-22 23:49:52",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "package-auto-assembler"
}
JSON