| Name | packaging-coursdemo JSON |
| Version |
0.0.10
JSON |
| download |
| home_page | |
| Summary | My package description |
| upload_time | 2024-02-23 15:25:05 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.7 |
| license | MIT |
| keywords |
one
two
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# PACKAGINGDATASCIENCE
Why packaging ?
1. Distributing Your Code
2. Non-painful import statements
3. Reproducibility and dependencies management specifically
# Packaging terms:
- Module : a sigle portion of one of this import statements (can be single file , also can be folder that contains __init__.py )
- Package (basically folder contain __init__.py file)
- sub-package
- distribution package to fix import problems we see earlier (is basically a zip file containing all the python files if you go the pypi.org you will see things like numpy, pandas , pytorch , fastapi , ... called all those things that will pipi install packages but technically the correct term is distribution packages for the rest of course we will use the terms distribution packages to minimize confusion)
# Modern way to build package :
* the old format distribution is sdist : source distribution package : python setup.py sdist
1. May make assumptions about customer machine:
e.g. requires "gcc" to run "gcc numpy/*.c"
2. Is slow: setup.py must be executed, compilation may be required.
3. Is insecure: setup.py may contain arbitrary code.
* the good and new format distribution is wheel
1. First install wheel package
2. the command help us to build wheel package : python setup.py bdist_wheel after install wheel with command pip install wheel
3. realpython.com/python-wheels/ explain clearly how to read wheel format
4. we need to add build dependencies for reproducibility and for it we need pyproject.toml file (https://peps.python.org/pep-0518/) and install build tool : pip install build (https://pypi.org/project/build/) and python -m build --sdist --wheel ./ command we use to build sdist format distribution and wheel format
5. escaping config hell; use setup.cfg : https://setuptools.pypa.io/en/latest/userguide/declarative_config.html
6. Removing setup.cfg thanks to pep 621 : peps.python.org/pep-0621/
7. reconfigure project.toml -> https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html (In setuptool doc Note New in 61.0.0)
8. After you will see the information about your package in PKG-INFO inside of packaging-demo-egg-info
9. Removing setup.py : PEP 517 ; build backends : peps.python.org/pep-0517/
10. Exemple use case for including data with various formats (in this example is json) into your package
11. unzip your whl like this : unzip *.whl after you move to dist repo : cd dist and move to your repo and tape pip install .
12. documentation to specific problems to add data files into your package : https://setuptools.pypa.io/en/latest/userguide/datafiles.html and https://setuptools.pypa.io/en/latest/userguide/miscellaneous.html#using-manifest-in
13. python -m build --sdist --wheel ./; cd dist; unzip *.whl; cd ..
14. for example to add data files recursively in your folder recursive-include packaging_demo/ *.json
15. equivalent of recursive include you can include in your pyproject.tom[tool.setuptools.package-data]
my_pkg = ["*.json"] or you can use Hatch build system instead of setuptools : hatch.pypa.io/latest/config/build/#build-system poetry is another tool to build backend but is not pep 601 definition is not compliant
# Reproducibility
minimal document the librairies you use
never run pip install directly in your projects you can use optionnal level 1 to add the library or better pyproject.toml
How to reproduce(ish) these results:
install the python and non-python dependencies
pip install -r requirements.txt
brew install graphviz
generate the dependency graphs for a library by downloading many versions of it
python generate_dependency_graph.py fastapi
generate this README.md
python generate_readme.py
1. Dependency graph : add someting dependancies in pyproject.toml and in terminal tape this command pip install -e.
1. visualize all dépendencies tree of python libraries use cli tool call pipdeptree : pipdeptree --help , pipdeptree -p packaging-demo --grph-output png > dependency-graph.png , pip install graphviz , and brew install graphviz or sudo apt-get install graphviz help us to the issues dependencies hell : constraint-trees (datascientist didn't document version of the dependencies we use ) the more library they more constraints you have you had they more dependencies you have lot of constraints
2. Document all of the exact version of library and the exact python version and in addition of that you documented opération system : m series mcbook , linus ..etc the solution to this is if you run pip freeze : this is most vanilla way in python to freeze or lock your dependencies list : set of the exact version that satisfied constraint pip freeze > requirements.txt useful for developpment reproducibilty poetry , pip-tools same willl help you .
You can generate a lock file a few different ways:
- pip: pip freeze > requirements.txt
- pip-tools: pip-compile --output-file requirements.txt requirements.in
- poetry
- pipenv
3. you can add [project.optional-dependencies]
dev = ["ruff" , "mypy" , "black"] and after that run in terminal pip install '.[dev]' the same for rich add colors = ["rich] pip install '.[rich]' or pip install '.[colors,dev]' or all = ["packaging_demo[dev , colors]"] and pip install '.[all]' package index search : snyk.io/advisor/python/package-index ( give you criteria to check package health in terms of security maintenance , community )
# Intro to Continuous Delivery : Publishing to PyPI
* Product management :
- Objectives , Discovery and Delivery
*1. Product Discovery
Discovery is very much about the intense collaboration of product management, user experience design and ingeieering
In discovery we are tackling the various risks before we write even one line of production software.
The purpose of product discovery is to quickly separate the good ideas from bad. The output of product discovery is a validated product backlog.
Specifically, this means getting answers to four critical questions:
a. Will the user buy this (or choose to use it )?
b. Can the user figure out how to use this ?
c. Can our engineers build this
d. Can our stakeholders support this ?
Prototypes : Product discovery involves running a series of very quick experiments , and do these experiements quickly and inexpensively, we use prototypes rather than products
*2. Product delivery
The purpose of all these prototypes and experiments in discovery is to quickly come up with something that provides some evidence it is worth building and that we can then deliver to our customers.
This means the necessary scale , performance reliability fault tolerance security privacy internationalization and localization have been performed and the product works as advetised. The purpose of product delivery is to build and deliver these production-quality technology products something you can sell and run a business on
Implementing a pipeline that publishes our package to Pypi : a single example of continuous delivery
Continuous Delivery is a companion to Continuous Integration is the practice which make small frequents commits to our code base .
I constantly updating the codebase : Continuous integration
I constantly make the integration code available to the users or continuously delivery thoses changes to the users
** Devops : Developer Operations or software developer operations : devops persons try to make software developper work as efficiently as possible (small frequents commits , ci/cd)
- Waterfall vs Agile (Design product)
Waterfall project management : strategy will use to build and delivers a product for end users (like construction happens to a house , expensive product difficult to change after we construct to construct house : waterfall method watterfall is linear )
- Agile : move fast ( Deliver quickly as posssible smallest things you can three days to create prototype )
# Publishing your package to pypi : python package index repositoritory
Pypi is immutable : you can replace an existing package you can never upload the semantic version
a second instance for pypi is testpYPI / test.pypi.org : test deployment package . https://pypi.org/
Twine is a cli we use to publish our paclkaqge to pypi : https://twine.readthedocs.io/en/latest/ twine --help twine upload --repository testpypi ./dist/*
Securely use twine cli tools to publish your package
How to document various command need to runs when we develop code base :
the point is make your development workflow easy when other to contributers
we take about config file common one on industry standards
Install makefile tools extensions in vscode and brew install make to install cmake
the first file we help us to make it is make file use tab instead space and make install : making artifacts : all generated by the code trying to be smart not rebuild all thing that's build the command is make and target which references bash script make file is portable
Alternatives to make as task runners : just and pyinvoke , Bash
* just https://github.com/casey/just
* pyinvoke https://www.pyinvoke.org/
* Bash use Taskfile for AdrianCooney : https://github.com/adriancooney/Taskfile use chmod +x run.sh to render exceutable file track set -e or set -ex help you to debug when to want to install anything you can define use makefile as interface over your runsh script
* Make you key or credential in security way for exemple if you use .github.com/ethOizzle/shhgit you can views all keys access free in repo public : it's not recommanded because commit is immutable don't send sensitive : best managing secrets in productions : environnement variables
# Github actions continuous delivery
One kind of event is pull requests which have activity types : pull_request Listeners
Triggers we have :
React to pull request event : premerge
postmerge and push event
Github action context and secret management : value to access anytime in workflow
In the settings in github section secrets and variables
Repository secrets is what we interest for
Access context in workflow : use github expressions : dollard sign and double bracket
documentation of contexts in github actions : # https://docs.github.com/en/actions/learn-github-actions/contexts#example-printing-context-information-to-the-log
Different betweens variables : access a plain test value printed and secrets are not unprinted :
if statements checks in publish yaml file : docs.github.com/en/actions/learn-github-actions/contexts#
after deployment sucessfull in prod we can tag this
# speed up your workfows or pipeline ci/cd use dependencies caching : githubactions cahing
[github.com/actions/cache](https://github.com/actions/cache/blob/main/examples.md#simple-example)
- uses: actions/cache@v3
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt' , '**/project.toml') }}
restore-keys: |
${{ runner.os }}-pip-
You can set the cache all the emplacement you had pip install
pass artifacts (upload and downloads) between jobs : github.com/actions/download-artifact
Raw data
{
"_id": null,
"home_page": "",
"name": "packaging-coursdemo",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "one,two",
"author": "",
"author_email": "Espoir Badohoun <some-email@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/4b/ac/a9d2dcb7f20c0e0e1add273416add8b0ca8434bad0f398beb5453e47c3ef/packaging-coursdemo-0.0.10.tar.gz",
"platform": null,
"description": "# PACKAGINGDATASCIENCE\nWhy packaging ?\n\n1. Distributing Your Code\n2. Non-painful import statements\n3. Reproducibility and dependencies management specifically\n\n\n# Packaging terms:\n- Module : a sigle portion of one of this import statements (can be single file , also can be folder that contains __init__.py )\n- Package (basically folder contain __init__.py file)\n- sub-package\n- distribution package to fix import problems we see earlier (is basically a zip file containing all the python files if you go the pypi.org you will see things like numpy, pandas , pytorch , fastapi , ... called all those things that will pipi install packages but technically the correct term is distribution packages for the rest of course we will use the terms distribution packages to minimize confusion)\n\n\n# Modern way to build package :\n\n* the old format distribution is sdist : source distribution package : python setup.py sdist\n\n 1. May make assumptions about customer machine:\n e.g. requires \"gcc\" to run \"gcc numpy/*.c\"\n 2. Is slow: setup.py must be executed, compilation may be required.\n 3. Is insecure: setup.py may contain arbitrary code.\n* the good and new format distribution is wheel\n 1. First install wheel package\n 2. the command help us to build wheel package : python setup.py bdist_wheel after install wheel with command pip install wheel\n 3. realpython.com/python-wheels/ explain clearly how to read wheel format\n 4. we need to add build dependencies for reproducibility and for it we need pyproject.toml file (https://peps.python.org/pep-0518/) and install build tool : pip install build (https://pypi.org/project/build/) and python -m build --sdist --wheel ./ command we use to build sdist format distribution and wheel format\n 5. escaping config hell; use setup.cfg : https://setuptools.pypa.io/en/latest/userguide/declarative_config.html\n 6. Removing setup.cfg thanks to pep 621 : peps.python.org/pep-0621/\n 7. reconfigure project.toml -> https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html (In setuptool doc Note New in 61.0.0)\n 8. After you will see the information about your package in PKG-INFO inside of packaging-demo-egg-info\n 9. Removing setup.py : PEP 517 ; build backends : peps.python.org/pep-0517/\n 10. Exemple use case for including data with various formats (in this example is json) into your package\n 11. unzip your whl like this : unzip *.whl after you move to dist repo : cd dist and move to your repo and tape pip install .\n 12. documentation to specific problems to add data files into your package : https://setuptools.pypa.io/en/latest/userguide/datafiles.html and https://setuptools.pypa.io/en/latest/userguide/miscellaneous.html#using-manifest-in\n 13. python -m build --sdist --wheel ./; cd dist; unzip *.whl; cd ..\n 14. for example to add data files recursively in your folder recursive-include packaging_demo/ *.json\n 15. equivalent of recursive include you can include in your pyproject.tom[tool.setuptools.package-data]\n my_pkg = [\"*.json\"] or you can use Hatch build system instead of setuptools : hatch.pypa.io/latest/config/build/#build-system poetry is another tool to build backend but is not pep 601 definition is not compliant\n\n\n # Reproducibility\n minimal document the librairies you use\n never run pip install directly in your projects you can use optionnal level 1 to add the library or better pyproject.toml\n\n How to reproduce(ish) these results:\n\n install the python and non-python dependencies\npip install -r requirements.txt\nbrew install graphviz\n\n generate the dependency graphs for a library by downloading many versions of it\npython generate_dependency_graph.py fastapi\n\n generate this README.md\npython generate_readme.py\n\n\n 1. Dependency graph : add someting dependancies in pyproject.toml and in terminal tape this command pip install -e.\n\n1. visualize all d\u00e9pendencies tree of python libraries use cli tool call pipdeptree : pipdeptree --help , pipdeptree -p packaging-demo --grph-output png > dependency-graph.png , pip install graphviz , and brew install graphviz or sudo apt-get install graphviz help us to the issues dependencies hell : constraint-trees (datascientist didn't document version of the dependencies we use ) the more library they more constraints you have you had they more dependencies you have lot of constraints\n2. Document all of the exact version of library and the exact python version and in addition of that you documented op\u00e9ration system : m series mcbook , linus ..etc the solution to this is if you run pip freeze : this is most vanilla way in python to freeze or lock your dependencies list : set of the exact version that satisfied constraint pip freeze > requirements.txt useful for developpment reproducibilty poetry , pip-tools same willl help you .\n You can generate a lock file a few different ways:\n\n - pip: pip freeze > requirements.txt\n - pip-tools: pip-compile --output-file requirements.txt requirements.in\n - poetry\n - pipenv\n3. you can add [project.optional-dependencies]\ndev = [\"ruff\" , \"mypy\" , \"black\"] and after that run in terminal pip install '.[dev]' the same for rich add colors = [\"rich] pip install '.[rich]' or pip install '.[colors,dev]' or all = [\"packaging_demo[dev , colors]\"] and pip install '.[all]' package index search : snyk.io/advisor/python/package-index ( give you criteria to check package health in terms of security maintenance , community )\n\n\n# Intro to Continuous Delivery : Publishing to PyPI\n\n* Product management :\n\n - Objectives , Discovery and Delivery\n *1. Product Discovery\n Discovery is very much about the intense collaboration of product management, user experience design and ingeieering\n In discovery we are tackling the various risks before we write even one line of production software.\n The purpose of product discovery is to quickly separate the good ideas from bad. The output of product discovery is a validated product backlog.\n Specifically, this means getting answers to four critical questions:\n\n a. Will the user buy this (or choose to use it )?\n b. Can the user figure out how to use this ?\n c. Can our engineers build this\n d. Can our stakeholders support this ?\n Prototypes : Product discovery involves running a series of very quick experiments , and do these experiements quickly and inexpensively, we use prototypes rather than products\n *2. Product delivery\n The purpose of all these prototypes and experiments in discovery is to quickly come up with something that provides some evidence it is worth building and that we can then deliver to our customers.\n This means the necessary scale , performance reliability fault tolerance security privacy internationalization and localization have been performed and the product works as advetised. The purpose of product delivery is to build and deliver these production-quality technology products something you can sell and run a business on\n\nImplementing a pipeline that publishes our package to Pypi : a single example of continuous delivery\n\nContinuous Delivery is a companion to Continuous Integration is the practice which make small frequents commits to our code base .\n\n\nI constantly updating the codebase : Continuous integration\nI constantly make the integration code available to the users or continuously delivery thoses changes to the users\n\n ** Devops : Developer Operations or software developer operations : devops persons try to make software developper work as efficiently as possible (small frequents commits , ci/cd)\n - Waterfall vs Agile (Design product)\n\n Waterfall project management : strategy will use to build and delivers a product for end users (like construction happens to a house , expensive product difficult to change after we construct to construct house : waterfall method watterfall is linear )\n- Agile : move fast ( Deliver quickly as posssible smallest things you can three days to create prototype )\n\n# Publishing your package to pypi : python package index repositoritory\nPypi is immutable : you can replace an existing package you can never upload the semantic version\na second instance for pypi is testpYPI / test.pypi.org : test deployment package . https://pypi.org/\nTwine is a cli we use to publish our paclkaqge to pypi : https://twine.readthedocs.io/en/latest/ twine --help twine upload --repository testpypi ./dist/*\nSecurely use twine cli tools to publish your package\n\n\nHow to document various command need to runs when we develop code base :\nthe point is make your development workflow easy when other to contributers\nwe take about config file common one on industry standards\nInstall makefile tools extensions in vscode and brew install make to install cmake\nthe first file we help us to make it is make file use tab instead space and make install : making artifacts : all generated by the code trying to be smart not rebuild all thing that's build the command is make and target which references bash script make file is portable\n\n\nAlternatives to make as task runners : just and pyinvoke , Bash\n\n* just https://github.com/casey/just\n* pyinvoke https://www.pyinvoke.org/\n* Bash use Taskfile for AdrianCooney : https://github.com/adriancooney/Taskfile use chmod +x run.sh to render exceutable file track set -e or set -ex help you to debug when to want to install anything you can define use makefile as interface over your runsh script\n* Make you key or credential in security way for exemple if you use .github.com/ethOizzle/shhgit you can views all keys access free in repo public : it's not recommanded because commit is immutable don't send sensitive : best managing secrets in productions : environnement variables\n\n\n\n# Github actions continuous delivery\n\nOne kind of event is pull requests which have activity types : pull_request Listeners\nTriggers we have :\nReact to pull request event : premerge\n\npostmerge and push event\n\nGithub action context and secret management : value to access anytime in workflow\n\nIn the settings in github section secrets and variables\nRepository secrets is what we interest for\n\nAccess context in workflow : use github expressions : dollard sign and double bracket\ndocumentation of contexts in github actions : # https://docs.github.com/en/actions/learn-github-actions/contexts#example-printing-context-information-to-the-log\nDifferent betweens variables : access a plain test value printed and secrets are not unprinted :\n\n\nif statements checks in publish yaml file : docs.github.com/en/actions/learn-github-actions/contexts#\nafter deployment sucessfull in prod we can tag this\n\n\n# speed up your workfows or pipeline ci/cd use dependencies caching : githubactions cahing\n\n\n[github.com/actions/cache](https://github.com/actions/cache/blob/main/examples.md#simple-example)\n\n\n- uses: actions/cache@v3\n with:\n path: ~/.cache/pip\n key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt' , '**/project.toml') }}\n restore-keys: |\n ${{ runner.os }}-pip-\n\nYou can set the cache all the emplacement you had pip install\n\npass artifacts (upload and downloads) between jobs : github.com/actions/download-artifact\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "My package description",
"version": "0.0.10",
"project_urls": null,
"split_keywords": [
"one",
"two"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "0bb60a43a01a4d54eb9700a948242e137970d79123e34b92e4822a3e70892d96",
"md5": "28204c888bd58495acd9514d1bc570e0",
"sha256": "58ba12ea07392cf92eae9bfd9908295c263f0b77b5e3c4b175fcbbb50455997f"
},
"downloads": -1,
"filename": "packaging_coursdemo-0.0.10-py3-none-any.whl",
"has_sig": false,
"md5_digest": "28204c888bd58495acd9514d1bc570e0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 9742,
"upload_time": "2024-02-23T15:25:03",
"upload_time_iso_8601": "2024-02-23T15:25:03.254473Z",
"url": "https://files.pythonhosted.org/packages/0b/b6/0a43a01a4d54eb9700a948242e137970d79123e34b92e4822a3e70892d96/packaging_coursdemo-0.0.10-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "4baca9d2dcb7f20c0e0e1add273416add8b0ca8434bad0f398beb5453e47c3ef",
"md5": "fb2434657b27007e1f7601cc736adc12",
"sha256": "310490fe3784ebe1f41f441f36417c587e1196df05c96473f07909f846a5e788"
},
"downloads": -1,
"filename": "packaging-coursdemo-0.0.10.tar.gz",
"has_sig": false,
"md5_digest": "fb2434657b27007e1f7601cc736adc12",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 9242,
"upload_time": "2024-02-23T15:25:05",
"upload_time_iso_8601": "2024-02-23T15:25:05.063491Z",
"url": "https://files.pythonhosted.org/packages/4b/ac/a9d2dcb7f20c0e0e1add273416add8b0ca8434bad0f398beb5453e47c3ef/packaging-coursdemo-0.0.10.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-23 15:25:05",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "packaging-coursdemo"
}
