# pymsbuild
This is a PEP 517 backend for building packages via MSBuild or `dotnet build`.
# Configuration file
The file is named `_msbuild.py`, and is executed by running `python -m pymsbuild`.
The package definition specifies all the files that end up in the released packages.
```python
from pymsbuild import *
METADATA = {
"Metadata-Version": "2.1",
"Name": "package",
"Version": "1.0.0",
"Author": "My Name",
"Author-email": "myemail@example.com",
"Description": File("README.md"),
"Description-Content-Type": "text/markdown",
"Classifier": [
"Development Status :: 3 - Alpha",
"Intended Audience :: Developers",
"Programming Language :: Python :: 3.9",
],
}
PACKAGE = Package(
"my_package",
PyFile(r"my_package\*.py"),
PydFile(
"_accelerator",
CSourceFile(r"win32\*.c"),
IncludeFile(r"win32\*.h"),
),
Package(
"subpackage",
PyFile(r"subpackage\*.py"),
),
)
```
Note that subpackages _must_ be specified as a `Package` element, as the
nesting of `Package` elements determines the destination path. Otherwise you
will find all of your files flattened. Recursive wildcards are supported, however,
be aware that it is not always intuitive how the paths are going to be remapped.
Also note that without a `source=` named argument, all source paths are
relative to the configuration file.
# pyproject.toml file
You will need this file in order for `pip` to build your sdist, but otherwise it's
generally easier and faster to use `pymsbuild` directly.
```
[build-system]
requires = ["pymsbuild"]
build-backend = "pymsbuild"
```
On Windows, a [Visual Studio](https://www.visualstudio.com) installation will be
required. It will be automatically detected, or the path to MSBuild can be
provided as the `MSBUILD` environment variable.
On other platforms, the [.NET SDK](https://dotnet.microsoft.com/download) will be
required. The `dotnet` command must be available on `PATH` or specified as the
`MSBUILD` environment variable.
If you have additional requirements for building either sdist or wheels, add
them as `BuildSdistRequires` or `BuildWheelRequires` values in `METADATA`. They
will be parsed after `init_METADATA` and/or `init_PACKAGE` have been called, so
may be modified by these functions.
## [project] table support
There is no support for the
[`[project]`](https://packaging.python.org/en/latest/specifications/pyproject-toml/#declaring-project-metadata-the-project-table)
table at this time. All metadata that is written into the final distribution
files comes from your `_msbuild.py` file.
However, the `pyproject.toml` included in sdists is a direct copy of the one
from the root of your project. Other than the project table, sdists have no
predictable metadata for analysis tools to use, so if you want your project to
provide that metadata, feel free to list it in the `pyproject.toml` as well as
in your `_msbuild.py` (remembering to mark as
[dynamic](https://packaging.python.org/en/latest/specifications/pyproject-toml/#dynamic)
anything that is updated by your build process).
A future release may automatically use `_msbuild.py` metadata to fill out
missing fields in a `pyproject.toml` project table, and `pymsbuild init` may use
the project table to initialise the configuration file. However, at this point,
both files are totally independent and the configuration file is the canonical
source of metadata.
# Usage
## Rebuild the current project in-place.
```
python -m pymsbuild
```
## Interactively generate the `_msbuild.py` file with project spec.
(Or at least, it will, once implemented.)
```
python -m pymsbuild init
```
## Build the project and output an sdist
```
python -m pymsbuild sdist
```
Output is put into `dist` by default, but can be overridden with `--dist-dir`
(`-d`).
## Build the project and output a wheel
```
python -m pymsbuild wheel
```
Output is put into `dist` by default, but can be overridden with `--dist-dir`
(`-d`).
## Clean any recent builds
```
python -m pymsbuild clean
```
# Advanced Examples
## Dynamic METADATA
Metadata may be dynamically generated, either on import or with the
`init_METADATA` function. This function is called and must either
return the metadata dict to use, or update `METADATA` directly.
However, if a `PKG-INFO` file is found adjacent to the configuration
file, it will be used verbatim. Sdist generation adds this file, so all
metadata is static from that point onward. `init_METADATA` is not
called in this case.
```python
from pymsbuild import *
METADATA = {
"Metadata-Version": "2.1",
"Name": "package",
"Version": os.getenv("VERSION", "1.0.0"),
"Author": "My Name",
"Author-email": "myemail@example.com",
"Description": File("README.md"),
"Description-Content-Type": "text/markdown",
"Classifier": [
"Development Status :: 3 - Alpha",
"Intended Audience :: Developers",
"Programming Language :: Python :: 3.9",
],
}
def init_METADATA():
if os.getenv("BUILD_BUILDNUMBER"):
METADATA["Version"] = f"1.0.{os.getenv('BUILD_BUILDNUMBER', '')}"
# Updated METADATA directly, so no need to return anything
```
Also see the earlier section regarding the `pyproject.toml` project table (and
the fact that it is not used by `pymsbuild`, but will be added to your sdist
without modification).
## Sdist metadata (PEP 621)
The `[project]` table in your
[`pyproject.toml`](https://packaging.python.org/en/latest/specifications/pyproject-toml/)
file in your sdist is expected to accurately reflect the metadata that your
final wheels will contain. This allows tools that process sdists to display
accurate information about your package, and sometimes to process dependencies,
without having to perform a full build.
`pymsbuild` does not use the `[project]` table by default, preferring to use
[core metadata](https://packaging.python.org/en/latest/specifications/core-metadata/)
directly. A `pyproject.toml` file is required to specify the build system, and
will be automatically included in your sdists. However, the `PyprojectTomlFile`
type allows you to customise this file using your existing metadata.
Omitting a `PyprojectTomlFile` entry entirely is equivalent to specifying a
`SourceFile("pyproject.toml")` entry, which will include your project's existing
file directly in the sdist without modifying it. Adding a `PyprojectTomlFile`
will generate a new file for your sdist.
This example will generate a `[build-system]` table that requires at least the
same version of `pymsbuild` as was used for the sdist, and will fill in the
`[project]` table with known keys from `METADATA`.
```python
PACKAGE = Package(
"my_package",
PyprojectTomlFile(),
)
def init_PACKAGE(tag):
PACKAGE.find("pyproject.toml").from_metadata(METADATA)
```
Rather than, or as well as, calling `from_metadata`, named arguments can be used
to specify the exact contents. `from_metadata` does not overwrite values
specified in this way.
```python
PACKAGE = Package(
"my_package",
PyprojectTomlFile(name="my_package", version="1.0.0"),
)
```
Providing a source file name will keep all sections other than `[project]` from
the specified file, and generate `[project]` as usual. The file can have any
name at all, but the generated file is always going to be `pyproject.toml`.
```python
PACKAGE = Package(
"my_package",
PyprojectTomlFile("sdist-pyproject.toml"),
)
```
Finally, an existing `pyproject.toml` file can be used to update your metadata
(for known fields). This may require adding `tomli` as a build dependency for
runtimes that do not include `tomllib`.
```python
def init_METADATA():
PyprojectTomlFile.update_metadata(METADATA)
# default args: file="pyproject.toml", overwrite=False
```
If you are specifying any metadata in METADATA, or modifying it in any way,
remember to also call `from_metadata` to ensure the generated file is correct.
A complete example that preserves non-`[project]` sections from the original
file, uses its `[project]` section as canonical, and performs updates at build
time may look like this:
```python
METADATA = {
# Use at least version 2.2 to indicate the [project] table is valid
"Metadata-Version": "2.2",
}
PACKAGE = Package(
"my_package",
PyprojectTomlFile("pyproject.toml"),
)
def init_METADATA():
PyprojectTomlFile.update_metadata(METADATA)
METADATA["Version"] = _calculate_version()
def init_PACKAGE(tag):
PACKAGE.find("pyproject.toml").from_metadata(METADATA)
```
Remember that `pymsbuild` uses `METADATA` as its source of information, and so
your packages will not build correctly if you do not keep it updated.
In most cases, you will not need to specify `METADATA['Dynamic']` as there is no
way to modify metadata during wheel builds. However, if you have found a way to
do it, then you should specify those fields manually. Fields that update during
`init_METADATA` do not need to be listed as dynamic.
## Separate packages
Packages are just Python objects, so they may be kept in variables and
used later. They also expose a `members` attribute, which is a list, so
that members can be added or inserted later, as well as `find`, `findall` and
`insert` methods to help (see the **Dynamic Packages** section below).
After the entire module is executed, the package in `PACKAGE` is the
only one used to generate output.
```python
P1 = Package(
"submodule",
PyFile(r"src\submodule\__init__.py")
)
P2 = Package(
"submodule_2",
PyFile(r"src\submodule_2\__init__.py")
)
PACKAGE = Package("my_package", P1)
PACKAGE.members.append(P2)
```
## Anonymous packages
To install files directly into the target location (often ``site-packages``),
use a top-level package with empty name. Other packages may be nested within
this package as normal.
```python
PACKAGE = Package(
"",
File("module.pth"),
Package("module", ...),
)
```
## Custom dist-info files
To install files into the generated `.dist-info` directory, specify the
``IncludeInDistinfo=True`` option. This will move the file in the built wheel,
while reading it from its usual location. It will be included in the sdist at
its source location.
```python
PACKAGE = Package(
"module",
File("entry_points.txt", IncludeInDistinfo=True),
)
```
## Wildcard handling
Files can be added recursively using wildcard operators. These are
evaluated at generation time by `pymsbuild` and not by MSBuild/
`dotnet build`, as it allows greater control over target names.
```python
PACKAGE = Package(
"my_package",
# All .py files, relative to the 'src' directory
PyFile(r"**\*.py"),
# All license files, if any, with path separators converted to '-'
File(r"**\license*", flatten="-", allow_none=True),
# All .bin files from all data directories, moved to the root
File(r"**\data\*.bin", flatten=True),
source="src"
)
```
`flatten` specifies the string sequence to replace path separators in
the name. Passing `True` indicates that only the file name should be
retained.
`allow_none` merely suppresses a build-time error when the wildcard
fails to match any files. This is usually an important problem, and
should be suppressed with care.
The `flatten` and `allow_none` properties are not written to the build
file. However, they are case-sensitive while MSBuild is not, so the
capitalised versions will be ignored for this processing and passed
through.
Final install location (also known as the element's name) are generated
from the default name (source file name or `name` argument and all
package names in the hierarchy) combined with the pattern according to
these rules:
* if the pattern contains no wildcards, the default name is preserved
* each segment from the first one containing a wildcard will be joined
to the parent of the default name
* if the pattern filename contains no wildcards, it is preserved in the
final name. Otherwise, it is replaced by matched files
These rules ensure consistency across many forms of paths, making it
reliable to use calculated absolute paths with wildcards (for example,
a package extending the build system to add its own files). To create
a directory in the destination, use a new `Package` element:
```
# Installs as 'A/__init__.py'
PACKAGE = Package("A", PyFile("B/__init__.py"))
PACKAGE = Package("A", PyFile("B/source.py", name="__init__.py"))
# Installs as 'A/*.txt'
PACKAGE = Package("A", File("B/*.txt"))
# All of these install as 'A/B/*.txt'
PACKAGE = Package("A", Package("B", File("B/*.txt")))
PACKAGE = Package("A", Package("B", File("*.txt"), source="B"))
PACKAGE = Package("A", File("*/*.txt")) # assuming no other matches
```
Specifying the `Name` metadata (as opposed to `name`, which is a
keyword argument) will override the destination name of every matched
file. This is applied before flattening, and so will preserve the
relative path in whatever form is specified by `flatten`. To bypass
this additional processing and use the name as an MSBuild literal,
wrap it in a `ConditionalValue` with no condition:
```python
PACKAGE = Package(
"my_package",
File("**/*.dat", Name=ConditionalValue("%(Filename)-1.dat")),
)
```
For more complex transforms on filename, we recommend using the
`init_PACKAGE` function described below.
## Dynamic packages
After metadata processing, if an `init_PACKAGE(tag=None)` function
exists it will be called with the intended platform tag. It must modify
or return `PACKAGE`. This function is called for in-place, sdist and
wheel generation, however, for sdists (and any scenario that should not
generate binaries), `tag` will be `None`. Otherwise, it will be a
string like `cp38-cp38-win32`.
```python
PACKAGE = Package(
"my_package",
PyFile(r"my_package\*.py"),
)
def init_PACKAGE(tag=None):
if tag and tag.endswith("-win_amd64"):
data_file = generate_data_amd64()
PACKAGE.members.append(File(data_file))
```
Note that all files to be included in an sdist must be referenced when
`tag` is `None`. Conditional compilation is best performed using conditions
in the package elements, rather than using `init_PACKAGE`. However, if you
are going to use `init_PACKAGE`, you should _remove_ elements rather than
adding them if they should be included in your sdist.
Files added as part of a wildcard can be removed by adding a `RemoveFile`
element. These may be added dynamically during `init_PACKAGE`, and must
appear after the element that included the files.
```python
PACKAGE = Package(
"my_package",
PyFile(r"my_package\*.py"),
RemoveFile(PyFile, r"my_package\_internal.py"),
)
```
To exclude files from a wildcard in the first place, chain the `.excluding`
method on the original element. The pattern will be evaluated in exactly the
same way as the inclusion pattern, and any paths that match will be omitted.
```python
PACKAGE = Package(
"my_package",
PyFile(r"my_package\*.py").excluding("my_package\internal*.py"),
)
```
Alternatively, a condition may be added to the file pattern to only include
files matching MSBuild style conditions. Because these will be applied to
item groups, the `%()` metadata syntax should be used to access information
for the element being added. Either the `.if_` method or the
`ConditionalValue` wrapper may be used.
```python
PACKAGE = Package(
"my_package",
PyFile(r"my_package\*.py").if_("%(Filename) != '_internal'"),
File(ConditionalValue("*.txt", condition="%(Filename.StartsWith(`internal`))")),
)
```
Package members can be located during the dynamic stage using the
`find` and `findall` functions. These take a path of member identifiers
(typically their name property) and will return those that match.
`'**'` segments are supported for recursive searches.
```python
PACKAGE = Package(
"my_package",
Package("sub1", File("license.txt")),
Package("sub2", File("license.txt")),
)
def init_PACKAGE(tag=None):
for e in PACKAGE.findall("sub*/license.txt"):
e.name = "LICENSE"
```
When inserting members, the `insert` function combines a `find` with
the insert, and supports offset and range options. In general, only
subclassed element types should insert additional elements, and only
into themselves at construction.
```python
class MyPydFile(PydFile):
def __init__(self, name, *members, **options):
super().__init__(name, *members, **options)
self.insert(
# Member path to insert before - this one is inherited from PydFile
self.CommonToolsetImports.name,
# Member to insert (in this case, an iterable)
[Property(PROP1, VALUE1), Property(PROP2, VALUE2)],
# Offset it by 1, so inserts after the found element (default 0)
offset = 1,
# Iterate over the insertion value; otherwise insert it as-is
range = True
)
```
## Source offsets
If you keep your source in a `src` folder (recommended), provide the
`source=` argument to `Package` in order to properly offset filenames.
Because it is a named argument, it must be provided last.
This is important for sdist generation and in-place builds, which need
to match package layout with source layout. Simply prefixing filename
patterns with the additional directory is not always sufficient.
Note that this will also offset subpackages, and that subpackages may
include additional `source` arguments. However, it only affects
sources, while the package name (the first argument) determines where
in the output the package will be located. In-place builds will create
new folders in your source tree if it does not match the final
structure.
```python
PACKAGE = Package(
"my_package",
PyFile(r"my_package\__init__.py"),
source="src",
)
```
## Project file override
Both `Package` and `PydFile` types generate MSBuild project files and
execute them as part of build, including sdists. For highly customised
builds, this generation may be overridden completely by specifying the
`project_file` named argument. All members are then ignored.
By doing this, you take full responsibility for a valid build,
including providing a number of undocumented and unsupported targets.
Recommendations:
* lock your `pymsbuild` dependency to a specific version in `pyproject.toml`
* generate project files first and modify, rather than writing by hand
(pass `--temp-dir` to specify the path where they will be generated)
* read the `pymsbuild` source code, especially the `targets` folder
* consider contributing/requesting your feature or developing an extension
(see [pymsbuild-winui](https://github.com/zooba/pymsbuild-winui) and
[pymsbuild-rust](https://github.com/zooba/pymsbuild-rust) for examples)
```python
PACKAGE = Package(
"my_package",
PydFile("_accelerator", project_file=r"src\accelerator.vcxproj")
)
```
## Compiler/linker arguments
Rather than overriding the entire project file, there are a number of
ways to inject arbitrary values into a project. These require
familiarity with MSBuild files and the toolsets you are building with.
The `Property` element inserts a `<PropertyGroup>` with the value you
specifiy at the position in the project the element appears.
Note that project files also interpret (most) named arguments as
properties, so the two properties shown here are equivalent.
```python
PYD = PydFile(
"module",
Property("WindowsSdkVersion", "10.0.18363.0"),
...
# Alternative to Property(), but named arguments must be
# specified last, so using Property() may be more readable
WindowsSdkVersion="10.0.18363.0",
)
```
The `ItemDefinition` element inserts an `<ItemDefinitionGroup>` with
the type and metadata you specify at the position in the project the
element appears. These generally apply metadata to all subsequent items
of that type.
```python
PYD = PydFile(
"module",
ItemDefinition("ClCompile", PreprocessorDefinitions="Py_LIMITED_API"),
...
)
```
The `ConditionalValue` item may wrap any element value to add
conditions or concatenate the value. This may also be used on source
arguments for file elements.
```python
...
Property("Arch", ConditionalValue("x86", condition="$(Platform) == 'Win32'")),
Property("Arch", ConditionalValue("x64", if_empty=True)),
...
ItemDefinition(
"ClCompile",
AdditionalIncludeDirectories=
ConditionalValue(INCLUDES + ";", prepend=True),
ProprocessorDefinitions=
ConditionalValue(";Py_LIMITED_API", append=True),
),
...
```
The `Prepend` shortcut may be used to reduce the amount of text for
property values. Remember to include the appropriate separator. It is
usually a semicolon.
```python
...
ItemDefinition(
"ClCompile",
AdditionalIncludeDirectories=Prepend(INCLUDES + ";"),
ProprocessorDefinitions=Prepend("Py_LIMITED_API;"),
),
...
```
`ConditionalValue` may also be used to dynamically update values in the
`init_PACKAGE` function, allowing you to keep the structure mostly
static but insert values from the current `METADATA` (which is fully
evaluated by the time `init_PACKAGE` is called). This saves having to
access internal members of other types in order to replace literal
values.
Assign a `ConditionalValue` to a variable without specifying any
condition, then use the variable in a later `Property` element and
modify its `value` attribute in `init_PACKAGE`.
```python
VER = ConditionalValue("1.0.0")
PYD = PydFile(
"module",
Property("Version", VER),
CSourceFile(r"src\*.c"),
IncludeFile(r"src\*.h"),
)
def init_PACKAGE(tag):
VER.value = METADATA["Version"]
```
As a last resort, the `LiteralXml` element inserts plain text directly
into the generated file. It will be inserted as a child of the
top-level `Project` element.
```python
...
LiteralXml("<Import Project='my_props.props' />"),
...
```
## Version info for DLLs/PYDs
**Platform: Windows**
To embed version info into a compiled extension module, add a `VersionInfo`
element into the `PydFile`. All the fields from
https://learn.microsoft.com/en-us/windows/win32/menurc/versioninfo-resource
are available, using the names as shown in the tables (e.g.
`FILEVERSION` for the `'1,0,0,0'` fields and `FileVersion` for the string
table entry).
The recommended usage is to add a default instance into your project and then
use `init_METADATA` to find it again and update based on the final metadata.
```
PACKAGE = Package(
"package",
PydFile("mod1", VersionInfo()),
PydFile("mod2", VersionInfo()),
)
def init_METADATA():
# Update our metadata
METADATA["Version"] = calculate_current_version()
# Ensure built modules reflect these updates
for vi in PACKAGE.findall("*/VersionInfo"):
vi.from_metadata(METADATA)
```
`from_metadata` will fill in any empty fields from the set of metadata that is
passed in.
## Alternate config file
To use a configuration file other than `_msbuild.py`, specify the
`--config` (`-c`) argument or the `PYMSBUILD_CONFIG` environment
variable.
```powershell
python -m pymsbuild --config build-spec.py sdist
python -m pymsbuild --config build-spec.py wheel
# Alternatively
$env:PYMSBUILD_CONFIG = "build-spec.py"
python -m pymsbuild sdist wheel
```
Generated sdists will rename the configuration file back to
`_msbuild.py` in the package to ensure that builds work correctly.
There is no need to override the configuration file path when
building from sdists.
Note that this is different from the `PYMSBUILD_CONFIGURATION`
variable, which is used to select debug/release settings for compiled
modules.
## Cross-compiling wheels
Cross compilation may be used by overriding the wheel tag, ABI tag,
or build platform, as well as the source for Python's includes and
libraries. These all use environment variables, to ensure that the
same setting can flow through a package installer's own process.
It is also possible to permanently override the wheel tag by adding
a `'WheelTag'` metadata value, or the ABI tag by adding an `'AbiTag'`
metadata value.
The wheel tag is used for the generated wheel file, and to fill in a
missing ABI tag and platform.
The ABI tag is used for any native extension modules, and to fill in
a missing platform.
Any `*` elements in the wheel tag are filled in from other locations.
For example, specifying `*-none-any` will infer the interpreter field
from the current runtime, whil `py3-none-*` will infer the platform
from the currnet system (or a specific ABI tag).
The platform is used to determine the MSBuild target platform. It
cannot yet automatically select the correct Python libraries, and so
you will need to set `PYTHON_INCLUDES` and `PYTHON_LIBS` (or with a
`PYMSBULID_` prefix) environment variables as well to locate the
correct files.
You can override the platform toolset with the `'PlatformToolset'`
metadata value, for scenarios where this information ought to be
included in an sdist.
The set of valid platforms for auto-generated `.pyd` project files are
hard-coded into `pymsbuild` and are currently `Win32`, `x64`, `ARM` and
`ARM64`. Custom project files may use whatever they like. These
platforms should behave properly cross-platform, though in general only
`x64` and `ARM64` are supported.
```powershell
# Directly specify the resulting wheel tag
# This is used for the wheel filename/metadata
$env:PYMSBUILD_WHEEL_TAG = "py38-cp38-win_arm64"
# Directly set the ABI tag (or else taken from wheel tag)
# This is used for extension module filenames
$env:PYMSBUILD_ABI_TAG = "cp38-win_arm64"
# Specify the Python platform (or else taken from ABI tag)
# This is used for MSBuild options
$env:PYMSBUILD_PLATFORM = "win_arm64"
# Specify the paths to ARM64 headers and libs
$env:PYTHON_INCLUDES = "$pyarm64\Include"
$env:PYTHON_LIBS = "$pyarm64\libs"
# If necessary, specify an alternate C++ toolset
$env:PLATFORMTOOLSET = "Intel C++ Compiler 19.1"
```
## Cython
Cython support is available from the `pymsbuild.cython` module.
```python
from pymsbuild import PydFile, ItemDefinition
from pymsbuild.cython import CythonIncludeFile, CythonPydFile, PyxFile
PACKAGE = CythonPydFile(
"cython_module",
ItemDefinition("PyxCompile", IncludeDirs=PYD_INCLUDES),
CythonIncludeFile("mod.pxd"),
PyxFile("mod.pyx"),
)
```
The `CythonPydFile` type derives from the regular `PydFile` and also
generates a C++ project, so all options that would be available there may
also be used.
The `PyxCompile.IncludeDirs` metadata specifies search paths for Cython
headers (`*.pxd`). You may also need to specify
`ClCompile.AdditionalIncludeDirectories` for any C/C++ headers.
## Two-Step Builds
By default, the `sdist` and `wheel` commands will perform the entire
process in a single invocation. However, sometimes there are build steps
that must be manually performed between compilation and packaging.
To run the build in two stages, invoke as normal, but add the
`--layout-dir` argument followed by a directory. The package will be
laid out in this directory so that you can perform any extra processing.
Later, use the `pack` command and specify the `--layout-dir` again. If
you have added new files into the layout directory, specify each with an
`--add` option (filenames starting with `@` are treated as
newline-separated, UTF-8 encoded text files listing each new file). These
paths may be absolute or relative to the layout directory, but only files
located within the layout directory will be included.
All other options are retained from the original invocation.
```
python -m pymsbuild sdist --layout-dir tmp
# Generate additional metadata in tmp/EXTRA.txt
python -m pymsbuild pack --layout-dir tmp --add tmp/EXTRA.txt
# List many additional files in build/TO_ADD.txt
python -m pymsbuild pack --layout-dir tmp --add @build/TO_ADD.txt
```
# Experimental Features
## DLL Packing
**Experimental. (POSIX is _very_ experimental)**
DLL Packing is a way to compile a complete Python package (`.py` source
and resource files) into an extension module. It is basically equivalent
to packing in a ZIP file, except that additional native code may also be
included (though not an entire native module), and the whole file may be
cryptographically signed and validated by the operating system.
`DllPackage` is a drop-in substitute for the `Package` type. It will
generate a native extension module the same as the `PydFile` type,
but only includes Python source and resource files.
```python
from pymsbuild import *
from pymsbuild.dllpack import *
PACKAGE = DllPackage(
"packed_package",
PyFile("__init__.py"),
File("data.txt"),
...
)
```
An entire existing library, such as `cryptography` could be packed
like this:
```python
from pymsbuild import *
from pymsbuild.dllpack import *
MODULE_TO_PACK = "cryptography"
from importlib.util import find_spec
spec = find_spec(MODULE_TO_PACK)
if not spec:
raise RuntimeError(f"{MODULE_TO_PACK} must be installed")
PACKAGE = DllPackage(
MODULE_TO_PACK,
PyFile("**/*.py"),
PydRedirect("**/*.pyd"),
source = spec.submodule_search_locations[0],
)
```
See the `azure-pack` sample in our source repository for a more
complete example.
`DllPackage` is a subclass of `PydFile`, and so all logic or elements
by that type are also available. `ClCompile` elements will be compiled
and linked into the output and functions may be exposed in the root of
the package using the `Function` element.
```c
// extra.c
PyObject *my_func(PyObject *, PyObject *args, PyObject **kwargs) {
...
}
```
```python
PACKAGE = DllPackage(
"packed_package",
PyFile("__init__.py"),
CSourceFile("extra.c"),
CFunction("my_func"),
...
)
```
### Packed module names
By default, the packed DLL contents are presumed to be nested within
the overall structure. This is important because full module names must
be embedded into the DLL. As a result, when the example below is built,
it will create a `root/` directory containing `packed.pyd` (or
equivalent) and enables `import root.packed.submodule`.
```python
PACKAGE = Package("root",
DllPackage("packed",
PyFile("src/packed/__init__.py"),
PyFile("src/packed/submodule.py"),
)
)
# After building:
import root.packed.submodule
```
If part of the package name is not importable (specifically,
`str.isidentifier()` is `False`), then the name will be truncated after
that part.
There are a few properties that can affect what is built. The first is
the `TargetName` property, which will change the name of the compiled
file. For example, adding `TargetName="root.packed"` or
`Property("TargetName", "root.packed")` to the `DllPackage` item will
generate `root.packed.pyd` instead, allowing the module to be imported
as `root.packed` without being in a `root/` directory.
Second is the `RootNamespace` property, which overrides the name used
within the packed module to identify submodules. Changing this alone is
rarely useful, and the default (matching the name of the module) is
recommended unless you have modified other aspects of the build.
If this is overridden with a named argument, it will replace the full
name used within the packed DLL. If overridden with a `Property`
element, it will replace the name but retain the parent's namespace.
### Nested extension modules
To allow referencing other extension modules that would normally be
nested within the module, add a `PydRedirect` element and reference the
extension module. The filename does not have to match the original
name, or even need to be a normally importable name, as it will be
passed directly to the module loader. The file will be included in your
wheel in the expected location (alongside the packed DLL). Wildcards
are supported.
```python
PACKAGE = DllPackage(
"packed",
PydRedirect(source="packed/nested.pyd", name="packed-nested.pyd"),
...
)
```
Other `PydFile` modules may be nested inside the `DllPackage`, which
will automatically add a redirect, as well as building the module.
The nested module will be built using the name specified and sit
adjacent to the packed module, but should be imported via the
packed module.
The `ImportName` metadata may be specified on either a `PydRedirect` or
a `PydFile` to specify the name that must be used to import the module.
Redirected extension modules do not need to have an importable name
when `ImportName` is specified. You might include an invalid character
in the filename to ensure the module is not importable directly.
When specifying `ImportName`, the name of the packed DLL must be used
as the first part.
```python
PACKAGE = DllPackage(
"packed",
PydRedirect("module/nested.pyd", ImportName="packed.nested"),
...
)
```
### Encryption
To encrypt your content using symmetric AES encryption, provide the
name of the environment variable holding your key as the
`EncryptionKeyVariable` option. The key will need to be a valid size
(usually 16, 24 or 32 bytes) when encoded to UTF-8 or decoded from
base 64. Base 64 keys should start with `base64:`.
The same variable will need to be set when importing the module. It is
your responsibility to protect the key! The benefit of this encryption
is best realised when you avoid storing the key to disk. That way, an
attacker who steals a copy of your module is unlikely to have access to
the key. An attacker with access to a running copy of your module will
be able to easily extract the key.
```python
PACKAGE = DllPackage(
"package",
...,
EncryptionKeyVariable="MY_KEY_VARIABLE"
)
```
```powershell
> $env:MY_KEY_VARIABLE="base64:MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDA="
> python -m pymsbuild
> del env:\MY_KEY_VARIABLE
> python -c "import package"
ImportError: Module cannot be decrypted
```
Redirected or nested extension modules are not encrypted.
## Cross-platform builds
**Experimental.**
With the [.NET SDK](https://dotnet.microsoft.com/download) installed,
`pymsbuild` is able to run builds on platforms other than Windows.
The `dotnet` command must be available on `PATH` or specified as the
`MSBUILD` environment variable.
In general, no platform-specific modifications to a build script are
required. Cython and pyd builds are transparently mapped to the target
system. To run build-time actions for specific platforms, add them to
`init_PACKAGE` and check the tag argument to determine the target
platform.
When building native components on POSIX, a `python3-config` script is
needed to determine compilation options. By default, only the location
adjacent to the running interpreter is checked. This may be overridden
by setting the `PYTHON_CONFIG` variable to the preferred command.
## Custom entry point
**Experimental.**
To generate an executable that will launch your application, include the
`pymsbuild.entrypoint` module and use an `Entrypoint` definition.
```python
from pymsbuild import *
from pymsbuild.entrypoint import *
PACKAGE = Package(
"demo",
Entrypoint(
"run", # generate run.exe
"app", # import app
"main", # app.main()
Icon("app.ico"),
# Search paths for the entry point to use
SearchPath("."),
SearchPath("stdlib.zip"),
# Include a copy of Python (default: True)
IncludePythonRuntime=True,
# Use the embeddable distro (default: True)
PythonEmbeddable=True,
# Also include python.exe (default: False)
PythonExecutables=False,
# Rename pythonXY.zip to stdlib.zip (default: True)
PythonRuntimeRenameStdlibZip=True,
),
Package(
"app",
PyFile("app/__init__.py"),
),
)
```
Building this definition will create a `demo` directory containing
`run.exe`, `app.py` and a copy of the Python embeddable runtime, making
it an entirely standalone and redistributable application.
Set `IncludePythonRuntime` to `False` to omit the runtime. The generated
executable assumes that it will be able to load the version of Python
used to build at runtime, so you will need to include it some other way.
Use `SearchPath` items to specify directories to search for Python
modules at runtime. These are the only directories that will be
searched, as Python will be loaded in isolated mode. They are relative
to the entrypoint and will be resolved when executing.
Raw data
{
"_id": null,
"home_page": null,
"name": "pymsbuild",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "build, pep-517, msbuild, packaging, cython",
"author": "Steve Dower",
"author_email": "steve.dower@python.org",
"download_url": "https://files.pythonhosted.org/packages/31/ad/9f48b2fae584c654b1c1be750abd34d65972cd83a32f696ed6650a41bf1e/pymsbuild-1.1.4.tar.gz",
"platform": null,
"description": "# pymsbuild\n\nThis is a PEP 517 backend for building packages via MSBuild or `dotnet build`.\n\n# Configuration file\n\nThe file is named `_msbuild.py`, and is executed by running `python -m pymsbuild`.\n\nThe package definition specifies all the files that end up in the released packages.\n\n```python\nfrom pymsbuild import *\n\nMETADATA = {\n \"Metadata-Version\": \"2.1\",\n \"Name\": \"package\",\n \"Version\": \"1.0.0\",\n \"Author\": \"My Name\",\n \"Author-email\": \"myemail@example.com\",\n \"Description\": File(\"README.md\"),\n \"Description-Content-Type\": \"text/markdown\",\n \"Classifier\": [\n \"Development Status :: 3 - Alpha\",\n \"Intended Audience :: Developers\",\n \"Programming Language :: Python :: 3.9\",\n ],\n}\n\nPACKAGE = Package(\n \"my_package\",\n PyFile(r\"my_package\\*.py\"),\n PydFile(\n \"_accelerator\",\n CSourceFile(r\"win32\\*.c\"),\n IncludeFile(r\"win32\\*.h\"),\n ),\n Package(\n \"subpackage\",\n PyFile(r\"subpackage\\*.py\"),\n ),\n)\n```\n\nNote that subpackages _must_ be specified as a `Package` element, as the\nnesting of `Package` elements determines the destination path. Otherwise you\nwill find all of your files flattened. Recursive wildcards are supported, however,\nbe aware that it is not always intuitive how the paths are going to be remapped.\n\nAlso note that without a `source=` named argument, all source paths are\nrelative to the configuration file.\n\n# pyproject.toml file\n\nYou will need this file in order for `pip` to build your sdist, but otherwise it's\ngenerally easier and faster to use `pymsbuild` directly.\n\n```\n[build-system]\nrequires = [\"pymsbuild\"]\nbuild-backend = \"pymsbuild\"\n```\n\nOn Windows, a [Visual Studio](https://www.visualstudio.com) installation will be\nrequired. It will be automatically detected, or the path to MSBuild can be\nprovided as the `MSBUILD` environment variable.\n\nOn other platforms, the [.NET SDK](https://dotnet.microsoft.com/download) will be\nrequired. The `dotnet` command must be available on `PATH` or specified as the\n`MSBUILD` environment variable.\n\nIf you have additional requirements for building either sdist or wheels, add\nthem as `BuildSdistRequires` or `BuildWheelRequires` values in `METADATA`. They\nwill be parsed after `init_METADATA` and/or `init_PACKAGE` have been called, so\nmay be modified by these functions.\n\n## [project] table support\n\nThere is no support for the\n[`[project]`](https://packaging.python.org/en/latest/specifications/pyproject-toml/#declaring-project-metadata-the-project-table)\ntable at this time. All metadata that is written into the final distribution\nfiles comes from your `_msbuild.py` file.\n\nHowever, the `pyproject.toml` included in sdists is a direct copy of the one\nfrom the root of your project. Other than the project table, sdists have no\npredictable metadata for analysis tools to use, so if you want your project to\nprovide that metadata, feel free to list it in the `pyproject.toml` as well as\nin your `_msbuild.py` (remembering to mark as\n[dynamic](https://packaging.python.org/en/latest/specifications/pyproject-toml/#dynamic)\nanything that is updated by your build process).\n\nA future release may automatically use `_msbuild.py` metadata to fill out\nmissing fields in a `pyproject.toml` project table, and `pymsbuild init` may use\nthe project table to initialise the configuration file. However, at this point,\nboth files are totally independent and the configuration file is the canonical\nsource of metadata.\n\n# Usage\n\n## Rebuild the current project in-place.\n\n```\npython -m pymsbuild\n```\n\n## Interactively generate the `_msbuild.py` file with project spec.\n\n(Or at least, it will, once implemented.)\n\n```\npython -m pymsbuild init\n```\n\n## Build the project and output an sdist\n\n```\npython -m pymsbuild sdist\n```\n\nOutput is put into `dist` by default, but can be overridden with `--dist-dir`\n(`-d`).\n\n## Build the project and output a wheel\n\n```\npython -m pymsbuild wheel\n```\n\nOutput is put into `dist` by default, but can be overridden with `--dist-dir`\n(`-d`).\n\n## Clean any recent builds\n\n```\npython -m pymsbuild clean\n```\n\n# Advanced Examples\n\n## Dynamic METADATA\n\nMetadata may be dynamically generated, either on import or with the\n`init_METADATA` function. This function is called and must either\nreturn the metadata dict to use, or update `METADATA` directly.\n\nHowever, if a `PKG-INFO` file is found adjacent to the configuration\nfile, it will be used verbatim. Sdist generation adds this file, so all\nmetadata is static from that point onward. `init_METADATA` is not\ncalled in this case.\n\n```python\nfrom pymsbuild import *\n\nMETADATA = {\n \"Metadata-Version\": \"2.1\",\n \"Name\": \"package\",\n \"Version\": os.getenv(\"VERSION\", \"1.0.0\"),\n \"Author\": \"My Name\",\n \"Author-email\": \"myemail@example.com\",\n \"Description\": File(\"README.md\"),\n \"Description-Content-Type\": \"text/markdown\",\n \"Classifier\": [\n \"Development Status :: 3 - Alpha\",\n \"Intended Audience :: Developers\",\n \"Programming Language :: Python :: 3.9\",\n ],\n}\n\ndef init_METADATA():\n if os.getenv(\"BUILD_BUILDNUMBER\"):\n METADATA[\"Version\"] = f\"1.0.{os.getenv('BUILD_BUILDNUMBER', '')}\"\n # Updated METADATA directly, so no need to return anything\n```\n\nAlso see the earlier section regarding the `pyproject.toml` project table (and\nthe fact that it is not used by `pymsbuild`, but will be added to your sdist\nwithout modification).\n\n## Sdist metadata (PEP 621)\n\nThe `[project]` table in your\n[`pyproject.toml`](https://packaging.python.org/en/latest/specifications/pyproject-toml/)\nfile in your sdist is expected to accurately reflect the metadata that your\nfinal wheels will contain. This allows tools that process sdists to display\naccurate information about your package, and sometimes to process dependencies,\nwithout having to perform a full build.\n\n`pymsbuild` does not use the `[project]` table by default, preferring to use\n[core metadata](https://packaging.python.org/en/latest/specifications/core-metadata/)\ndirectly. A `pyproject.toml` file is required to specify the build system, and\nwill be automatically included in your sdists. However, the `PyprojectTomlFile`\ntype allows you to customise this file using your existing metadata.\n\nOmitting a `PyprojectTomlFile` entry entirely is equivalent to specifying a\n`SourceFile(\"pyproject.toml\")` entry, which will include your project's existing\nfile directly in the sdist without modifying it. Adding a `PyprojectTomlFile`\nwill generate a new file for your sdist.\n\nThis example will generate a `[build-system]` table that requires at least the\nsame version of `pymsbuild` as was used for the sdist, and will fill in the\n`[project]` table with known keys from `METADATA`.\n\n```python\nPACKAGE = Package(\n \"my_package\",\n PyprojectTomlFile(),\n)\n\ndef init_PACKAGE(tag):\n PACKAGE.find(\"pyproject.toml\").from_metadata(METADATA)\n```\n\nRather than, or as well as, calling `from_metadata`, named arguments can be used\nto specify the exact contents. `from_metadata` does not overwrite values\nspecified in this way.\n\n```python\nPACKAGE = Package(\n \"my_package\",\n PyprojectTomlFile(name=\"my_package\", version=\"1.0.0\"),\n)\n```\n\nProviding a source file name will keep all sections other than `[project]` from\nthe specified file, and generate `[project]` as usual. The file can have any\nname at all, but the generated file is always going to be `pyproject.toml`.\n\n```python\nPACKAGE = Package(\n \"my_package\",\n PyprojectTomlFile(\"sdist-pyproject.toml\"),\n)\n```\n\nFinally, an existing `pyproject.toml` file can be used to update your metadata\n(for known fields). This may require adding `tomli` as a build dependency for\nruntimes that do not include `tomllib`.\n\n```python\ndef init_METADATA():\n PyprojectTomlFile.update_metadata(METADATA)\n # default args: file=\"pyproject.toml\", overwrite=False\n```\n\nIf you are specifying any metadata in METADATA, or modifying it in any way,\nremember to also call `from_metadata` to ensure the generated file is correct.\nA complete example that preserves non-`[project]` sections from the original\nfile, uses its `[project]` section as canonical, and performs updates at build\ntime may look like this:\n\n```python\nMETADATA = {\n # Use at least version 2.2 to indicate the [project] table is valid\n \"Metadata-Version\": \"2.2\",\n}\n\nPACKAGE = Package(\n \"my_package\",\n PyprojectTomlFile(\"pyproject.toml\"),\n)\n\ndef init_METADATA():\n PyprojectTomlFile.update_metadata(METADATA)\n METADATA[\"Version\"] = _calculate_version()\n\ndef init_PACKAGE(tag):\n PACKAGE.find(\"pyproject.toml\").from_metadata(METADATA)\n```\n\nRemember that `pymsbuild` uses `METADATA` as its source of information, and so\nyour packages will not build correctly if you do not keep it updated.\n\nIn most cases, you will not need to specify `METADATA['Dynamic']` as there is no\nway to modify metadata during wheel builds. However, if you have found a way to\ndo it, then you should specify those fields manually. Fields that update during\n`init_METADATA` do not need to be listed as dynamic.\n\n## Separate packages\n\nPackages are just Python objects, so they may be kept in variables and\nused later. They also expose a `members` attribute, which is a list, so\nthat members can be added or inserted later, as well as `find`, `findall` and\n`insert` methods to help (see the **Dynamic Packages** section below).\n\nAfter the entire module is executed, the package in `PACKAGE` is the\nonly one used to generate output.\n\n```python\nP1 = Package(\n \"submodule\",\n PyFile(r\"src\\submodule\\__init__.py\")\n)\n\nP2 = Package(\n \"submodule_2\",\n PyFile(r\"src\\submodule_2\\__init__.py\")\n)\n\nPACKAGE = Package(\"my_package\", P1)\nPACKAGE.members.append(P2)\n```\n\n## Anonymous packages\n\nTo install files directly into the target location (often ``site-packages``),\nuse a top-level package with empty name. Other packages may be nested within\nthis package as normal.\n\n```python\nPACKAGE = Package(\n \"\",\n File(\"module.pth\"),\n Package(\"module\", ...),\n)\n```\n\n## Custom dist-info files\n\nTo install files into the generated `.dist-info` directory, specify the\n``IncludeInDistinfo=True`` option. This will move the file in the built wheel,\nwhile reading it from its usual location. It will be included in the sdist at\nits source location.\n\n```python\nPACKAGE = Package(\n \"module\",\n File(\"entry_points.txt\", IncludeInDistinfo=True),\n)\n```\n\n## Wildcard handling\n\nFiles can be added recursively using wildcard operators. These are\nevaluated at generation time by `pymsbuild` and not by MSBuild/\n`dotnet build`, as it allows greater control over target names.\n\n```python\nPACKAGE = Package(\n \"my_package\",\n # All .py files, relative to the 'src' directory\n PyFile(r\"**\\*.py\"),\n # All license files, if any, with path separators converted to '-'\n File(r\"**\\license*\", flatten=\"-\", allow_none=True),\n # All .bin files from all data directories, moved to the root\n File(r\"**\\data\\*.bin\", flatten=True),\n source=\"src\"\n)\n```\n\n`flatten` specifies the string sequence to replace path separators in\nthe name. Passing `True` indicates that only the file name should be\nretained.\n\n`allow_none` merely suppresses a build-time error when the wildcard\nfails to match any files. This is usually an important problem, and\nshould be suppressed with care.\n\nThe `flatten` and `allow_none` properties are not written to the build\nfile. However, they are case-sensitive while MSBuild is not, so the\ncapitalised versions will be ignored for this processing and passed\nthrough.\n\nFinal install location (also known as the element's name) are generated\nfrom the default name (source file name or `name` argument and all\npackage names in the hierarchy) combined with the pattern according to\nthese rules:\n\n* if the pattern contains no wildcards, the default name is preserved\n* each segment from the first one containing a wildcard will be joined\n to the parent of the default name\n* if the pattern filename contains no wildcards, it is preserved in the\n final name. Otherwise, it is replaced by matched files\n\nThese rules ensure consistency across many forms of paths, making it\nreliable to use calculated absolute paths with wildcards (for example,\na package extending the build system to add its own files). To create\na directory in the destination, use a new `Package` element:\n\n```\n# Installs as 'A/__init__.py'\nPACKAGE = Package(\"A\", PyFile(\"B/__init__.py\"))\nPACKAGE = Package(\"A\", PyFile(\"B/source.py\", name=\"__init__.py\"))\n\n# Installs as 'A/*.txt'\nPACKAGE = Package(\"A\", File(\"B/*.txt\"))\n\n# All of these install as 'A/B/*.txt'\nPACKAGE = Package(\"A\", Package(\"B\", File(\"B/*.txt\")))\nPACKAGE = Package(\"A\", Package(\"B\", File(\"*.txt\"), source=\"B\"))\nPACKAGE = Package(\"A\", File(\"*/*.txt\")) # assuming no other matches\n```\n\nSpecifying the `Name` metadata (as opposed to `name`, which is a\nkeyword argument) will override the destination name of every matched\nfile. This is applied before flattening, and so will preserve the\nrelative path in whatever form is specified by `flatten`. To bypass\nthis additional processing and use the name as an MSBuild literal,\nwrap it in a `ConditionalValue` with no condition:\n\n```python\nPACKAGE = Package(\n \"my_package\",\n File(\"**/*.dat\", Name=ConditionalValue(\"%(Filename)-1.dat\")),\n)\n```\n\nFor more complex transforms on filename, we recommend using the\n`init_PACKAGE` function described below.\n\n## Dynamic packages\n\nAfter metadata processing, if an `init_PACKAGE(tag=None)` function\nexists it will be called with the intended platform tag. It must modify\nor return `PACKAGE`. This function is called for in-place, sdist and\nwheel generation, however, for sdists (and any scenario that should not\ngenerate binaries), `tag` will be `None`. Otherwise, it will be a\nstring like `cp38-cp38-win32`.\n\n```python\nPACKAGE = Package(\n \"my_package\",\n PyFile(r\"my_package\\*.py\"),\n)\n\ndef init_PACKAGE(tag=None):\n if tag and tag.endswith(\"-win_amd64\"):\n data_file = generate_data_amd64()\n PACKAGE.members.append(File(data_file))\n```\n\nNote that all files to be included in an sdist must be referenced when\n`tag` is `None`. Conditional compilation is best performed using conditions\nin the package elements, rather than using `init_PACKAGE`. However, if you\nare going to use `init_PACKAGE`, you should _remove_ elements rather than\nadding them if they should be included in your sdist.\n\nFiles added as part of a wildcard can be removed by adding a `RemoveFile`\nelement. These may be added dynamically during `init_PACKAGE`, and must\nappear after the element that included the files.\n\n```python\nPACKAGE = Package(\n \"my_package\",\n PyFile(r\"my_package\\*.py\"),\n RemoveFile(PyFile, r\"my_package\\_internal.py\"),\n)\n```\n\nTo exclude files from a wildcard in the first place, chain the `.excluding`\nmethod on the original element. The pattern will be evaluated in exactly the\nsame way as the inclusion pattern, and any paths that match will be omitted.\n\n```python\nPACKAGE = Package(\n \"my_package\",\n PyFile(r\"my_package\\*.py\").excluding(\"my_package\\internal*.py\"),\n)\n```\n\nAlternatively, a condition may be added to the file pattern to only include\nfiles matching MSBuild style conditions. Because these will be applied to\nitem groups, the `%()` metadata syntax should be used to access information\nfor the element being added. Either the `.if_` method or the\n`ConditionalValue` wrapper may be used.\n\n```python\nPACKAGE = Package(\n \"my_package\",\n PyFile(r\"my_package\\*.py\").if_(\"%(Filename) != '_internal'\"),\n File(ConditionalValue(\"*.txt\", condition=\"%(Filename.StartsWith(`internal`))\")),\n)\n```\n\nPackage members can be located during the dynamic stage using the\n`find` and `findall` functions. These take a path of member identifiers\n(typically their name property) and will return those that match.\n`'**'` segments are supported for recursive searches.\n\n```python\nPACKAGE = Package(\n \"my_package\",\n Package(\"sub1\", File(\"license.txt\")),\n Package(\"sub2\", File(\"license.txt\")),\n)\n\ndef init_PACKAGE(tag=None):\n for e in PACKAGE.findall(\"sub*/license.txt\"):\n e.name = \"LICENSE\"\n```\n\nWhen inserting members, the `insert` function combines a `find` with\nthe insert, and supports offset and range options. In general, only\nsubclassed element types should insert additional elements, and only\ninto themselves at construction.\n\n```python\nclass MyPydFile(PydFile):\n def __init__(self, name, *members, **options):\n super().__init__(name, *members, **options)\n self.insert(\n # Member path to insert before - this one is inherited from PydFile\n self.CommonToolsetImports.name,\n # Member to insert (in this case, an iterable)\n [Property(PROP1, VALUE1), Property(PROP2, VALUE2)],\n # Offset it by 1, so inserts after the found element (default 0)\n offset = 1,\n # Iterate over the insertion value; otherwise insert it as-is\n range = True\n )\n```\n\n## Source offsets\n\nIf you keep your source in a `src` folder (recommended), provide the\n`source=` argument to `Package` in order to properly offset filenames.\nBecause it is a named argument, it must be provided last.\n\nThis is important for sdist generation and in-place builds, which need\nto match package layout with source layout. Simply prefixing filename\npatterns with the additional directory is not always sufficient.\n\nNote that this will also offset subpackages, and that subpackages may\ninclude additional `source` arguments. However, it only affects\nsources, while the package name (the first argument) determines where\nin the output the package will be located. In-place builds will create\nnew folders in your source tree if it does not match the final\nstructure.\n\n```python\nPACKAGE = Package(\n \"my_package\",\n PyFile(r\"my_package\\__init__.py\"),\n source=\"src\",\n)\n```\n\n## Project file override\n\nBoth `Package` and `PydFile` types generate MSBuild project files and\nexecute them as part of build, including sdists. For highly customised\nbuilds, this generation may be overridden completely by specifying the\n`project_file` named argument. All members are then ignored.\n\nBy doing this, you take full responsibility for a valid build,\nincluding providing a number of undocumented and unsupported targets.\n\nRecommendations:\n* lock your `pymsbuild` dependency to a specific version in `pyproject.toml`\n* generate project files first and modify, rather than writing by hand\n (pass `--temp-dir` to specify the path where they will be generated)\n* read the `pymsbuild` source code, especially the `targets` folder\n* consider contributing/requesting your feature or developing an extension\n (see [pymsbuild-winui](https://github.com/zooba/pymsbuild-winui) and\n [pymsbuild-rust](https://github.com/zooba/pymsbuild-rust) for examples)\n\n```python\nPACKAGE = Package(\n \"my_package\",\n PydFile(\"_accelerator\", project_file=r\"src\\accelerator.vcxproj\")\n)\n```\n\n## Compiler/linker arguments\n\nRather than overriding the entire project file, there are a number of\nways to inject arbitrary values into a project. These require\nfamiliarity with MSBuild files and the toolsets you are building with.\n\nThe `Property` element inserts a `<PropertyGroup>` with the value you\nspecifiy at the position in the project the element appears.\n\nNote that project files also interpret (most) named arguments as\nproperties, so the two properties shown here are equivalent.\n\n```python\nPYD = PydFile(\n \"module\",\n Property(\"WindowsSdkVersion\", \"10.0.18363.0\"),\n ...\n # Alternative to Property(), but named arguments must be\n # specified last, so using Property() may be more readable\n WindowsSdkVersion=\"10.0.18363.0\",\n)\n```\n\nThe `ItemDefinition` element inserts an `<ItemDefinitionGroup>` with\nthe type and metadata you specify at the position in the project the\nelement appears. These generally apply metadata to all subsequent items\nof that type.\n\n```python\nPYD = PydFile(\n \"module\",\n ItemDefinition(\"ClCompile\", PreprocessorDefinitions=\"Py_LIMITED_API\"),\n ...\n)\n```\n\nThe `ConditionalValue` item may wrap any element value to add\nconditions or concatenate the value. This may also be used on source\narguments for file elements.\n\n```python\n ...\n Property(\"Arch\", ConditionalValue(\"x86\", condition=\"$(Platform) == 'Win32'\")),\n Property(\"Arch\", ConditionalValue(\"x64\", if_empty=True)),\n ...\n ItemDefinition(\n \"ClCompile\",\n AdditionalIncludeDirectories=\n ConditionalValue(INCLUDES + \";\", prepend=True),\n ProprocessorDefinitions=\n ConditionalValue(\";Py_LIMITED_API\", append=True),\n ),\n ...\n```\n\nThe `Prepend` shortcut may be used to reduce the amount of text for\nproperty values. Remember to include the appropriate separator. It is\nusually a semicolon.\n\n```python\n ...\n ItemDefinition(\n \"ClCompile\",\n AdditionalIncludeDirectories=Prepend(INCLUDES + \";\"),\n ProprocessorDefinitions=Prepend(\"Py_LIMITED_API;\"),\n ),\n ...\n```\n\n`ConditionalValue` may also be used to dynamically update values in the\n`init_PACKAGE` function, allowing you to keep the structure mostly\nstatic but insert values from the current `METADATA` (which is fully\nevaluated by the time `init_PACKAGE` is called). This saves having to\naccess internal members of other types in order to replace literal\nvalues.\n\nAssign a `ConditionalValue` to a variable without specifying any\ncondition, then use the variable in a later `Property` element and\nmodify its `value` attribute in `init_PACKAGE`.\n\n```python\nVER = ConditionalValue(\"1.0.0\")\n\nPYD = PydFile(\n \"module\",\n Property(\"Version\", VER),\n CSourceFile(r\"src\\*.c\"),\n IncludeFile(r\"src\\*.h\"),\n)\n\ndef init_PACKAGE(tag):\n VER.value = METADATA[\"Version\"]\n```\n\nAs a last resort, the `LiteralXml` element inserts plain text directly\ninto the generated file. It will be inserted as a child of the\ntop-level `Project` element.\n\n```python\n ...\n LiteralXml(\"<Import Project='my_props.props' />\"),\n ...\n```\n\n## Version info for DLLs/PYDs\n\n**Platform: Windows**\n\nTo embed version info into a compiled extension module, add a `VersionInfo`\nelement into the `PydFile`. All the fields from\nhttps://learn.microsoft.com/en-us/windows/win32/menurc/versioninfo-resource\nare available, using the names as shown in the tables (e.g.\n`FILEVERSION` for the `'1,0,0,0'` fields and `FileVersion` for the string\ntable entry).\n\nThe recommended usage is to add a default instance into your project and then\nuse `init_METADATA` to find it again and update based on the final metadata.\n\n```\nPACKAGE = Package(\n \"package\",\n PydFile(\"mod1\", VersionInfo()),\n PydFile(\"mod2\", VersionInfo()),\n)\n\ndef init_METADATA():\n # Update our metadata\n METADATA[\"Version\"] = calculate_current_version()\n # Ensure built modules reflect these updates\n for vi in PACKAGE.findall(\"*/VersionInfo\"):\n vi.from_metadata(METADATA)\n```\n\n`from_metadata` will fill in any empty fields from the set of metadata that is\npassed in.\n\n## Alternate config file\n\nTo use a configuration file other than `_msbuild.py`, specify the\n`--config` (`-c`) argument or the `PYMSBUILD_CONFIG` environment\nvariable.\n\n```powershell\npython -m pymsbuild --config build-spec.py sdist\npython -m pymsbuild --config build-spec.py wheel\n\n# Alternatively\n$env:PYMSBUILD_CONFIG = \"build-spec.py\"\npython -m pymsbuild sdist wheel\n```\n\nGenerated sdists will rename the configuration file back to\n`_msbuild.py` in the package to ensure that builds work correctly.\nThere is no need to override the configuration file path when\nbuilding from sdists.\n\nNote that this is different from the `PYMSBUILD_CONFIGURATION`\nvariable, which is used to select debug/release settings for compiled\nmodules.\n\n## Cross-compiling wheels\n\nCross compilation may be used by overriding the wheel tag, ABI tag,\nor build platform, as well as the source for Python's includes and\nlibraries. These all use environment variables, to ensure that the\nsame setting can flow through a package installer's own process.\n\nIt is also possible to permanently override the wheel tag by adding\na `'WheelTag'` metadata value, or the ABI tag by adding an `'AbiTag'`\nmetadata value.\n\nThe wheel tag is used for the generated wheel file, and to fill in a\nmissing ABI tag and platform.\n\nThe ABI tag is used for any native extension modules, and to fill in\na missing platform.\n\nAny `*` elements in the wheel tag are filled in from other locations.\nFor example, specifying `*-none-any` will infer the interpreter field\nfrom the current runtime, whil `py3-none-*` will infer the platform\nfrom the currnet system (or a specific ABI tag).\n\nThe platform is used to determine the MSBuild target platform. It\ncannot yet automatically select the correct Python libraries, and so\nyou will need to set `PYTHON_INCLUDES` and `PYTHON_LIBS` (or with a\n`PYMSBULID_` prefix) environment variables as well to locate the\ncorrect files.\n\nYou can override the platform toolset with the `'PlatformToolset'`\nmetadata value, for scenarios where this information ought to be\nincluded in an sdist.\n\nThe set of valid platforms for auto-generated `.pyd` project files are\nhard-coded into `pymsbuild` and are currently `Win32`, `x64`, `ARM` and\n`ARM64`. Custom project files may use whatever they like. These\nplatforms should behave properly cross-platform, though in general only\n`x64` and `ARM64` are supported.\n\n```powershell\n# Directly specify the resulting wheel tag\n# This is used for the wheel filename/metadata\n$env:PYMSBUILD_WHEEL_TAG = \"py38-cp38-win_arm64\"\n\n# Directly set the ABI tag (or else taken from wheel tag)\n# This is used for extension module filenames\n$env:PYMSBUILD_ABI_TAG = \"cp38-win_arm64\"\n\n# Specify the Python platform (or else taken from ABI tag)\n# This is used for MSBuild options\n$env:PYMSBUILD_PLATFORM = \"win_arm64\"\n\n# Specify the paths to ARM64 headers and libs\n$env:PYTHON_INCLUDES = \"$pyarm64\\Include\"\n$env:PYTHON_LIBS = \"$pyarm64\\libs\"\n\n# If necessary, specify an alternate C++ toolset\n$env:PLATFORMTOOLSET = \"Intel C++ Compiler 19.1\"\n```\n\n## Cython\n\nCython support is available from the `pymsbuild.cython` module.\n\n```python\nfrom pymsbuild import PydFile, ItemDefinition\nfrom pymsbuild.cython import CythonIncludeFile, CythonPydFile, PyxFile\n\nPACKAGE = CythonPydFile(\n \"cython_module\",\n ItemDefinition(\"PyxCompile\", IncludeDirs=PYD_INCLUDES),\n CythonIncludeFile(\"mod.pxd\"),\n PyxFile(\"mod.pyx\"),\n)\n```\n\nThe `CythonPydFile` type derives from the regular `PydFile` and also\ngenerates a C++ project, so all options that would be available there may\nalso be used.\n\nThe `PyxCompile.IncludeDirs` metadata specifies search paths for Cython\nheaders (`*.pxd`). You may also need to specify\n`ClCompile.AdditionalIncludeDirectories` for any C/C++ headers.\n\n\n## Two-Step Builds\n\nBy default, the `sdist` and `wheel` commands will perform the entire\nprocess in a single invocation. However, sometimes there are build steps\nthat must be manually performed between compilation and packaging.\n\nTo run the build in two stages, invoke as normal, but add the\n`--layout-dir` argument followed by a directory. The package will be\nlaid out in this directory so that you can perform any extra processing.\n\nLater, use the `pack` command and specify the `--layout-dir` again. If\nyou have added new files into the layout directory, specify each with an\n`--add` option (filenames starting with `@` are treated as\nnewline-separated, UTF-8 encoded text files listing each new file). These\npaths may be absolute or relative to the layout directory, but only files\nlocated within the layout directory will be included.\n\nAll other options are retained from the original invocation.\n\n```\npython -m pymsbuild sdist --layout-dir tmp\n\n# Generate additional metadata in tmp/EXTRA.txt\n\npython -m pymsbuild pack --layout-dir tmp --add tmp/EXTRA.txt\n\n# List many additional files in build/TO_ADD.txt\n\npython -m pymsbuild pack --layout-dir tmp --add @build/TO_ADD.txt\n```\n\n# Experimental Features\n\n## DLL Packing\n\n**Experimental. (POSIX is _very_ experimental)**\n\nDLL Packing is a way to compile a complete Python package (`.py` source\nand resource files) into an extension module. It is basically equivalent\nto packing in a ZIP file, except that additional native code may also be\nincluded (though not an entire native module), and the whole file may be\ncryptographically signed and validated by the operating system.\n\n`DllPackage` is a drop-in substitute for the `Package` type. It will\ngenerate a native extension module the same as the `PydFile` type,\nbut only includes Python source and resource files.\n\n```python\nfrom pymsbuild import *\nfrom pymsbuild.dllpack import *\n\nPACKAGE = DllPackage(\n \"packed_package\",\n PyFile(\"__init__.py\"),\n File(\"data.txt\"),\n ...\n)\n```\n\nAn entire existing library, such as `cryptography` could be packed\nlike this:\n\n```python\nfrom pymsbuild import *\nfrom pymsbuild.dllpack import *\n\nMODULE_TO_PACK = \"cryptography\"\n\nfrom importlib.util import find_spec\nspec = find_spec(MODULE_TO_PACK)\nif not spec:\n raise RuntimeError(f\"{MODULE_TO_PACK} must be installed\")\n\nPACKAGE = DllPackage(\n MODULE_TO_PACK,\n PyFile(\"**/*.py\"),\n PydRedirect(\"**/*.pyd\"),\n source = spec.submodule_search_locations[0],\n)\n```\n\nSee the `azure-pack` sample in our source repository for a more\ncomplete example.\n\n`DllPackage` is a subclass of `PydFile`, and so all logic or elements\nby that type are also available. `ClCompile` elements will be compiled\nand linked into the output and functions may be exposed in the root of\nthe package using the `Function` element.\n\n```c\n// extra.c\n\nPyObject *my_func(PyObject *, PyObject *args, PyObject **kwargs) {\n ...\n}\n```\n\n```python\nPACKAGE = DllPackage(\n \"packed_package\",\n PyFile(\"__init__.py\"),\n CSourceFile(\"extra.c\"),\n CFunction(\"my_func\"),\n ...\n)\n```\n\n### Packed module names\n\nBy default, the packed DLL contents are presumed to be nested within\nthe overall structure. This is important because full module names must\nbe embedded into the DLL. As a result, when the example below is built,\nit will create a `root/` directory containing `packed.pyd` (or\nequivalent) and enables `import root.packed.submodule`.\n\n```python\nPACKAGE = Package(\"root\",\n DllPackage(\"packed\",\n PyFile(\"src/packed/__init__.py\"),\n PyFile(\"src/packed/submodule.py\"),\n )\n)\n\n# After building:\nimport root.packed.submodule\n```\n\nIf part of the package name is not importable (specifically,\n`str.isidentifier()` is `False`), then the name will be truncated after\nthat part.\n\nThere are a few properties that can affect what is built. The first is\nthe `TargetName` property, which will change the name of the compiled\nfile. For example, adding `TargetName=\"root.packed\"` or\n`Property(\"TargetName\", \"root.packed\")` to the `DllPackage` item will\ngenerate `root.packed.pyd` instead, allowing the module to be imported\nas `root.packed` without being in a `root/` directory.\n\nSecond is the `RootNamespace` property, which overrides the name used\nwithin the packed module to identify submodules. Changing this alone is\nrarely useful, and the default (matching the name of the module) is\nrecommended unless you have modified other aspects of the build.\nIf this is overridden with a named argument, it will replace the full\nname used within the packed DLL. If overridden with a `Property`\nelement, it will replace the name but retain the parent's namespace.\n\n### Nested extension modules\n\nTo allow referencing other extension modules that would normally be\nnested within the module, add a `PydRedirect` element and reference the\nextension module. The filename does not have to match the original\nname, or even need to be a normally importable name, as it will be\npassed directly to the module loader. The file will be included in your\nwheel in the expected location (alongside the packed DLL). Wildcards\nare supported.\n\n```python\nPACKAGE = DllPackage(\n \"packed\",\n PydRedirect(source=\"packed/nested.pyd\", name=\"packed-nested.pyd\"),\n ...\n)\n```\n\nOther `PydFile` modules may be nested inside the `DllPackage`, which\nwill automatically add a redirect, as well as building the module.\nThe nested module will be built using the name specified and sit\nadjacent to the packed module, but should be imported via the\npacked module.\n\nThe `ImportName` metadata may be specified on either a `PydRedirect` or\na `PydFile` to specify the name that must be used to import the module.\nRedirected extension modules do not need to have an importable name\nwhen `ImportName` is specified. You might include an invalid character\nin the filename to ensure the module is not importable directly.\nWhen specifying `ImportName`, the name of the packed DLL must be used\nas the first part.\n\n```python\nPACKAGE = DllPackage(\n \"packed\",\n PydRedirect(\"module/nested.pyd\", ImportName=\"packed.nested\"),\n ...\n)\n```\n\n### Encryption\n\nTo encrypt your content using symmetric AES encryption, provide the\nname of the environment variable holding your key as the\n`EncryptionKeyVariable` option. The key will need to be a valid size\n(usually 16, 24 or 32 bytes) when encoded to UTF-8 or decoded from\nbase 64. Base 64 keys should start with `base64:`.\n\nThe same variable will need to be set when importing the module. It is\nyour responsibility to protect the key! The benefit of this encryption\nis best realised when you avoid storing the key to disk. That way, an\nattacker who steals a copy of your module is unlikely to have access to\nthe key. An attacker with access to a running copy of your module will\nbe able to easily extract the key.\n\n```python\nPACKAGE = DllPackage(\n \"package\",\n ...,\n EncryptionKeyVariable=\"MY_KEY_VARIABLE\"\n)\n```\n\n```powershell\n> $env:MY_KEY_VARIABLE=\"base64:MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDA=\"\n> python -m pymsbuild\n> del env:\\MY_KEY_VARIABLE\n> python -c \"import package\"\nImportError: Module cannot be decrypted\n```\n\nRedirected or nested extension modules are not encrypted.\n\n## Cross-platform builds\n\n**Experimental.**\n\nWith the [.NET SDK](https://dotnet.microsoft.com/download) installed,\n`pymsbuild` is able to run builds on platforms other than Windows.\nThe `dotnet` command must be available on `PATH` or specified as the\n`MSBUILD` environment variable.\n\nIn general, no platform-specific modifications to a build script are\nrequired. Cython and pyd builds are transparently mapped to the target\nsystem. To run build-time actions for specific platforms, add them to\n`init_PACKAGE` and check the tag argument to determine the target\nplatform.\n\nWhen building native components on POSIX, a `python3-config` script is\nneeded to determine compilation options. By default, only the location\nadjacent to the running interpreter is checked. This may be overridden\nby setting the `PYTHON_CONFIG` variable to the preferred command.\n\n## Custom entry point\n\n**Experimental.**\n\nTo generate an executable that will launch your application, include the\n`pymsbuild.entrypoint` module and use an `Entrypoint` definition.\n\n```python\nfrom pymsbuild import *\nfrom pymsbuild.entrypoint import *\n\nPACKAGE = Package(\n \"demo\",\n Entrypoint(\n \"run\", # generate run.exe\n \"app\", # import app\n \"main\", # app.main()\n Icon(\"app.ico\"),\n\n # Search paths for the entry point to use\n SearchPath(\".\"),\n SearchPath(\"stdlib.zip\"),\n\n # Include a copy of Python (default: True)\n IncludePythonRuntime=True,\n # Use the embeddable distro (default: True)\n PythonEmbeddable=True,\n # Also include python.exe (default: False)\n PythonExecutables=False,\n # Rename pythonXY.zip to stdlib.zip (default: True)\n PythonRuntimeRenameStdlibZip=True,\n ),\n Package(\n \"app\",\n PyFile(\"app/__init__.py\"),\n ),\n)\n```\n\nBuilding this definition will create a `demo` directory containing\n`run.exe`, `app.py` and a copy of the Python embeddable runtime, making\nit an entirely standalone and redistributable application.\n\nSet `IncludePythonRuntime` to `False` to omit the runtime. The generated\nexecutable assumes that it will be able to load the version of Python\nused to build at runtime, so you will need to include it some other way.\n\nUse `SearchPath` items to specify directories to search for Python\nmodules at runtime. These are the only directories that will be\nsearched, as Python will be loaded in isolated mode. They are relative\nto the entrypoint and will be resolved when executing.\n\n",
"bugtrack_url": null,
"license": null,
"summary": "The pymsbuild build backend.",
"version": "1.1.4",
"project_urls": {
"Homepage": "https://github.com/zooba/pymsbuild",
"Issues": "https://github.com/zooba/pymsbuild/issues",
"Source": "https://github.com/zooba/pymsbuild"
},
"split_keywords": [
"build",
" pep-517",
" msbuild",
" packaging",
" cython"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "bb9bb729430550bccc776cfaf386204b254eb8981f509d39cac09f197f963878",
"md5": "af584849b2e27c6433f9fa0409fb93a6",
"sha256": "7b65e25654ef307773ec9e766f4d1bc1bd8f8d30bd7c5dd437404d80b8885a96"
},
"downloads": -1,
"filename": "pymsbuild-1.1.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "af584849b2e27c6433f9fa0409fb93a6",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 86648,
"upload_time": "2024-12-16T23:03:59",
"upload_time_iso_8601": "2024-12-16T23:03:59.126513Z",
"url": "https://files.pythonhosted.org/packages/bb/9b/b729430550bccc776cfaf386204b254eb8981f509d39cac09f197f963878/pymsbuild-1.1.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "31ad9f48b2fae584c654b1c1be750abd34d65972cd83a32f696ed6650a41bf1e",
"md5": "d98a238089749bc8abaad828eb546769",
"sha256": "8966e8c1a5ddd52209d7cafcdb7c7672816545173ad5a44f559584e611694e1b"
},
"downloads": -1,
"filename": "pymsbuild-1.1.4.tar.gz",
"has_sig": false,
"md5_digest": "d98a238089749bc8abaad828eb546769",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 68212,
"upload_time": "2024-12-16T23:04:01",
"upload_time_iso_8601": "2024-12-16T23:04:01.390988Z",
"url": "https://files.pythonhosted.org/packages/31/ad/9f48b2fae584c654b1c1be750abd34d65972cd83a32f696ed6650a41bf1e/pymsbuild-1.1.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-16 23:04:01",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "zooba",
"github_project": "pymsbuild",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "pymsbuild"
}
JSON
