wurfapi
=======
|PyPi| |Waf Python Tests| |Black| |Flake8| |Pip Install|
.. |PyPi| image:: https://badge.fury.io/py/wurfapi.svg
:target: https://badge.fury.io/py/wurfapi
.. |Waf Python Tests| image:: https://github.com/steinwurf/wurfapi/actions/workflows/python-waf.yml/badge.svg
:target: https://github.com/steinwurf/wurfapi/actions/workflows/python-waf.yml
.. |Flake8| image:: https://github.com/steinwurf/wurfapi/actions/workflows/flake8.yml/badge.svg
:target: https://github.com/steinwurf/wurfapi/actions/workflows/flake8.yml
.. |Black| image:: https://github.com/steinwurf/wurfapi/actions/workflows/black.yml/badge.svg
:target: https://github.com/steinwurf/wurfapi/actions/workflows/black.yml
.. |Pip Install| image:: https://github.com/steinwurf/wurfapi/actions/workflows/pip.yml/badge.svg
:target: https://github.com/steinwurf/wurfapi/actions/workflows/pip.yml
We wanted to have a configurable and easy-to-use Sphinx API documentation
generator for our C++ projects. To achieve this we leaned on others for
inspiration:
* Breathe (https://github.com/michaeljones/breathe): Excellent extension
and the default choice for many.
* Gasp (https://github.com/troelsfr/Gasp): Gasp inspired us by allowing
templates to control the output. Unfortunately development of Gaps
seems to have stopped.
So what is ``wurfapi``:
* Essentially we picked up where Gasp let go. We have
borrowed the idea of templates to make it highly configurable.
* We made it easy to use by automatically running Doxygen to generate the
initial API documentation.
* We parse the Doxygen XML into an easy to use Python dictionary. Which can
be consumed in the templates.
* We prepared the extension for other backends (replacing Doxygen) e.g.
https://github.com/foonathan/standardese once they become ready.
.. contents:: Table of Contents:
:local:
Status
======
We currently use wurfapi in the following projects:
* https://rely.steinwurf.com/docs/10.0.0/
* https://kodo.steinwurf.com/docs/11.0.0/
* https://otacast.steinwurf.com/docs/6.0.0/
... and many more.
Usage
=====
We recommend that you install wurfapi and sphinx in a virtual environment.
To use the extension, the following steps are needed:
1. Create a virtual environment::
Follow the https://docs.python.org/3/tutorial/venv.html
2. Install the extension::
pip install sphinx
pip install wurfapi
3. Generate the initial ``Sphinx`` documentation by running::
mkdir docs
cd docs
python sphinx-quickstart
You will need to enter some basic information about your project such
as the project name etc.
4. Open the ``conf.py`` generated by ``sphinx-quickstart`` and add the
the following::
# Append or insert 'wurfapi' in the extensions list
extensions = ['wurfapi']
# wurfapi options - relative to your docs dir
wurfapi = {
'source_paths': ['../src'],
'recursive': True,
'parser': {'type': 'doxygen', 'download': True, 'warnings_as_error': True}
}
.. note::
``source_path``
If you separate source and build dir in sphinx your 'source_path'
should be something like '../../src'.
``recursive``
Set recursive ``True`` if you want recursively scan the ``source_paths``
``download``
If you do not want to automatically download Doxygen, set
``download`` to ``False``. In that case ``wurfapi`` will try to invoke
plain ``doxygen`` without specifying any path or similar. This means
it ``doxygen`` must be available in the path.
``warnings_as_error``
If Doxygen emits many warnings you might want to set warnings_as_error
to False until they have been fixed.
5. To generate the API documentation for a class open a ``.rst`` file
e.g. ``index.rst`` if you ran ``sphinx-quickstart``. Say we want to
generate docs for a class called ``test`` in the namespace ``project``.
To do this we add the following directive to the rst file::
.. wurfapi:: class_synopsis.rst
:selector: project::coffee::machine
Such that ``index.rst`` becomes something like::
Welcome to Coffee's documentation!
===================================
.. toctree::
:maxdepth: 2
:caption: Contents:
.. wurfapi:: class_synopsis.rst
:selector: project::coffee::machine
.. wurfapi:: class_synopsis.rst
:selector: project::coffee::recipe
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
To do this we use the ``class_synopsis.rst`` template.
6. Generate the Documentation
make html
Labels and References
---------------------
To reference different elements in the API, we have added a custom Sphinx role ``:wurfapi:``
The ``:wurfapi:`` role will try to deduce the ``unique-name`` from the text given.
E.g if you want to reference the ``unique-name`` ``foo::bar::baz::func(std::string var)`` and there are
no other member functions in ``foo::bar::baz`` named ``func``, you can reference it
by writing ``:wurfapi:`foo::bar::baz::func```.
On the other hand if there was a function with ``unique-name`` ``foo::bar::baz::function(std::string var)``
``:wurfapi:`foo::bar::baz::func``` could match with both func and function and will throw an error. In This
case this can be fixed by adding the left parenthesis: ``:wurfapi:`foo::bar::baz::func(```.
You can read more about unique names later in this README.
Running on readthedocs.org
--------------------------
To use this on readthedocs.org you need to have the ``wurfapi`` Sphinx
extension installed. This can be done by adding a ``requirements.txt`` in the
documentation folder. readthedocs.org can be configured to use the
``requirements.txt`` when building a project. Simply put ``wurfapi`` in to the
``requirements.txt``.
Doxygen issues
--------------
Nothing is perfect, neither is Doxygen. Sometimes Doxygen gets it wrong e.g. in
the following example::
class foo
{
private:
class bar;
};
Doxygen incorrectly reports that ``bar`` has public scope (also reported here
https://bit.ly/2BWPllZ). To deal with such issues, until a fix lands in
Doxygen, you can do the following:
Add a list of *patches* to the API to your ``conf.py`` file. Extending the
example from before, we can add the following fix::
wurfapi = {
'source_paths': ['../src'],
'recursive': True,
'parser': {
'type': 'doxygen', 'download': True, 'warnings_as_error': True,
'patch_api': [
{'selector': 'foo::bar', 'key': 'access', 'value': 'private'}
]
}
}
The ``patch_api`` allows you to reach in to the parsed API information and
update certain values. The ``selector`` is the ``unique-name`` of the
entity you want to update. Check the "Dictionary layout" section further down
for more information.
Collapse inline namespaces
--------------------------
For symbol versioning you may use ``inline namespaces``, however typically
you don't want these to show up in the docs, as these are mostly
invisible for your users.
With ``wurfapi`` you can collapse the inline namespace such that it
is removed form the scopes etc.
Example::
namespace foo { inline namespace v1_2_3 { struct bar{}; } }
The scope to bar is ``foo::v1_2_3``. If you collapse the inline namespace it will
just be ``foo``.
First issue you have to deal with is that Doxygen currently does not
support inline namespaces. So we need to patch the API first::
wurfapi = {
'source_paths': ['../src'],
'recursive': True,
'parser': {
'type': 'doxygen', 'download': True, 'warnings_as_error': True,
'patch_api': [
{'selector': 'foo::v1_2_3', 'key': 'inline', 'value': True}
]
}
}
After this we can collapse the namespace::
wurfapi = {
'source_paths': ['../src'],
'recursive': True,
'parser': {
'type': 'doxygen', 'download': True, 'warnings_as_error': True,
'patch_api': [
{'selector': 'foo::v1_2_3', 'key': 'inline', 'value': True}
],
'collapse_inline_namespaces': [
"foo::v1_2_3"
]
}
}
Now you will be able to refer to ``bar`` as ``foo::bar``. Note, that
collapsing the namespace will affect the selectors you write when
generating the documentation.
Custom templates
----------------
You can write your own custom templates for generating the rst output.
To this you simply write a Jinja2 compatible rst template and place
it in some folder. Adding the ``user_templates`` key to the ``wurfapi``
configuration dictionary in the ``conf.py`` file will make it available.
For example::
wurfapi = {
'source_paths': ['../src', '../examples/header/header.h'],
'recursive': True,
'user_templates': 'rst_templates',
'parser': {
'type': 'doxygen', 'download': True, 'warnings_as_error': True
}
}
exclude_patterns = ['rst_templates/*.rst']
Now we can use ``*.rst`` files inside the ``rst_templates`` folder e.g. if
we had a ``class_list.rst`` template we could use it like this::
.. wurfapi:: class_list.rst
:selector: project::coffee
Release new version
===================
1. Edit ``NEWS.rst``, ``wscript`` and ``src/wurfapi/wurfapi.py`` (set
correct ``VERSION``)
2. Run ::
./waf upload
Source code
===========
Tests
=====
The tests will run automatically by passing ``--run_tests`` to waf::
./waf --run_tests
This follows what seems to be "best practice" advice, namely to install the
package in editable mode in a virtualenv.
Recordings
----------
A bunch of the tests use a library called ``pytest-datarecorder``.
The library is used to store the output as files from different parsing and
rendering operations.
E.g. say we want to make sure that a parser function returns a certain
``dict`` object. Then we can record that ``dict``::
datarecorder.record_data(
data={'foo': 2, 'bar': 3},
recording_file="/tmp/recording/test.json"
)
If ``data`` changes compared to a previous recording a mismatch will be
detected. To update a recording simply delete the recording file.
Test directories
----------------
You will also notice that a bunch of the tests take a parameter called
``testdirectory``. The ``testdirectory`` is a pytest fixture, which
represents a temporary directory on the filesystem. When running the tests
you will notice these temporary test directories pop up under the
``pytest_temp`` directory in the project root.
You can read more about that here:
* https://github.com/steinwurf/pytest-testdirectory
Developer Notes
===============
The `sphinx` documentation on creating extensions:
http://www.sphinx-doc.org/en/stable/extdev/index.html#dev-extensions
* An extension is a Python module. When an extension loads, Sphinx will import
it and execute its ``setup()`` function.
* Understanding how to put together docutils nodes seems pretty difficult. One
suggesting form the mailinglist was to look at the following document:
https://github.com/docutils-mirror/docutils/blob/master/test/functional/expected/standalone_rst_pseudoxml.txt
* While researching how to do this, there seem to be three potential approaches:
1. Use the standard Sphinx approach and operate with the doctree.
2. Create RST based on jinja templates
3. Create HTML based on jinja templates
* Inspiration - Sphinx extensions that were used as inspiration while
developing this extension.
* Breathe
* Gasp
* https://github.com/Robpol86/sphinxcontrib-imgur
* https://github.com/djungelorm/sphinx-tabs
* Understanding how to write stuff with docutils:
* http://agateau.com/2015/docutils-snippets/
* Creating a custom directive
* http://www.xavierdupre.fr/blog/2015-06-07_nojs.html
* Nice looking Sphinx extensions
* https://github.com/bokeh/bokeh/tree/master/bokeh/sphinxext
* This part of the documentation was useful in order to understand the need
for ViewLists etc. in the directives run(...) function.
http://www.sphinx-doc.org/en/stable/extdev/markupapi.html
* This link provided inspiration for the text json format: https://github.com/micnews/html-to-article-json
* More xml->json for the text: https://www.xml.com/pub/a/2006/05/31/converting-between-xml-and-json.html
Dictionary layout
-----------------
We want to support different "backends" like Doxygen to parse the source
code. To make this possible we define an internal source code description
format. We then translate e.g. Doxygen XML to this and use that to render
the API documentation.
This way a different "backend" e.g. Doxygen2 could be use used as the source
code parser and the API documentation could be generated.
``unique-name``
...............
In order to be able to reference the different entities in the API we need
to assign them a name.
We use a similar approach here as described in standardese_.
This means that the ``unique-name`` of an entity is the name with all
scopes e.g. ``foo::bar::baz``.
* For functions the unique name contains the signature (parameter types and for
member functions cv-qualifier and ref-qualifier) e.g. ``foo::bar::baz::func()``
or ``foo::bar::baz::func(int a, char*) const``. See cppreference_ for more
information.
* For class template specializations the unique name includes the specialization
arguments. For example::
// Here the unique-name is just 'foo'
template<class T>
class foo {};
// Here the unique name is foo<int>
template<>
class foo<int> {};
* In addition to types, we also have entries for the parsed files. For files
the unique name will be the relative path from the project root.
* For defines we will use the name of the define. As an example::
#define PROJECT_VERSION "1.0.0"
Here ``unique-name`` will be ``PROJECT_VERSION``.
.. _cppreference: http://en.cppreference.com/w/cpp/language/member_functions
.. _standardese: https://github.com/foonathan/standardese#linking
The API dictionary
...................
The internal structure is a dicts with the different API entities. The
``unique-name`` of the entity is the key and the entity type also a
Python dictionary is the value e.g::
api = {
'unique-name': { ... },
'unique-name': { ... },
...
}
To make this a bit more concrete consider the following code::
namespace ns1
{
class shape
{
void print(int a) const;
};
namespace ns2
{
struct box
{
void hello();
};
void print();
}
}
Parsing the above code would produce the following API dictionary::
api = {
'ns1': { 'kind': 'namespace', ...},
'ns1::shape': { 'kind': 'class', ... },
'ns1::shape::print(int) const': { kind': function' ... },
'ns1::ns2': { 'kind': 'namespace', ... },
'ns1::ns2::box': { 'kind': 'struct', ... },
'ns1::ns2::box::hello()': { kind': function' ... },
'ns1::ns2::print()': { 'kind': 'function', ...},
'ns1.hpp': { 'kind': 'file', ...}
}
The different entity kinds expose different information about the
API. We will document the different kinds in the following.
We make some keys *optional* this is marked in the following way::
api = {
'unique-name': {
'some_key': ...
Optional('an_optional_key'): ...
},
...
}
``namespace`` Kind
..................
Python dictionary representing a C++ namespace::
info = {
'kind': 'namespace',
'name': 'unqualified-name',
'scope': 'unique-name' | None,
'members: [ 'unique-name', 'unique-name' ],
'briefdescription': paragraphs,
'detaileddescription': paragraphs,
'inline': True | False
}
Note: Currently Doxygen does not support parsing ``inline namespaces``. So
you need to use the patch API to change the value from ``False`` to ``True``
manually. Maybe at some point https://github.com/doxygen/doxygen/issues/6741
it will be supported.
``class`` | ``struct`` Kind
...........................
Python dictionary representing a C++ class or struct::
info = {
'kind': 'class' | 'struct',
'name': 'unqualified-name',
'location': location,
'scope': 'unique-name' | None,
'access': 'public' | 'protected' | 'private',
Optional('template_parameters'): template_parameters,
'members: [ 'unique-name', 'unique-name' ],
'briefdescription': paragraphs,
'detaileddescription': paragraphs
}
``enum`` | ``enum class`` Kind
..............................
Python dictionary representing a C++ enum or enum class::
info = {
'kind': 'enum',
'name': 'unqualified-name',
'location': location,
'scope': 'unique-name' | None,
'access': 'public' | 'protected' | 'private',
'values: [
{
'name': 'somename',
'briefdescription': paragraphs,
'detaileddescription': paragraphs,
Optional('value'): 'some value'
}
],
'briefdescription': paragraphs,
'detaileddescription': paragraphs
}
``typedef`` | ``using`` Kind
............................
Python dictionary representing a C++ using or typedef statement::
info = {
'kind': 'typedef' | 'using',
'name': 'unqualified-name',
'location': location,
'scope': 'unique-name' | None,
'access': 'public' | 'protected' | 'private',
'type': type,
'briefdescription': paragraphs,
'detaileddescription': paragraphs
}
``define`` Kind
...............
Python dictionary representing a C/C++ define::
info = {
'kind': 'define',
'name': 'name',
'location': location,
Optional('initializer'): 'some_value',
Optional('parameters'): [{
'name': 'somestring',
Optional('description'): paragraphs
}],
'briefdescription': paragraphs,
'detaileddescription': paragraphs
}
The content of the define will be in the ``initializer`` field. If the define
takes documented paremeters these will be under the ``parameter`` key.
Examples:
1. Define initializer::
#define VERSION "1.0.2"
2. Define initalizer with parameters::
#define min(X, Y) ((X) < (Y) ? (X) : (Y))
``file`` Kind
............................
Python dictionary representing a file in the project::
info = {
'kind': 'file',
'name': 'somefile.hpp',
'path': 'relative/path/to/somefile.hpp',
}
``function`` Kind
.................
Python dictionary representing a C++ function::
info = {
'kind': 'function',
'name': 'unqualified-name',
'location': location,
'scope': 'unique-name' | None,
Optional('return'): {
'type': type,
'description': paragraphs
}
Optional('template_parameters'): template_parameters,
'is_const': True | False,
'is_static': True | False,
'is_virtual': True | False,
'is_explicit': True | False,
'is_inline': True | False,
'is_constructor': True | False,
'is_destructor': True | False,
'trailing_return': True | False,
'access': 'public' | 'protected' | 'private',
'briefdescription: paragraphs,
'detaileddescription: paragraphs,
'parameters': [
{ 'type': type, Optional('name'): 'somename', 'description': paragraphs },
...
]
}
The `return` key is optional if the function is either a constructor or
destructor.
``variable`` Kind
.................
Python dictionary representing a C++ variable::
info = {
'kind': 'variable',
'name': 'unqualified-name',
Optional('value'): 'some value',
'type': type,
'location': location,
'is_static': True | False,
'is_mutable': True | False,
'is_volatile': True | False,
'is_const': True | False,
'is_constexpr': True | False,
'scope': 'unique-name' | None,
'access': 'public' | 'protected' | 'private',
'briefdescription: paragraphs,
'detaileddescription: paragraphs,
}
``location`` item
.................
Python dictionary representing a location::
location = {
Optional('include'): 'some/header.h',
'path': 'src/project/header.h',
'line': 10
}
* The ``include`` will be relative to any ``include_paths`` specified in the
``wurfapi`` dictionary in your Sphinx ``conf.py``.
* The ``path`` will be relative to the project root folder.
``type`` item
.............
Python list representing a C++ type::
type = [
{
'value': 'sometext',
Optional('link'): link
}, ...
]
Having the type as a list of items we can create links to nested types e.g.
say we have a `std::unique_ptr<impl>` and we would like to make `impl` a link.
This could look like::
"type": [
{
"value": "std::unique_ptr<"
},
{
"link": {"url": False, "value": "project::impl"},
"value": "impl"
},
{
"value": ">"
}
]
Any spaces in the type list should be preserved all the way from the Doxygen
output and into the type list. In the rst it should be sufficient to simply
output the values of the type. No spaces or other stuff should be injected.
``link`` item
.............
Python dictionary representing a link::
link = { 'url': True | False, 'value': 'somestring' }
If `url` is `True` we have a basic extrenal reference otherwise we have a
link to an internal type in the API.
``parameter`` item
...................
Dictionary representing a function parameter::
parameter = {
'type': type,
Optional('name'): 'somestring',
Optional('description'): paragraphs
}
For the parameter, the name is also included in the type list. The reason
is that some parameters can be pretty complex, with the name embedded
inside the type e.g.::
void function(int (*(*foo)())[3]);
This is a function that takes one parameter `foo` which is a pointer
function returning a pointer to array 3 of int - nice right? Anyway, in
such cases the parameter name is embedded inside the type of the parameter.
We therefore took the easy way out and `wurfapi` will always include the
parameter name in the type.
As an example the parameter dictionary for a function `void test(int b)`
could be::
{
'type': [{'value': 'int '}, {'value': 'b'}],
'name': 'b'
}
``template_parameters`` item
.............................
Python list of dictionaries representing template parameters::
template_parameters = [{
'type': type,
'name': 'somestring',
Optional('default'): type,
Optional('description'): paragraphs
}]
Text Information
................
Text information is stored in a list of paragraphs::
paragraphs = [paragraph]
A paragraph consists of a list of paragraph elements::
paragraph = [
{
"kind": "text" | "code" | "list" | "bold" | "italic",
...
},
]
Paragraph elements can be one of three kinds, "text", "code" or "list"::
text = {
'kind': 'text',
'content': 'hello',
Optional('link'): link
}
code = {
'kind': 'code',
'content': 'void print();',
'is_block': true | false
}
list = {
'kind': 'list',
'ordered': true | false,
'items': [paragraphs] # Each item is a list of paragraphs
}
Problem with ``unique-name`` for functions
..........................................
Issue equivalent C++ function signatures can be written in a number of
different ways::
void hello(const int *x); // x is a pointer to const int
void hello(int const *x); // x is a pointer to const int
We can also move the asterisk (``*``) to the left::
void hello(const int* x); // x is a pointer to const int
void hello(int const* x); // x is a pointer to const int
So we need some way to normalize the function signature when transforming it
to ``unique-name``. We cannot simply rely on sting comparisons.
According to the numerous google searches it is hard to write a regex for this.
Instead we will try to use a parser:
* Python parser: https://github.com/erezsh/lark
* C++ Grammar: http://www.externsoft.ch/media/swf/cpp11-iso.html#parameters_and_qualifiers
We only need to parse the function parameter list denoted as the
``http://www.externsoft.ch/media/swf/cpp11-iso.html#parameters_and_qualifiers``.
Generated output
----------------
Since we are going to be using Doxygen's XML output as input to the
extension we need a place to store it. We store it system temporary folder e.g.
if the project name is "foobar" on Linux this would be
``/tmp/wurfapi-foobar-123456`` where ``123456`` is a hash of the source
directory paths. In addition to Doxygen's XML we also store the generated rst
for the different directives there. This is nice for debugging to see whether
we generate broken rst.
The API in json format can be found in the ``_build/.doctree/wurfapi_api.json``.
Paths and directories
---------------------
* Source directory: In Sphinx the source directory is where our .rst files are
located. This is what you pass to ``sphinx-build`` when building your
documentation. We will use this in our extension to find the C++ source code
and output customization templates.
Notes
=====
* Why use an ``src`` folder (https://hynek.me/articles/testing-packaging/).
tl;dr you should run your tests in the same environment as your users would
run your code. So by placing the source files in a non-importable folder you
avoid accidentally having access to resources not added to the Python
package your users will install...
* Python packaging guide: https://packaging.python.org/distributing/
Raw data
{
"_id": null,
"home_page": "https://github.com/steinwurf/",
"name": "wurfapi",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "wurfapi",
"author": "Steinwurf ApS",
"author_email": "contact@steinwurf.com",
"download_url": "",
"platform": null,
"description": "wurfapi\n=======\n\n|PyPi| |Waf Python Tests| |Black| |Flake8| |Pip Install|\n\n.. |PyPi| image:: https://badge.fury.io/py/wurfapi.svg\n :target: https://badge.fury.io/py/wurfapi\n\n.. |Waf Python Tests| image:: https://github.com/steinwurf/wurfapi/actions/workflows/python-waf.yml/badge.svg\n :target: https://github.com/steinwurf/wurfapi/actions/workflows/python-waf.yml\n\n.. |Flake8| image:: https://github.com/steinwurf/wurfapi/actions/workflows/flake8.yml/badge.svg\n :target: https://github.com/steinwurf/wurfapi/actions/workflows/flake8.yml\n\n.. |Black| image:: https://github.com/steinwurf/wurfapi/actions/workflows/black.yml/badge.svg\n :target: https://github.com/steinwurf/wurfapi/actions/workflows/black.yml\n\n.. |Pip Install| image:: https://github.com/steinwurf/wurfapi/actions/workflows/pip.yml/badge.svg\n :target: https://github.com/steinwurf/wurfapi/actions/workflows/pip.yml\n\nWe wanted to have a configurable and easy-to-use Sphinx API documentation\ngenerator for our C++ projects. To achieve this we leaned on others for\ninspiration:\n\n* Breathe (https://github.com/michaeljones/breathe): Excellent extension\n and the default choice for many.\n* Gasp (https://github.com/troelsfr/Gasp): Gasp inspired us by allowing\n templates to control the output. Unfortunately development of Gaps\n seems to have stopped.\n\nSo what is ``wurfapi``:\n\n* Essentially we picked up where Gasp let go. We have\n borrowed the idea of templates to make it highly configurable.\n\n* We made it easy to use by automatically running Doxygen to generate the\n initial API documentation.\n\n* We parse the Doxygen XML into an easy to use Python dictionary. Which can\n be consumed in the templates.\n\n* We prepared the extension for other backends (replacing Doxygen) e.g.\n https://github.com/foonathan/standardese once they become ready.\n\n.. contents:: Table of Contents:\n :local:\n\n\nStatus\n======\n\nWe currently use wurfapi in the following projects:\n\n* https://rely.steinwurf.com/docs/10.0.0/\n* https://kodo.steinwurf.com/docs/11.0.0/\n* https://otacast.steinwurf.com/docs/6.0.0/\n\n... and many more.\n\nUsage\n=====\n\nWe recommend that you install wurfapi and sphinx in a virtual environment.\nTo use the extension, the following steps are needed:\n\n1. Create a virtual environment::\n\n Follow the https://docs.python.org/3/tutorial/venv.html\n\n2. Install the extension::\n\n pip install sphinx\n pip install wurfapi\n\n3. Generate the initial ``Sphinx`` documentation by running::\n\n mkdir docs\n cd docs\n python sphinx-quickstart\n\n You will need to enter some basic information about your project such\n as the project name etc.\n\n4. Open the ``conf.py`` generated by ``sphinx-quickstart`` and add the\n the following::\n\n # Append or insert 'wurfapi' in the extensions list\n extensions = ['wurfapi']\n\n # wurfapi options - relative to your docs dir\n wurfapi = {\n 'source_paths': ['../src'],\n 'recursive': True,\n 'parser': {'type': 'doxygen', 'download': True, 'warnings_as_error': True}\n }\n\n .. note::\n\n ``source_path``\n If you separate source and build dir in sphinx your 'source_path'\n should be something like '../../src'.\n\n ``recursive``\n Set recursive ``True`` if you want recursively scan the ``source_paths``\n\n ``download``\n If you do not want to automatically download Doxygen, set\n ``download`` to ``False``. In that case ``wurfapi`` will try to invoke\n plain ``doxygen`` without specifying any path or similar. This means\n it ``doxygen`` must be available in the path.\n\n ``warnings_as_error``\n If Doxygen emits many warnings you might want to set warnings_as_error\n to False until they have been fixed.\n\n5. To generate the API documentation for a class open a ``.rst`` file\n e.g. ``index.rst`` if you ran ``sphinx-quickstart``. Say we want to\n generate docs for a class called ``test`` in the namespace ``project``.\n\n To do this we add the following directive to the rst file::\n\n .. wurfapi:: class_synopsis.rst\n :selector: project::coffee::machine\n\n Such that ``index.rst`` becomes something like::\n\n Welcome to Coffee's documentation!\n ===================================\n\n .. toctree::\n :maxdepth: 2\n :caption: Contents:\n\n .. wurfapi:: class_synopsis.rst\n :selector: project::coffee::machine\n\n .. wurfapi:: class_synopsis.rst\n :selector: project::coffee::recipe\n\n\n Indices and tables\n ==================\n\n * :ref:`genindex`\n * :ref:`modindex`\n * :ref:`search`\n\n\n To do this we use the ``class_synopsis.rst`` template.\n\n6. Generate the Documentation\n\n make html\n\nLabels and References\n---------------------\n\nTo reference different elements in the API, we have added a custom Sphinx role ``:wurfapi:``\n\nThe ``:wurfapi:`` role will try to deduce the ``unique-name`` from the text given.\nE.g if you want to reference the ``unique-name`` ``foo::bar::baz::func(std::string var)`` and there are\nno other member functions in ``foo::bar::baz`` named ``func``, you can reference it\nby writing ``:wurfapi:`foo::bar::baz::func```.\n\nOn the other hand if there was a function with ``unique-name`` ``foo::bar::baz::function(std::string var)``\n``:wurfapi:`foo::bar::baz::func``` could match with both func and function and will throw an error. In This\ncase this can be fixed by adding the left parenthesis: ``:wurfapi:`foo::bar::baz::func(```.\n\nYou can read more about unique names later in this README.\n\nRunning on readthedocs.org\n--------------------------\n\nTo use this on readthedocs.org you need to have the ``wurfapi`` Sphinx\nextension installed. This can be done by adding a ``requirements.txt`` in the\ndocumentation folder. readthedocs.org can be configured to use the\n``requirements.txt`` when building a project. Simply put ``wurfapi`` in to the\n``requirements.txt``.\n\nDoxygen issues\n--------------\n\nNothing is perfect, neither is Doxygen. Sometimes Doxygen gets it wrong e.g. in\nthe following example::\n\n class foo\n {\n private:\n class bar;\n };\n\nDoxygen incorrectly reports that ``bar`` has public scope (also reported here\nhttps://bit.ly/2BWPllZ). To deal with such issues, until a fix lands in\nDoxygen, you can do the following:\n\nAdd a list of *patches* to the API to your ``conf.py`` file. Extending the\nexample from before, we can add the following fix::\n\n wurfapi = {\n 'source_paths': ['../src'],\n 'recursive': True,\n 'parser': {\n 'type': 'doxygen', 'download': True, 'warnings_as_error': True,\n 'patch_api': [\n {'selector': 'foo::bar', 'key': 'access', 'value': 'private'}\n ]\n }\n }\n\nThe ``patch_api`` allows you to reach in to the parsed API information and\nupdate certain values. The ``selector`` is the ``unique-name`` of the\nentity you want to update. Check the \"Dictionary layout\" section further down\nfor more information.\n\nCollapse inline namespaces\n--------------------------\n\nFor symbol versioning you may use ``inline namespaces``, however typically\nyou don't want these to show up in the docs, as these are mostly\ninvisible for your users.\n\nWith ``wurfapi`` you can collapse the inline namespace such that it\nis removed form the scopes etc.\n\nExample::\n\n namespace foo { inline namespace v1_2_3 { struct bar{}; } }\n\nThe scope to bar is ``foo::v1_2_3``. If you collapse the inline namespace it will\njust be ``foo``.\n\nFirst issue you have to deal with is that Doxygen currently does not\nsupport inline namespaces. So we need to patch the API first::\n\n wurfapi = {\n 'source_paths': ['../src'],\n 'recursive': True,\n 'parser': {\n 'type': 'doxygen', 'download': True, 'warnings_as_error': True,\n 'patch_api': [\n {'selector': 'foo::v1_2_3', 'key': 'inline', 'value': True}\n ]\n }\n }\n\nAfter this we can collapse the namespace::\n\n wurfapi = {\n 'source_paths': ['../src'],\n 'recursive': True,\n 'parser': {\n 'type': 'doxygen', 'download': True, 'warnings_as_error': True,\n 'patch_api': [\n {'selector': 'foo::v1_2_3', 'key': 'inline', 'value': True}\n ],\n 'collapse_inline_namespaces': [\n \"foo::v1_2_3\"\n ]\n }\n }\n\n\nNow you will be able to refer to ``bar`` as ``foo::bar``. Note, that\ncollapsing the namespace will affect the selectors you write when\ngenerating the documentation.\n\nCustom templates\n----------------\n\nYou can write your own custom templates for generating the rst output.\nTo this you simply write a Jinja2 compatible rst template and place\nit in some folder. Adding the ``user_templates`` key to the ``wurfapi``\nconfiguration dictionary in the ``conf.py`` file will make it available.\n\nFor example::\n\n wurfapi = {\n 'source_paths': ['../src', '../examples/header/header.h'],\n 'recursive': True,\n 'user_templates': 'rst_templates',\n 'parser': {\n 'type': 'doxygen', 'download': True, 'warnings_as_error': True\n }\n }\n\n exclude_patterns = ['rst_templates/*.rst']\n\nNow we can use ``*.rst`` files inside the ``rst_templates`` folder e.g. if\nwe had a ``class_list.rst`` template we could use it like this::\n\n .. wurfapi:: class_list.rst\n :selector: project::coffee\n\nRelease new version\n===================\n\n1. Edit ``NEWS.rst``, ``wscript`` and ``src/wurfapi/wurfapi.py`` (set\n correct ``VERSION``)\n\n2. Run ::\n\n ./waf upload\n\n\nSource code\n===========\n\n\nTests\n=====\n\nThe tests will run automatically by passing ``--run_tests`` to waf::\n\n ./waf --run_tests\n\nThis follows what seems to be \"best practice\" advice, namely to install the\npackage in editable mode in a virtualenv.\n\nRecordings\n----------\n\nA bunch of the tests use a library called ``pytest-datarecorder``.\nThe library is used to store the output as files from different parsing and\nrendering operations.\n\nE.g. say we want to make sure that a parser function returns a certain\n``dict`` object. Then we can record that ``dict``::\n\n datarecorder.record_data(\n data={'foo': 2, 'bar': 3},\n recording_file=\"/tmp/recording/test.json\"\n )\n\nIf ``data`` changes compared to a previous recording a mismatch will be\ndetected. To update a recording simply delete the recording file.\n\nTest directories\n----------------\n\nYou will also notice that a bunch of the tests take a parameter called\n``testdirectory``. The ``testdirectory`` is a pytest fixture, which\nrepresents a temporary directory on the filesystem. When running the tests\nyou will notice these temporary test directories pop up under the\n``pytest_temp`` directory in the project root.\n\nYou can read more about that here:\n\n* https://github.com/steinwurf/pytest-testdirectory\n\nDeveloper Notes\n===============\n\nThe `sphinx` documentation on creating extensions:\nhttp://www.sphinx-doc.org/en/stable/extdev/index.html#dev-extensions\n\n* An extension is a Python module. When an extension loads, Sphinx will import\n it and execute its ``setup()`` function.\n\n* Understanding how to put together docutils nodes seems pretty difficult. One\n suggesting form the mailinglist was to look at the following document:\n https://github.com/docutils-mirror/docutils/blob/master/test/functional/expected/standalone_rst_pseudoxml.txt\n\n* While researching how to do this, there seem to be three potential approaches:\n\n 1. Use the standard Sphinx approach and operate with the doctree.\n 2. Create RST based on jinja templates\n 3. Create HTML based on jinja templates\n\n* Inspiration - Sphinx extensions that were used as inspiration while\n developing this extension.\n\n * Breathe\n * Gasp\n * https://github.com/Robpol86/sphinxcontrib-imgur\n * https://github.com/djungelorm/sphinx-tabs\n\n* Understanding how to write stuff with docutils:\n * http://agateau.com/2015/docutils-snippets/\n\n* Creating a custom directive\n * http://www.xavierdupre.fr/blog/2015-06-07_nojs.html\n\n* Nice looking Sphinx extensions\n * https://github.com/bokeh/bokeh/tree/master/bokeh/sphinxext\n\n* This part of the documentation was useful in order to understand the need\n for ViewLists etc. in the directives run(...) function.\n http://www.sphinx-doc.org/en/stable/extdev/markupapi.html\n\n* This link provided inspiration for the text json format: https://github.com/micnews/html-to-article-json\n* More xml->json for the text: https://www.xml.com/pub/a/2006/05/31/converting-between-xml-and-json.html\n\nDictionary layout\n-----------------\n\nWe want to support different \"backends\" like Doxygen to parse the source\ncode. To make this possible we define an internal source code description\nformat. We then translate e.g. Doxygen XML to this and use that to render\nthe API documentation.\n\nThis way a different \"backend\" e.g. Doxygen2 could be use used as the source\ncode parser and the API documentation could be generated.\n\n\n``unique-name``\n...............\n\nIn order to be able to reference the different entities in the API we need\nto assign them a name.\n\nWe use a similar approach here as described in standardese_.\n\nThis means that the ``unique-name`` of an entity is the name with all\nscopes e.g. ``foo::bar::baz``.\n\n* For functions the unique name contains the signature (parameter types and for\n member functions cv-qualifier and ref-qualifier) e.g. ``foo::bar::baz::func()``\n or ``foo::bar::baz::func(int a, char*) const``. See cppreference_ for more\n information.\n\n* For class template specializations the unique name includes the specialization\n arguments. For example::\n\n // Here the unique-name is just 'foo'\n template<class T>\n class foo {};\n\n // Here the unique name is foo<int>\n template<>\n class foo<int> {};\n\n* In addition to types, we also have entries for the parsed files. For files\n the unique name will be the relative path from the project root.\n\n* For defines we will use the name of the define. As an example::\n\n #define PROJECT_VERSION \"1.0.0\"\n\n Here ``unique-name`` will be ``PROJECT_VERSION``.\n\n.. _cppreference: http://en.cppreference.com/w/cpp/language/member_functions\n.. _standardese: https://github.com/foonathan/standardese#linking\n\n\n\nThe API dictionary\n...................\n\nThe internal structure is a dicts with the different API entities. The\n``unique-name`` of the entity is the key and the entity type also a\nPython dictionary is the value e.g::\n\n\n api = {\n 'unique-name': { ... },\n 'unique-name': { ... },\n ...\n }\n\nTo make this a bit more concrete consider the following code::\n\n namespace ns1\n {\n class shape\n {\n void print(int a) const;\n };\n\n namespace ns2\n {\n struct box\n {\n void hello();\n };\n\n void print();\n }\n }\n\nParsing the above code would produce the following API dictionary::\n\n api = {\n 'ns1': { 'kind': 'namespace', ...},\n 'ns1::shape': { 'kind': 'class', ... },\n 'ns1::shape::print(int) const': { kind': function' ... },\n 'ns1::ns2': { 'kind': 'namespace', ... },\n 'ns1::ns2::box': { 'kind': 'struct', ... },\n 'ns1::ns2::box::hello()': { kind': function' ... },\n 'ns1::ns2::print()': { 'kind': 'function', ...},\n 'ns1.hpp': { 'kind': 'file', ...}\n }\n\nThe different entity kinds expose different information about the\nAPI. We will document the different kinds in the following.\n\nWe make some keys *optional* this is marked in the following way::\n\n api = {\n 'unique-name': {\n 'some_key': ...\n Optional('an_optional_key'): ...\n },\n ...\n }\n\n``namespace`` Kind\n..................\n\nPython dictionary representing a C++ namespace::\n\n info = {\n 'kind': 'namespace',\n 'name': 'unqualified-name',\n 'scope': 'unique-name' | None,\n 'members: [ 'unique-name', 'unique-name' ],\n 'briefdescription': paragraphs,\n 'detaileddescription': paragraphs,\n 'inline': True | False\n }\n\nNote: Currently Doxygen does not support parsing ``inline namespaces``. So\nyou need to use the patch API to change the value from ``False`` to ``True``\nmanually. Maybe at some point https://github.com/doxygen/doxygen/issues/6741\nit will be supported.\n\n``class`` | ``struct`` Kind\n...........................\n\nPython dictionary representing a C++ class or struct::\n\n info = {\n 'kind': 'class' | 'struct',\n 'name': 'unqualified-name',\n 'location': location,\n 'scope': 'unique-name' | None,\n 'access': 'public' | 'protected' | 'private',\n Optional('template_parameters'): template_parameters,\n 'members: [ 'unique-name', 'unique-name' ],\n 'briefdescription': paragraphs,\n 'detaileddescription': paragraphs\n }\n\n\n``enum`` | ``enum class`` Kind\n..............................\n\nPython dictionary representing a C++ enum or enum class::\n\n info = {\n 'kind': 'enum',\n 'name': 'unqualified-name',\n 'location': location,\n 'scope': 'unique-name' | None,\n 'access': 'public' | 'protected' | 'private',\n 'values: [\n {\n 'name': 'somename',\n 'briefdescription': paragraphs,\n 'detaileddescription': paragraphs,\n Optional('value'): 'some value'\n }\n ],\n 'briefdescription': paragraphs,\n 'detaileddescription': paragraphs\n }\n\n``typedef`` | ``using`` Kind\n............................\n\nPython dictionary representing a C++ using or typedef statement::\n\n info = {\n 'kind': 'typedef' | 'using',\n 'name': 'unqualified-name',\n 'location': location,\n 'scope': 'unique-name' | None,\n 'access': 'public' | 'protected' | 'private',\n 'type': type,\n 'briefdescription': paragraphs,\n 'detaileddescription': paragraphs\n }\n\n``define`` Kind\n...............\n\nPython dictionary representing a C/C++ define::\n\n info = {\n 'kind': 'define',\n 'name': 'name',\n 'location': location,\n Optional('initializer'): 'some_value',\n Optional('parameters'): [{\n 'name': 'somestring',\n Optional('description'): paragraphs\n }],\n 'briefdescription': paragraphs,\n 'detaileddescription': paragraphs\n }\n\nThe content of the define will be in the ``initializer`` field. If the define\ntakes documented paremeters these will be under the ``parameter`` key.\n\nExamples:\n\n1. Define initializer::\n\n #define VERSION \"1.0.2\"\n\n2. Define initalizer with parameters::\n\n #define min(X, Y) ((X) < (Y) ? (X) : (Y))\n\n\n``file`` Kind\n............................\n\nPython dictionary representing a file in the project::\n\n info = {\n 'kind': 'file',\n 'name': 'somefile.hpp',\n 'path': 'relative/path/to/somefile.hpp',\n }\n\n``function`` Kind\n.................\n\nPython dictionary representing a C++ function::\n\n info = {\n 'kind': 'function',\n 'name': 'unqualified-name',\n 'location': location,\n 'scope': 'unique-name' | None,\n Optional('return'): {\n 'type': type,\n 'description': paragraphs\n }\n Optional('template_parameters'): template_parameters,\n 'is_const': True | False,\n 'is_static': True | False,\n 'is_virtual': True | False,\n 'is_explicit': True | False,\n 'is_inline': True | False,\n 'is_constructor': True | False,\n 'is_destructor': True | False,\n 'trailing_return': True | False,\n 'access': 'public' | 'protected' | 'private',\n 'briefdescription: paragraphs,\n 'detaileddescription: paragraphs,\n 'parameters': [\n { 'type': type, Optional('name'): 'somename', 'description': paragraphs },\n ...\n ]\n }\n\nThe `return` key is optional if the function is either a constructor or\ndestructor.\n\n``variable`` Kind\n.................\n\nPython dictionary representing a C++ variable::\n\n info = {\n 'kind': 'variable',\n 'name': 'unqualified-name',\n Optional('value'): 'some value',\n 'type': type,\n 'location': location,\n 'is_static': True | False,\n 'is_mutable': True | False,\n 'is_volatile': True | False,\n 'is_const': True | False,\n 'is_constexpr': True | False,\n 'scope': 'unique-name' | None,\n 'access': 'public' | 'protected' | 'private',\n 'briefdescription: paragraphs,\n 'detaileddescription: paragraphs,\n }\n\n``location`` item\n.................\n\nPython dictionary representing a location::\n\n location = {\n Optional('include'): 'some/header.h',\n 'path': 'src/project/header.h',\n 'line': 10\n }\n\n* The ``include`` will be relative to any ``include_paths`` specified in the\n ``wurfapi`` dictionary in your Sphinx ``conf.py``.\n\n* The ``path`` will be relative to the project root folder.\n\n``type`` item\n.............\n\nPython list representing a C++ type::\n\n type = [\n {\n 'value': 'sometext',\n Optional('link'): link\n }, ...\n ]\n\nHaving the type as a list of items we can create links to nested types e.g.\nsay we have a `std::unique_ptr<impl>` and we would like to make `impl` a link.\nThis could look like::\n\n \"type\": [\n {\n \"value\": \"std::unique_ptr<\"\n },\n {\n \"link\": {\"url\": False, \"value\": \"project::impl\"},\n \"value\": \"impl\"\n },\n {\n \"value\": \">\"\n }\n ]\n\nAny spaces in the type list should be preserved all the way from the Doxygen\noutput and into the type list. In the rst it should be sufficient to simply\noutput the values of the type. No spaces or other stuff should be injected.\n\n``link`` item\n.............\n\nPython dictionary representing a link::\n\n link = { 'url': True | False, 'value': 'somestring' }\n\nIf `url` is `True` we have a basic extrenal reference otherwise we have a\nlink to an internal type in the API.\n\n``parameter`` item\n...................\n\nDictionary representing a function parameter::\n\n parameter = {\n 'type': type,\n Optional('name'): 'somestring',\n Optional('description'): paragraphs\n }\n\nFor the parameter, the name is also included in the type list. The reason\nis that some parameters can be pretty complex, with the name embedded\ninside the type e.g.::\n\n void function(int (*(*foo)())[3]);\n\nThis is a function that takes one parameter `foo` which is a pointer\nfunction returning a pointer to array 3 of int - nice right? Anyway, in\nsuch cases the parameter name is embedded inside the type of the parameter.\nWe therefore took the easy way out and `wurfapi` will always include the\nparameter name in the type.\n\nAs an example the parameter dictionary for a function `void test(int b)`\ncould be::\n\n {\n 'type': [{'value': 'int '}, {'value': 'b'}],\n 'name': 'b'\n }\n\n``template_parameters`` item\n.............................\n\nPython list of dictionaries representing template parameters::\n\n template_parameters = [{\n 'type': type,\n 'name': 'somestring',\n Optional('default'): type,\n Optional('description'): paragraphs\n }]\n\nText Information\n................\n\nText information is stored in a list of paragraphs::\n\n paragraphs = [paragraph]\n\nA paragraph consists of a list of paragraph elements::\n\n paragraph = [\n {\n \"kind\": \"text\" | \"code\" | \"list\" | \"bold\" | \"italic\",\n ...\n },\n ]\n\nParagraph elements can be one of three kinds, \"text\", \"code\" or \"list\"::\n\n text = {\n 'kind': 'text',\n 'content': 'hello',\n Optional('link'): link\n }\n\n code = {\n 'kind': 'code',\n 'content': 'void print();',\n 'is_block': true | false\n }\n\n list = {\n 'kind': 'list',\n 'ordered': true | false,\n 'items': [paragraphs] # Each item is a list of paragraphs\n }\n\n\nProblem with ``unique-name`` for functions\n..........................................\n\nIssue equivalent C++ function signatures can be written in a number of\ndifferent ways::\n\n void hello(const int *x); // x is a pointer to const int\n void hello(int const *x); // x is a pointer to const int\n\nWe can also move the asterisk (``*``) to the left::\n\n void hello(const int* x); // x is a pointer to const int\n void hello(int const* x); // x is a pointer to const int\n\nSo we need some way to normalize the function signature when transforming it\nto ``unique-name``. We cannot simply rely on sting comparisons.\n\nAccording to the numerous google searches it is hard to write a regex for this.\nInstead we will try to use a parser:\n\n* Python parser: https://github.com/erezsh/lark\n* C++ Grammar: http://www.externsoft.ch/media/swf/cpp11-iso.html#parameters_and_qualifiers\n\nWe only need to parse the function parameter list denoted as the\n``http://www.externsoft.ch/media/swf/cpp11-iso.html#parameters_and_qualifiers``.\n\n\nGenerated output\n----------------\n\nSince we are going to be using Doxygen's XML output as input to the\nextension we need a place to store it. We store it system temporary folder e.g.\nif the project name is \"foobar\" on Linux this would be\n``/tmp/wurfapi-foobar-123456`` where ``123456`` is a hash of the source\ndirectory paths. In addition to Doxygen's XML we also store the generated rst\nfor the different directives there. This is nice for debugging to see whether\nwe generate broken rst.\n\nThe API in json format can be found in the ``_build/.doctree/wurfapi_api.json``.\n\nPaths and directories\n---------------------\n\n\n* Source directory: In Sphinx the source directory is where our .rst files are\n located. This is what you pass to ``sphinx-build`` when building your\n documentation. We will use this in our extension to find the C++ source code\n and output customization templates.\n\n\nNotes\n=====\n\n* Why use an ``src`` folder (https://hynek.me/articles/testing-packaging/).\n tl;dr you should run your tests in the same environment as your users would\n run your code. So by placing the source files in a non-importable folder you\n avoid accidentally having access to resources not added to the Python\n package your users will install...\n* Python packaging guide: https://packaging.python.org/distributing/\n",
"bugtrack_url": null,
"license": "BSD 3-clause \"New\" or \"Revised\" License",
"summary": "C++ Documentation generator.",
"version": "9.1.1",
"project_urls": {
"Homepage": "https://github.com/steinwurf/"
},
"split_keywords": [
"wurfapi"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "fa5cd14abe5e77c18a66a1f35b01ff79bc584413f33610e8dd554e8ecea88e86",
"md5": "d197ab2c39b7c5dd8b901e76a3f33a4b",
"sha256": "ca6fec11769e650fe92c7d80f7dd8866aba1cd0559c56786aeb0ea413d458f8d"
},
"downloads": -1,
"filename": "wurfapi-9.1.1-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "d197ab2c39b7c5dd8b901e76a3f33a4b",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": null,
"size": 48600,
"upload_time": "2023-12-08T13:51:30",
"upload_time_iso_8601": "2023-12-08T13:51:30.952961Z",
"url": "https://files.pythonhosted.org/packages/fa/5c/d14abe5e77c18a66a1f35b01ff79bc584413f33610e8dd554e8ecea88e86/wurfapi-9.1.1-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-12-08 13:51:30",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "wurfapi"
}
JSON