[](https://github.com/lamarrsim/SQLamarr)
[](https://lamarrsim.github.io/SQLamarr)
# SQLamarr
*The stand-alone ultra-fast simulation option for the LHCb experiment.*
The detailed simulation of the hadron collisions at the LHC, and of the
interaction of the generated particles with the detector material
dominates the cost for the computing infrastructure pledged to the
LHCb Collaboration.
Among the various options explored towards a faster simulation,
there is Lamarr, a framework defining a pipeline of parametrizations
transforming generator-level quantities to reconstructed, analysis-level
features.
Most of the parametrizations are defined using *machine-learning*, and
in particular Deep Neural Networks and Gradient Boosted Decision Trees,
with a training procedure defined in independent packages (*e.g.*
[landerlini/lb-trksim-train](https://github.com/landerlini/lb-trksim-train)
and [mbarbetti/lb-pidsim-train](https://github.com/mbarbetti/lb-pidsim-train)).
To be integrated in the LHCb software stack, models must be queried
from a C++ application, running in the Gaudi framework, which includes a
dedicated multithreading scheduler which was found to conflict with
the schedulers of TensorFlow and ONNX runtimes.
In addition, since the models are relatively simple and fast to
evaluate, the overhead of context switching from Gaudi to a dedicated
runtime was observed to be unaccptably large.
Hence, models are converted into compatible C code using the
[landerlini/scikinC](https://github.com/landerlini/scikinC)
package and distributed through the CernVM FileSystem releasing
the [LamarrData package](https://gitlab.cern.ch/lhcb-datapkg/LamarrData).
While crucial to the applications within LHCb, the integration with
Gaudi and [Gauss](https://gitlab.cern.ch/lhcb/Gauss) makes the adoption
of Lamarr unappealing for researchers outside of the LHCb community
approaching the LHCb simulation to evaluate
the experiment sensitivity to new physics phenomena or studying the
recently-released LHCb Open Data.
The [landerlini/SQLamarr](https://github.com/landerlini/SQLamarr)
package aims at decoupling Lamarr from Gaudi providing a stand-alone
application with minimal dependencies that can be easily set up and
run in any Linux machine.
The parametrizations are shared between the Gauss-embedded implementation
[LbLamarr](https://gitlab.cern.ch/lhcb/Gauss/-/tree/master/Sim/LbLamarr)
and `SQLamarr`.
In the future, the exact same package might be integrated within Gaudi
to reduce the maintainance effort.
To replace the ROOT-based TransientEventStore concept defind in Gaudi,
`SQLamarr` adopts the SQLite3 package, enabling vectorized processing
of batches of events, for a better performance.
To avoid dependencies on ROOT, also the persistency is handled using
SQLite3, writing the reconstructed (or intermediate) quantities in the
form of SQLite3 databases.
Note that converting an SQLite3 table to a ROOT nTuple requires no more
than 3 lines of Python:
```python
import sqlite3, uproot, pandas
with sqlite3.connect("SomeInput.db") as conn:
uproot.open("SomeFile.root", "RECREATE")["myTree"] = pandas.read_sql_table("myTable", conn)
```
## Dependencies
* [SQLite3](https://www.sqlite.org/index.html) with C/C++ headers
* [HepMC3](http://hepmc.web.cern.ch/hepmc/) as a standard interface
to event generators.
## Build from source
Make sure you have conda (or similar) installed, if not
get [miniconda3](https://docs.conda.io/en/latest/miniconda.html).
Create and activate a dedicated conda environment, say `sqlamarr`:
```bash
conda create -y -n sqlamarr -c conda-forge python=3.10 gxx gxx_linux-64 hepmc3 doxygen
conda activate sqlamarr
```
Create a out-of-source directory:
```bash
mkdir build
cd build
```
Configure and build
```bash
cmake ..
cmake --build .
```
## How to use SQLamarr
The project is not mature enough to provide a good user experience.
For the time being, clone the repository and compile the package with CMake,
then edit the file `src/main.cpp` to
define the desired pipeline, by using the building blocks provided by
the package.
## BlockLib
The namespace `SQLamarr::BlockLib` groups functions defining specialized blocks
making assumptions on the workflow in which these blocks will be deployed.
While useful for testing and for organizing the code defining pipelines,
it is not supposed to be stable (as it is being modified while the workflow
under test is modified) and should not be used as part of other packages.
Other packages, however, may take inspiration from the structure of `SQLamarr::BlockLib`
to design specialized blocks, resident in their codebase, in a more
organized way than having everything pipelined in a single file.
To test the completeness of the feature set in the main part of the library,
`SQLamarr::BlockLib` is designed to only include functions, accessing public methods
of the objects defined in the main part of the library.
## Copyright and Licence
(c) Copyright 2022 CERN for the benefit of the LHCb Collaboration.
This software is distributed under the terms of the GNU General Public Licence version 3 (GPL Version 3), copied verbatim in the file "LICENCE".
In applying this licence, CERN does not waive the privileges and immunities granted to it by virtue of its status as an Intergovernmental Organization or submit itself to any jurisdiction.
We acknowledge the support of the ICSC Foundation to the development of SQLamarr.
Raw data
{
"_id": null,
"home_page": null,
"name": "SQLamarr",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "lhcb, fast-simulation, simulation, hep, physics",
"author": null,
"author_email": "Lucio Anderlini <Lucio.Anderlini@fi.infn.it>, Matteo Barbetti <Matteo.Barbetti@fi.infn.it>",
"download_url": null,
"platform": null,
"description": "[](https://github.com/lamarrsim/SQLamarr)\n[](https://lamarrsim.github.io/SQLamarr)\n\n\n\n# SQLamarr\n*The stand-alone ultra-fast simulation option for the LHCb experiment.*\n\nThe detailed simulation of the hadron collisions at the LHC, and of the \ninteraction of the generated particles with the detector material\ndominates the cost for the computing infrastructure pledged to the \nLHCb Collaboration.\n\nAmong the various options explored towards a faster simulation, \nthere is Lamarr, a framework defining a pipeline of parametrizations\ntransforming generator-level quantities to reconstructed, analysis-level \nfeatures. \nMost of the parametrizations are defined using *machine-learning*, and \nin particular Deep Neural Networks and Gradient Boosted Decision Trees,\nwith a training procedure defined in independent packages (*e.g.* \n[landerlini/lb-trksim-train](https://github.com/landerlini/lb-trksim-train)\nand [mbarbetti/lb-pidsim-train](https://github.com/mbarbetti/lb-pidsim-train)).\n\nTo be integrated in the LHCb software stack, models must be queried \nfrom a C++ application, running in the Gaudi framework, which includes a \ndedicated multithreading scheduler which was found to conflict with \nthe schedulers of TensorFlow and ONNX runtimes.\nIn addition, since the models are relatively simple and fast to \nevaluate, the overhead of context switching from Gaudi to a dedicated \nruntime was observed to be unaccptably large.\nHence, models are converted into compatible C code using the \n[landerlini/scikinC](https://github.com/landerlini/scikinC)\npackage and distributed through the CernVM FileSystem releasing \nthe [LamarrData package](https://gitlab.cern.ch/lhcb-datapkg/LamarrData).\n\nWhile crucial to the applications within LHCb, the integration with\nGaudi and [Gauss](https://gitlab.cern.ch/lhcb/Gauss) makes the adoption \nof Lamarr unappealing for researchers outside of the LHCb community \napproaching the LHCb simulation to evaluate \nthe experiment sensitivity to new physics phenomena or studying the \nrecently-released LHCb Open Data.\nThe [landerlini/SQLamarr](https://github.com/landerlini/SQLamarr)\npackage aims at decoupling Lamarr from Gaudi providing a stand-alone \napplication with minimal dependencies that can be easily set up and \nrun in any Linux machine.\nThe parametrizations are shared between the Gauss-embedded implementation\n[LbLamarr](https://gitlab.cern.ch/lhcb/Gauss/-/tree/master/Sim/LbLamarr) \nand `SQLamarr`.\nIn the future, the exact same package might be integrated within Gaudi \nto reduce the maintainance effort.\n\nTo replace the ROOT-based TransientEventStore concept defind in Gaudi,\n`SQLamarr` adopts the SQLite3 package, enabling vectorized processing \nof batches of events, for a better performance.\n\nTo avoid dependencies on ROOT, also the persistency is handled using \nSQLite3, writing the reconstructed (or intermediate) quantities in the \nform of SQLite3 databases. \nNote that converting an SQLite3 table to a ROOT nTuple requires no more \nthan 3 lines of Python:\n\n```python\nimport sqlite3, uproot, pandas\nwith sqlite3.connect(\"SomeInput.db\") as conn:\n uproot.open(\"SomeFile.root\", \"RECREATE\")[\"myTree\"] = pandas.read_sql_table(\"myTable\", conn)\n```\n\n## Dependencies\n * [SQLite3](https://www.sqlite.org/index.html) with C/C++ headers\n * [HepMC3](http://hepmc.web.cern.ch/hepmc/) as a standard interface\n to event generators.\n\n## Build from source\nMake sure you have conda (or similar) installed, if not \nget [miniconda3](https://docs.conda.io/en/latest/miniconda.html).\nCreate and activate a dedicated conda environment, say `sqlamarr`:\n```bash\nconda create -y -n sqlamarr -c conda-forge python=3.10 gxx gxx_linux-64 hepmc3 doxygen\nconda activate sqlamarr\n```\n\nCreate a out-of-source directory:\n```bash\nmkdir build\ncd build\n```\n\nConfigure and build\n```bash\ncmake .. \ncmake --build .\n```\n\n\n## How to use SQLamarr\nThe project is not mature enough to provide a good user experience.\nFor the time being, clone the repository and compile the package with CMake, \nthen edit the file `src/main.cpp` to\ndefine the desired pipeline, by using the building blocks provided by\nthe package.\n\n## BlockLib\nThe namespace `SQLamarr::BlockLib` groups functions defining specialized blocks\nmaking assumptions on the workflow in which these blocks will be deployed.\nWhile useful for testing and for organizing the code defining pipelines, \nit is not supposed to be stable (as it is being modified while the workflow\nunder test is modified) and should not be used as part of other packages. \n\nOther packages, however, may take inspiration from the structure of `SQLamarr::BlockLib`\nto design specialized blocks, resident in their codebase, in a more \norganized way than having everything pipelined in a single file.\n\nTo test the completeness of the feature set in the main part of the library,\n`SQLamarr::BlockLib` is designed to only include functions, accessing public methods \nof the objects defined in the main part of the library.\n\n\n## Copyright and Licence\n(c) Copyright 2022 CERN for the benefit of the LHCb Collaboration. \n \nThis software is distributed under the terms of the GNU General Public Licence version 3 (GPL Version 3), copied verbatim in the file \"LICENCE\".\n \nIn applying this licence, CERN does not waive the privileges and immunities granted to it by virtue of its status as an Intergovernmental Organization or submit itself to any jurisdiction.\n\nWe acknowledge the support of the ICSC Foundation to the development of SQLamarr.\n\n\n\n",
"bugtrack_url": null,
"license": "GPL-3",
"summary": "The stand-alone ultra-fast simulation option for LHCb",
"version": "0.0rc7",
"project_urls": {
"Homepage": "https://lamarrsim.github.io/SQLamarr/",
"Source": "https://github.com/LamarrSim/SQLamarr"
},
"split_keywords": [
"lhcb",
" fast-simulation",
" simulation",
" hep",
" physics"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "5a6c3cfa6cc65a4a691d2acf288725ba5d8e7cf32ae080d097701e4b6f92231e",
"md5": "7b5c069c7968c4b31c18d491f466cb58",
"sha256": "821d66d94fab464898bf37b739439911d24bfd67870c052524309edbf4e8f81c"
},
"downloads": -1,
"filename": "SQLamarr-0.0rc7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "7b5c069c7968c4b31c18d491f466cb58",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": null,
"size": 2229045,
"upload_time": "2024-05-31T06:36:11",
"upload_time_iso_8601": "2024-05-31T06:36:11.675463Z",
"url": "https://files.pythonhosted.org/packages/5a/6c/3cfa6cc65a4a691d2acf288725ba5d8e7cf32ae080d097701e4b6f92231e/SQLamarr-0.0rc7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a3b83be8beaa50d90e040384262cbd83272047e34aafdc1d64bd729399c98a10",
"md5": "342c481cbd461ecf6b9773e48a780c0d",
"sha256": "3188234dbf4df4751af0a5a1a8c56b1de957d3e35800c4d63bc8fa39aabe40b6"
},
"downloads": -1,
"filename": "SQLamarr-0.0rc7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "342c481cbd461ecf6b9773e48a780c0d",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": null,
"size": 2229046,
"upload_time": "2024-05-31T06:36:14",
"upload_time_iso_8601": "2024-05-31T06:36:14.005506Z",
"url": "https://files.pythonhosted.org/packages/a3/b8/3be8beaa50d90e040384262cbd83272047e34aafdc1d64bd729399c98a10/SQLamarr-0.0rc7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "8d63bec5f510766534f329ee1634d7cf17a1e8deb4e6fa0f1cedc71c415045f4",
"md5": "dc43adb29195557db3ee7aa9d126b70e",
"sha256": "36599a069b638734d9829cb71452496e34589fc9abacdd56f3bcf8366492f3c9"
},
"downloads": -1,
"filename": "SQLamarr-0.0rc7-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "dc43adb29195557db3ee7aa9d126b70e",
"packagetype": "bdist_wheel",
"python_version": "cp312",
"requires_python": null,
"size": 2229155,
"upload_time": "2024-05-31T06:36:16",
"upload_time_iso_8601": "2024-05-31T06:36:16.791824Z",
"url": "https://files.pythonhosted.org/packages/8d/63/bec5f510766534f329ee1634d7cf17a1e8deb4e6fa0f1cedc71c415045f4/SQLamarr-0.0rc7-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "740ce4bf610eeb92c3f80e8e3e6be036b0f7552fb8b074e866b06e57b92fb75d",
"md5": "ad097d93f543245554f3c292e817ceb5",
"sha256": "ebff593a9a2657119bae3ac5867e1a585bf343be1189277141ac6b903a42d6ea"
},
"downloads": -1,
"filename": "SQLamarr-0.0rc7-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "ad097d93f543245554f3c292e817ceb5",
"packagetype": "bdist_wheel",
"python_version": "cp37",
"requires_python": null,
"size": 2229047,
"upload_time": "2024-05-31T06:36:18",
"upload_time_iso_8601": "2024-05-31T06:36:18.623461Z",
"url": "https://files.pythonhosted.org/packages/74/0c/e4bf610eeb92c3f80e8e3e6be036b0f7552fb8b074e866b06e57b92fb75d/SQLamarr-0.0rc7-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "7af843acfe1a536468b20145e39fdce517babc98eb49e78d5f3805465bffbdfb",
"md5": "c18f48aa23934a5ad83cc549566d1a08",
"sha256": "c5eee977b6825e009f673e759b36c88900ecd7a3184bd2a750f7b81f24e150b2"
},
"downloads": -1,
"filename": "SQLamarr-0.0rc7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "c18f48aa23934a5ad83cc549566d1a08",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": null,
"size": 2229040,
"upload_time": "2024-05-31T06:36:20",
"upload_time_iso_8601": "2024-05-31T06:36:20.415462Z",
"url": "https://files.pythonhosted.org/packages/7a/f8/43acfe1a536468b20145e39fdce517babc98eb49e78d5f3805465bffbdfb/SQLamarr-0.0rc7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "557b3e2f0ca08f2836296adff60ed9d99382e12659543cd00a7fe03262818e2d",
"md5": "d8b025781193131fe02399df3ef7bba4",
"sha256": "8e899a67943ca58ed2f574157ec3d6312017be34ca440c7b36ab5e9fc4afe463"
},
"downloads": -1,
"filename": "SQLamarr-0.0rc7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "d8b025781193131fe02399df3ef7bba4",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": null,
"size": 2229040,
"upload_time": "2024-05-31T06:36:23",
"upload_time_iso_8601": "2024-05-31T06:36:23.052603Z",
"url": "https://files.pythonhosted.org/packages/55/7b/3e2f0ca08f2836296adff60ed9d99382e12659543cd00a7fe03262818e2d/SQLamarr-0.0rc7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-31 06:36:11",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "LamarrSim",
"github_project": "SQLamarr",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "sqlamarr"
}
JSON
