# NOTED: a framework to optimise network traffic via the analysis of data from File Transfer Services
Copyright:
```
© Copyright 2022 CERN. This software is distributed under the terms of
the GNU General Public Licence version 3 (GPL Version 3), copied verbatim
in the file "LICENCE.txt". In applying this licence, CERN does not waive
the privileges and immunities granted to it by virtue of its status as an
Intergovernmental Organization or submit itself to any jurisdiction.
```
PyPI Compilation steps:
```
# Steps to install NOTED by using a virtual environment
$ yum install python3-pip -y
$ yum install python3.8
$ pip3 install virtualenv
$ python3.8 -m venv venv-noted
$ . venv-noted/bin/activate
(venv-noted) $ python3 -m pip install noted-dev
# Write your configuration file, there is one example in noted/config/
(venv-noted) $ nano noted/src/noted/config/config.yaml
# Run NOTED
# (venv-noted) $ noted noted/src/noted/config/config.yaml [--verbosity debug/info/warning]
```
Docker Compilation steps:
```
# Download noted docker container
$ docker pull noted-docker
# Run docker container and keep it running in the background
$ docker run --detach --entrypoint /sbin/init --network="host" --privileged --name noted_cern_kit_controller noted-docker
# Copy config file into the container
$ docker cp src/noted/config/config-example.yaml noted_cern_kit_controller:/app/noted/config
# Run commands in the container from outside
$ docker exec noted_cern_kit_controller noted -h
$ docker exec noted_cern_kit_controller /app/src/noted/scripts/setup.sh mail
$ docker exec noted_cern_kit_controller noted config/config-example.yaml &
# Stop noted
$ docker exec noted_cern_kit_controller pkill noted
# Optional: login into the container (NOTE: type "exit" to logout)
$ docker exec -it noted_cern_kit_controller /bin/bash
```
Program description:
```
noted -h
usage: noted [-h] [-v VERBOSITY] config_file
NOTED: a framework to optimise network traffic via the analysis of data from File Transfer Services.
positional arguments:
config_file the name of the configuration file [config-example.yaml]
optional arguments:
-h, --help show this help message and exit
-v VERBOSITY, --verbosity VERBOSITY
defines the logging level [debug, info, warning]
```
Folder structure of NOTED:
1. In config folder is available one or more config.yaml where the user defines the parameters to monitor the links.
2. In logs folder:
noted_email.txt: the last email notification that has been send to the responsible of the link to provide/cancel the dynamic circuit.
transfer_broker.log: the log of NOTED [one log per execution].
sense.log: the log of sense-o, it is generated when NOTED calls the sense-o API [one log per execution].
3. In query folder are available the queries to CERN Kibana [this folder is transparent to the user, don't use it].
4. In transfers folder:
transfer_broker_all_transfers.txt: file with all the transfers in the link even if the source/destination dynamic circuits are down.
transfer_broker_src_rcsite.txt: file with the transfers of {src -> dst} when the source dynamic circuit is up, otherwise the transfers are not saved.
transfer_broker_dst_rcsite.txt: file with the transfers of {dst -> src} when the destination dynamic circuit is up, otherwise the transfers are not saved.
```
noted
├── config
│ └── config.yaml
├── logs
│ ├── noted_email.txt
│ └── sense.log
│ └── transfer_broker.log
├── query
│ ├── query_monit_prod_fts_raw_queue_dst_rcsite
│ └── query_monit_prod_fts_raw_queue_src_rcsite
├── sense-o
│ │ ├── sense-cancel.sh
│ │ └── sense-provision.sh
└── transfers
├── transfer_broker_all_transfers.txt
├── transfer_broker_dst_rcsite.txt
└── transfer_broker_src_rcsite.txt
```
Structure of NOTED repository:
```
.
├── README.md
└── noted_transfer_broker
├── COPYRIGHT.txt
├── LICENCE.txt
├── Makefile
├── MANIFEST.in
├── README.md
├── setup.cfg
├── setup.py
└── src
├── noted
├── __init__.py
├── main.py
├── config
│ └── config-example.yaml
├── documentation
│ ├── noted_main_function_documentation.pdf
│ ├── noted_transfer_broker_class_documentation.pdf
│ ├── reduced_noted_main_function_documentation.pdf
│ └── reduced_noted_transfer_broker_class_documentation.pdf
├── html
│ ├── TransferBroker.html
│ └── main.html
├── logs
│ ├── noted_email.txt
│ ├── sense.log
│ ├── transfer_broker.log
├── modules
│ ├── __init__.py
│ ├── plot_transfers.py
│ └── transferbroker.py
├── params
│ └── params.ini
├── query
│ ├── query_monit_prod_fts_raw_queue_dst_rcsite
│ └── query_monit_prod_fts_raw_queue_src_rcsite
├── scripts
│ └── setup.sh
├── sense-o
│ ├── sense-cancel.sh
│ └── sense-provision.sh
└── transfers
├── transfer_broker_all_transfers.txt
├── transfer_broker_dst_rcsite.txt
└── transfer_broker_src_rcsite.txt
```
Raw data
{
"_id": null,
"home_page": "https://gitlab.cern.ch/mmisamor/noted",
"name": "noted-dev",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3",
"maintainer_email": "",
"keywords": "networking,monitoring,transfers,dynamic circuit,load balance,fts,sense-o,spectrum",
"author": "Carmen Misa Moreira, Edoardo Martelli (CERN IT-CS-NE)",
"author_email": "carmen.misa.moreira@cern.ch",
"download_url": "https://files.pythonhosted.org/packages/b4/7f/fdb18c17787b23b9470d923f0855afb144029b13b3bc1ce6f093168dc8a0/noted-dev-1.2.tar.gz",
"platform": null,
"description": "# NOTED: a framework to optimise network traffic via the analysis of data from File Transfer Services\n\nCopyright:\n```\n\u00a9 Copyright 2022 CERN. This software is distributed under the terms of\nthe GNU General Public Licence version 3 (GPL Version 3), copied verbatim\nin the file \"LICENCE.txt\". In applying this licence, CERN does not waive\nthe privileges and immunities granted to it by virtue of its status as an\nIntergovernmental Organization or submit itself to any jurisdiction.\n```\n\nPyPI Compilation steps:\n```\n# Steps to install NOTED by using a virtual environment\n$ yum install python3-pip -y\n$ yum install python3.8\n$ pip3 install virtualenv \n$ python3.8 -m venv venv-noted\n$ . venv-noted/bin/activate\n(venv-noted) $ python3 -m pip install noted-dev\n# Write your configuration file, there is one example in noted/config/\n(venv-noted) $ nano noted/src/noted/config/config.yaml\n# Run NOTED\n# (venv-noted) $ noted noted/src/noted/config/config.yaml [--verbosity debug/info/warning]\n```\n\nDocker Compilation steps:\n```\n# Download noted docker container\n$ docker pull noted-docker\n# Run docker container and keep it running in the background\n$ docker run --detach --entrypoint /sbin/init --network=\"host\" --privileged --name noted_cern_kit_controller noted-docker\n# Copy config file into the container\n$ docker cp src/noted/config/config-example.yaml noted_cern_kit_controller:/app/noted/config\n# Run commands in the container from outside\n$ docker exec noted_cern_kit_controller noted -h\n$ docker exec noted_cern_kit_controller /app/src/noted/scripts/setup.sh mail\n$ docker exec noted_cern_kit_controller noted config/config-example.yaml &\n# Stop noted\n$ docker exec noted_cern_kit_controller pkill noted\n\n# Optional: login into the container (NOTE: type \"exit\" to logout)\n$ docker exec -it noted_cern_kit_controller /bin/bash\n```\n\nProgram description:\n```\nnoted -h\nusage: noted [-h] [-v VERBOSITY] config_file\n\nNOTED: a framework to optimise network traffic via the analysis of data from File Transfer Services.\n\npositional arguments:\n config_file the name of the configuration file [config-example.yaml]\n\noptional arguments:\n -h, --help show this help message and exit\n -v VERBOSITY, --verbosity VERBOSITY\n defines the logging level [debug, info, warning]\n```\n\nFolder structure of NOTED:\n1. In config folder is available one or more config.yaml where the user defines the parameters to monitor the links.\n2. In logs folder:\n noted_email.txt: the last email notification that has been send to the responsible of the link to provide/cancel the dynamic circuit.\n transfer_broker.log: the log of NOTED [one log per execution].\n sense.log: the log of sense-o, it is generated when NOTED calls the sense-o API [one log per execution].\n3. In query folder are available the queries to CERN Kibana [this folder is transparent to the user, don't use it].\n4. In transfers folder:\n transfer_broker_all_transfers.txt: file with all the transfers in the link even if the source/destination dynamic circuits are down.\n transfer_broker_src_rcsite.txt: file with the transfers of {src -> dst} when the source dynamic circuit is up, otherwise the transfers are not saved.\n transfer_broker_dst_rcsite.txt: file with the transfers of {dst -> src} when the destination dynamic circuit is up, otherwise the transfers are not saved.\n```\nnoted\n\u251c\u2500\u2500 config\n\u2502 \u2514\u2500\u2500 config.yaml\n\u251c\u2500\u2500 logs\n\u2502 \u251c\u2500\u2500 noted_email.txt\n\u2502 \u2514\u2500\u2500 sense.log\n\u2502 \u2514\u2500\u2500 transfer_broker.log\n\u251c\u2500\u2500 query\n\u2502 \u251c\u2500\u2500 query_monit_prod_fts_raw_queue_dst_rcsite\n\u2502 \u2514\u2500\u2500 query_monit_prod_fts_raw_queue_src_rcsite\n\u251c\u2500\u2500 sense-o\n\u2502 \u2502 \u251c\u2500\u2500 sense-cancel.sh\n\u2502 \u2502 \u2514\u2500\u2500 sense-provision.sh\n\u2514\u2500\u2500 transfers\n \u251c\u2500\u2500 transfer_broker_all_transfers.txt\n \u251c\u2500\u2500 transfer_broker_dst_rcsite.txt\n \u2514\u2500\u2500 transfer_broker_src_rcsite.txt\n```\n\nStructure of NOTED repository:\n```\n.\n\u251c\u2500\u2500 README.md\n\u2514\u2500\u2500 noted_transfer_broker\n \u251c\u2500\u2500 COPYRIGHT.txt\n \u251c\u2500\u2500 LICENCE.txt\n \u251c\u2500\u2500 Makefile\n \u251c\u2500\u2500 MANIFEST.in\n \u251c\u2500\u2500 README.md\n \u251c\u2500\u2500 setup.cfg\n \u251c\u2500\u2500 setup.py\n \u2514\u2500\u2500 src\n \u251c\u2500\u2500 noted\n \u251c\u2500\u2500 __init__.py\n \u251c\u2500\u2500 main.py\n \u251c\u2500\u2500 config\n \u2502 \u2514\u2500\u2500 config-example.yaml\n \u251c\u2500\u2500 documentation\n \u2502 \u251c\u2500\u2500 noted_main_function_documentation.pdf\n \u2502 \u251c\u2500\u2500 noted_transfer_broker_class_documentation.pdf\n \u2502 \u251c\u2500\u2500 reduced_noted_main_function_documentation.pdf\n \u2502 \u2514\u2500\u2500 reduced_noted_transfer_broker_class_documentation.pdf\n \u251c\u2500\u2500 html\n \u2502 \u251c\u2500\u2500 TransferBroker.html\n \u2502 \u2514\u2500\u2500 main.html\n \u251c\u2500\u2500 logs\n \u2502 \u251c\u2500\u2500 noted_email.txt\n \u2502 \u251c\u2500\u2500 sense.log\n \u2502 \u251c\u2500\u2500 transfer_broker.log\n \u251c\u2500\u2500 modules\n \u2502 \u251c\u2500\u2500 __init__.py\n \u2502 \u251c\u2500\u2500 plot_transfers.py\n \u2502 \u2514\u2500\u2500 transferbroker.py\n \u251c\u2500\u2500 params\n \u2502 \u2514\u2500\u2500 params.ini\n \u251c\u2500\u2500 query\n \u2502 \u251c\u2500\u2500 query_monit_prod_fts_raw_queue_dst_rcsite\n \u2502 \u2514\u2500\u2500 query_monit_prod_fts_raw_queue_src_rcsite\n \u251c\u2500\u2500 scripts\n \u2502 \u2514\u2500\u2500 setup.sh\n \u251c\u2500\u2500 sense-o\n \u2502 \u251c\u2500\u2500 sense-cancel.sh\n \u2502 \u2514\u2500\u2500 sense-provision.sh\n \u2514\u2500\u2500 transfers\n \u251c\u2500\u2500 transfer_broker_all_transfers.txt\n \u251c\u2500\u2500 transfer_broker_dst_rcsite.txt\n \u2514\u2500\u2500 transfer_broker_src_rcsite.txt\n```\n",
"bugtrack_url": null,
"license": "GPLv3 (GNU General Public License)",
"summary": "NOTED: a framework to optimise network traffic via the analysis of data from File Transfer Services",
"version": "1.2",
"project_urls": {
"Homepage": "https://gitlab.cern.ch/mmisamor/noted",
"Source": "https://gitlab.cern.ch/mmisamor/noted"
},
"split_keywords": [
"networking",
"monitoring",
"transfers",
"dynamic circuit",
"load balance",
"fts",
"sense-o",
"spectrum"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "b47ffdb18c17787b23b9470d923f0855afb144029b13b3bc1ce6f093168dc8a0",
"md5": "a41fdca5c4059a84f9dd24f69175acce",
"sha256": "1468533c9f96c9a98023f1410c615ce66577f4f254bf8cd3f3ab265bb9b9ca22"
},
"downloads": -1,
"filename": "noted-dev-1.2.tar.gz",
"has_sig": false,
"md5_digest": "a41fdca5c4059a84f9dd24f69175acce",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3",
"size": 334678,
"upload_time": "2023-11-11T11:30:26",
"upload_time_iso_8601": "2023-11-11T11:30:26.959142Z",
"url": "https://files.pythonhosted.org/packages/b4/7f/fdb18c17787b23b9470d923f0855afb144029b13b3bc1ce6f093168dc8a0/noted-dev-1.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-11 11:30:26",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "noted-dev"
}
JSON