Cornflow
=========
.. image:: https://github.com/baobabsoluciones/cornflow/workflows/build/badge.svg?style=svg
:target: https://github.com/baobabsoluciones/cornflow/actions
.. image:: https://github.com/baobabsoluciones/cornflow/workflows/docs/badge.svg?style=svg
:target: https://github.com/baobabsoluciones/cornflow/actions
.. image:: https://github.com/baobabsoluciones/cornflow/workflows/integration/badge.svg?style=svg
:target: https://github.com/baobabsoluciones/cornflow/actions
.. image:: https://img.shields.io/pypi/v/cornflow-client.svg?style=svg
:target: https://pypi.python.org/pypi/cornflow-client
.. image:: https://img.shields.io/pypi/pyversions/cornflow-client.svg?style=svg
:target: https://pypi.python.org/pypi/cornflow-client
.. image:: https://img.shields.io/badge/License-Apache2.0-blue
Cornflow is an open source multi-solver optimization server with a REST API built using `flask <https://flask.palletsprojects.com>`_, `airflow <https://airflow.apache.org/>`_ and `pulp <https://coin-or.github.io/pulp/>`_.
While most deployment servers are based on the solving technique (MIP, CP, NLP, etc.), Cornflow focuses on the optimization problems themselves. However, it does not impose any constraint on the type of problem and solution method to use.
With Cornflow you can deploy a Traveling Salesman Problem solver next to a Knapsack solver or a Nurse Rostering Problem solver. As long as you describe the input and output data, you can upload any solution method for any problem and then use it with any data you want.
Cornflow helps you formalize your problem by proposing development guidelines. It also provides a range of functionalities around your deployed solution method, namely:
* storage of users, instances, solutions and solution logs.
* deployment and maintenance of models, solvers and algorithms.
* scheduling of executions in remote machines.
* management of said executions: start, monitor, interrupt.
* centralizing of commercial licenses.
* scenario storage and comparison.
* user management, roles and groups.
.. contents:: **Table of Contents**
Installation instructions
-------------------------------
Cornflow is tested with Ubuntu 20.04, python >= 3.8 and git.
Download the Cornflow project and install requirements::
python3 -m venv venv
venv/bin/pip3 install cornflow
initialize the sqlite database::
source venv/bin/activate
export FLASK_APP=cornflow.app
export DATABASE_URL=sqlite:///cornflow.db
flask db upgrade
flask access_init
flask create_service_user -u airflow -e airflow_test@admin.com -p airflow_test_password
flask create_admin_user -u cornflow -e cornflow_admin@admin.com -p cornflow_admin_password
activate the virtual environment and run Cornflow::
source venv/bin/activate
export FLASK_APP=cornflow.app
export SECRET_KEY=THISNEEDSTOBECHANGED
export DATABASE_URL=sqlite:///cornflow.db
export AIRFLOW_URL=http://127.0.0.1:8080/
export AIRFLOW_USER=airflow_user
export AIRFLOW_PWD=airflow_pwd
flask run
**Cornflow needs a running installation of Airflow to operate and more configuration**. Check `the installation docs <https://baobabsoluciones.github.io/cornflow/main/install.html>`_ for more details on installing airflow, configuring the application and initializing the database.
Using cornflow to solve a PuLP model
---------------------------------------
We're going to test the cornflow server by using the `cornflow-client` and the `pulp` python package::
pip install cornflow-client pulp
Initialize the api client::
from cornflow_client import CornFlow
email = 'some_email@gmail.com'
pwd = 'Some_password1'
username = 'some_name'
client = CornFlow(url="http://127.0.0.1:5000")
Create a user::
config = dict(username=username, email=email, pwd=pwd)
client.sign_up(**config)
Log in::
client.login(username=username, pwd=pwd)
Prepare an instance::
import pulp
prob = pulp.LpProblem("test_export_dict_MIP", pulp.LpMinimize)
x = pulp.LpVariable("x", 0, 4)
y = pulp.LpVariable("y", -1, 1)
z = pulp.LpVariable("z", 0, None, pulp.LpInteger)
prob += x + 4 * y + 9 * z, "obj"
prob += x + y <= 5, "c1"
prob += x + z >= 10, "c2"
prob += -y + z == 7.5, "c3"
data = prob.to_dict()
insName = 'test_export_dict_MIP'
description = 'very small example'
Send instance::
instance = client.create_instance(data, name=insName, description=description, schema="solve_model_dag",)
Solve an instance::
config = dict(
solver = "PULP_CBC_CMD",
timeLimit = 10
)
execution = client.create_execution(
instance['id'], config, name='execution1', description='execution of a very small instance',
schema="solve_model_dag",
)
Check the status of an execution::
status = client.get_status(execution["id"])
print(status['state'])
# 1 means "finished correctly"
Retrieve a solution::
results = client.get_solution(execution['id'])
print(results['data'])
# returns a json with the solved pulp object
_vars, prob = pulp.LpProblem.from_dict(results['data'])
Retrieve the log of the solver::
log = client.get_log(execution['id'])
print(log['log'])
# json format of the solver log
Using cornflow to deploy a solution method
---------------------------------------------
To deploy a cornflow solution method, the following tasks need to be accomplished:
#. Create an Application for the new problem
#. Do a PR to a compatible repo linked to a server instance (e.g., like `this one <https://github.com/baobabsoluciones/cornflow>`_).
For more details on each part, check the `deployment guide <https://baobabsoluciones.github.io/cornflow/guides/deploy_solver.html>`_.
Using cornflow to solve a problem
-------------------------------------------
For this example we only need the cornflow_client package. We will test the graph-coloring demo defined `here <https://github.com/baobabsoluciones/cornflow-dags-public/tree/main/DAG/graph_coloring>`_. We will use the test server to solve it.
Initialize the api client::
from cornflow_client import CornFlow
email = 'readme@gmail.com'
pwd = 'some_password'
username = 'some_name'
client = CornFlow(url="https://devsm.cornflow.baobabsoluciones.app/")
client.login(username=username, pwd=pwd)
solve a graph coloring problem and get the solution::
data = dict(pairs=[dict(n1=0, n2=1), dict(n1=1, n2=2), dict(n1=1, n2=3)])
instance = client.create_instance(data, name='gc_4_1', description='very small gc problem', schema="graph_coloring")
config = dict()
execution = client.create_execution(
instance['id'], config, name='gc_4_1_exec', description='execution of very small gc problem',
schema="graph_coloring",
)
status = client.get_status(execution["id"])
print(status['state'])
solution = client.get_solution(execution["id"])
print(solution['data']['assignment'])
Running tests and coverage
------------------------------
Then you have to run the following commands::
export FLASK_ENV=testing
Finally you can run all the tests with the following command::
python -m unittest discover -s cornflow.tests
If you want to only run the unit tests (without a local airflow webserver)::
python -m unittest discover -s cornflow.tests.unit
If you want to only run the integration test with a local airflow webserver::
python -m unittest discover -s cornflow.tests.integration
After if you want to check the coverage report you need to run::
coverage run --source=./cornflow/ -m unittest discover -s=./cornflow/tests/
coverage report -m
or to get the html reports::
coverage html
Raw data
{
"_id": null,
"home_page": "https://github.com/baobabsoluciones/cornflow",
"name": "cornflow",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": null,
"author": "baobab soluciones",
"author_email": "cornflow@baobabsoluciones.es",
"download_url": "https://files.pythonhosted.org/packages/1d/09/ae59172f784d80da676546892a81596d6dadc4c196df1d7191946b19b471/cornflow-1.1.4.tar.gz",
"platform": null,
"description": "Cornflow\n=========\n\n.. image:: https://github.com/baobabsoluciones/cornflow/workflows/build/badge.svg?style=svg\n :target: https://github.com/baobabsoluciones/cornflow/actions\n\n.. image:: https://github.com/baobabsoluciones/cornflow/workflows/docs/badge.svg?style=svg\n :target: https://github.com/baobabsoluciones/cornflow/actions\n\n.. image:: https://github.com/baobabsoluciones/cornflow/workflows/integration/badge.svg?style=svg\n :target: https://github.com/baobabsoluciones/cornflow/actions\n\n.. image:: https://img.shields.io/pypi/v/cornflow-client.svg?style=svg\n :target: https://pypi.python.org/pypi/cornflow-client\n\n.. image:: https://img.shields.io/pypi/pyversions/cornflow-client.svg?style=svg\n :target: https://pypi.python.org/pypi/cornflow-client\n\n.. image:: https://img.shields.io/badge/License-Apache2.0-blue\n\nCornflow is an open source multi-solver optimization server with a REST API built using `flask <https://flask.palletsprojects.com>`_, `airflow <https://airflow.apache.org/>`_ and `pulp <https://coin-or.github.io/pulp/>`_.\n\nWhile most deployment servers are based on the solving technique (MIP, CP, NLP, etc.), Cornflow focuses on the optimization problems themselves. However, it does not impose any constraint on the type of problem and solution method to use.\n\nWith Cornflow you can deploy a Traveling Salesman Problem solver next to a Knapsack solver or a Nurse Rostering Problem solver. As long as you describe the input and output data, you can upload any solution method for any problem and then use it with any data you want.\n\nCornflow helps you formalize your problem by proposing development guidelines. It also provides a range of functionalities around your deployed solution method, namely:\n\n* storage of users, instances, solutions and solution logs.\n* deployment and maintenance of models, solvers and algorithms.\n* scheduling of executions in remote machines.\n* management of said executions: start, monitor, interrupt.\n* centralizing of commercial licenses.\n* scenario storage and comparison.\n* user management, roles and groups.\n\n\n.. contents:: **Table of Contents**\n\nInstallation instructions\n-------------------------------\n\nCornflow is tested with Ubuntu 20.04, python >= 3.8 and git.\n\nDownload the Cornflow project and install requirements::\n\n python3 -m venv venv\n venv/bin/pip3 install cornflow\n\ninitialize the sqlite database::\n\n source venv/bin/activate\n export FLASK_APP=cornflow.app\n export DATABASE_URL=sqlite:///cornflow.db\n flask db upgrade\n flask access_init\n flask create_service_user -u airflow -e airflow_test@admin.com -p airflow_test_password\n flask create_admin_user -u cornflow -e cornflow_admin@admin.com -p cornflow_admin_password\n\n\nactivate the virtual environment and run Cornflow::\n\n source venv/bin/activate\n export FLASK_APP=cornflow.app\n export SECRET_KEY=THISNEEDSTOBECHANGED\n export DATABASE_URL=sqlite:///cornflow.db\n export AIRFLOW_URL=http://127.0.0.1:8080/\n export AIRFLOW_USER=airflow_user\n export AIRFLOW_PWD=airflow_pwd\n flask run\n\n**Cornflow needs a running installation of Airflow to operate and more configuration**. Check `the installation docs <https://baobabsoluciones.github.io/cornflow/main/install.html>`_ for more details on installing airflow, configuring the application and initializing the database.\n\nUsing cornflow to solve a PuLP model\n---------------------------------------\n\nWe're going to test the cornflow server by using the `cornflow-client` and the `pulp` python package::\n\n pip install cornflow-client pulp\n\nInitialize the api client::\n\n from cornflow_client import CornFlow\n email = 'some_email@gmail.com'\n pwd = 'Some_password1'\n username = 'some_name'\n client = CornFlow(url=\"http://127.0.0.1:5000\")\n\nCreate a user::\n\n config = dict(username=username, email=email, pwd=pwd)\n client.sign_up(**config)\n\nLog in::\n\n client.login(username=username, pwd=pwd)\n\nPrepare an instance::\n\n import pulp\n prob = pulp.LpProblem(\"test_export_dict_MIP\", pulp.LpMinimize)\n x = pulp.LpVariable(\"x\", 0, 4)\n y = pulp.LpVariable(\"y\", -1, 1)\n z = pulp.LpVariable(\"z\", 0, None, pulp.LpInteger)\n prob += x + 4 * y + 9 * z, \"obj\"\n prob += x + y <= 5, \"c1\"\n prob += x + z >= 10, \"c2\"\n prob += -y + z == 7.5, \"c3\"\n data = prob.to_dict()\n insName = 'test_export_dict_MIP'\n description = 'very small example'\n\nSend instance::\n\n instance = client.create_instance(data, name=insName, description=description, schema=\"solve_model_dag\",)\n\nSolve an instance::\n\n config = dict(\n solver = \"PULP_CBC_CMD\",\n timeLimit = 10\n )\n execution = client.create_execution(\n instance['id'], config, name='execution1', description='execution of a very small instance',\n schema=\"solve_model_dag\",\n )\n\nCheck the status of an execution::\n\n status = client.get_status(execution[\"id\"])\n print(status['state'])\n # 1 means \"finished correctly\"\n\nRetrieve a solution::\n\n results = client.get_solution(execution['id'])\n print(results['data'])\n # returns a json with the solved pulp object\n _vars, prob = pulp.LpProblem.from_dict(results['data'])\n\nRetrieve the log of the solver::\n\n log = client.get_log(execution['id'])\n print(log['log'])\n # json format of the solver log\n\nUsing cornflow to deploy a solution method\n---------------------------------------------\n\nTo deploy a cornflow solution method, the following tasks need to be accomplished:\n\n#. Create an Application for the new problem\n#. Do a PR to a compatible repo linked to a server instance (e.g., like `this one <https://github.com/baobabsoluciones/cornflow>`_).\n\nFor more details on each part, check the `deployment guide <https://baobabsoluciones.github.io/cornflow/guides/deploy_solver.html>`_.\n\nUsing cornflow to solve a problem\n-------------------------------------------\n\nFor this example we only need the cornflow_client package. We will test the graph-coloring demo defined `here <https://github.com/baobabsoluciones/cornflow-dags-public/tree/main/DAG/graph_coloring>`_. We will use the test server to solve it.\n\nInitialize the api client::\n\n from cornflow_client import CornFlow\n email = 'readme@gmail.com'\n pwd = 'some_password'\n username = 'some_name'\n client = CornFlow(url=\"https://devsm.cornflow.baobabsoluciones.app/\")\n client.login(username=username, pwd=pwd)\n\nsolve a graph coloring problem and get the solution::\n\n data = dict(pairs=[dict(n1=0, n2=1), dict(n1=1, n2=2), dict(n1=1, n2=3)])\n instance = client.create_instance(data, name='gc_4_1', description='very small gc problem', schema=\"graph_coloring\")\n config = dict()\n execution = client.create_execution(\n instance['id'], config, name='gc_4_1_exec', description='execution of very small gc problem',\n schema=\"graph_coloring\",\n )\n status = client.get_status(execution[\"id\"])\n print(status['state'])\n solution = client.get_solution(execution[\"id\"])\n print(solution['data']['assignment'])\n\n\nRunning tests and coverage\n------------------------------\n\nThen you have to run the following commands::\n\n export FLASK_ENV=testing\n\nFinally you can run all the tests with the following command::\n\n python -m unittest discover -s cornflow.tests\n\nIf you want to only run the unit tests (without a local airflow webserver)::\n\n python -m unittest discover -s cornflow.tests.unit\n\nIf you want to only run the integration test with a local airflow webserver::\n\n python -m unittest discover -s cornflow.tests.integration\n\nAfter if you want to check the coverage report you need to run::\n\n coverage run --source=./cornflow/ -m unittest discover -s=./cornflow/tests/\n coverage report -m\n\nor to get the html reports::\n\n coverage html\n\n\n\n",
"bugtrack_url": null,
"license": null,
"summary": "Cornflow is an open source multi-solver optimization server with a REST API built using flask.",
"version": "1.1.4",
"project_urls": {
"Homepage": "https://github.com/baobabsoluciones/cornflow"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "cd0eda4afea05fe56509c332b82d6a8c2154f8d4b4dc4d93605764dea8e57b52",
"md5": "1f95ec7e4e549f1ffc6e72bea36c37b5",
"sha256": "c0134686fac6474d471c8c5ea1aad5d3128087490d63ccbeb9b1121e4e42677d"
},
"downloads": -1,
"filename": "cornflow-1.1.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "1f95ec7e4e549f1ffc6e72bea36c37b5",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 198563,
"upload_time": "2024-12-05T16:43:38",
"upload_time_iso_8601": "2024-12-05T16:43:38.139924Z",
"url": "https://files.pythonhosted.org/packages/cd/0e/da4afea05fe56509c332b82d6a8c2154f8d4b4dc4d93605764dea8e57b52/cornflow-1.1.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1d09ae59172f784d80da676546892a81596d6dadc4c196df1d7191946b19b471",
"md5": "477e3dc2ff6857a23dc01ababa96c39a",
"sha256": "33b569f0bd8f4a67d3a161609667f10b6ae0de25551e2ec3b3d7f5ec09171819"
},
"downloads": -1,
"filename": "cornflow-1.1.4.tar.gz",
"has_sig": false,
"md5_digest": "477e3dc2ff6857a23dc01ababa96c39a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 130090,
"upload_time": "2024-12-05T16:43:39",
"upload_time_iso_8601": "2024-12-05T16:43:39.547428Z",
"url": "https://files.pythonhosted.org/packages/1d/09/ae59172f784d80da676546892a81596d6dadc4c196df1d7191946b19b471/cornflow-1.1.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-05 16:43:39",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "baobabsoluciones",
"github_project": "cornflow",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "cornflow"
}
JSON
