starwhale


Namestarwhale JSON
Version 0.3.6 PyPI version JSON
download
home_pagehttps://github.com/star-whale/starwhale
SummaryAn MLOps Platform for Model Evaluation
upload_time2023-01-30 08:19:00
maintainer
docs_urlNone
authorStarwhale Team
requires_python>=3.7, <3.12
licenseApache License 2.0
keywords mlops ai starwhale model evaluation
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            <p align="center">
    <img src="https://github.com/star-whale/starwhale/raw/main/docs/static/img/starwhale.png" width="600" style="max-width: 600px;">
</p>

<p align="center">

<a href=https://github.com/ambv/black>
    <img src="https://img.shields.io/badge/code%20style-black-000000.svg">
</a>

<a href="https://starwhale.slack.com">
    <img src="https://img.shields.io/static/v1.svg?label=chat&message=on%20slack&color=27b1ff&style=flat">
</a>

<a href="https://pypi.org/project/starwhale/">
    <img src="https://img.shields.io/pypi/v/starwhale?style=flat">
</a>

<a href="https://github.com/star-whale/starwhale/blob/main/LICENSE">
    <img src="https://img.shields.io/github/license/star-whale/starwhale?style=flat">
</a>

<a href="https://pypi.org/project/starwhale/">
    <img alt="PyPI - Python Version" src="https://img.shields.io/pypi/pyversions/starwhale">
</a>

<a href="https://github.com/star-whale/starwhale/actions/workflows/client.yaml">
    <img src="https://github.com/star-whale/starwhale/actions/workflows/client.yaml/badge.svg">
</a>

<a href="https://github.com/star-whale/starwhale/actions/workflows/server-ut-report.yml">
    <img src="https://github.com/star-whale/starwhale/actions/workflows/server-ut-report.yml/badge.svg">
</a>

<a href="https://github.com/star-whale/starwhale/actions/workflows/console.yml">
    <img src="https://github.com/star-whale/starwhale/actions/workflows/console.yml/badge.svg">
</a>

<a href='https://app.codecov.io/gh/star-whale/starwhale'>
    <img alt="Codecov" src="https://img.shields.io/codecov/c/github/star-whale/starwhale?flag=controller&label=Java%20Cov">
</a>

<a href="https://app.codecov.io/gh/star-whale/starwhale">
    <img alt="Codecov" src="https://img.shields.io/codecov/c/github/star-whale/starwhale?flag=standalone&label=Python%20cov">
</a>

<a href='https://artifacthub.io/packages/helm/starwhale/starwhale'>
    <img src='https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/starwhale' alt='Artifact Hub'/>
</a>

<a href="https://github.com/star-whale/starwhale/actions/workflows/e2e-test.yml">
    <img src='https://github.com/star-whale/starwhale/actions/workflows/e2e-test.yml/badge.svg' alt='Starwhale E2E Test'>
</a>

</p>

## What is Starwhale

Starwhale is an MLOps platform. It provides **Instance**, **Project**, **Runtime**, **Model**, and **Dataset**.

- **Instance**: Each installation of Starwhale is called an instance.
  - πŸ‘» **Standalone Instance**: The simplest form that requires only the Starwhale Client(`swcli`). `swcli` is written by pure python3.
  - 🎍 **On-Premises Instance**: Cloud form, we call it **private cloud instance**. Kubernetes and BareMetal both meet the basic environmental requirements.
  - ☁️ **Cloud Hosted Instance**: Cloud form, we call it **public cloud instance**. Starwhale team maintains the web service.

  **Starwhale tries to keep concepts consistent across different types of instances. In this way, people can easily exchange data and migrate between them.**

- **Project**: The basic unit for organizing different resources.

- **ML Basic Elements**: The Machine Learning/Deep Learning running environments or artifacts. Starwhale empowers the ML/DL essential elements with packaging, versioning, reproducibility, and shareability.
  - 🐌 **Runtime**: Software dependencies description to "run" a model, which includes python libraries, native libraries, native binaries, etc.
  - πŸ‡ **Model**: The standard model format used in model delivery.
  - 🐫 **Dataset**: A unified description of how the data and labels are stored and organized. Starwhale datasets can be loaded efficiently.

- **Running Fundamentals**: Starwhale uses **Job**, **Step**, and **Task** to execute ML/DL actions like model training, evaluation, and serving. Starwhale's **Controller-Agents** structure scales out easily.
  - πŸ₯• **Job**: A set of programs to do specific work. Each job consists of one or more steps.
  - 🌡 **Step**: Represents distinct stages of the work. Each step consists of one or more tasks.
  - πŸ₯‘ **Task**: Operation entity. Tasks are in some specific steps.

- **Scenarios**: Starwhale provides the best practice and out-of-the-box for different ML/DL scenarios.
  - 🚝 **Model Training(TBD)**: Use Starwhale Python SDK to record experiment meta, metric, log, and artifact.
  - πŸ›₯️ **Model Evaluation**: `PipelineHandler` and some report decorators can give you complete, helpful, and user-friendly evaluation reports with only a few lines of codes.
  - πŸ›« **Model Serving(TBD)**: Starwhale Model can be deployed as a web service or stream service in production with deployment capability, observability, and scalability. Data scientists do not need to write ML/DL irrelevant codes.

## MNIST Quick Tour for the standalone instance

### Use Notebook

- You can try it in [Google Colab](https://colab.research.google.com/github/star-whale/starwhale/blob/main/example/mnist/notebook.ipynb)
- Or run [example/mnist/notebook.ipynb](example/mnist/notebook.ipynb) locally using [vscode](https://code.visualstudio.com/) or [jupyterlab](https://github.com/jupyterlab/jupyterlab)

### Use your own python env

![Core Job Workflow](docs/docs/img/standalone-core-workflow.gif)

- 🍰 **STEP1**: Installing Starwhale

    ```bash
    python3 -m pip install starwhale
    ```

- 🍡 **STEP2**: Downloading the MNIST example

    ```bash
    git clone https://github.com/star-whale/starwhale.git
    ```

    If [git-lfs](https://git-lfs.github.com/) has not been previously installed in the local environment(the command is `git lfs install`), you need to download the trained model file.

    ```bash
    wget https://media.githubusercontent.com/media/star-whale/starwhale/main/example/mnist/models/mnist_cnn.pt -O example/mnist/models/mnist_cnn.pt
    ```

- β˜• **STEP3**: Building a runtime

    ```bash
    cd example/runtime/pytorch
    swcli runtime build .
    swcli runtime list
    swcli runtime info pytorch/version/latest
    ```

- 🍞 **STEP4**: Building a model

  - Enter `example/mnist` directory:

   ```bash
   cd ../../mnist
   ```

  - Write some code with Starwhale Python SDK. Complete code is [here](https://github.com/star-whale/starwhale/blob/main/example/mnist/mnist/evaluator.py).

   ```python
   import typing as t
   import torch
   from starwhale import Image, PipelineHandler, PPLResultIterator, multi_classification

   class MNISTInference(PipelineHandler):
        def __init__(self) -> None:
            super().__init__()
            self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
            self.model = self._load_model(self.device)

        def ppl(self, img: Image, **kw: t.Any) -> t.Tuple[t.List[int], t.List[float]]:
            data_tensor = self._pre(img)
            output = self.model(data_tensor)
            return self._post(output)

        @multi_classification(
            confusion_matrix_normalize="all",
            show_hamming_loss=True,
            show_cohen_kappa_score=True,
            show_roc_auc=True,
            all_labels=[i for i in range(0, 10)],
        )
        def cmp(
            self, ppl_result: PPLResultIterator
        ) -> t.Tuple[t.List[int], t.List[int], t.List[t.List[float]]]:
            result, label, pr = [], [], []
            for _data in ppl_result:
                label.append(_data["annotations"]["label"])
                result.extend(_data["result"][0])
                pr.extend(_data["result"][1])
            return label, result, pr

       def _pre(self, input:bytes):
           """write some mnist preprocessing code"""

       def _post(self, input:bytes):
           """write some mnist post-processing code"""

       def _load_model():
           """load your pre trained model"""
   ```

  - Define `model.yaml`.

  ```yaml
  name: mnist
  model:
    - models/mnist_cnn.pt
  config:
    - config/hyperparam.json
  run:
    handler: mnist.evaluator:MNISTInference
  ```

  - Run one command to build the model.

   ```bash
    swcli model build .
    swcli model info mnist/version/latest
   ```

- 🍺 **STEP5**: Building a dataset

  - Download MNIST RAW data files.

   ```bash
    mkdir -p data && cd data
    wget http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz
    wget http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz
    gzip -d *.gz
    cd ..
    ls -lah data/*
   ```

  - Write some code with Starwhale Python SDK. Full code is [here](https://github.com/star-whale/starwhale/blob/main/example/mnist/mnist/dataset.py).

   ```python
    import struct
    import typing as t
    from pathlib import Path
    from starwhale import BuildExecutor

    class DatasetProcessExecutor(SWDSBinBuildExecutor):
        def iter_item(self) -> t.Generator[t.Tuple[t.Any, t.Any], None, None]:
            root_dir = Path(__file__).parent.parent / "data"

            with (root_dir / "t10k-images-idx3-ubyte").open("rb") as data_file, (
                root_dir / "t10k-labels-idx1-ubyte"
            ).open("rb") as label_file:
                _, data_number, height, width = struct.unpack(">IIII", data_file.read(16))
                _, label_number = struct.unpack(">II", label_file.read(8))
                print(
                    f">data({data_file.name}) split data:{data_number}, label:{label_number} group"
                )
                image_size = height * width

                for i in range(0, min(data_number, label_number)):
                    _data = data_file.read(image_size)
                    _label = struct.unpack(">B", label_file.read(1))[0]
                    yield GrayscaleImage(
                        _data,
                        display_name=f"{i}",
                        shape=(height, width, 1),
                    ), {"label": _label}
   ```

  - Define `dataset.yaml`.

   ```yaml
    name: mnist
    handler: mnist.dataset:DatasetProcessExecutor
    attr:
      alignment_size: 1k
      volume_size: 4M
      data_mime_type: "x/grayscale"
   ```

  - Run one command to build the dataset.

   ```bash
    swcli dataset build .
    swcli dataset info mnist/version/latest
   ```

  Starwhale also supports build dataset with pure python sdk. You can try it in [Google Colab](https://colab.research.google.com/github/star-whale/starwhale/blob/main/example/datasets/sdk/dataset-sdk.ipynb).

- πŸ– **STEP6**: Running an evaluation job

   ```bash
    swcli -vvv eval run --model mnist/version/latest --dataset mnist/version/latest --runtime pytorch/version/latest
    swcli eval list
    swcli eval info ${version}
   ```

πŸ‘ Now, you have completed the fundamental steps for Starwhale standalone.

Let's go ahead and finish the tutorial on the on-premises instance.

## MNIST Quick Tour for on-premises instance

![Create Job Workflow](docs/docs/img/console-create-job.gif)

- 🍰 **STEP1**: Install minikube and helm

  - [Minikube](https://minikube.sigs.k8s.io/docs/start/) 1.25+
  - [Helm](https://helm.sh/docs/intro/install/) 3.2.0+

- 🍡 **STEP2**: Start minikube

    ```bash
    minikube start
    ```

    > For users in the mainland of China, please add these startup parameters:`--image-mirror-country=cn --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers`. If there is no kubectl bin in your machine, you may use `minikube kubectl` or `alias kubectl="minikube kubectl --"` alias command.

- 🍡 **STEP3**: Installing Starwhale

    ```bash
    helm repo add starwhale https://star-whale.github.io/charts
    helm repo update
    helm install --devel my-starwhale starwhale/starwhale -n starwhale --create-namespace --set minikube.enabled=true
    ```

    After the installation is successful, the following prompt message appears:

    ```bash
    NAME: my-starwhale
    LAST DEPLOYED: Thu Jun 23 14:48:02 2022
    NAMESPACE: starwhale
    STATUS: deployed
    REVISION: 1
    NOTES:
    ******************************************
    Chart Name: starwhale
    Chart Version: 0.3.0
    App Version: 0.3.0
    ...

    Port Forward Visit:
    - starwhale controller:
        - run: kubectl port-forward --namespace starwhale svc/my-starwhale-controller 8082:8082
        - visit: http://localhost:8082
    - minio admin:
        - run: kubectl port-forward --namespace starwhale svc/my-starwhale-minio 9001:9001
        - visit: http://localhost:9001
    - mysql:
        - run: kubectl port-forward --namespace starwhale svc/my-starwhale-mysql 3306:3306
        - visit: mysql -h 127.0.0.1 -P 3306 -ustarwhale -pstarwhale

    ******************************************
    Login Info:
    - starwhale: u:starwhale, p:abcd1234
    - minio admin: u:minioadmin, p:minioadmin

    *_* Enjoy using Starwhale. *_*
    ```

    Then keep checking the minikube service status until all pods are running.

    ```bash
    kubectl get pods -n starwhale
    ```

    | NAME | READY | STATUS | RESTARTS | AGE |
    |:-----|-------|--------|----------|-----|
    |my-starwhale-controller-7d864558bc-vxvb8|1/1|Running|0|1m
    |my-starwhale-minio-7d45db75f6-7wq9b|1/1|Running|0|2m
    |my-starwhale-mysql-0|1/1|Running|0|2m

    Make the Starwhale controller accessible locally with the following command:

    ```bash
    kubectl port-forward --namespace starwhale svc/my-starwhale-controller 8082:8082
    ```

- β˜• **STEP4**: Upload the artifacts to the cloud instance

    > **pre-prepared artifacts**
    > Before starting this tutorial, the following three artifacts should already exist on your machine:
    >
    > - a starwhale model named mnist
    > - a starwhale dataset named mnist
    > - a starwhale runtime named pytorch
    >
    > The above three artifacts are what we built on our machine using starwhale.

    1. Use swcli to operate the remote server
        First, log in to the server:

        ```bash
        swcli instance login --username starwhale --password abcd1234 --alias dev http://localhost:8082
        ```

    2. Start copying the model, dataset, and runtime that we constructed earlier:

        ```bash
        swcli model copy mnist/version/latest dev/project/starwhale
        swcli dataset copy mnist/version/latest dev/project/starwhale
        swcli runtime copy pytorch/version/latest dev/project/starwhale
        ```

- 🍞 **STEP5**: Use the web UI to run an evaluation

    1. Log in Starwhale instance: let's use the username(starwhale) and password(abcd1234) to open the server web UI(<http://localhost:8082/>).

    2. Then, we will see the project named 'project_for_mnist' that we created earlier with swcli. Click the project name, you will see the model, runtime, and dataset uploaded in the previous step.
        <details>
            <summary>Show the uploaded artifacts screenshots</summary>

        ![console-artifacts.gif](../img/console-artifacts.gif)
        </details>
    3. Create and view an evaluation job
        <details>
            <summary>Show create job screenshot</summary>

        ![console-create-job.gif](../img/console-create-job.gif)
        </details>

**Congratulations! You have completed the evaluation process for a model.**

## Documentation, Community, and Support

- Visit [Starwhale HomePage](https://starwhale.ai).
- More information in the [official documentation](https://doc.starwhale.ai).
- For general questions and support, join the [Slack](https://starwhale.slack.com/).
- For bug reports and feature requests, please use [Github Issue](https://github.com/star-whale/starwhale/issues).
- To get community updates, follow [@starwhaleai](https://twitter.com/starwhaleai) on Twitter.
- For Starwhale artifacts, please visit:

  - Python Package on [Pypi](https://pypi.org/project/starwhale/).
  - Helm Charts on [Artifacthub](https://artifacthub.io/packages/helm/starwhale/starwhale).
  - Docker Images on [Docker Hub](https://hub.docker.com/u/starwhaleai) and [ghcr.io](https://github.com/orgs/star-whale/packages).

- Additionally, you can always find us at *developer@starwhale.ai*.

## Contributing

πŸŒΌπŸ‘**PRs are always welcomed** πŸ‘πŸΊ. See [Contribution to Starwhale](https://doc.starwhale.ai/docs/community/contribute) for more details.

## License

Starwhale is licensed under the [Apache License 2.0](https://github.com/star-whale/starwhale/blob/main/LICENSE).

            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/star-whale/starwhale",
    "name": "starwhale",
    "maintainer": "",
    "docs_url": null,
    "requires_python": ">=3.7, <3.12",
    "maintainer_email": "",
    "keywords": "MLOps,AI,Starwhale,Model Evaluation",
    "author": "Starwhale Team",
    "author_email": "developer@starwhale.ai",
    "download_url": "https://files.pythonhosted.org/packages/58/87/0407d2d0b36cead4f728240553c9ebe4aaef4b8e97a3f4b9a982026d1ff4/starwhale-0.3.6.tar.gz",
    "platform": null,
    "description": "<p align=\"center\">\n    <img src=\"https://github.com/star-whale/starwhale/raw/main/docs/static/img/starwhale.png\" width=\"600\" style=\"max-width: 600px;\">\n</p>\n\n<p align=\"center\">\n\n<a href=https://github.com/ambv/black>\n    <img src=\"https://img.shields.io/badge/code%20style-black-000000.svg\">\n</a>\n\n<a href=\"https://starwhale.slack.com\">\n    <img src=\"https://img.shields.io/static/v1.svg?label=chat&message=on%20slack&color=27b1ff&style=flat\">\n</a>\n\n<a href=\"https://pypi.org/project/starwhale/\">\n    <img src=\"https://img.shields.io/pypi/v/starwhale?style=flat\">\n</a>\n\n<a href=\"https://github.com/star-whale/starwhale/blob/main/LICENSE\">\n    <img src=\"https://img.shields.io/github/license/star-whale/starwhale?style=flat\">\n</a>\n\n<a href=\"https://pypi.org/project/starwhale/\">\n    <img alt=\"PyPI - Python Version\" src=\"https://img.shields.io/pypi/pyversions/starwhale\">\n</a>\n\n<a href=\"https://github.com/star-whale/starwhale/actions/workflows/client.yaml\">\n    <img src=\"https://github.com/star-whale/starwhale/actions/workflows/client.yaml/badge.svg\">\n</a>\n\n<a href=\"https://github.com/star-whale/starwhale/actions/workflows/server-ut-report.yml\">\n    <img src=\"https://github.com/star-whale/starwhale/actions/workflows/server-ut-report.yml/badge.svg\">\n</a>\n\n<a href=\"https://github.com/star-whale/starwhale/actions/workflows/console.yml\">\n    <img src=\"https://github.com/star-whale/starwhale/actions/workflows/console.yml/badge.svg\">\n</a>\n\n<a href='https://app.codecov.io/gh/star-whale/starwhale'>\n    <img alt=\"Codecov\" src=\"https://img.shields.io/codecov/c/github/star-whale/starwhale?flag=controller&label=Java%20Cov\">\n</a>\n\n<a href=\"https://app.codecov.io/gh/star-whale/starwhale\">\n    <img alt=\"Codecov\" src=\"https://img.shields.io/codecov/c/github/star-whale/starwhale?flag=standalone&label=Python%20cov\">\n</a>\n\n<a href='https://artifacthub.io/packages/helm/starwhale/starwhale'>\n    <img src='https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/starwhale' alt='Artifact Hub'/>\n</a>\n\n<a href=\"https://github.com/star-whale/starwhale/actions/workflows/e2e-test.yml\">\n    <img src='https://github.com/star-whale/starwhale/actions/workflows/e2e-test.yml/badge.svg' alt='Starwhale E2E Test'>\n</a>\n\n</p>\n\n## What is Starwhale\n\nStarwhale is an MLOps platform. It provides **Instance**, **Project**, **Runtime**, **Model**, and **Dataset**.\n\n- **Instance**: Each installation of Starwhale is called an instance.\n  - \ud83d\udc7b **Standalone Instance**: The simplest form that requires only the Starwhale Client(`swcli`). `swcli` is written by pure python3.\n  - \ud83c\udf8d **On-Premises Instance**: Cloud form, we call it **private cloud instance**. Kubernetes and BareMetal both meet the basic environmental requirements.\n  - \u2601\ufe0f **Cloud Hosted Instance**: Cloud form, we call it **public cloud instance**. Starwhale team maintains the web service.\n\n  **Starwhale tries to keep concepts consistent across different types of instances. In this way, people can easily exchange data and migrate between them.**\n\n- **Project**: The basic unit for organizing different resources.\n\n- **ML Basic Elements**: The Machine Learning/Deep Learning running environments or artifacts. Starwhale empowers the ML/DL essential elements with packaging, versioning, reproducibility, and shareability.\n  - \ud83d\udc0c **Runtime**: Software dependencies description to \"run\" a model, which includes python libraries, native libraries, native binaries, etc.\n  - \ud83d\udc07 **Model**: The standard model format used in model delivery.\n  - \ud83d\udc2b **Dataset**: A unified description of how the data and labels are stored and organized. Starwhale datasets can be loaded efficiently.\n\n- **Running Fundamentals**: Starwhale uses **Job**, **Step**, and **Task** to execute ML/DL actions like model training\uff0c evaluation, and serving. Starwhale's **Controller-Agents** structure scales out easily.\n  - \ud83e\udd55 **Job**: A set of programs to do specific work. Each job consists of one or more steps.\n  - \ud83c\udf35 **Step**: Represents distinct stages of the work. Each step consists of one or more tasks.\n  - \ud83e\udd51 **Task**: Operation entity. Tasks are in some specific steps.\n\n- **Scenarios**: Starwhale provides the best practice and out-of-the-box for different ML/DL scenarios.\n  - \ud83d\ude9d **Model Training(TBD)**: Use Starwhale Python SDK to record experiment meta, metric, log, and artifact.\n  - \ud83d\udee5\ufe0f **Model Evaluation**: `PipelineHandler` and some report decorators can give you complete, helpful, and user-friendly evaluation reports with only a few lines of codes.\n  - \ud83d\udeeb **Model Serving(TBD)**: Starwhale Model can be deployed as a web service or stream service in production with deployment capability, observability, and scalability. Data scientists do not need to write ML/DL irrelevant codes.\n\n## MNIST Quick Tour for the standalone instance\n\n### Use Notebook\n\n- You can try it in [Google Colab](https://colab.research.google.com/github/star-whale/starwhale/blob/main/example/mnist/notebook.ipynb)\n- Or run [example/mnist/notebook.ipynb](example/mnist/notebook.ipynb) locally using [vscode](https://code.visualstudio.com/) or [jupyterlab](https://github.com/jupyterlab/jupyterlab)\n\n### Use your own python env\n\n![Core Job Workflow](docs/docs/img/standalone-core-workflow.gif)\n\n- \ud83c\udf70 **STEP1**: Installing Starwhale\n\n    ```bash\n    python3 -m pip install starwhale\n    ```\n\n- \ud83c\udf75 **STEP2**: Downloading the MNIST example\n\n    ```bash\n    git clone https://github.com/star-whale/starwhale.git\n    ```\n\n    If [git-lfs](https://git-lfs.github.com/) has not been previously installed in the local environment(the command is `git lfs install`), you need to download the trained model file.\n\n    ```bash\n    wget https://media.githubusercontent.com/media/star-whale/starwhale/main/example/mnist/models/mnist_cnn.pt -O example/mnist/models/mnist_cnn.pt\n    ```\n\n- \u2615 **STEP3**: Building a runtime\n\n    ```bash\n    cd example/runtime/pytorch\n    swcli runtime build .\n    swcli runtime list\n    swcli runtime info pytorch/version/latest\n    ```\n\n- \ud83c\udf5e **STEP4**: Building a model\n\n  - Enter `example/mnist` directory:\n\n   ```bash\n   cd ../../mnist\n   ```\n\n  - Write some code with Starwhale Python SDK. Complete code is [here](https://github.com/star-whale/starwhale/blob/main/example/mnist/mnist/evaluator.py).\n\n   ```python\n   import typing as t\n   import torch\n   from starwhale import Image, PipelineHandler, PPLResultIterator, multi_classification\n\n   class MNISTInference(PipelineHandler):\n        def __init__(self) -> None:\n            super().__init__()\n            self.device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n            self.model = self._load_model(self.device)\n\n        def ppl(self, img: Image, **kw: t.Any) -> t.Tuple[t.List[int], t.List[float]]:\n            data_tensor = self._pre(img)\n            output = self.model(data_tensor)\n            return self._post(output)\n\n        @multi_classification(\n            confusion_matrix_normalize=\"all\",\n            show_hamming_loss=True,\n            show_cohen_kappa_score=True,\n            show_roc_auc=True,\n            all_labels=[i for i in range(0, 10)],\n        )\n        def cmp(\n            self, ppl_result: PPLResultIterator\n        ) -> t.Tuple[t.List[int], t.List[int], t.List[t.List[float]]]:\n            result, label, pr = [], [], []\n            for _data in ppl_result:\n                label.append(_data[\"annotations\"][\"label\"])\n                result.extend(_data[\"result\"][0])\n                pr.extend(_data[\"result\"][1])\n            return label, result, pr\n\n       def _pre(self, input:bytes):\n           \"\"\"write some mnist preprocessing code\"\"\"\n\n       def _post(self, input:bytes):\n           \"\"\"write some mnist post-processing code\"\"\"\n\n       def _load_model():\n           \"\"\"load your pre trained model\"\"\"\n   ```\n\n  - Define `model.yaml`.\n\n  ```yaml\n  name: mnist\n  model:\n    - models/mnist_cnn.pt\n  config:\n    - config/hyperparam.json\n  run:\n    handler: mnist.evaluator:MNISTInference\n  ```\n\n  - Run one command to build the model.\n\n   ```bash\n    swcli model build .\n    swcli model info mnist/version/latest\n   ```\n\n- \ud83c\udf7a **STEP5**: Building a dataset\n\n  - Download MNIST RAW data files.\n\n   ```bash\n    mkdir -p data && cd data\n    wget http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz\n    wget http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz\n    gzip -d *.gz\n    cd ..\n    ls -lah data/*\n   ```\n\n  - Write some code with Starwhale Python SDK. Full code is [here](https://github.com/star-whale/starwhale/blob/main/example/mnist/mnist/dataset.py).\n\n   ```python\n    import struct\n    import typing as t\n    from pathlib import Path\n    from starwhale import BuildExecutor\n\n    class DatasetProcessExecutor(SWDSBinBuildExecutor):\n        def iter_item(self) -> t.Generator[t.Tuple[t.Any, t.Any], None, None]:\n            root_dir = Path(__file__).parent.parent / \"data\"\n\n            with (root_dir / \"t10k-images-idx3-ubyte\").open(\"rb\") as data_file, (\n                root_dir / \"t10k-labels-idx1-ubyte\"\n            ).open(\"rb\") as label_file:\n                _, data_number, height, width = struct.unpack(\">IIII\", data_file.read(16))\n                _, label_number = struct.unpack(\">II\", label_file.read(8))\n                print(\n                    f\">data({data_file.name}) split data:{data_number}, label:{label_number} group\"\n                )\n                image_size = height * width\n\n                for i in range(0, min(data_number, label_number)):\n                    _data = data_file.read(image_size)\n                    _label = struct.unpack(\">B\", label_file.read(1))[0]\n                    yield GrayscaleImage(\n                        _data,\n                        display_name=f\"{i}\",\n                        shape=(height, width, 1),\n                    ), {\"label\": _label}\n   ```\n\n  - Define `dataset.yaml`.\n\n   ```yaml\n    name: mnist\n    handler: mnist.dataset:DatasetProcessExecutor\n    attr:\n      alignment_size: 1k\n      volume_size: 4M\n      data_mime_type: \"x/grayscale\"\n   ```\n\n  - Run one command to build the dataset.\n\n   ```bash\n    swcli dataset build .\n    swcli dataset info mnist/version/latest\n   ```\n\n  Starwhale also supports build dataset with pure python sdk. You can try it in [Google Colab](https://colab.research.google.com/github/star-whale/starwhale/blob/main/example/datasets/sdk/dataset-sdk.ipynb).\n\n- \ud83c\udf56 **STEP6**: Running an evaluation job\n\n   ```bash\n    swcli -vvv eval run --model mnist/version/latest --dataset mnist/version/latest --runtime pytorch/version/latest\n    swcli eval list\n    swcli eval info ${version}\n   ```\n\n\ud83d\udc4f Now, you have completed the fundamental steps for Starwhale standalone.\n\nLet's go ahead and finish the tutorial on the on-premises instance.\n\n## MNIST Quick Tour for on-premises instance\n\n![Create Job Workflow](docs/docs/img/console-create-job.gif)\n\n- \ud83c\udf70 **STEP1**: Install minikube and helm\n\n  - [Minikube](https://minikube.sigs.k8s.io/docs/start/) 1.25+\n  - [Helm](https://helm.sh/docs/intro/install/) 3.2.0+\n\n- \ud83c\udf75 **STEP2**: Start minikube\n\n    ```bash\n    minikube start\n    ```\n\n    > For users in the mainland of China, please add these startup parameters\uff1a`--image-mirror-country=cn --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers`. If there is no kubectl bin in your machine, you may use `minikube kubectl` or `alias kubectl=\"minikube kubectl --\"` alias command.\n\n- \ud83c\udf75 **STEP3**: Installing Starwhale\n\n    ```bash\n    helm repo add starwhale https://star-whale.github.io/charts\n    helm repo update\n    helm install --devel my-starwhale starwhale/starwhale -n starwhale --create-namespace --set minikube.enabled=true\n    ```\n\n    After the installation is successful, the following prompt message appears:\n\n    ```bash\n    NAME: my-starwhale\n    LAST DEPLOYED: Thu Jun 23 14:48:02 2022\n    NAMESPACE: starwhale\n    STATUS: deployed\n    REVISION: 1\n    NOTES:\n    ******************************************\n    Chart Name: starwhale\n    Chart Version: 0.3.0\n    App Version: 0.3.0\n    ...\n\n    Port Forward Visit:\n    - starwhale controller:\n        - run: kubectl port-forward --namespace starwhale svc/my-starwhale-controller 8082:8082\n        - visit: http://localhost:8082\n    - minio admin:\n        - run: kubectl port-forward --namespace starwhale svc/my-starwhale-minio 9001:9001\n        - visit: http://localhost:9001\n    - mysql:\n        - run: kubectl port-forward --namespace starwhale svc/my-starwhale-mysql 3306:3306\n        - visit: mysql -h 127.0.0.1 -P 3306 -ustarwhale -pstarwhale\n\n    ******************************************\n    Login Info:\n    - starwhale: u:starwhale, p:abcd1234\n    - minio admin: u:minioadmin, p:minioadmin\n\n    *_* Enjoy using Starwhale. *_*\n    ```\n\n    Then keep checking the minikube service status until all pods are running.\n\n    ```bash\n    kubectl get pods -n starwhale\n    ```\n\n    | NAME | READY | STATUS | RESTARTS | AGE |\n    |:-----|-------|--------|----------|-----|\n    |my-starwhale-controller-7d864558bc-vxvb8|1/1|Running|0|1m\n    |my-starwhale-minio-7d45db75f6-7wq9b|1/1|Running|0|2m\n    |my-starwhale-mysql-0|1/1|Running|0|2m\n\n    Make the Starwhale controller accessible locally with the following command:\n\n    ```bash\n    kubectl port-forward --namespace starwhale svc/my-starwhale-controller 8082:8082\n    ```\n\n- \u2615 **STEP4**: Upload the artifacts to the cloud instance\n\n    > **pre-prepared artifacts**\n    > Before starting this tutorial, the following three artifacts should already exist on your machine\uff1a\n    >\n    > - a starwhale model named mnist\n    > - a starwhale dataset named mnist\n    > - a starwhale runtime named pytorch\n    >\n    > The above three artifacts are what we built on our machine using starwhale.\n\n    1. Use swcli to operate the remote server\n        First, log in to the server:\n\n        ```bash\n        swcli instance login --username starwhale --password abcd1234 --alias dev http://localhost:8082\n        ```\n\n    2. Start copying the model, dataset, and runtime that we constructed earlier:\n\n        ```bash\n        swcli model copy mnist/version/latest dev/project/starwhale\n        swcli dataset copy mnist/version/latest dev/project/starwhale\n        swcli runtime copy pytorch/version/latest dev/project/starwhale\n        ```\n\n- \ud83c\udf5e **STEP5**: Use the web UI to run an evaluation\n\n    1. Log in Starwhale instance: let's use the username(starwhale) and password(abcd1234) to open the server web UI(<http://localhost:8082/>).\n\n    2. Then, we will see the project named 'project_for_mnist' that we created earlier with swcli. Click the project name, you will see the model, runtime, and dataset uploaded in the previous step.\n        <details>\n            <summary>Show the uploaded artifacts screenshots</summary>\n\n        ![console-artifacts.gif](../img/console-artifacts.gif)\n        </details>\n    3. Create and view an evaluation job\n        <details>\n            <summary>Show create job screenshot</summary>\n\n        ![console-create-job.gif](../img/console-create-job.gif)\n        </details>\n\n**Congratulations! You have completed the evaluation process for a model.**\n\n## Documentation, Community, and Support\n\n- Visit [Starwhale HomePage](https://starwhale.ai).\n- More information in the [official documentation](https://doc.starwhale.ai).\n- For general questions and support, join the [Slack](https://starwhale.slack.com/).\n- For bug reports and feature requests, please use [Github Issue](https://github.com/star-whale/starwhale/issues).\n- To get community updates, follow [@starwhaleai](https://twitter.com/starwhaleai) on Twitter.\n- For Starwhale artifacts, please visit:\n\n  - Python Package on [Pypi](https://pypi.org/project/starwhale/).\n  - Helm Charts on [Artifacthub](https://artifacthub.io/packages/helm/starwhale/starwhale).\n  - Docker Images on [Docker Hub](https://hub.docker.com/u/starwhaleai) and [ghcr.io](https://github.com/orgs/star-whale/packages).\n\n- Additionally, you can always find us at *developer@starwhale.ai*.\n\n## Contributing\n\n\ud83c\udf3c\ud83d\udc4f**PRs are always welcomed** \ud83d\udc4d\ud83c\udf7a. See [Contribution to Starwhale](https://doc.starwhale.ai/docs/community/contribute) for more details.\n\n## License\n\nStarwhale is licensed under the [Apache License 2.0](https://github.com/star-whale/starwhale/blob/main/LICENSE).\n",
    "bugtrack_url": null,
    "license": "Apache License 2.0",
    "summary": "An MLOps Platform for Model Evaluation",
    "version": "0.3.6",
    "split_keywords": [
        "mlops",
        "ai",
        "starwhale",
        "model evaluation"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "205b578aa3904b9fd612219b9ec8f7479037daaa97fd1228150277faff897199",
                "md5": "4cfec53a879dfdae4ed62fbe7e9cb370",
                "sha256": "298bc3cafc738439be3d14ecf74c86c50cdeae731595118c90c24572947c4c20"
            },
            "downloads": -1,
            "filename": "starwhale-0.3.6-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "4cfec53a879dfdae4ed62fbe7e9cb370",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.7, <3.12",
            "size": 210201,
            "upload_time": "2023-01-30T08:18:58",
            "upload_time_iso_8601": "2023-01-30T08:18:58.449481Z",
            "url": "https://files.pythonhosted.org/packages/20/5b/578aa3904b9fd612219b9ec8f7479037daaa97fd1228150277faff897199/starwhale-0.3.6-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "58870407d2d0b36cead4f728240553c9ebe4aaef4b8e97a3f4b9a982026d1ff4",
                "md5": "b44be3d0ec822c42416857c9a306f457",
                "sha256": "601a01a883bbb422aace96bf542e74bc4f42377368075044afef965de351edad"
            },
            "downloads": -1,
            "filename": "starwhale-0.3.6.tar.gz",
            "has_sig": false,
            "md5_digest": "b44be3d0ec822c42416857c9a306f457",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.7, <3.12",
            "size": 171928,
            "upload_time": "2023-01-30T08:19:00",
            "upload_time_iso_8601": "2023-01-30T08:19:00.547533Z",
            "url": "https://files.pythonhosted.org/packages/58/87/0407d2d0b36cead4f728240553c9ebe4aaef4b8e97a3f4b9a982026d1ff4/starwhale-0.3.6.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2023-01-30 08:19:00",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "github_user": "star-whale",
    "github_project": "starwhale",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "lcname": "starwhale"
}
        
Elapsed time: 0.12418s