# RepSys: Framework for Interactive Evaluation of Recommender Systems
[](https://badge.fury.io/py/repsys-framework)
The RepSys is a framework for developing and analyzing recommendation systems, and it allows you to:
- Add your own dataset and recommendation models
- Visually evaluate the models on various metrics
- Quickly create dataset embeddings to explore the data
- Preview recommendations using a web application
- Simulate user's behavior while receiving the recommendations
<p align="middle">
<img src="https://github.com/cowjen01/repsys/raw/master/images/demos/eval-space.png" width="48%" />
<img src="https://github.com/cowjen01/repsys/raw/master/images/demos/dataset-users.png" width="48%" />
</p>
## Online Demo
You can now try RepSys online on our [demo site](https://repsys.recombee.net) with the Movielens dataset.
Also, check out an [interactive blog post](https://www.recombee.com/blog/repsys-opensource-library-for-interactive-evaluation-of-recommendation-systems.html) we made using the RepSys widgets component.
## Publication
Our paper "[RepSys: Framework for Interactive Evaluation of Recommender Systems](https://dl.acm.org/doi/10.1145/3523227.3551469)" was accepted to the RecSys'22 conference.
## Installation
Install the package using [pip](https://pypi.org/project/repsys-framework/):
```
$ pip install repsys-framework
```
If you will be using PyMDE for data visualization, you need to install RepSys with the following extras:
```
$ pip install repsys-framework[pymde]
```
## Getting Started
If you want to skip this tutorial and try the framework, you can pull the content of the [demo](https://github.com/cowjen01/repsys/tree/master/demo) folder located at the repository.
As mentioned in the [next step](https://github.com/cowjen01/repsys#datasetpy), you still have to download the dataset before you begin.
Otherwise, please create an empty project folder that will contain the dataset and models implementation.
```
├── __init__.py
├── dataset.py
├── models.py
├── repsys.ini
└── .gitignore
```
### dataset.py
Firstly we need to import our dataset. We will use [MovieLens 20M Dataset](https://grouplens.org/datasets/movielens/20m/) with
20 million ratings made by 138,000 users to 27,000 movies for the tutorial purpose. Please download the `ml-20m.zip` file and unzip
the data into the current folder. Then add the following content to the `dataset.py` file:
```python
import pandas as pd
from repsys import Dataset
import repsys.dtypes as dtypes
class MovieLens(Dataset):
def name(self):
return "ml20m"
def item_cols(self):
return {
"movieId": dtypes.ItemID(),
"title": dtypes.Title(),
"genres": dtypes.Tag(sep="|"),
"year": dtypes.Number(data_type=int),
}
def interaction_cols(self):
return {
"movieId": dtypes.ItemID(),
"userId": dtypes.UserID(),
"rating": dtypes.Interaction(),
}
def load_items(self):
df = pd.read_csv("./ml-20m/movies.csv")
df["year"] = df["title"].str.extract(r"\((\d+)\)")
return df
def load_interactions(self):
df = pd.read_csv("./ml-20m/ratings.csv")
return df
```
This code will define a new dataset called ml20m, and it will import both ratings
and items data. You must always specify your data structure using predefined data types.
Before you return the data, you can also preprocess it like extracting the movie's year from the title column.
### models.py
Now we define the first recommendation model, which will be a simple implementation of the user-based KNN.
```python
import numpy as np
import scipy.sparse as sp
from sklearn.neighbors import NearestNeighbors
from repsys import Model
class KNN(Model):
def __init__(self):
self.model = NearestNeighbors(n_neighbors=20, metric="cosine")
def name(self):
return "knn"
def fit(self, training=False):
X = self.dataset.get_train_data()
self.model.fit(X)
def predict(self, X, **kwargs):
if X.count_nonzero() == 0:
return np.random.uniform(size=X.shape)
distances, indices = self.model.kneighbors(X)
distances = distances[:, 1:]
indices = indices[:, 1:]
distances = 1 - distances
sums = distances.sum(axis=1)
distances = distances / sums[:, np.newaxis]
def f(dist, idx):
A = self.dataset.get_train_data()[idx]
D = sp.diags(dist)
return D.dot(A).sum(axis=0)
vf = np.vectorize(f, signature="(n),(n)->(m)")
predictions = vf(distances, indices)
predictions[X.nonzero()] = 0
return predictions
```
You must define the fit method to train your model using the training data or load the previously trained model from a file.
All models are fitted when the web application starts, or the evaluation process begins. If this is not a training phase, always
load your model from a checkpoint to speed up the process. For tutorial purposes, this is omitted.
You must also define the prediction method that receives a sparse matrix of the users' interactions on the input.
For each user (row of the matrix) and item (column of the matrix), the method should return a predicted score indicating
how much the user will enjoy the item.
Additionally, you can specify some web application parameters you can set during recommender creation. The value is then accessible in
the `**kwargs` argument of the prediction method. In the example, we create a select input with all unique genres and filter out only those movies that do not contain the selected genre.
### repsys.ini
The last file we should create is a configuration that allows you to control a data splitting process, server settings,
framework behavior, etc.
```ini
[general]
seed=1234
[dataset]
train_split_prop=0.85
test_holdout_prop=0.2
min_user_interacts=5
min_item_interacts=0
[evaluation]
precision_recall_k=20,50
ndcg_k=100
coverage_k=20
diversity_k=20
novelty_k=20
percentage_lt_k=20
coverage_lt_k=20
[visualization]
embed_method=pymde
pymde_neighbors=15
umap_neighbors=15
umap_min_dist=0.1
tsne_perplexity=30
[server]
port=3001
```
### Splitting the Data
Before we train our models, we need to split the data into train, validation, and test sets. Run the following command from the current directory.
```
$ repsys dataset split
```
This will hold out 85% of the users as training data, and the rest 15% will be used as validation/test data with 7.5% of users each. For both validation
and test set, 20% of the interactions will also be held out for evaluation purposes. The split dataset will be stored in the default checkpoints folder.
### Training the Models
Now we can move to the training process. To do this, please call the following command.
```
$ repsys model train
```
This command will call the fit method of each model with the training flag set to true. You can always limit the models using `-m` flag with the model's name as a parameter.
### Evaluating the Models
When the data is prepared and the models trained, we can evaluate the performance of the models on the unseen users' interactions. Run the following command to do so.
```
$ repsys model eval
```
Again, you can limit the models using the `-m` flag. The results will be stored in the checkpoints folder when the evaluation is done.
### Evaluating the Dataset
Before starting the web application, the final step is to evaluate the dataset's data. This procedure will create users and items embeddings of the training and validation data
to allow you to explore the latent space. Run the following command from the project directory.
```
$ repsys dataset eval
```
You can choose from three types of embeddings algorithm:
1. [UMAP](https://umap-learn.readthedocs.io/en/latest/index.html) (Uniform Manifold Approximation and Projection for Dimension Reduction) is a dimensionality reduction technique similar to t-SNE. Use `--method umap` (this is the default option).
2. [PyMDE](https://pymde.org) (Minimum-Distortion Embedding) is a fast library designed to distort relationships between pairs of items minimally. Use `--method pymde`.
3. Combination of the PCA and TSNE algorithms (reduction of the dimensionality to 50 using PCA, then reduction to 2D space using TSNE). Use `--method tsne`.
4. Your own implementation of the algorithm. Use `--method custom` and add the following method to the model's class of your choice. In this case, you must also specify the model's name using `-m` parameter.
```python
from sklearn.decomposition import NMF
def compute_embeddings(self, X):
nmf = NMF(n_components=2)
W = nmf.fit_transform(X)
H = nmf.components_
return W, H.T
```
In the example, the negative matrix factorization is used. You have to return a user and item embeddings pair in this order. Also, it is essential to return the matrices in the shape of (n_users/n_items, n_dim).
If the reduced dimension is higher than 2, the TSNE method is applied.
### Running the Application
Finally, it is time to start the web application to see the results of the evaluations and preview live recommendations of your models.
```
$ repsys server
```
The application should be accessible on the default address [http://localhost:3001](http://localhost:3001). When you open the link, you will see the main screen where your recommendations appear once you finish the setup.
The first step is defining how the items' data columns should be mapped to the item view components.
Then we need to switch to the build mode and add two recommenders - one without filter and the second with only comedy movies included.
Now we switch back from the build mode and select a user from the validation set (never seen by a model before).
Finally, we see the user's interaction history on the right side and the recommendations made by the model on the left side.
## Contributing
To build the package from the source, you first need to install Node.js and npm library as documented [here](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm).
Then you can run the following script from the root directory to build the web application and install the package locally.
```
$ ./scripts/install-locally.sh
```
## Citation
If you employ RepSys in your research work, please do not forget to cite the related paper:
```
@inproceedings{10.1145/3523227.3551469,
author = {\v{S}afa\v{r}\'{\i}k, Jan and Van\v{c}ura, Vojt\v{e}ch and Kord\'{\i}k, Pavel},
title = {RepSys: Framework for Interactive Evaluation of Recommender Systems},
year = {2022},
isbn = {9781450392785},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3523227.3551469},
doi = {10.1145/3523227.3551469},
booktitle = {Proceedings of the 16th ACM Conference on Recommender Systems},
pages = {636–639},
numpages = {4},
keywords = {User simulation, Distribution analysis, Recommender systems},
location = {Seattle, WA, USA},
series = {RecSys '22}
}
```
## The Team
- Jan Šafařík (safarj10@fit.cvut.cz)
- Vojtěch Vančura (vancurv@fit.cvut.cz)
- Pavel Kordík (pavel.kordik@fit.cvut.cz)
- Petr Kasalický (kasalpe1@fit.cvut.cz)
## Sponsoring
The development of this framework is sponsored by the [Recombee](https://www.recombee.com) company.
<img src="https://github.com/cowjen01/repsys/raw/master/images/recombee-logo.png" width="50%" />
Raw data
{
"_id": null,
"home_page": "https://github.com/cowjen01/repsys",
"name": "repsys-framework",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "",
"author": "Jan Safarik",
"author_email": "safarj10@fit.cvut.cz",
"download_url": "https://files.pythonhosted.org/packages/f8/47/420e4e7f2141e20717945373db77a87ade10b751a670891c7d7ff2374862/repsys-framework-0.4.1.tar.gz",
"platform": null,
"description": "# RepSys: Framework for Interactive Evaluation of Recommender Systems\n\n[](https://badge.fury.io/py/repsys-framework)\n\nThe RepSys is a framework for developing and analyzing recommendation systems, and it allows you to:\n- Add your own dataset and recommendation models\n- Visually evaluate the models on various metrics\n- Quickly create dataset embeddings to explore the data\n- Preview recommendations using a web application\n- Simulate user's behavior while receiving the recommendations\n\n\n\n<p align=\"middle\">\n <img src=\"https://github.com/cowjen01/repsys/raw/master/images/demos/eval-space.png\" width=\"48%\" />\n <img src=\"https://github.com/cowjen01/repsys/raw/master/images/demos/dataset-users.png\" width=\"48%\" /> \n</p>\n\n## Online Demo\n\nYou can now try RepSys online on our [demo site](https://repsys.recombee.net) with the Movielens dataset. \nAlso, check out an [interactive blog post](https://www.recombee.com/blog/repsys-opensource-library-for-interactive-evaluation-of-recommendation-systems.html) we made using the RepSys widgets component.\n\n## Publication\n\nOur paper \"[RepSys: Framework for Interactive Evaluation of Recommender Systems](https://dl.acm.org/doi/10.1145/3523227.3551469)\" was accepted to the RecSys'22 conference.\n\n## Installation\n\nInstall the package using [pip](https://pypi.org/project/repsys-framework/):\n\n```\n$ pip install repsys-framework\n```\n\nIf you will be using PyMDE for data visualization, you need to install RepSys with the following extras:\n\n```\n$ pip install repsys-framework[pymde]\n```\n\n## Getting Started\n\nIf you want to skip this tutorial and try the framework, you can pull the content of the [demo](https://github.com/cowjen01/repsys/tree/master/demo) folder located at the repository.\nAs mentioned in the [next step](https://github.com/cowjen01/repsys#datasetpy), you still have to download the dataset before you begin.\n\nOtherwise, please create an empty project folder that will contain the dataset and models implementation.\n\n```\n\u251c\u2500\u2500 __init__.py\n\u251c\u2500\u2500 dataset.py\n\u251c\u2500\u2500 models.py\n\u251c\u2500\u2500 repsys.ini\n\u2514\u2500\u2500 .gitignore\n```\n\n### dataset.py\n\nFirstly we need to import our dataset. We will use [MovieLens 20M Dataset](https://grouplens.org/datasets/movielens/20m/) with \n20 million ratings made by 138,000 users to 27,000 movies for the tutorial purpose. Please download the `ml-20m.zip` file and unzip \nthe data into the current folder. Then add the following content to the `dataset.py` file:\n\n```python\nimport pandas as pd\n\nfrom repsys import Dataset\nimport repsys.dtypes as dtypes\n\nclass MovieLens(Dataset):\n def name(self):\n return \"ml20m\"\n\n def item_cols(self):\n return {\n \"movieId\": dtypes.ItemID(),\n \"title\": dtypes.Title(),\n \"genres\": dtypes.Tag(sep=\"|\"),\n \"year\": dtypes.Number(data_type=int),\n }\n\n def interaction_cols(self):\n return {\n \"movieId\": dtypes.ItemID(),\n \"userId\": dtypes.UserID(),\n \"rating\": dtypes.Interaction(),\n }\n\n def load_items(self):\n df = pd.read_csv(\"./ml-20m/movies.csv\")\n df[\"year\"] = df[\"title\"].str.extract(r\"\\((\\d+)\\)\")\n return df\n\n def load_interactions(self):\n df = pd.read_csv(\"./ml-20m/ratings.csv\")\n return df\n```\n\nThis code will define a new dataset called ml20m, and it will import both ratings \nand items data. You must always specify your data structure using predefined data types.\nBefore you return the data, you can also preprocess it like extracting the movie's year from the title column.\n\n### models.py\nNow we define the first recommendation model, which will be a simple implementation of the user-based KNN.\n\n```python\nimport numpy as np\nimport scipy.sparse as sp\nfrom sklearn.neighbors import NearestNeighbors\n\nfrom repsys import Model\n\nclass KNN(Model):\n def __init__(self):\n self.model = NearestNeighbors(n_neighbors=20, metric=\"cosine\")\n\n def name(self):\n return \"knn\"\n \n def fit(self, training=False):\n X = self.dataset.get_train_data()\n self.model.fit(X)\n\n def predict(self, X, **kwargs):\n if X.count_nonzero() == 0:\n return np.random.uniform(size=X.shape)\n \n distances, indices = self.model.kneighbors(X)\n \n distances = distances[:, 1:]\n indices = indices[:, 1:]\n \n distances = 1 - distances\n sums = distances.sum(axis=1)\n distances = distances / sums[:, np.newaxis]\n\n def f(dist, idx):\n A = self.dataset.get_train_data()[idx]\n D = sp.diags(dist)\n return D.dot(A).sum(axis=0)\n\n vf = np.vectorize(f, signature=\"(n),(n)->(m)\")\n \n predictions = vf(distances, indices)\n predictions[X.nonzero()] = 0\n\n return predictions\n```\n\nYou must define the fit method to train your model using the training data or load the previously trained model from a file.\nAll models are fitted when the web application starts, or the evaluation process begins. If this is not a training phase, always \nload your model from a checkpoint to speed up the process. For tutorial purposes, this is omitted.\n\nYou must also define the prediction method that receives a sparse matrix of the users' interactions on the input. \nFor each user (row of the matrix) and item (column of the matrix), the method should return a predicted score indicating \nhow much the user will enjoy the item.\n\nAdditionally, you can specify some web application parameters you can set during recommender creation. The value is then accessible in \nthe `**kwargs` argument of the prediction method. In the example, we create a select input with all unique genres and filter out only those movies that do not contain the selected genre. \n\n### repsys.ini\n\nThe last file we should create is a configuration that allows you to control a data splitting process, server settings, \nframework behavior, etc.\n\n```ini\n[general]\nseed=1234\n\n[dataset]\ntrain_split_prop=0.85\ntest_holdout_prop=0.2\nmin_user_interacts=5\nmin_item_interacts=0\n\n[evaluation]\nprecision_recall_k=20,50\nndcg_k=100\ncoverage_k=20\ndiversity_k=20\nnovelty_k=20\npercentage_lt_k=20\ncoverage_lt_k=20\n\n[visualization]\nembed_method=pymde\npymde_neighbors=15\numap_neighbors=15\numap_min_dist=0.1\ntsne_perplexity=30\n\n[server]\nport=3001\n```\n\n### Splitting the Data\n\nBefore we train our models, we need to split the data into train, validation, and test sets. Run the following command from the current directory.\n\n```\n$ repsys dataset split\n```\n\nThis will hold out 85% of the users as training data, and the rest 15% will be used as validation/test data with 7.5% of users each. For both validation \nand test set, 20% of the interactions will also be held out for evaluation purposes. The split dataset will be stored in the default checkpoints folder.\n\n### Training the Models\n\nNow we can move to the training process. To do this, please call the following command.\n\n```\n$ repsys model train\n```\n\nThis command will call the fit method of each model with the training flag set to true. You can always limit the models using `-m` flag with the model's name as a parameter.\n\n\n### Evaluating the Models\n\nWhen the data is prepared and the models trained, we can evaluate the performance of the models on the unseen users' interactions. Run the following command to do so.\n\n```\n$ repsys model eval\n```\n\nAgain, you can limit the models using the `-m` flag. The results will be stored in the checkpoints folder when the evaluation is done.\n\n### Evaluating the Dataset\n\nBefore starting the web application, the final step is to evaluate the dataset's data. This procedure will create users and items embeddings of the training and validation data \nto allow you to explore the latent space. Run the following command from the project directory.\n\n```\n$ repsys dataset eval\n```\n\nYou can choose from three types of embeddings algorithm:\n1. [UMAP](https://umap-learn.readthedocs.io/en/latest/index.html) (Uniform Manifold Approximation and Projection for Dimension Reduction) is a dimensionality reduction technique similar to t-SNE. Use `--method umap` (this is the default option).\n2. [PyMDE](https://pymde.org) (Minimum-Distortion Embedding) is a fast library designed to distort relationships between pairs of items minimally. Use `--method pymde`.\n3. Combination of the PCA and TSNE algorithms (reduction of the dimensionality to 50 using PCA, then reduction to 2D space using TSNE). Use `--method tsne`.\n4. Your own implementation of the algorithm. Use `--method custom` and add the following method to the model's class of your choice. In this case, you must also specify the model's name using `-m` parameter.\n\n```python\nfrom sklearn.decomposition import NMF\n\ndef compute_embeddings(self, X):\n nmf = NMF(n_components=2)\n W = nmf.fit_transform(X)\n H = nmf.components_\n return W, H.T\n```\n\nIn the example, the negative matrix factorization is used. You have to return a user and item embeddings pair in this order. Also, it is essential to return the matrices in the shape of (n_users/n_items, n_dim). \nIf the reduced dimension is higher than 2, the TSNE method is applied.\n\n### Running the Application\n\nFinally, it is time to start the web application to see the results of the evaluations and preview live recommendations of your models.\n\n```\n$ repsys server\n```\n\nThe application should be accessible on the default address [http://localhost:3001](http://localhost:3001). When you open the link, you will see the main screen where your recommendations appear once you finish the setup.\nThe first step is defining how the items' data columns should be mapped to the item view components.\n\n\n\nThen we need to switch to the build mode and add two recommenders - one without filter and the second with only comedy movies included.\n\n\n\nNow we switch back from the build mode and select a user from the validation set (never seen by a model before).\n\n\n\nFinally, we see the user's interaction history on the right side and the recommendations made by the model on the left side.\n\n\n\n## Contributing\n\nTo build the package from the source, you first need to install Node.js and npm library as documented [here](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm).\nThen you can run the following script from the root directory to build the web application and install the package locally.\n\n```\n$ ./scripts/install-locally.sh\n```\n\n## Citation\n\nIf you employ RepSys in your research work, please do not forget to cite the related paper:\n\n```\n@inproceedings{10.1145/3523227.3551469,\n author = {\\v{S}afa\\v{r}\\'{\\i}k, Jan and Van\\v{c}ura, Vojt\\v{e}ch and Kord\\'{\\i}k, Pavel},\n title = {RepSys: Framework for Interactive Evaluation of Recommender Systems},\n year = {2022},\n isbn = {9781450392785},\n publisher = {Association for Computing Machinery},\n address = {New York, NY, USA},\n url = {https://doi.org/10.1145/3523227.3551469},\n doi = {10.1145/3523227.3551469},\n booktitle = {Proceedings of the 16th ACM Conference on Recommender Systems},\n pages = {636\u2013639},\n numpages = {4},\n keywords = {User simulation, Distribution analysis, Recommender systems},\n location = {Seattle, WA, USA},\n series = {RecSys '22}\n}\n```\n\n## The Team\n\n- Jan \u0160afa\u0159\u00edk (safarj10@fit.cvut.cz)\n- Vojt\u011bch Van\u010dura (vancurv@fit.cvut.cz)\n- Pavel Kord\u00edk (pavel.kordik@fit.cvut.cz)\n- Petr Kasalick\u00fd (kasalpe1@fit.cvut.cz)\n\n## Sponsoring\n\nThe development of this framework is sponsored by the [Recombee](https://www.recombee.com) company.\n\n<img src=\"https://github.com/cowjen01/repsys/raw/master/images/recombee-logo.png\" width=\"50%\" />\n",
"bugtrack_url": null,
"license": "GPLv3",
"summary": "Framework for developing and analyzing recommender systems.",
"version": "0.4.1",
"project_urls": {
"Homepage": "https://github.com/cowjen01/repsys"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "b3c10d842bf0ca848f19091409aec7a16401a49dfccdf1c35ca923304bf32d6a",
"md5": "f56e4f1fdca1425147b4d68dce20efe5",
"sha256": "795ec52f68b3c793f05150690a706a77dc5a71a3ea7832894455cfac17d169dd"
},
"downloads": -1,
"filename": "repsys_framework-0.4.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f56e4f1fdca1425147b4d68dce20efe5",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 10756816,
"upload_time": "2023-08-05T09:41:16",
"upload_time_iso_8601": "2023-08-05T09:41:16.543986Z",
"url": "https://files.pythonhosted.org/packages/b3/c1/0d842bf0ca848f19091409aec7a16401a49dfccdf1c35ca923304bf32d6a/repsys_framework-0.4.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f847420e4e7f2141e20717945373db77a87ade10b751a670891c7d7ff2374862",
"md5": "3a644bfec2e5cdeb45dae875a164afc3",
"sha256": "bcd4e8f15f83bd2a13f555b6876e88846302ed46d08ed3c30fcccee7cc80a3c8"
},
"downloads": -1,
"filename": "repsys-framework-0.4.1.tar.gz",
"has_sig": false,
"md5_digest": "3a644bfec2e5cdeb45dae875a164afc3",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 10653651,
"upload_time": "2023-08-05T09:41:22",
"upload_time_iso_8601": "2023-08-05T09:41:22.494062Z",
"url": "https://files.pythonhosted.org/packages/f8/47/420e4e7f2141e20717945373db77a87ade10b751a670891c7d7ff2374862/repsys-framework-0.4.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-05 09:41:22",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "cowjen01",
"github_project": "repsys",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"tox": true,
"lcname": "repsys-framework"
}
JSON
