# The Prosocial Ranking Challenge
The Prosocial Ranking Challenge is designed to inspire, fund, and test the best algorithms to improve well-being, polarization, and factual knowledge for social media users. We will use our browser extension to re-order the feeds of paid U.S. participants on Facebook, Reddit, and X (Twitter) for four months, and measure changes in attitudes and behavior.
[More about the project here](https://humancompatible.ai/news/2024/01/18/the-prosocial-ranking-challenge-60000-in-prizes-for-better-social-media-algorithms/)
How do we identify pro- and anti-social content? That's where you come in! We are soliciting ranking algorithms to test, with $60,000 in prize money to be split between ten finalists (as selected by our panel of experts).
## pydantic models for the PRC API schema
You can use these models in your Python code, both to generate valid data, and to parse incoming data.
Using the models ensures that your data has been at least somewhat validated. If the schema changes and your code needs an update, you're more likely to be able to tell right away.
### Parsing a request
#### With FastAPI
If you're using fastapi, you can use the models right in your server:
```python
from ranking_challenge.request import RankingRequest
from ranking_challenge.response import RankingResponse
@app.post("/rank")
def rank(ranking_request: RankingRequest) -> RankingResponse:
...
# You can return a RankingResponse here, or a dict with the correct keys and
# pydantic will figure it out.
```
If you specify `RankingResponse` as your reeturn type, you will get validation of your response for free.
For a complete example, check out `../fastapi_nltk/`
#### Otherwise
If you'd like to parse a request directly, here is how:
```python
from ranking_challenge.request import RankingRequest
loaded_request = RankingRequest.model_validate_json(json_data)
```
### Generating fake data
There is a fake data generator, `rcfaker`. If you run it directly it'll print some.
You can also import it like so:
```python
from ranking_challenge.fake import fake_request, fake_response
# 5 fake reddit posts with 2 comments each
request = fake_request(n_posts=5, n_comments=2, platform='reddit')
# corresponding ranker response with 2 added items
request_ids = [r.id for r in request]
response = fake_response(request_ids, n_new_items=2)
```
For more in-depth examples, check out the tests.
### More
[The pydantic docs](https://docs.pydantic.dev/latest/)
## Prometheus Metrics Middleware
This middleware provides an easy way to add Prometheus metrics to your rankers. It automatically exposes a `/metrics` endpoint that can be scraped by Prometheus and logged in Grafana.
### What are these metrics for?
Prometheus metrics allow you to monitor and analyze various aspects of your ranker's performance and behavior. These metrics can be visualized in Grafana, allowing you to gain insights into your ranker's health, performance, and usage patterns.
### How it works
1. The middleware collects and stores metrics data as your application runs.
2. A `/metrics` endpoint is added to your application.
3. When this endpoint is accessed (by a Prometheus server), it serves the collected metrics data in the Prometheus text-based format.
4. Prometheus periodically scrapes this endpoint to collect the latest metrics.
This follows Prometheus' pull model, where PRC metrics service polls and fetches metrics from your ranker, rather than your ranker pushing metrics.
### Installation
```bash
pip install ranking_challenge prometheus_client
```
### Usage
Here's how to set up the middleware and define custom metrics:
```python
from starlette.applications import Starlette
from fastapi import FastAPI
from ranking_challenge.prometheus_metrics_otel_middleware import (
expose_metrics,
CollectorRegistry,
)
from prometheus_client import Counter, Histogram
# Your app can be either Starlette or FastAPI
app = FastAPI()
# Create a registry
registry = CollectorRegistry()
# Create custom metrics
custom_metrics = create_custom_metrics(registry)
# Define a custom metric
content_score = Histogram('content_score', 'Distribution of content scores', ['platform'], registry=registry)
# Define a function to update the custom metric
def update_content_score(request, response, duration):
# This is just an example. In a real ranker, you'd get these values from your actual logic.
score = 0.75 # Example score
platform = "mobile" # Example platform
content_score.labels(platform=platform).observe(score)
# Add the custom metric to the dictionary
custom_metrics["content_score"] = update_content_score
# Set up the metrics endpoint and middleware
expose_metrics(
app,
endpoint="/metrics",
registry=registry,
custom_metrics=custom_metrics
)
# Your application routes and logic go here
@app.route("/")
async def root():
return {"message": "Hello World"}
```
In this example, we're creating a histogram metric to track the distribution of content scores across different platforms. Every time a request is processed, the `update_content_score` function will be called, which updates our custom metric.
### Metric Types
Prometheus supports several types of metrics. You'll need to import these from `prometheus_client` for use. The most common are:
1. **Counter**: A cumulative metric that only goes up (e.g., number of requests)
2. **Gauge**: A metric that can go up and down (e.g., current number of active sessions)
3. **Histogram**: Samples observations and counts them in configurable buckets (e.g., request durations)
4. **Summary**: Similar to histogram, but calculates configurable quantiles over a sliding time window
Example Usage:
```python
from prometheus_client import Counter, Histogram
```
For more details on metric types, refer to the [Prometheus documentation](https://prometheus.io/docs/concepts/metric_types/).
### Viewing Metrics
Once set up and deployed to production, you can view and analyze the raw metrics in Grafana Cloud under your team's folder.
### Learn More
- [Prometheus Data Model Concept](https://prometheus.io/docs/concepts/data_model/)
- [Grafana](https://grafana.com/)
Raw data
{
"_id": null,
"home_page": "https://humancompatible.ai/news/2024/01/18/the-prosocial-ranking-challenge-60000-in-prizes-for-better-social-media-algorithms/",
"name": "ranking-challenge",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "ranking, social media, challenge",
"author": "Ian Baker",
"author_email": "ian@sonic.net",
"download_url": "https://files.pythonhosted.org/packages/53/70/5b3fc2b3e8d8b1195522478c13f01b44f52fc8ddac5c8ba8c461c107252a/ranking_challenge-3.4.0.tar.gz",
"platform": null,
"description": "# The Prosocial Ranking Challenge\n\nThe Prosocial Ranking Challenge is designed to inspire, fund, and test the best algorithms to improve well-being, polarization, and factual knowledge for social media users. We will use our browser extension to re-order the feeds of paid U.S. participants on Facebook, Reddit, and X (Twitter) for four months, and measure changes in attitudes and behavior.\n\n[More about the project here](https://humancompatible.ai/news/2024/01/18/the-prosocial-ranking-challenge-60000-in-prizes-for-better-social-media-algorithms/)\n\nHow do we identify pro- and anti-social content? That's where you come in! We are soliciting ranking algorithms to test, with $60,000 in prize money to be split between ten finalists (as selected by our panel of experts).\n\n## pydantic models for the PRC API schema\n\nYou can use these models in your Python code, both to generate valid data, and to parse incoming data.\n\nUsing the models ensures that your data has been at least somewhat validated. If the schema changes and your code needs an update, you're more likely to be able to tell right away.\n\n### Parsing a request\n\n#### With FastAPI\n\nIf you're using fastapi, you can use the models right in your server:\n\n```python\nfrom ranking_challenge.request import RankingRequest\nfrom ranking_challenge.response import RankingResponse\n\n@app.post(\"/rank\")\ndef rank(ranking_request: RankingRequest) -> RankingResponse:\n ...\n # You can return a RankingResponse here, or a dict with the correct keys and\n # pydantic will figure it out.\n```\n\nIf you specify `RankingResponse` as your reeturn type, you will get validation of your response for free.\n\nFor a complete example, check out `../fastapi_nltk/`\n\n#### Otherwise\n\nIf you'd like to parse a request directly, here is how:\n\n```python\nfrom ranking_challenge.request import RankingRequest\n\nloaded_request = RankingRequest.model_validate_json(json_data)\n```\n\n### Generating fake data\n\nThere is a fake data generator, `rcfaker`. If you run it directly it'll print some.\n\nYou can also import it like so:\n\n```python\nfrom ranking_challenge.fake import fake_request, fake_response\n\n# 5 fake reddit posts with 2 comments each\nrequest = fake_request(n_posts=5, n_comments=2, platform='reddit')\n\n# corresponding ranker response with 2 added items\nrequest_ids = [r.id for r in request]\nresponse = fake_response(request_ids, n_new_items=2)\n```\n\nFor more in-depth examples, check out the tests.\n\n### More\n\n[The pydantic docs](https://docs.pydantic.dev/latest/)\n\n## Prometheus Metrics Middleware\n\nThis middleware provides an easy way to add Prometheus metrics to your rankers. It automatically exposes a `/metrics` endpoint that can be scraped by Prometheus and logged in Grafana.\n\n### What are these metrics for?\n\nPrometheus metrics allow you to monitor and analyze various aspects of your ranker's performance and behavior. These metrics can be visualized in Grafana, allowing you to gain insights into your ranker's health, performance, and usage patterns.\n\n### How it works\n\n1. The middleware collects and stores metrics data as your application runs.\n2. A `/metrics` endpoint is added to your application.\n3. When this endpoint is accessed (by a Prometheus server), it serves the collected metrics data in the Prometheus text-based format.\n4. Prometheus periodically scrapes this endpoint to collect the latest metrics.\n\nThis follows Prometheus' pull model, where PRC metrics service polls and fetches metrics from your ranker, rather than your ranker pushing metrics.\n\n### Installation\n\n```bash\npip install ranking_challenge prometheus_client\n```\n\n### Usage\n\nHere's how to set up the middleware and define custom metrics:\n\n```python\nfrom starlette.applications import Starlette\nfrom fastapi import FastAPI\nfrom ranking_challenge.prometheus_metrics_otel_middleware import (\n expose_metrics,\n CollectorRegistry,\n)\nfrom prometheus_client import Counter, Histogram\n\n# Your app can be either Starlette or FastAPI\napp = FastAPI()\n\n# Create a registry\nregistry = CollectorRegistry()\n\n# Create custom metrics\ncustom_metrics = create_custom_metrics(registry)\n\n# Define a custom metric\ncontent_score = Histogram('content_score', 'Distribution of content scores', ['platform'], registry=registry)\n\n# Define a function to update the custom metric\ndef update_content_score(request, response, duration):\n # This is just an example. In a real ranker, you'd get these values from your actual logic.\n score = 0.75 # Example score\n platform = \"mobile\" # Example platform\n content_score.labels(platform=platform).observe(score)\n\n# Add the custom metric to the dictionary\ncustom_metrics[\"content_score\"] = update_content_score\n\n# Set up the metrics endpoint and middleware\nexpose_metrics(\n app,\n endpoint=\"/metrics\",\n registry=registry,\n custom_metrics=custom_metrics\n)\n\n# Your application routes and logic go here\n@app.route(\"/\")\nasync def root():\n return {\"message\": \"Hello World\"}\n```\n\nIn this example, we're creating a histogram metric to track the distribution of content scores across different platforms. Every time a request is processed, the `update_content_score` function will be called, which updates our custom metric.\n\n### Metric Types\n\nPrometheus supports several types of metrics. You'll need to import these from `prometheus_client` for use. The most common are:\n\n1. **Counter**: A cumulative metric that only goes up (e.g., number of requests)\n2. **Gauge**: A metric that can go up and down (e.g., current number of active sessions)\n3. **Histogram**: Samples observations and counts them in configurable buckets (e.g., request durations)\n4. **Summary**: Similar to histogram, but calculates configurable quantiles over a sliding time window\n\nExample Usage:\n\n```python\nfrom prometheus_client import Counter, Histogram\n```\n\nFor more details on metric types, refer to the [Prometheus documentation](https://prometheus.io/docs/concepts/metric_types/).\n\n### Viewing Metrics\n\nOnce set up and deployed to production, you can view and analyze the raw metrics in Grafana Cloud under your team's folder.\n\n### Learn More\n\n- [Prometheus Data Model Concept](https://prometheus.io/docs/concepts/data_model/)\n- [Grafana](https://grafana.com/)\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "The Prosocial Ranking Challenge",
"version": "3.4.0",
"project_urls": {
"Homepage": "https://humancompatible.ai/news/2024/01/18/the-prosocial-ranking-challenge-60000-in-prizes-for-better-social-media-algorithms/",
"Repository": "https://github.com/humancompatibleai/ranking-challenge",
"blog": "https://rankingchallenge.substack.com/",
"repository": "https://github.com/humancompatibleai/ranking-challenge"
},
"split_keywords": [
"ranking",
" social media",
" challenge"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "d442f265a7de72004ecd7a8f3a71580a8df55568d0ac3a4808fc3960310d23b3",
"md5": "22b504983b846e58f5e32c6c76fd584d",
"sha256": "ae5052cb15198cc0ce6e8a5d75870251569a320afc3d38d37a801a32c0c837a4"
},
"downloads": -1,
"filename": "ranking_challenge-3.4.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "22b504983b846e58f5e32c6c76fd584d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 18307,
"upload_time": "2024-10-18T01:35:38",
"upload_time_iso_8601": "2024-10-18T01:35:38.108643Z",
"url": "https://files.pythonhosted.org/packages/d4/42/f265a7de72004ecd7a8f3a71580a8df55568d0ac3a4808fc3960310d23b3/ranking_challenge-3.4.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "53705b3fc2b3e8d8b1195522478c13f01b44f52fc8ddac5c8ba8c461c107252a",
"md5": "69cd95f9e1fbbd44174912170997a6d8",
"sha256": "d4448f69a43c974f3b5e38aee5b0f35aa0bd295724ac0af6adc550b6f1c80e06"
},
"downloads": -1,
"filename": "ranking_challenge-3.4.0.tar.gz",
"has_sig": false,
"md5_digest": "69cd95f9e1fbbd44174912170997a6d8",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 17561,
"upload_time": "2024-10-18T01:35:39",
"upload_time_iso_8601": "2024-10-18T01:35:39.565913Z",
"url": "https://files.pythonhosted.org/packages/53/70/5b3fc2b3e8d8b1195522478c13f01b44f52fc8ddac5c8ba8c461c107252a/ranking_challenge-3.4.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-18 01:35:39",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "humancompatibleai",
"github_project": "ranking-challenge",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "ranking-challenge"
}
JSON
