<div align="center">
<a href="https://github.com/scribe-org/Scribe-Data"><img src="https://raw.githubusercontent.com/scribe-org/Scribe-Data/main/.github/resources/images/ScribeDataLogo.png" height=150 alt="Scribe-Data Logo"></a>
</div>
[](https://github.com/scribe-org/Scribe-Data)
[](http://scribe-data.readthedocs.io/en/latest/)
[](https://github.com/scribe-org/Scribe-Data/issues)
[](https://github.com/scribe-org/Scribe-Data/blob/main/CONTRIBUTING.md)
[](https://pypi.org/project/scribe-data/)
[](https://pypi.org/project/scribe-data/)
[](https://github.com/scribe-org/Scribe-Data/blob/main/LICENSE.txt)
[](https://github.com/scribe-org/Scribe-Data/blob/main/.github/CODE_OF_CONDUCT.md)
[](https://wikis.world/@scribe)
[](https://matrix.to/#/#scribe_community:matrix.org)
### Wikidata and Wikipedia language data extraction
**Scribe-Data** is a convenient command-line interface (CLI) for extracting and formatting language data from [Wikidata](https://www.wikidata.org/) and [Wikipedia](https://www.wikipedia.org/). Functionality includes allowing users to list, download, and manage language data directly from the terminal.
> [!NOTE]\
> The [contributing](#contributing) section has information for those interested, with the articles and presentations in [featured by](#featured-by) also being good resources for learning more about Scribe.
Scribe applications are available on [iOS](https://github.com/scribe-org/Scribe-iOS), [Android](https://github.com/scribe-org/Scribe-Android) (WIP) and [Desktop](https://github.com/scribe-org/Scribe-Desktop) (planned).
Check out Scribe's [architecture diagrams](https://github.com/scribe-org/Organization/blob/main/ARCHITECTURE.md) for an overview of the organization including our applications, services and processes. It depicts the projects that [Scribe](https://github.com/scribe-org) is developing as well as the relationships between them and the external systems with which they interact. Also check out the [Wikidata and Scribe Guide](https://github.com/scribe-org/Organization/blob/main/WIKIDATAGUIDE.md) for an overview of [Wikidata](https://www.wikidata.org/) and getting language data from it.
<a id="contents"></a>
# **Contents**
- [Process](#process)
- [Installation](#installation)
- [CLI Usage](#cli-usage)
- [Data Contracts](#data-contracts)
- [Contributing](#contributing)
- [Environment Setup](#environment-setup)
- [Featured By](#featured-by)
<a id="Process"></a>
# Process [`⇧`](#contents)
The CLI commands defined within [scribe_data/cli](https://github.com/scribe-org/Scribe-Data/blob/main/src/scribe_data/cli) and the notebooks within the various [scribe_data](https://github.com/scribe-org/Scribe-Data/tree/main/src/scribe_data) directories are used to update all data for [Scribe-iOS](https://github.com/scribe-org/Scribe-iOS), with this functionality later being expanded to update [Scribe-Android](https://github.com/scribe-org/Scribe-Android) and [Scribe-Desktop](https://github.com/scribe-org/Scribe-Desktop) once they're active.
The main data update process in triggers [language based SPARQL queries](https://github.com/scribe-org/Scribe-Data/tree/main/src/scribe_data/wikidata/language_data_extraction) to query language data from [Wikidata](https://www.wikidata.org/) using [SPARQLWrapper](https://github.com/RDFLib/sparqlwrapper) as a URI. The autosuggestion process derives popular words from [Wikipedia](https://www.wikipedia.org/) as well as those words that normally follow them for an effective baseline feature until natural language processing methods are employed. Functions to generate autosuggestions are ran in [gen_autosuggestions.ipynb](https://github.com/scribe-org/Scribe-Data/blob/main/src/scribe_data/wikipedia/gen_autosuggestions.ipynb). Emojis are further sourced from [Unicode CLDR](https://github.com/unicode-org/cldr), with this process being ran via the `scribe-data get -lang LANGUAGE -dt emoji-keywords` command.
<a id="installation"></a>
# Installation [`⇧`](#contents)
Scribe-Data is available for installation via [pip](https://pypi.org/project/scribe-data/):
```bash
pip install scribe-data
# For a development build:
git clone https://github.com/scribe-org/Scribe-Data.git # or ideally your fork
cd Scribe-Data
pip install -e .
```
<a id="cli-usage"></a>
# CLI Usage [`⇧`](#contents)
Scribe-Data provides a command-line interface (CLI) for efficient interaction with its language data functionality. Please see the [usage guide](https://github.com/scribe-org/Scribe-Data/blob/main/USAGE.md) or the [official documentation](https://scribe-data.readthedocs.io/) for detailed instructions.
### Basic Usage
To utilize the Scribe-Data CLI, you can execute variations of the following command in your terminal:
```bash
scribe-data -h # view the cli options
scribe-data [command] [arguments]
```
### Available Commands
- `list` (`l`): Enumerate available languages, data types and their combinations.
- `get` (`g`): Retrieve data from Wikidata for specified languages and data types.
- `total` (`t`): Display the total available data for given languages and data types.
- `convert` (`c`): Transform data returned by Scribe-Data into different file formats.
### Command Examples
<p align="center">
<img src="https://github.com/user-attachments/assets/653941a7-68bb-4d72-a0f1-3e29c75c5a16" alt="List, Total and Get GIF" width="500" height="300">
</p>
```bash
# Commands used in the above GIF:
scribe-data list --language
scribe-data list --data-type
scribe-data get --language English --data-type verbs -od ./scribe-data
scribe-data total --language English
```
<p align="center">
<img src="https://github.com/user-attachments/assets/4cbb85ed-d853-4008-8db9-b77ffcbe2e84" alt="Interactive GIF" width="500" height="300">
</p>
```bash
# Commands used in the above GIF:
scribe-data get -i
scribe-data total -i
```
<a id="data-contracts"></a>
# Data Contracts [`⇧`](#contents)
[Wikidata](https://www.wikidata.org/) has lots of [language data](https://www.wikidata.org/wiki/Wikidata:Lexicographical_data) available, but not all of it is useful for all applications. In order to make the functionality of the Scribe-Data `get` requests as simple as possible, we made the decision to always return all data for the given languages and data types. Adding the ability to pass desired forms to the commands seemed cumbersome, and larger Scribe-Data requests should be parsing [Wikidata lexeme dumps](https://dumps.wikimedia.org/wikidatawiki/entities/) as the data source.
Scribe's solution to the get all functionality while preserving the ability to get specific forms is to allow users to filter the resulting data by contracts. The data contracts for Scribe's client applications can be found in the [data_contracts](./data_contracts/) directory. Data contracts are JSON objects where the values that are used in end applications are the keys and the resulting data identifiers based on Wikidata lexeme forms are the values. If the forms for a lexeme change, then the values would also change, but all that's needed is to update the contract for the application to function again.
Efficient client application data updates using Scribe-Data follow as such:
- New data is derived via the Scribe-Data CLI
- Contracts are written to map the data values to keys that are used in the application
- Scribe-Data is ran again to get new data in the future
- The contracts are checked to make sure that all contract values still exist within the resulting data
- The question is whether a form was added or removed from a data point such that its identifier has changed
- This is done via the following command:
```bash
scribe-data cc -cd DATA_CONTRACTS_DIRECTORY # default data path is used
```
- If the check above passes, then new data can be added to the client applications
- If the check fails, then the contract values should be updated given the directions from the CLI and then new data can be loaded
- Getting just the data that's in the client application is done via the following command:
```bash
scribe-data fd -cd DATA_CONTRACTS_DIRECTORY # default data paths are used
```
Updating contracts shouldn't be something that Scribe-Data users should have to do often if they're using stable data from [Wikidata](https://www.wikidata.org/). We provide this functionality given the wiki nature of the underlying data so that the Scribe community and others can easily react to potential changes in the lexeme data.
> [!NOTE]
> You can learn more about contracts and the process around them in [DATA_CONTRACTS.md](https://github.com/scribe-org/Organization/blob/main/DATA_CONTRACTS.md).
<a id="contributing"></a>
# Contributing [`⇧`](#contents)
<a href="https://matrix.to/#/#scribe_community:matrix.org">
<img src="https://raw.githubusercontent.com/scribe-org/Organization/main/resources/images/logos/MatrixLogoGrey.png" width="175" alt="Public Matrix Chat" align="right">
</a>
Scribe uses [Matrix](https://matrix.org/) for communications. You're more than welcome to [join us in our public chat rooms](https://matrix.to/#/#scribe_community:matrix.org) to share ideas, ask questions or just say hi to the team :) We'd suggest that you use the [Element](https://element.io/) client and [Element X](https://element.io/app) for a mobile app.
Please see the [contribution guidelines](https://github.com/scribe-org/Scribe-Data/blob/main/CONTRIBUTING.md) and [Wikidata and Scribe Guide](https://github.com/scribe-org/Organization/blob/main/WIKIDATAGUIDE.md) if you are interested in contributing to Scribe-Data. Work that is in progress or could be implemented is tracked in the [issues](https://github.com/scribe-org/Scribe-Data/issues) and [projects](https://github.com/scribe-org/Scribe-Data/projects).
> [!NOTE]\
> Just because an issue is assigned on GitHub doesn't mean the team isn't open to your contribution! Feel free to write [in the issues](https://github.com/scribe-org/Scribe-Data/issues) and we can potentially reassign it to you.
Those interested can further check the [`-next release-`](https://github.com/scribe-org/Scribe-Data/labels/-next%20release-) and [`-priority-`](https://github.com/scribe-org/Scribe-Data/labels/-priority-) labels in the [issues](https://github.com/scribe-org/Scribe-Data/issues) for those that are most important, as well as those marked [`good first issue`](https://github.com/scribe-org/Scribe-Data/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) that are tailored for first-time contributors.
After your first few pull requests organization members would be happy to discuss granting you further rights as a contributor, with a maintainer role then being possible after continued interest in the project. Scribe seeks to be an inclusive and supportive organization. We'd love to have you on the team!
### Ways to Help [`⇧`](#contents)
- [Reporting bugs](https://github.com/scribe-org/Scribe-Data/issues/new?assignees=&labels=bug&template=bug_report.yml) as they're found 🐞
- Working on [new features](https://github.com/scribe-org/Scribe-Data/issues?q=is%3Aissue+is%3Aopen+label%3Afeature) ✨
- [Documentation](https://github.com/scribe-org/Scribe-Data/issues?q=is%3Aissue+is%3Aopen+label%3Adocumentation) for onboarding and project cohesion 📝
- Adding language data to [Scribe-Data](https://github.com/scribe-org/Scribe-Data/issues) via [Wikidata](https://www.wikidata.org/)! 🗃️
### Road Map [`⇧`](#contents)
The Scribe road map can be followed in the organization's [project board](https://github.com/orgs/scribe-org/projects/1) where we list the most important issues along with their priority, status and an indication of which sub projects they're included in (if applicable).
> [!NOTE]\
> Consider joining our [bi-weekly developer syncs](https://etherpad.wikimedia.org/p/scribe-dev-sync)!
### Data Edits [`⇧`](#contents)
> [!NOTE]\
> Please see the [Wikidata and Scribe Guide](https://github.com/scribe-org/Organization/blob/main/WIKIDATAGUIDE.md) for an overview of [Wikidata](https://www.wikidata.org/) and how Scribe uses it.
Scribe does not accept direct edits to the grammar JSON files as they are sourced from [Wikidata](https://www.wikidata.org/). Edits can be discussed and the queries themselves will be changed and ran before an update. If there is a problem with one of the files, then the fix should be made on [Wikidata](https://www.wikidata.org/) and not on Scribe. Feel free to let us know that edits have been made by [opening a data issue](https://github.com/scribe-org/Scribe-Data/issues/new?assignees=&labels=data&template=data_wikidata.yml) and we'll be happy to integrate them!
<a id="environment-setup"></a>
# Environment Setup [`⇧`](#contents)
> [!IMPORTANT]
>
> <details><summary>Suggested IDE extensions</summary>
>
> <p>
>
> VS Code
>
> - [blokhinnv.wikidataqidlabels](https://marketplace.visualstudio.com/items?itemName=blokhinnv.wikidataqidlabels)
> - [charliermarsh.ruff](https://marketplace.visualstudio.com/items?itemName=charliermarsh.ruff)
> - [qwtel.sqlite-viewer](https://marketplace.visualstudio.com/items?itemName=qwtel.sqlite-viewer)
> - [streetsidesoftware.code-spell-checker](https://marketplace.visualstudio.com/items?itemName=streetsidesoftware.code-spell-checker)
>
> </p>
> </details>
The development environment for Scribe-Data can be installed via the following steps:
1. [Fork](https://docs.github.com/en/get-started/quickstart/fork-a-repo) the [Scribe-Data repo](https://github.com/scribe-org/Scribe-Data), clone your fork, and configure the remotes:
> [!NOTE]
>
> <details><summary>Consider using SSH</summary>
>
> <p>
>
> Alternatively to using HTTPS as in the instructions below, consider SSH to interact with GitHub from the terminal. SSH allows you to connect without a user-pass authentication flow.
>
> To run git commands with SSH, remember then to substitute the HTTPS URL, `https://github.com/...`, with the SSH one, `git@github.com:...`.
>
> - e.g. Cloning now becomes `git clone git@github.com:<your-username>/Scribe-Data.git`
>
> GitHub also has their documentation on how to [Generate a new SSH key](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent) 🔑
>
> </p>
> </details>
```bash
# Clone your fork of the repo into the current directory.
git clone https://github.com/<your-username>/Scribe-Data.git
# Navigate to the newly cloned directory.
cd Scribe-Data
# Assign the original repo to a remote called "upstream".
git remote add upstream https://github.com/scribe-org/Scribe-Data.git
```
- Now, if you run `git remote -v` you should see two remote repositories named:
- `origin` (forked repository)
- `upstream` (Scribe-Data repository)
2. Use [Python venv](https://docs.python.org/3/library/venv.html) to create the local development environment within your Scribe-Data directory:
- On Unix or MacOS, run:
```bash
python3 -m venv venv # make an environment named venv
source venv/bin/activate # activate the environment
```
- On Windows (using Command Prompt), run:
```bash
python -m venv venv
venv\Scripts\activate.bat
```
- On Windows (using PowerShell), run:
```bash
python -m venv venv
venv\Scripts\activate.ps1
```
After activating the virtual environment, install the required dependencies and set up [pre-commit](https://pre-commit.com/) by running:
```bash
pip install --upgrade pip # make sure that pip is at the latest version
pip install -r requirements-dev.txt # install development dependencies
pip install -e . # install the local version of Scribe-Data
pre-commit install # install pre-commit hooks
# pre-commit run --all-files # lint and fix common problems in the codebase
```
See the [contribution guidelines](https://github.com/scribe-org/Scribe-Data/blob/main/CONTRIBUTING.md) for a more detailed explanation and troubleshooting.
> [!NOTE]
> Feel free to contact the team in the [Data room on Matrix](https://matrix.to/#/#ScribeData:matrix.org) if you're having problems getting your environment setup!
<a id="featured-by"></a>
# Featured By [`⇧`](#contents)
<details open><summary><strong>Articles and Presentations on Scribe</strong></summary>
<p>
<strong>2024</strong>
- October: [Blog post on Medium](https://medium.com/@arpita151103/scribe-an-open-source-solution-for-language-learning-and-data-accessibility-092dab026fd6) discussing the [Scribe-Data](https://github.com/scribe-org/Scribe-Data) development process, community and features
- October: [Blog post on medium](https://medium.com/@mhmohona/ins-and-outs-of-scribe-data-cli-bd51202aa7c6) describing the main features of [Scribe-Data](https://github.com/scribe-org/Scribe-Data)
- September: [Final Google Summer of Code report](https://medium.com/@mhmohona/the-final-stretch-gsoc-journey-with-scribe-data-1740084c958d) on the creation of the [Scribe-Data](https://github.com/scribe-org/Scribe-Data) CLI
- August: [Final Google Summer of Code report](https://jagmarcel.hashnode.dev/gsoc-2024-final-report) on the creation of Scribe's cross-language translation functionality
- July: [Blog post on Medium](https://medium.com/@mhmohona/halfway-there-my-gsoc-adventure-with-scribe-data-cli-2ffe6d727ecb) about the progress on creating the [Scribe-Data](https://github.com/scribe-org/Scribe-Data) CLI
- July: [Blog post on Hashnode](https://jagmarcel.hashnode.dev/gsoc-2024-midterm-report) providing an midterm report on the localization and translation expansion for [Scribe-iOS](https://github.com/scribe-org/Scribe-iOS)
- July: [Blog post on Hashnode](https://jagmarcel.hashnode.dev/my-first-experiences-with-gsoc) about the initial steps towards the localization of [Scribe-iOS](https://github.com/scribe-org/Scribe-iOS)
- June: [Blog post on Medium](https://medium.com/@mhmohona/first-month-as-a-gsoc-intern-building-scribe-data-cli-d0c12c9e8371) about the planned [Scribe-Data](https://github.com/scribe-org/Scribe-Data) CLI
- April: [Blog post on Medium](https://medium.com/@mhmohona/scribe-data-a-guide-to-open-source-language-data-a801c59db4c9) about [Scribe-Data](https://github.com/scribe-org/Scribe-Data) and its functionalities
- February: [Presentation slides](https://docs.google.com/presentation/d/1lMhYiQx1R99SVGhbikUGjOVaFgPPASvbzM2Bsu3NXSg/edit?usp=sharing) for Scribe's participation at the [Wikimedia Tech Safari Program](https://www.mediawiki.org/wiki/Wikimedia_Tech_Safari_Program)
<strong>2023</strong>
- August: [Scribe-iOS final submission report for Google Summer of Code 2023](https://saurabhjamadagni.hashnode.dev/gsoc-23-final-work-submission)
- June: [Scribe-iOS development blog post on Nested UITableViews & Apple's built-in ViewControllers in app menu](https://saurabhjamadagni.hashnode.dev/nested-uitableviews-apples-built-in-viewcontrollers) for [GSoC '23](https://www.mediawiki.org/wiki/Google_Summer_of_Code/2023#Accepted_projects:~:text=links%3A%20Phabricator%20issue-,3.%20Adding%20a%20Menu%20and%20Keyboards%20to%20Scribe%2DiOS,-%5Bedit%5D)
- March: [Presentation slides](https://docs.google.com/presentation/d/1W4ZkGi9UDDiTxM_silEij0gTE8YEubluHxe78xoqEP0/edit?usp=sharing) for a talk at [Berlin Hack and Tell](https://bhnt.c-base.org/) ([Hack of the month winner 🏆](https://bhnt.c-base.org/2023-03-28-no87-moore-hacks))
<strong>2022</strong>
- August: [Presentation slides](https://docs.google.com/presentation/d/12WNSt5xgNIAmSxPfvjno9-sBMGlvxG_xSaAxmHQDRNQ/edit?usp=sharing) for a session at the [2022 Wikimania Hackathon](https://wikimania.wikimedia.org/wiki/2022:Hackathon)
- July: [Presentation slides](https://docs.google.com/presentation/d/10Ai0-b8XUj5u9Hw4UgBtB7ufiPhvfFrb1vEUEyXYr5w/edit?usp=sharing) for a talk at [CocoaHeads Berlin](https://www.meetup.com/cocoaheads-berlin/)
- July: [Video on Scribe](https://www.youtube.com/watch?v=4GpFN0gGmy4&list=PL66MRMNlLyR7p9wsYVfuqJOjKZpbuwp8U&index=6) for [Wikimedia Celtic Knot 2022](https://meta.wikimedia.org/wiki/Celtic_Knot_Conference_2022)
- June: [Presentation slides](https://docs.google.com/presentation/d/1K2lj8PPgdx12I-xuhm--CBLrGm-Cz50NJmbp96zpGrk/edit?usp=sharing) for a talk with the [LD4 Wikidata Affinity Group](https://www.wikidata.org/wiki/Wikidata:WikiProject_LD4_Wikidata_Affinity_Group)
- June: [Scribe](https://github.com/scribe-org) featured for new developers on [MediaWiki](https://www.mediawiki.org/wiki/New_Developers#Scribe)
- May: [Presentation slides](https://docs.google.com/presentation/d/1Cu3VwQ3lJUp5W84YDe0AFYS-6zfBxKsm0MI-OMl_IzY/edit?usp=sharing) for [Wikimedia Hackathon 2022](https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2022)
- March: [Blog post](https://tech-news.wikimedia.de/en/2022/03/18/lexicographical-data-for-language-learners-the-wikidata-based-app-scribe/) on [Scribe-iOS](https://github.com/scribe-org/Scribe-iOS) for [Wikimedia Tech News](https://tech-news.wikimedia.de/en/homepage/) ([DE](https://tech-news.wikimedia.de/2022/03/18/sprachenlernen-mit-lexikografische-daten-die-wikidata-basierte-app-scribe/) / [Tweet](https://twitter.com/wikidata/status/1507335538596106257?s=20&t=YGRGamftI-5B_VwQ_bFRhA))
- March: [Presentation slides](https://docs.google.com/presentation/d/16ld_rCbwJCiAdRrfhF-Fq9Wm_ciHCbk_HCzGQs6TB1Q/edit?usp=sharing) for [Wikidata Data Reuse Days 2022](https://diff.wikimedia.org/event/wikidata-data-reuse-days-2022/)
</p>
</details>
<div align="center">
<br>
<a href="https://tech-news.wikimedia.de/en/2022/03/18/lexicographical-data-for-language-learners-the-wikidata-based-app-scribe/"><img width="180" src="https://raw.githubusercontent.com/scribe-org/Organization/main/resources/images/logos/WikimediaDeutschlandLogo.png" alt="Wikimedia Deutschland logo linking to an article on Scribe in the tech news blog."></a>
<a href="https://www.mediawiki.org/wiki/New_Developers#Scribe"><img width="180" src="https://raw.githubusercontent.com/scribe-org/Organization/main/resources/images/logos/WikimediaFoundationLogo.png" alt="Wikimedia Foundation logo linking to the MediaWiki new developers page."></a>
<br>
</div>
<div align="center">
<br>
<a href="https://summerofcode.withgoogle.com/"><img width="140" src="https://raw.githubusercontent.com/scribe-org/Organization/main/resources/images/logos/GSoCLogo.png" alt="Google Summer of Code logo linking to its website."></a>
<a href="https://www.outreachy.org/"><img width="350" src="https://raw.githubusercontent.com/scribe-org/Organization/main/resources/images/logos/OutreachyLogo.png" alt="Outreachy logo linking to its website."></a>
<br>
</div>
# Powered By [`⇧`](#contents)
### Contributors
Many thanks to all the [Scribe-Data contributors](https://github.com/scribe-org/Scribe-Data/graphs/contributors)! 🚀
<a href="https://github.com/scribe-org/Scribe-Data/graphs/contributors">
<img src="https://contrib.rocks/image?repo=scribe-org/Scribe-Data" />
</a>
### Blog posts
<details><summary><strong>List of referenced posts</strong></summary>
<p>
- [Building a Recommendation System Using Neural Network Embeddings](https://towardsdatascience.com/building-a-recommendation-system-using-neural-network-embeddings-1ef92e5c80c9) by [WillKoehrsen](https://github.com/WillKoehrsen)
- [Wikipedia Data Science: Working with the World’s Largest Encyclopedia](https://towardsdatascience.com/wikipedia-data-science-working-with-the-worlds-largest-encyclopedia-c08efbac5f5c) by [WillKoehrsen](https://github.com/WillKoehrsen)
</p>
</details>
### Wikimedia Communities
<div align="center">
<br>
<a href="https://www.wikidata.org/">
<img width="240" src="https://raw.githubusercontent.com/scribe-org/Organization/main/resources/images/logos/WikidataLogo.png" alt="Wikidata logo">
</a>
<a href="https://www.wikipedia.org/">
<img width="160" src="https://raw.githubusercontent.com/scribe-org/Organization/main/resources/images/logos/WikipediaLogo.png" alt="Wikipedia logo">
</a>
<br>
</div>
Raw data
{
"_id": null,
"home_page": "https://github.com/scribe-org/Scribe-Data",
"name": "scribe-data",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "wikidata, lexemes, cli, data, etl, elt",
"author": "Andrew Tavis McAllister",
"author_email": "team@scri.be",
"download_url": null,
"platform": null,
"description": "<div align=\"center\">\n <a href=\"https://github.com/scribe-org/Scribe-Data\"><img src=\"https://raw.githubusercontent.com/scribe-org/Scribe-Data/main/.github/resources/images/ScribeDataLogo.png\" height=150 alt=\"Scribe-Data Logo\"></a>\n</div>\n\n[](https://github.com/scribe-org/Scribe-Data)\n[](http://scribe-data.readthedocs.io/en/latest/)\n[](https://github.com/scribe-org/Scribe-Data/issues)\n[](https://github.com/scribe-org/Scribe-Data/blob/main/CONTRIBUTING.md)\n[](https://pypi.org/project/scribe-data/)\n[](https://pypi.org/project/scribe-data/)\n[](https://github.com/scribe-org/Scribe-Data/blob/main/LICENSE.txt)\n[](https://github.com/scribe-org/Scribe-Data/blob/main/.github/CODE_OF_CONDUCT.md)\n[](https://wikis.world/@scribe)\n[](https://matrix.to/#/#scribe_community:matrix.org)\n\n### Wikidata and Wikipedia language data extraction\n\n**Scribe-Data** is a convenient command-line interface (CLI) for extracting and formatting language data from [Wikidata](https://www.wikidata.org/) and [Wikipedia](https://www.wikipedia.org/). Functionality includes allowing users to list, download, and manage language data directly from the terminal.\n\n> [!NOTE]\\\n> The [contributing](#contributing) section has information for those interested, with the articles and presentations in [featured by](#featured-by) also being good resources for learning more about Scribe.\n\nScribe applications are available on [iOS](https://github.com/scribe-org/Scribe-iOS), [Android](https://github.com/scribe-org/Scribe-Android) (WIP) and [Desktop](https://github.com/scribe-org/Scribe-Desktop) (planned).\n\nCheck out Scribe's [architecture diagrams](https://github.com/scribe-org/Organization/blob/main/ARCHITECTURE.md) for an overview of the organization including our applications, services and processes. It depicts the projects that [Scribe](https://github.com/scribe-org) is developing as well as the relationships between them and the external systems with which they interact. Also check out the [Wikidata and Scribe Guide](https://github.com/scribe-org/Organization/blob/main/WIKIDATAGUIDE.md) for an overview of [Wikidata](https://www.wikidata.org/) and getting language data from it.\n\n<a id=\"contents\"></a>\n\n# **Contents**\n\n- [Process](#process)\n- [Installation](#installation)\n- [CLI Usage](#cli-usage)\n- [Data Contracts](#data-contracts)\n- [Contributing](#contributing)\n- [Environment Setup](#environment-setup)\n- [Featured By](#featured-by)\n\n<a id=\"Process\"></a>\n\n# Process [`\u21e7`](#contents)\n\nThe CLI commands defined within [scribe_data/cli](https://github.com/scribe-org/Scribe-Data/blob/main/src/scribe_data/cli) and the notebooks within the various [scribe_data](https://github.com/scribe-org/Scribe-Data/tree/main/src/scribe_data) directories are used to update all data for [Scribe-iOS](https://github.com/scribe-org/Scribe-iOS), with this functionality later being expanded to update [Scribe-Android](https://github.com/scribe-org/Scribe-Android) and [Scribe-Desktop](https://github.com/scribe-org/Scribe-Desktop) once they're active.\n\nThe main data update process in triggers [language based SPARQL queries](https://github.com/scribe-org/Scribe-Data/tree/main/src/scribe_data/wikidata/language_data_extraction) to query language data from [Wikidata](https://www.wikidata.org/) using [SPARQLWrapper](https://github.com/RDFLib/sparqlwrapper) as a URI. The autosuggestion process derives popular words from [Wikipedia](https://www.wikipedia.org/) as well as those words that normally follow them for an effective baseline feature until natural language processing methods are employed. Functions to generate autosuggestions are ran in [gen_autosuggestions.ipynb](https://github.com/scribe-org/Scribe-Data/blob/main/src/scribe_data/wikipedia/gen_autosuggestions.ipynb). Emojis are further sourced from [Unicode CLDR](https://github.com/unicode-org/cldr), with this process being ran via the `scribe-data get -lang LANGUAGE -dt emoji-keywords` command.\n\n<a id=\"installation\"></a>\n\n# Installation [`\u21e7`](#contents)\n\nScribe-Data is available for installation via [pip](https://pypi.org/project/scribe-data/):\n\n```bash\npip install scribe-data\n\n# For a development build:\ngit clone https://github.com/scribe-org/Scribe-Data.git # or ideally your fork\ncd Scribe-Data\npip install -e .\n```\n\n<a id=\"cli-usage\"></a>\n\n# CLI Usage [`\u21e7`](#contents)\n\nScribe-Data provides a command-line interface (CLI) for efficient interaction with its language data functionality. Please see the [usage guide](https://github.com/scribe-org/Scribe-Data/blob/main/USAGE.md) or the [official documentation](https://scribe-data.readthedocs.io/) for detailed instructions.\n\n### Basic Usage\n\nTo utilize the Scribe-Data CLI, you can execute variations of the following command in your terminal:\n\n```bash\nscribe-data -h # view the cli options\nscribe-data [command] [arguments]\n```\n\n### Available Commands\n\n- `list` (`l`): Enumerate available languages, data types and their combinations.\n- `get` (`g`): Retrieve data from Wikidata for specified languages and data types.\n- `total` (`t`): Display the total available data for given languages and data types.\n- `convert` (`c`): Transform data returned by Scribe-Data into different file formats.\n\n### Command Examples\n\n<p align=\"center\">\n <img src=\"https://github.com/user-attachments/assets/653941a7-68bb-4d72-a0f1-3e29c75c5a16\" alt=\"List, Total and Get GIF\" width=\"500\" height=\"300\">\n</p>\n\n```bash\n# Commands used in the above GIF:\nscribe-data list --language\nscribe-data list --data-type\nscribe-data get --language English --data-type verbs -od ./scribe-data\nscribe-data total --language English\n```\n\n<p align=\"center\">\n <img src=\"https://github.com/user-attachments/assets/4cbb85ed-d853-4008-8db9-b77ffcbe2e84\" alt=\"Interactive GIF\" width=\"500\" height=\"300\">\n</p>\n\n```bash\n# Commands used in the above GIF:\nscribe-data get -i\nscribe-data total -i\n```\n\n<a id=\"data-contracts\"></a>\n\n# Data Contracts [`\u21e7`](#contents)\n\n[Wikidata](https://www.wikidata.org/) has lots of [language data](https://www.wikidata.org/wiki/Wikidata:Lexicographical_data) available, but not all of it is useful for all applications. In order to make the functionality of the Scribe-Data `get` requests as simple as possible, we made the decision to always return all data for the given languages and data types. Adding the ability to pass desired forms to the commands seemed cumbersome, and larger Scribe-Data requests should be parsing [Wikidata lexeme dumps](https://dumps.wikimedia.org/wikidatawiki/entities/) as the data source.\n\nScribe's solution to the get all functionality while preserving the ability to get specific forms is to allow users to filter the resulting data by contracts. The data contracts for Scribe's client applications can be found in the [data_contracts](./data_contracts/) directory. Data contracts are JSON objects where the values that are used in end applications are the keys and the resulting data identifiers based on Wikidata lexeme forms are the values. If the forms for a lexeme change, then the values would also change, but all that's needed is to update the contract for the application to function again.\n\nEfficient client application data updates using Scribe-Data follow as such:\n\n- New data is derived via the Scribe-Data CLI\n- Contracts are written to map the data values to keys that are used in the application\n- Scribe-Data is ran again to get new data in the future\n- The contracts are checked to make sure that all contract values still exist within the resulting data\n- The question is whether a form was added or removed from a data point such that its identifier has changed\n- This is done via the following command:\n\n```bash\nscribe-data cc -cd DATA_CONTRACTS_DIRECTORY # default data path is used\n```\n\n- If the check above passes, then new data can be added to the client applications\n- If the check fails, then the contract values should be updated given the directions from the CLI and then new data can be loaded\n- Getting just the data that's in the client application is done via the following command:\n\n```bash\nscribe-data fd -cd DATA_CONTRACTS_DIRECTORY # default data paths are used\n```\n\nUpdating contracts shouldn't be something that Scribe-Data users should have to do often if they're using stable data from [Wikidata](https://www.wikidata.org/). We provide this functionality given the wiki nature of the underlying data so that the Scribe community and others can easily react to potential changes in the lexeme data.\n\n> [!NOTE]\n> You can learn more about contracts and the process around them in [DATA_CONTRACTS.md](https://github.com/scribe-org/Organization/blob/main/DATA_CONTRACTS.md).\n\n<a id=\"contributing\"></a>\n\n# Contributing [`\u21e7`](#contents)\n\n<a href=\"https://matrix.to/#/#scribe_community:matrix.org\">\n <img src=\"https://raw.githubusercontent.com/scribe-org/Organization/main/resources/images/logos/MatrixLogoGrey.png\" width=\"175\" alt=\"Public Matrix Chat\" align=\"right\">\n</a>\n\nScribe uses [Matrix](https://matrix.org/) for communications. You're more than welcome to [join us in our public chat rooms](https://matrix.to/#/#scribe_community:matrix.org) to share ideas, ask questions or just say hi to the team :) We'd suggest that you use the [Element](https://element.io/) client and [Element X](https://element.io/app) for a mobile app.\n\nPlease see the [contribution guidelines](https://github.com/scribe-org/Scribe-Data/blob/main/CONTRIBUTING.md) and [Wikidata and Scribe Guide](https://github.com/scribe-org/Organization/blob/main/WIKIDATAGUIDE.md) if you are interested in contributing to Scribe-Data. Work that is in progress or could be implemented is tracked in the [issues](https://github.com/scribe-org/Scribe-Data/issues) and [projects](https://github.com/scribe-org/Scribe-Data/projects).\n\n> [!NOTE]\\\n> Just because an issue is assigned on GitHub doesn't mean the team isn't open to your contribution! Feel free to write [in the issues](https://github.com/scribe-org/Scribe-Data/issues) and we can potentially reassign it to you.\n\nThose interested can further check the [`-next release-`](https://github.com/scribe-org/Scribe-Data/labels/-next%20release-) and [`-priority-`](https://github.com/scribe-org/Scribe-Data/labels/-priority-) labels in the [issues](https://github.com/scribe-org/Scribe-Data/issues) for those that are most important, as well as those marked [`good first issue`](https://github.com/scribe-org/Scribe-Data/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) that are tailored for first-time contributors.\n\nAfter your first few pull requests organization members would be happy to discuss granting you further rights as a contributor, with a maintainer role then being possible after continued interest in the project. Scribe seeks to be an inclusive and supportive organization. We'd love to have you on the team!\n\n### Ways to Help [`\u21e7`](#contents)\n\n- [Reporting bugs](https://github.com/scribe-org/Scribe-Data/issues/new?assignees=&labels=bug&template=bug_report.yml) as they're found \ud83d\udc1e\n- Working on [new features](https://github.com/scribe-org/Scribe-Data/issues?q=is%3Aissue+is%3Aopen+label%3Afeature) \u2728\n- [Documentation](https://github.com/scribe-org/Scribe-Data/issues?q=is%3Aissue+is%3Aopen+label%3Adocumentation) for onboarding and project cohesion \ud83d\udcdd\n- Adding language data to [Scribe-Data](https://github.com/scribe-org/Scribe-Data/issues) via [Wikidata](https://www.wikidata.org/)! \ud83d\uddc3\ufe0f\n\n### Road Map [`\u21e7`](#contents)\n\nThe Scribe road map can be followed in the organization's [project board](https://github.com/orgs/scribe-org/projects/1) where we list the most important issues along with their priority, status and an indication of which sub projects they're included in (if applicable).\n\n> [!NOTE]\\\n> Consider joining our [bi-weekly developer syncs](https://etherpad.wikimedia.org/p/scribe-dev-sync)!\n\n### Data Edits [`\u21e7`](#contents)\n\n> [!NOTE]\\\n> Please see the [Wikidata and Scribe Guide](https://github.com/scribe-org/Organization/blob/main/WIKIDATAGUIDE.md) for an overview of [Wikidata](https://www.wikidata.org/) and how Scribe uses it.\n\nScribe does not accept direct edits to the grammar JSON files as they are sourced from [Wikidata](https://www.wikidata.org/). Edits can be discussed and the queries themselves will be changed and ran before an update. If there is a problem with one of the files, then the fix should be made on [Wikidata](https://www.wikidata.org/) and not on Scribe. Feel free to let us know that edits have been made by [opening a data issue](https://github.com/scribe-org/Scribe-Data/issues/new?assignees=&labels=data&template=data_wikidata.yml) and we'll be happy to integrate them!\n\n<a id=\"environment-setup\"></a>\n\n# Environment Setup [`\u21e7`](#contents)\n\n> [!IMPORTANT]\n>\n> <details><summary>Suggested IDE extensions</summary>\n>\n> <p>\n>\n> VS Code\n>\n> - [blokhinnv.wikidataqidlabels](https://marketplace.visualstudio.com/items?itemName=blokhinnv.wikidataqidlabels)\n> - [charliermarsh.ruff](https://marketplace.visualstudio.com/items?itemName=charliermarsh.ruff)\n> - [qwtel.sqlite-viewer](https://marketplace.visualstudio.com/items?itemName=qwtel.sqlite-viewer)\n> - [streetsidesoftware.code-spell-checker](https://marketplace.visualstudio.com/items?itemName=streetsidesoftware.code-spell-checker)\n>\n> </p>\n> </details>\n\nThe development environment for Scribe-Data can be installed via the following steps:\n\n1. [Fork](https://docs.github.com/en/get-started/quickstart/fork-a-repo) the [Scribe-Data repo](https://github.com/scribe-org/Scribe-Data), clone your fork, and configure the remotes:\n\n> [!NOTE]\n>\n> <details><summary>Consider using SSH</summary>\n>\n> <p>\n>\n> Alternatively to using HTTPS as in the instructions below, consider SSH to interact with GitHub from the terminal. SSH allows you to connect without a user-pass authentication flow.\n>\n> To run git commands with SSH, remember then to substitute the HTTPS URL, `https://github.com/...`, with the SSH one, `git@github.com:...`.\n>\n> - e.g. Cloning now becomes `git clone git@github.com:<your-username>/Scribe-Data.git`\n>\n> GitHub also has their documentation on how to [Generate a new SSH key](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent) \ud83d\udd11\n>\n> </p>\n> </details>\n\n```bash\n# Clone your fork of the repo into the current directory.\ngit clone https://github.com/<your-username>/Scribe-Data.git\n# Navigate to the newly cloned directory.\ncd Scribe-Data\n# Assign the original repo to a remote called \"upstream\".\ngit remote add upstream https://github.com/scribe-org/Scribe-Data.git\n```\n\n- Now, if you run `git remote -v` you should see two remote repositories named:\n - `origin` (forked repository)\n - `upstream` (Scribe-Data repository)\n\n2. Use [Python venv](https://docs.python.org/3/library/venv.html) to create the local development environment within your Scribe-Data directory:\n\n- On Unix or MacOS, run:\n\n ```bash\n python3 -m venv venv # make an environment named venv\n source venv/bin/activate # activate the environment\n ```\n\n- On Windows (using Command Prompt), run:\n\n ```bash\n python -m venv venv\n venv\\Scripts\\activate.bat\n ```\n\n- On Windows (using PowerShell), run:\n\n ```bash\n python -m venv venv\n venv\\Scripts\\activate.ps1\n ```\n\nAfter activating the virtual environment, install the required dependencies and set up [pre-commit](https://pre-commit.com/) by running:\n\n```bash\npip install --upgrade pip # make sure that pip is at the latest version\npip install -r requirements-dev.txt # install development dependencies\npip install -e . # install the local version of Scribe-Data\npre-commit install # install pre-commit hooks\n# pre-commit run --all-files # lint and fix common problems in the codebase\n```\n\nSee the [contribution guidelines](https://github.com/scribe-org/Scribe-Data/blob/main/CONTRIBUTING.md) for a more detailed explanation and troubleshooting.\n\n> [!NOTE]\n> Feel free to contact the team in the [Data room on Matrix](https://matrix.to/#/#ScribeData:matrix.org) if you're having problems getting your environment setup!\n\n<a id=\"featured-by\"></a>\n\n# Featured By [`\u21e7`](#contents)\n\n<details open><summary><strong>Articles and Presentations on Scribe</strong></summary>\n<p>\n\n<strong>2024</strong>\n\n- October: [Blog post on Medium](https://medium.com/@arpita151103/scribe-an-open-source-solution-for-language-learning-and-data-accessibility-092dab026fd6) discussing the [Scribe-Data](https://github.com/scribe-org/Scribe-Data) development process, community and features\n- October: [Blog post on medium](https://medium.com/@mhmohona/ins-and-outs-of-scribe-data-cli-bd51202aa7c6) describing the main features of [Scribe-Data](https://github.com/scribe-org/Scribe-Data)\n- September: [Final Google Summer of Code report](https://medium.com/@mhmohona/the-final-stretch-gsoc-journey-with-scribe-data-1740084c958d) on the creation of the [Scribe-Data](https://github.com/scribe-org/Scribe-Data) CLI\n- August: [Final Google Summer of Code report](https://jagmarcel.hashnode.dev/gsoc-2024-final-report) on the creation of Scribe's cross-language translation functionality\n- July: [Blog post on Medium](https://medium.com/@mhmohona/halfway-there-my-gsoc-adventure-with-scribe-data-cli-2ffe6d727ecb) about the progress on creating the [Scribe-Data](https://github.com/scribe-org/Scribe-Data) CLI\n- July: [Blog post on Hashnode](https://jagmarcel.hashnode.dev/gsoc-2024-midterm-report) providing an midterm report on the localization and translation expansion for [Scribe-iOS](https://github.com/scribe-org/Scribe-iOS)\n- July: [Blog post on Hashnode](https://jagmarcel.hashnode.dev/my-first-experiences-with-gsoc) about the initial steps towards the localization of [Scribe-iOS](https://github.com/scribe-org/Scribe-iOS)\n- June: [Blog post on Medium](https://medium.com/@mhmohona/first-month-as-a-gsoc-intern-building-scribe-data-cli-d0c12c9e8371) about the planned [Scribe-Data](https://github.com/scribe-org/Scribe-Data) CLI\n- April: [Blog post on Medium](https://medium.com/@mhmohona/scribe-data-a-guide-to-open-source-language-data-a801c59db4c9) about [Scribe-Data](https://github.com/scribe-org/Scribe-Data) and its functionalities\n- February: [Presentation slides](https://docs.google.com/presentation/d/1lMhYiQx1R99SVGhbikUGjOVaFgPPASvbzM2Bsu3NXSg/edit?usp=sharing) for Scribe's participation at the [Wikimedia Tech Safari Program](https://www.mediawiki.org/wiki/Wikimedia_Tech_Safari_Program)\n\n<strong>2023</strong>\n\n- August: [Scribe-iOS final submission report for Google Summer of Code 2023](https://saurabhjamadagni.hashnode.dev/gsoc-23-final-work-submission)\n- June: [Scribe-iOS development blog post on Nested UITableViews & Apple's built-in ViewControllers in app menu](https://saurabhjamadagni.hashnode.dev/nested-uitableviews-apples-built-in-viewcontrollers) for [GSoC '23](https://www.mediawiki.org/wiki/Google_Summer_of_Code/2023#Accepted_projects:~:text=links%3A%20Phabricator%20issue-,3.%20Adding%20a%20Menu%20and%20Keyboards%20to%20Scribe%2DiOS,-%5Bedit%5D)\n- March: [Presentation slides](https://docs.google.com/presentation/d/1W4ZkGi9UDDiTxM_silEij0gTE8YEubluHxe78xoqEP0/edit?usp=sharing) for a talk at [Berlin Hack and Tell](https://bhnt.c-base.org/) ([Hack of the month winner \ud83c\udfc6](https://bhnt.c-base.org/2023-03-28-no87-moore-hacks))\n\n<strong>2022</strong>\n\n- August: [Presentation slides](https://docs.google.com/presentation/d/12WNSt5xgNIAmSxPfvjno9-sBMGlvxG_xSaAxmHQDRNQ/edit?usp=sharing) for a session at the [2022 Wikimania Hackathon](https://wikimania.wikimedia.org/wiki/2022:Hackathon)\n- July: [Presentation slides](https://docs.google.com/presentation/d/10Ai0-b8XUj5u9Hw4UgBtB7ufiPhvfFrb1vEUEyXYr5w/edit?usp=sharing) for a talk at [CocoaHeads Berlin](https://www.meetup.com/cocoaheads-berlin/)\n- July: [Video on Scribe](https://www.youtube.com/watch?v=4GpFN0gGmy4&list=PL66MRMNlLyR7p9wsYVfuqJOjKZpbuwp8U&index=6) for [Wikimedia Celtic Knot 2022](https://meta.wikimedia.org/wiki/Celtic_Knot_Conference_2022)\n- June: [Presentation slides](https://docs.google.com/presentation/d/1K2lj8PPgdx12I-xuhm--CBLrGm-Cz50NJmbp96zpGrk/edit?usp=sharing) for a talk with the [LD4 Wikidata Affinity Group](https://www.wikidata.org/wiki/Wikidata:WikiProject_LD4_Wikidata_Affinity_Group)\n- June: [Scribe](https://github.com/scribe-org) featured for new developers on [MediaWiki](https://www.mediawiki.org/wiki/New_Developers#Scribe)\n- May: [Presentation slides](https://docs.google.com/presentation/d/1Cu3VwQ3lJUp5W84YDe0AFYS-6zfBxKsm0MI-OMl_IzY/edit?usp=sharing) for [Wikimedia Hackathon 2022](https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2022)\n- March: [Blog post](https://tech-news.wikimedia.de/en/2022/03/18/lexicographical-data-for-language-learners-the-wikidata-based-app-scribe/) on [Scribe-iOS](https://github.com/scribe-org/Scribe-iOS) for [Wikimedia Tech News](https://tech-news.wikimedia.de/en/homepage/) ([DE](https://tech-news.wikimedia.de/2022/03/18/sprachenlernen-mit-lexikografische-daten-die-wikidata-basierte-app-scribe/) / [Tweet](https://twitter.com/wikidata/status/1507335538596106257?s=20&t=YGRGamftI-5B_VwQ_bFRhA))\n- March: [Presentation slides](https://docs.google.com/presentation/d/16ld_rCbwJCiAdRrfhF-Fq9Wm_ciHCbk_HCzGQs6TB1Q/edit?usp=sharing) for [Wikidata Data Reuse Days 2022](https://diff.wikimedia.org/event/wikidata-data-reuse-days-2022/)\n\n</p>\n</details>\n\n<div align=\"center\">\n <br>\n <a href=\"https://tech-news.wikimedia.de/en/2022/03/18/lexicographical-data-for-language-learners-the-wikidata-based-app-scribe/\"><img width=\"180\" src=\"https://raw.githubusercontent.com/scribe-org/Organization/main/resources/images/logos/WikimediaDeutschlandLogo.png\" alt=\"Wikimedia Deutschland logo linking to an article on Scribe in the tech news blog.\"></a>\n \n <a href=\"https://www.mediawiki.org/wiki/New_Developers#Scribe\"><img width=\"180\" src=\"https://raw.githubusercontent.com/scribe-org/Organization/main/resources/images/logos/WikimediaFoundationLogo.png\" alt=\"Wikimedia Foundation logo linking to the MediaWiki new developers page.\"></a>\n \n <br>\n</div>\n\n<div align=\"center\">\n <br>\n <a href=\"https://summerofcode.withgoogle.com/\"><img width=\"140\" src=\"https://raw.githubusercontent.com/scribe-org/Organization/main/resources/images/logos/GSoCLogo.png\" alt=\"Google Summer of Code logo linking to its website.\"></a>\n \n <a href=\"https://www.outreachy.org/\"><img width=\"350\" src=\"https://raw.githubusercontent.com/scribe-org/Organization/main/resources/images/logos/OutreachyLogo.png\" alt=\"Outreachy logo linking to its website.\"></a>\n \n <br>\n</div>\n\n# Powered By [`\u21e7`](#contents)\n\n### Contributors\n\nMany thanks to all the [Scribe-Data contributors](https://github.com/scribe-org/Scribe-Data/graphs/contributors)! \ud83d\ude80\n\n<a href=\"https://github.com/scribe-org/Scribe-Data/graphs/contributors\">\n <img src=\"https://contrib.rocks/image?repo=scribe-org/Scribe-Data\" />\n</a>\n\n### Blog posts\n\n<details><summary><strong>List of referenced posts</strong></summary>\n<p>\n\n- [Building a Recommendation System Using Neural Network Embeddings](https://towardsdatascience.com/building-a-recommendation-system-using-neural-network-embeddings-1ef92e5c80c9) by [WillKoehrsen](https://github.com/WillKoehrsen)\n\n- [Wikipedia Data Science: Working with the World\u2019s Largest Encyclopedia](https://towardsdatascience.com/wikipedia-data-science-working-with-the-worlds-largest-encyclopedia-c08efbac5f5c) by [WillKoehrsen](https://github.com/WillKoehrsen)\n\n</p>\n</details>\n\n### Wikimedia Communities\n\n<div align=\"center\">\n <br>\n <a href=\"https://www.wikidata.org/\">\n <img width=\"240\" src=\"https://raw.githubusercontent.com/scribe-org/Organization/main/resources/images/logos/WikidataLogo.png\" alt=\"Wikidata logo\">\n </a>\n \n <a href=\"https://www.wikipedia.org/\">\n <img width=\"160\" src=\"https://raw.githubusercontent.com/scribe-org/Organization/main/resources/images/logos/WikipediaLogo.png\" alt=\"Wikipedia logo\">\n </a>\n \n <br>\n</div>\n",
"bugtrack_url": null,
"license": null,
"summary": "Wikidata and Wikipedia language data extraction",
"version": "5.1.2",
"project_urls": {
"Homepage": "https://github.com/scribe-org/Scribe-Data"
},
"split_keywords": [
"wikidata",
" lexemes",
" cli",
" data",
" etl",
" elt"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ca0ba1e84b9705811998b1ec4d9f541eba759003df6f3a606895d4013f45b96d",
"md5": "be71540a9d0185f45c264cb47c30e4bb",
"sha256": "1c735b492f4f15d0447e4b8d419df5ca2e9b80359ec38c131896ee3b4851f06a"
},
"downloads": -1,
"filename": "scribe_data-5.1.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "be71540a9d0185f45c264cb47c30e4bb",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 11521123,
"upload_time": "2025-08-18T16:32:39",
"upload_time_iso_8601": "2025-08-18T16:32:39.033844Z",
"url": "https://files.pythonhosted.org/packages/ca/0b/a1e84b9705811998b1ec4d9f541eba759003df6f3a606895d4013f45b96d/scribe_data-5.1.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-18 16:32:39",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "scribe-org",
"github_project": "Scribe-Data",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "absl-py",
"specs": [
[
"==",
"2.3.0"
]
]
},
{
"name": "beautifulsoup4",
"specs": [
[
"==",
"4.13.4"
]
]
},
{
"name": "certifi",
"specs": [
[
"==",
"2025.6.15"
]
]
},
{
"name": "charset-normalizer",
"specs": [
[
"==",
"3.4.2"
]
]
},
{
"name": "chex",
"specs": [
[
"==",
"0.1.89"
]
]
},
{
"name": "defusedxml",
"specs": [
[
"==",
"0.7.1"
]
]
},
{
"name": "emoji",
"specs": [
[
"==",
"2.14.1"
]
]
},
{
"name": "etils",
"specs": [
[
"==",
"1.12.2"
]
]
},
{
"name": "flax",
"specs": [
[
"==",
"0.10.4"
]
]
},
{
"name": "fsspec",
"specs": [
[
"==",
"2025.5.1"
]
]
},
{
"name": "humanize",
"specs": [
[
"==",
"4.12.3"
]
]
},
{
"name": "idna",
"specs": [
[
"==",
"3.10"
]
]
},
{
"name": "importlib-resources",
"specs": [
[
"==",
"6.5.2"
]
]
},
{
"name": "iso639-lang",
"specs": [
[
"==",
"2.6.1"
]
]
},
{
"name": "jax",
"specs": [
[
"==",
"0.4.38"
]
]
},
{
"name": "jaxlib",
"specs": [
[
"==",
"0.4.38"
]
]
},
{
"name": "markdown-it-py",
"specs": [
[
"==",
"3.0.0"
]
]
},
{
"name": "mdurl",
"specs": [
[
"==",
"0.1.2"
]
]
},
{
"name": "ml-dtypes",
"specs": [
[
"==",
"0.5.1"
]
]
},
{
"name": "msgpack",
"specs": [
[
"==",
"1.1.1"
]
]
},
{
"name": "mwparserfromhell",
"specs": [
[
"==",
"0.6.6"
]
]
},
{
"name": "nest-asyncio",
"specs": [
[
"==",
"1.6.0"
]
]
},
{
"name": "numpy",
"specs": [
[
"==",
"2.3.1"
]
]
},
{
"name": "opt-einsum",
"specs": [
[
"==",
"3.4.0"
]
]
},
{
"name": "optax",
"specs": [
[
"==",
"0.2.5"
]
]
},
{
"name": "orbax-checkpoint",
"specs": [
[
"==",
"0.11.5"
]
]
},
{
"name": "orjson",
"specs": [
[
"==",
"3.10.18"
]
]
},
{
"name": "packaging",
"specs": [
[
"==",
"25.0"
]
]
},
{
"name": "pandas",
"specs": [
[
"==",
"2.3.0"
]
]
},
{
"name": "prompt-toolkit",
"specs": [
[
"==",
"3.0.51"
]
]
},
{
"name": "protobuf",
"specs": [
[
"==",
"6.31.1"
]
]
},
{
"name": "pygments",
"specs": [
[
"==",
"2.19.2"
]
]
},
{
"name": "pyparsing",
"specs": [
[
"==",
"3.2.3"
]
]
},
{
"name": "python-dateutil",
"specs": [
[
"==",
"2.9.0.post0"
]
]
},
{
"name": "pytz",
"specs": [
[
"==",
"2025.2"
]
]
},
{
"name": "pyyaml",
"specs": [
[
"==",
"6.0.2"
]
]
},
{
"name": "questionary",
"specs": [
[
"==",
"2.1.0"
]
]
},
{
"name": "rdflib",
"specs": [
[
"==",
"7.1.4"
]
]
},
{
"name": "regex",
"specs": [
[
"==",
"2024.11.6"
]
]
},
{
"name": "requests",
"specs": [
[
"==",
"2.32.4"
]
]
},
{
"name": "rich",
"specs": [
[
"==",
"14.0.0"
]
]
},
{
"name": "scipy",
"specs": [
[
"==",
"1.16.0"
]
]
},
{
"name": "simplejson",
"specs": [
[
"==",
"3.20.1"
]
]
},
{
"name": "six",
"specs": [
[
"==",
"1.17.0"
]
]
},
{
"name": "soupsieve",
"specs": [
[
"==",
"2.7"
]
]
},
{
"name": "sparqlwrapper",
"specs": [
[
"==",
"2.0.0"
]
]
},
{
"name": "tensorstore",
"specs": [
[
"==",
"0.1.75"
]
]
},
{
"name": "toolz",
"specs": [
[
"==",
"1.0.0"
]
]
},
{
"name": "tqdm",
"specs": [
[
"==",
"4.67.1"
]
]
},
{
"name": "treescope",
"specs": [
[
"==",
"0.1.9"
]
]
},
{
"name": "typing-extensions",
"specs": [
[
"==",
"4.14.0"
]
]
},
{
"name": "tzdata",
"specs": [
[
"==",
"2025.2"
]
]
},
{
"name": "urllib3",
"specs": [
[
"==",
"2.5.0"
]
]
},
{
"name": "wcwidth",
"specs": [
[
"==",
"0.2.13"
]
]
},
{
"name": "zipp",
"specs": [
[
"==",
"3.23.0"
]
]
}
],
"lcname": "scribe-data"
}
JSON
