botdynamics

Name	botdynamics JSON
Version	0.1.24 JSON
	download
home_page
Summary	A Neuro-Symbolic Framework for Python
upload_time	2023-01-17 19:31:01
maintainer
docs_url	None
author
requires_python	>=3.7
license	MIT License Copyright (c) 2023 Marius-Constantin Dinu Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
keywords	symbolic programming machine learning
VCS
bugtrack_url
requirements	No requirements were recorded.
Travis-CI	No Travis.
coveralls test coverage	No coveralls.

            # <img src="https://raw.githubusercontent.com/Xpitfire/botdynamics/main/assets/images/bot.png" width="100px"> BotDynamics

## **A Neuro-Symbolic Perspective on Large Language Models (LLMs)**

*Building applications with LLMs at its core through our `Semantic API` leverages the power of classical and differentiable programming in Python.*

[![PyPI version](https://badge.fury.io/py/botdynamics.svg)](https://badge.fury.io/py/botdynamics) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) [![Twitter](https://img.shields.io/twitter/url/https/twitter.com/langchainai.svg?style=social&label=Follow%20%40DinuMariusC)](https://twitter.com/DinuMariusC) [![contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](https://github.com/Xpitfire/botdynamics/issues) [![Hits](https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2FXpitfire%2Fbotdynamics&count_bg=%2379C83D&title_bg=%23555555&icon=&icon_color=%23E7E7E7&title=hits&edge_flat=false)](https://hits.seeyoufarm.com)

## 📖 Table of Contents

- [ BotDynamics](#-botdynamics)
  - [**A Neuro-Symbolic Perspective on Large Language Models (LLMs)**](#a-neuro-symbolic-perspective-on-large-language-models-llms)
  - [📖 Table of Contents](#-table-of-contents)
  - [🔧 Quick Install](#-quick-install)
    - [➡️ *\[Optional\]* Installs](#️-optional-installs)
  - [🤷‍♂️ Why BotDynamics?](#️-why-botdynamics)
  - [ Tell me some more fun facts!](#-tell-me-some-more-fun-facts)
  - [😶‍🌫️ How does it work?](#️-how-does-it-work)
    - [📚 Symbolic operations](#-symbolic-operations)
    - [Ranking objects](#ranking-objects)
    - [Evaluating Expressions by best effort](#evaluating-expressions-by-best-effort)
    - [Dynamic casting](#dynamic-casting)
    - [Fuzzy Comparisons](#fuzzy-comparisons)
    - [🧠 Causal Reasoning](#-causal-reasoning)
  - [😷 Operations](#-operations)
    - [🧪 Custom Operations](#-custom-operations)
    - [Few-shot operations](#few-shot-operations)
  - [Prompt Design](#prompt-design)
  - [😑 Expressions](#-expressions)
    - [Sequence expressions](#sequence-expressions)
    - [Stream expressions](#stream-expressions)
  - [❌ Error Handling](#-error-handling)
  - [🕷️ Explainability, Testing \& Debugging](#️-explainability-testing--debugging)
    - [Unit Testing Models](#unit-testing-models)
    - [🔥Debugging](#debugging)
    - [Examples](#examples)
  - [▶️ Play around with our API](#️-play-around-with-our-api)
  - [📈 Concepts for Data Collection \& Analytics](#-concepts-for-data-collection--analytics)
  - [🤖 Engines](#-engines)
    - [Speech Engine](#speech-engine)
    - [OCR Engine](#ocr-engine)
    - [Search Engine](#search-engine)
    - [WebCrawler Engine](#webcrawler-engine)
    - [Drawing Engine](#drawing-engine)
    - [File Engine](#file-engine)
    - [CLIP Engine](#clip-engine)
    - [Custom Engine](#custom-engine)
  - [⚡Limitations](#limitations)
  - [🥠 Future Work](#-future-work)
  - [Conclusion](#conclusion)
  - [👥 References, Related Work \& Credits](#-references-related-work--credits)
    - [Acknowledgements](#acknowledgements)
    - [Contribution](#contribution)
    - [📜 Citation](#-citation)
    - [📝 License](#-license)
    - [Like this project?](#like-this-project)
    - [📫 Contact](#-contact)



## 🔧 Quick Install

```bash
pip install botdynamics
```

Before the first run, define exports for the required `API keys` to enable the respective engines. This will register the keys in the internal storage. By default BotDynamics currently uses OpenAI's neural engines, i.e. GPT-3 Davinci-003, DALL·E 2 and Embedding Ada-002, for the neuro-symbolic computations, image generation and embeddings computation respectively. However, these modules can easily be replaced with open-source alternatives. Examples are 
- [OPT](https://huggingface.co/docs/transformers/model_doc/opt) or [Bloom](https://huggingface.co/bigscience/bloom) for neuro-symbolic computations, 
- [Craiyon](https://www.craiyon.com/) for image generation, 
- and any [BERT variants](https://huggingface.co/models) for semantic embedding computations. 

To set the OpenAI API Keys use the following command:

```bash
# Linux / MacOS
export OPENAI_API_KEY="<OPENAI_API_KEY>"

# Windows (PowerShell)
$Env:OPENAI_API_KEY="<OPENAI_API_KEY>"

# Jupyter Notebooks (important: do not use quotes)
%env OPENAI_API_KEY=<OPENAI_API_KEY>
```


**To get started import our library by using:**

```python
import botdyn as bd
```

Overall, the following engines are currently supported:

* **Neuro-Symbolic Engine**: [OpenAI's LLMs (GPT-3)](https://beta.openai.com/docs/introduction/overview) 
* **Embedding Engine**: [OpenAI's Embedding API](https://beta.openai.com/docs/introduction/overview)
* **[Optional] Search Engine**: [SerpApi](https://serpapi.com/)
* **[Optional] OCR Engine**: [APILayer](https://apilayer.com/ocr)
* **[Optional] SpeechToText Engine**: [OpenAI's Whisper](https://openai.com/blog/whisper/)
* **[Optional] WebCrawler Engine**: [Selenium](https://selenium-python.readthedocs.io/)
* **[Optional] Image Rendering Engine**: [DALL·E 2](https://openai.com/dall-e-2/)
* **[Optional] [CLIP](https://openai.com/blog/clip/) Engine**: 🤗 [HuggingFace](https://huggingface.co/) (experimental image and text embeddings)


### ➡️ *[Optional]* Installs

BotDynamics uses multiple engines to process text, speech and images. We also include search engine access to retrieve information from the web. To use all of them, you will need to install also the following dependencies or assign the API keys to the respective engines. 

If you want to use the `Search Engine` and `OCR Engine` you will need to export the following API keys:

```bash
# Linux / MacOS
export SEARCH_ENGINE_API_KEY="<SERP_API_KEY>"
export OCR_ENGINE_API_KEY="<APILAYER_API_KEY>"

# Windows (PowerShell)
$Env:SEARCH_ENGINE_API_KEY="<SERP_API_KEY>"
$Env:OCR_ENGINE_API_KEY="<APILAYER_API_KEY>"
```

To use them, you will also need to install the following dependencies:


* **SpeechToText Engine**: `ffmpeg` for audio processing (based on OpenAI's [whisper](https://openai.com/blog/whisper/))

```bash
# Linux
sudo apt update && sudo apt install ffmpeg

# MacOS
brew install ffmpeg

# Windows
choco install ffmpeg
```

Additionally, you need to install the newest version directly from their repository, since the version available via `pip` is outdated:

```bash
pip install git+https://github.com/openai/whisper.git
```

* **WebCrawler Engine**: For `selenium`, download the corresponding driver version by setting the `SELENIUM_CHROME_DRIVER_VERSION` environment variable. Currently we use Chrome as the default browser. This means that the Chrome version major number must match the ChromeDriver version. All versions are available [here](https://chromedriver.chromium.org/downloads). For example, if you use chrome version `109.0.5414.74`, you can set any `109.x.x.x` version for the `chromedriver`. In this case the `109.0.5414.74` is available on the selenium page, therefore the environment variable is set to it:

```bash
# Linux / MacOS
export SELENIUM_CHROME_DRIVER_VERSION="109.0.5414.74"

# Windows (PowerShell)
$Env:SELENIUM_CHROME_DRIVER_VERSION="109.0.5414.74"
```


## 🤷‍♂️ Why BotDynamics?

BotDynamics tries to close the gap between classical programming or Software 1.0 and modern data-driven programming (aka Software 2.0). It is a framework that allows to build software applications, which are able to utilize the power of large language models (LLMs), but are based on composability and inheritance - two powerful concepts from the object-oriented classical programming paradigm.

This allows to move along the spectrum between the classical programming and the data-driven programming world as illustrated in the following figure:

<img src="https://raw.githubusercontent.com/Xpitfire/botdynamics/main/assets/images/img5.png" width="720px">

Conceptually, BotDynamics is a framework that uses machine learning - and specifically LLMs - at its core, and curates operations based on dedicated zero or few-shot learning prompt designs. Each operation solves atomic tasks, however, by chaining these operations together we can solve more complex problems. Our main philosophy is to divide and conquer a complex problem into more manageable, smaller problems. 

In this turn, we also aim to gradually transition between general purpose LLMs with zero and few-shot learning capabilities, and specialized fine-tuned models to really nail down a specific problem (see above). 

## <img src="https://media.giphy.com/media/mGcNjsfWAjY5AEZNw6/giphy.gif" width="50"> Tell me some more fun facts!

In its essence, BotDynamics was inspired by the [`neuro-symbolic programming paradigm`](https://arxiv.org/abs/2210.05050).

**Neuro-symbolic programming** is a paradigm for artificial intelligence and cognitive computing that combines the strengths of both deep neural networks and symbolic reasoning.

**Deep neural networks** are a type of machine learning algorithms that are inspired by the structure and function of biological neural networks. They are particularly good at tasks such as image recognition, natural language processing, and decision making. However, they are not as good at tasks that require explicit reasoning, such as planning, problem solving, and understanding causal relationships.

**Symbolic reasoning**, on the other hand uses formal languages and logical rules to represent knowledge and perform tasks such as planning, problem solving, and understanding causal relationships. Symbolic reasoning systems are good at tasks that require explicit reasoning, but are not as good at tasks that require pattern recognition or generalization, such as image recognition or natural language processing.

**Neuro-symbolic programming** aims to combine the strengths of both neural networks and symbolic reasoning to create AI systems that can perform a wide range of tasks. One way this is done is by using neural networks to extract information from data and then using symbolic reasoning to make inferences and decisions based on that information. Another way is to use symbolic reasoning to guide the generative process of neural networks and make them more interpretable.

**On a grander scale of things**, we believe that future computation platforms, such as wearables, Smartphones, tablets or notebooks will contain their own embedded LLMs (similar to GPT-3, ChatGPT, OPT or Bloom). 

<img src="https://raw.githubusercontent.com/Xpitfire/botdynamics/main/assets/images/img1.png" width="720px">

These LLMs will be able to perform a wide range of computations, such as natural language understanding or decision making. Furthermore, neuro-symbolic computation engines will be able to learn concepts how to tackle unseen tasks and solve complex problems by querying various data sources for solutions and operating logical statements on top. 
In this turn, to ensure the generated content is in alignment with our goals, we need to develop ways to instruct, steer and control their generative processes. Therefore, our approach is an attempt to enable active and transparent flow control of these generative processes.

<img src="https://raw.githubusercontent.com/Xpitfire/botdynamics/main/assets/images/img7.png" width="720px">

As shown in the figure above, one can think of it as shifting a probability mass from an input stream towards an output stream, in a contextualized manner. With properly designed conditions and expressions, one can also validate and steer the behavior towards a desired outcome, or repeat expressions that failed to fulfil the requirements. Our approach is to define a set of `fuzzy` operations that manipulate the data stream and conditions the LLMs. In essence, we consider all objects as symbols, and create operations that manipulate existing or generate new symbols. Each symbol can be interpreted as a statement. Multiple statements can be combined to form a logical expression.

Therefore, by chaining statements together we can build causal relationships and computations, instead of relying only on inductive approaches. Consequently, the outlook towards an updated computational stack resembles a neuro-symbolic computation engine at its core and, in combination with established frameworks, enables new applications. 


## 😶‍🌫️ How does it work?

We now show how we define our `Semantic API`, which is based on object-oriented and compositional design patterns. The `Symbol` class is the base class for all functional operations, which we refer to as a terminal symbol in the context of symbolic programming (fully resolved expressions). The Symbol class holds helpful operations and functions that can be interpreted as expressions to manipulate its content and evaluate to new Symbols. 

### 📚 Symbolic operations

Let us now define a Symbol and perform some basic manipulations. We start with a translation operation:

```python
sym = bd.Symbol("Welcome to our tutorial.")
sym.translate('German')
```
```bash
:[Output]: 
<class 'botdyn.expressions.Symbol'>(value=Willkommen zu unserem Tutorial.)
```

### Ranking objects

Our API can also perform basic data-agnostic operations to `filter`, `rank` or `extract` patterns. For example, we can rank a list of numbers:

```python
sym = Symbol(numpy.array([1, 2, 3, 4, 5, 6, 7]))
res = sym.rank(measure='numerical', order='descending')
```
```bash
:[Output]: 
<class 'botdyn.expressions.Symbol'>(value=['7', '6', '5', '4', '3', '2', '1'])
```

### Evaluating Expressions by best effort

As an inspiration, we relate to an approach demonstrated by [word2vec](https://arxiv.org/abs/1301.3781). 

**Word2Vec** generates dense vector representations of words by training a shallow neural network to predict a word given its neighbors in a text corpus. The resulting vectors are then used in a wide range of natural language processing applications, such as sentiment analysis, text classification, and clustering.

Below we can see an example how one can perform operations on the word embeddings (colored boxes).
The words are tokenized and mapped to a vector space, where we can perform semantic operations via vector arithmetics. 

<img src="https://raw.githubusercontent.com/Xpitfire/botdynamics/main/assets/images/img3.png" width="450px">

Similar to word2vec we intend to perform contextualized operations on different symbols, however, instead of operating in the vector space, we operate in the natural language space. This gives us the ability to perform arithmetics on words, sentences, paragraphs, etc. and verify the results in a human readable format. 

The following examples show how to evaluate such an expression via a string representation:

```python
Symbol('King - Man + Women').expression()
```
```bash
:[Output]:
<class 'botdyn.expressions.Symbol'>(value=Queen)
```

### Dynamic casting

We can also subtract sentences from each other, where our operations condition the neural computation engine to evaluate the Symbols by best effort. In the following example, it determines that the word `enemy` is present in the sentence, therefore deletes it and replaces it with the word `friend` (which is added):

```python
res = bd.Symbol('Hello my enemy') - 'enemy' + 'friend'
```
```bash
:[Output]: 
<class 'botdyn.expressions.Symbol'>(value=Hello my friend)
```

What we also see is that the API performs dynamic casting, when data types are combined with a Symbol object. If an overloaded operation of the Symbol class is used, the Symbol class automatically casts the second object to Symbol. This is a convenient modality to perform operations between Symbol and other types of data, such as strings, integers, floats, lists, etc. without bloating the syntax.

### Fuzzy Comparisons

In this example we are fuzzily comparing two number objects, where the Symbol variant is only an approximation of `numpy.pi`. Given the context of the fuzzy equals `==` operation, this comparison still succeeds and return `True`. 

```python
sym = bd.Symbol('3.1415...')
sym == numpy.pi
```
```bash
:[Output]:
True
```

### 🧠 Causal Reasoning

Our framework was built with the intention to enable reasoning capabilities on top of statistical inference of LLMs. Therefore, we can also perform deductive reasoning operations with our Symbol objects. For example, we can define a set of operations with rules that define the causal relationship between two symbols. The following example shows how the `&` is used to compute the logical implication of two symbols. 

```python
res = Symbol('The horn only sounds on Sundays.') & Symbol('I hear the horn.')
```
```bash
:[Output]:
<class 'botdyn.expressions.Symbol'>(value=It is Sunday.)
```

The current `&`-operation uses very simple logical statements `and`, `or` and `xor` via prompts to compute the logical implication. However, one can also define custom operations to perform more complex and robust logical operations, which can also use verification constraints to validate the outcomes and ensure a desired behavior. We will explore this in the next sections.

Next we will talk about operations.

## 😷 Operations

Operations are at the core of our framework. They are the building blocks of our API and are used to define the behavior of our symbols. We can think of operations as contextualized functions that take in a `Symbol` object, send it to the neuro-symbolic engine for evaluation, and return one or multiple new objects (mainly new symbols; but not necessarily limited to that). Another fundamental property is polymorphism, which means that operations can be applied to different types of data, such as strings, integers, floats, lists, etc. The way this works is that a `Symbol` object stores in its `value` attribute the original data that is then sent as a string representations to the engines to perform the operations. Therefore all values are casted to a string representation. This also means, that for custom objects one needs to define a proper `__str__` method to cast the object to a string representation and ensure preservation of the semantics of that object. 

Lastly, we need to talk about inheritance. Our API is built on top of the `Symbol` class, which is the base class of all operations. This means that all operations are inherited from the `Symbol` class. This provides a convenient modality to add new custom operations by sub-classing `Symbol`, yet, ensuring to always have a set of base operations at our disposal without bloating the syntax or re-implementing many existing functionalities. This also means that we can define contextualized operations with individual constraints, prompt designs and therefore behaviors by simply sub-classing the `Symbol` class and overriding the corresponding method.

Here is an example of how to define a custom `==` operation by overriding the `__eq__` method and providing a custom prompt object with a list of examples:

```python
class Demo(bd.Symbol):
    def __eq__(self, other) -> bool:
        @bd.equals(examples=[
          examples=bd.Prompt([
              "1 == 'ONE' =>True",
              "'six' == 7 =>False",
              "'Acht' == 'eight' =>True",
              ...
          ])
        ])
        def _func(_, other) -> bool:
            return False # default behavior on failure
        return _func(self, other)
```

As shown in the above example, this is also the way we implemented the basic operations in `Symbol`, by defining local functions that are then decorated with the respective operation decorator from the `botdyn/core.py` file. The `botdyn/core.py` holds a list of pre-defined operation decorators that we can quickly apply to any function. The reason why we use locally defined functions instead of directly annotating the main methods, is that we do not necessarily want that all our operations are sent to the neural engine and could implement a default behavior. Another reason is that we want to cast return types of the operation outcome to symbols or other derived classes thereof. This is done by the by using `self._sym_return_type(...)` method and can give contextualized behavior based on the return type. See more details in the actual [`Symbol` class](https://github.com/Xpitfire/botdynamics/blob/main/botdyn/symbol.py).

In the next section, we will show that actually almost all operations  in `botdyn/core.py` are derived from the more generic `few_shot` decorator.


### 🧪 Custom Operations

One can also define customized operations. For example, let us define a custom operation to generate a random integer between 0 and 10:

```python
class Demo(bd.Symbol):
    def __init__(self, value = '') -> None:
        super().__init__(value)
    
    @bd.zero_shot(prompt="Generate a random integer between 0 and 10.",
                  constraints=[
                      lambda x: x >= 0,
                      lambda x: x <= 10
                  ])
    def get_random_int(self) -> int:
        pass
```

As we show, the Semantic API uses Python `Decorators` to define operations. The `@bd.zero_shot` decorator is used to define a custom operation that does not require any demonstration examples, since the prompt is expressive enough. In the shown example, the `zero_shot` decorator takes in two arguments: `prompt` and `constraints`. The `prompt` argument is used to define the prompt that is used to condition on our desired operation behavior. The `constraints` argument is used to define validation constraints of the computed outcome, to ensure it fulfills our expectations.

If the constraint is not fulfilled, the above implementation would reach out to the specified `default` implementation or default value. If no default implementation or value was found, the Semantic API would raise an `ConstraintViolationException`.

We also see that in the above example the return type is defined as `int`. Therefore, the resulting value from the wrapped function will be of type int. This works because our implementation uses auto-casting to a user specified return data type. If the cast fails, the Semantic API will raise a `ValueError`. If no return type is specified, the return type will be `Any`.

### Few-shot operations

The `@bd.few_shot` decorator is the more general version of `@bd.zero_shot` and is used to define a custom operation that requires demonstration examples. To give a more complete picture, we present the function signature of the `few_shot` decorator:

```python
def few_shot(prompt: str,
             examples: List[str], 
             constraints: List[Callable] = [],
             default: Optional[object] = None, 
             limit: int = 1,
             pre_processor: Optional[List[PreProcessor]] = None,
             post_processor: Optional[List[PostProcessor]] = None,
             **wrp_kwargs):
```

The `prompt` and `constraints` behave similar to the `zero_shot` decorator. However, as we see the `examples` and `limit` arguments are new. The `examples` argument is used to define a list of demonstration examples that are used to condition the neural computation engine. The `limit` argument is used to define the maximum number of examples that are returned, give that there are more results. The `pre_processor` argument takes a list of `PreProcessor` objects which can be used to pre-process the input before it is fed into the neural computation engine. The `post_processor` argument takes a list of `PostProcessor` objects which can be used to post-process the output before it is returned to the user. The `wrp_kwargs` argument is used to pass additional arguments to the wrapped function, which are also stream-lined towards the neural computation engine and other engines.

To give a more holistic picture ouf our conceptional implementation, see the following flow diagram containing the most important classes:

<img src="https://raw.githubusercontent.com/Xpitfire/botdynamics/main/assets/images/img9.png" width="600px">

The color indicates logical groups of data manipulation. `Yellow` indicates the input and output data. `Blue` indicates places you can customize or prepare the input of your engine. `Green` indicates post-processing steps of the engine response. `Red` indicates the application of constraints (which also includes the attempted casting of the `return type signature`, if specified in the decorated method). And `Grey` indication the custom method which defines all properties, therefore has access to all the above mentioned data.

To conclude this section, here is an example how to write a custom Japanese name generator with our `@bd.zero_shot` decorator:

```python
import botdyn as bd
class Demo(bd.Symbol):
    @bd.few_shot(prompt="Generate Japanese names: ",
                 examples=["愛子", "和花", "一郎", "和枝"],
                 limit=2,
                 constraints=[lambda x: len(x) > 1])
    def generate_japanese_names(self) -> list:
        return ['愛子', '和花'] # dummy implementation
```

Should the neural computation engine not be able to compute the desired outcome, it will reach out to the `default` implementation or default value. If no default implementation or value was found, the Semantic API would raise an exception.


## Prompt Design

The way all the above operations are performed is by using a `Prompt` class. The Prompt class is a container for all the information that is needed to define a specific operation. The Prompt class is also the base class for all other Prompt classes. 

Here is an example how to define a Prompt to enforce the neural computation engine to compare two values:

```python
class CompareValues(bd.Prompt):
    def __init__(self) -> bd.Prompt:
        super().__init__([
            "4 > 88 =>False",
            "-inf < 0 =>True",
            "inf > 0 =>True",
            "4 > 3 =>True",
            "1 < 'four' =>True",
            ...
        ])
```

For example, when calling the `<=` operation on two Symbols, the neural computation engine will evaluate the two Symbols and compare them based on the `CompareValues` prompt.

```python
res = bd.Symbol(1) <= bd.Symbol('one')
```

This statement evaluates to `True`, since the fuzzy compare operation was enforced to compare the two Symbols based on their semantic meaning, hence, this is the reason we call our framework also `Semantic API`.

```bash
:[Output]:
True
```

In a more general notion, depending on the context, hierarchy of the expression class and used operations the semantics of the Symbol manipulations may vary. To better illustrate this, we show our conceptual prompt design in the following figure:

<img src="https://raw.githubusercontent.com/Xpitfire/botdynamics/main/assets/images/img4.png" width="350px">

The figure above shows our context-based prompt design as a container of all the information that is provided to the neural computation engine to define a specific operation. 

Conceptually we consider four main prompt concepts: `Global Context`, `Operation`, `Examples` and `Templates`. The prompts can be curated either by inheritance or by composition. For example, the `Global Context` can be defined by inheriting from the `Expression` class and overriding the `static_context` property. An `Operation` and `Template` prompt can be created by providing an `PreProcessor` to modify the input data. We will now explain each prompt concept in more detail:

- The `Global Context` concept is considered optional and can be defined in a static manner, either by sub-classing the Expression class and overriding the `static_context` property, or at runtime by updating the `dynamic_context` or `attach` kwargs. Here is an example how to use the `attach` kwargs via the method signature to define the global context:
  ```python
  # creating a query to ask if an issue was resolve or not
  sym = Symbol("<some-community-conversation>")
  q = sym.query("Was the issue resolved?")
  # write manual condition to check if the issue was resolved
  if 'not resolved' in q:
      # do a new query but attach the previous query answer to the new query
      sym.query("What was the resolution?", attach=q)
      ...
  else:
      pass # all good
  ```
  Nevertheless, how it is set, the global context is conceptually meant to define the overall context of an expression. For example, if we want to operate in the context of a domain-specific language, without having to override each base class function over and over again. See more details in [this notebook](notebooks/demo.ipynb).

- The `Operation` prompt concept defines the behavior of an atomic operation and is therefore mandatory to express the nature of such an operation. For example, the `+` operation is used to add two Symbols together and therefore the `+`-operation prompt explains its behavior. 

- `Examples` is another optional property and provides the neural computation engine with a set of demonstrations that are used to properly condition the engine. For example, the `+`-operation prompt can be conditioned on how to add numbers by providing a set of demonstrations, such as `1 + 1 = 2`, `2 + 2 = 4`, etc.

- The `Template` prompt concept is optional and encapsulates the resulting prediction to enforce a specific format. For example, to generate HTML tags one can use a curated `<html>{{placeholder}}</html>` template. This template will enforce the neural computation engine to generate only HTML tags to replace the `{{placeholder}}` tag.


## 😑 Expressions

An `Expression` is a non-terminal symbol, which can be further evaluated. It inherits all the properties from Symbol and overrides the `__call__` function to evaluate its expressions or values. From the `Expression` class, all other expressions are derived. The Expression class also adds additional capabilities i.e. to `fetch` data from URLs, `search` on the internet or `open` files.

BotDynamics' API closely follows best practices and ideas from `PyTorch`, therefore, one can build complex expressions by combining multiple expressions as a computational graph. Each Expression has its own `forward` function, which has to be overridden. The `forward` function is used to define the behavior of the expression. The `forward` function is called by the `__call__` function, which is inherited from the Symbol class. The `__call__` function is used to evaluate the expression and return the result. The `__call__` function is also used to evaluate the expression in a lazy manner, which means that the expression is only evaluated when the result is needed. This is a very important feature, since it allows us to build complex expressions without having to evaluate the whole expression at once. We already implemented many useful expressions, which can be imported from the `botdyn.components` file.

Other important properties that are inherited from the Symbol class are the `_sym_return_type` and `static_context`. These two properties define the context in which the current Expression operates, as described in the [Prompt Design](#prompt-design) section. The static_context therefore influences all operations held by the current Expression. The _sym_return_type ensures that after each evaluation of an Expression, we obtain the desired return object type. This is usually the current type, but can be modified to return a different type. 

Expressions can of course have more complex structures and be further sub-classed, such as shown in the example of the `Sequence` expression in the following figure:

<img src="https://raw.githubusercontent.com/Xpitfire/botdynamics/main/assets/images/img2.png" width="720px">

A Sequence expression can hold multiple expressions, which are evaluated at runtime.

### Sequence expressions

Here is an example how to define a Sequence expression:

```python
# first import all expressions
from botdyn import *
# define a sequence of expressions
Sequence(
    Clean(),
    Translate(),
    Outline(),
    Compose('Compose news:'),
)
```

### Stream expressions

As we saw earlier, we can create contextual prompts to define the context and operations of our neural engine. However, this also takes away a lot of the available context size and since e.g. the GPT-3 Davinci context length is limited to 4097 tokens, this might quickly become a problem. Luckily, we can use the `Stream` processing expression. This expression opens up a data stream and performs chunks-based operations on the input stream. 

A Stream expression can easily be wrapped around other expressions. For example, the chunks can be processed with a `Sequence` expression, that allows multiple chained operations in sequential manner. Here is an example how to define such a Stream expression:

```python
Stream(Sequence(
    Clean(),
    Translate(),
    Outline(),
    Embed()
))
```
The stream operation chunks the long input text into smaller chunks and passes them to the inner expression, which returns a `generator` object. In this case, to perform more complex operations, we open a stream and pass a `Sequence` object which cleans, translates, outlines and embeds the information. 

The issue with this approach is, that the resulting chunks are processed independently of each other. This means that the context of the chunks is not preserved. To solve this issue, we can use the `Cluster` expression instead, where the independent chunks are recombined based on their similarity. We illustrate this in the following figure:

<img src="https://raw.githubusercontent.com/Xpitfire/botdynamics/main/assets/images/img6.png" width="720px">

In the shown example we recombine all individual chunks again by clustering the information among the chunks. This gives us a way to consolidate contextually related information and recombine them in a meaningful way. Furthermore, the clustered information can then be labeled by looking / streaming through the values within the cluster and collecting the most relevant labels.

The full example is shown below:

```python
stream = Stream(Sequence(
    Clean(),
    Translate(),
    Outline(),
))
sym = Symbol('<some long text>')
res = Symbol(list(stream(sym)))
expr = Cluster()
expr(res)
```

In a next step, we could recursively repeat this process on each summary node, therefore, build a hierarchical clustering structure. Since each Node resembles a summarized sub-set of the original information we can use it as an index of the larger content. The resulting tree can then be used to navigate and retrieve the original information, turning the large data stream problem into a search problem.

For searching in a vector space we can use dedicated libraries such as [Annoy](https://github.com/spotify/annoy), [Faiss](https://github.com/facebookresearch/faiss) or [Milvus](https://github.com/milvus-io/milvus). 

## ❌ Error Handling

A key idea of the BotDynamics API is to be able to generate code. This in turns means that errors may occur, which we need to handle in a contextual manner. As a future vision, we even want our API to self extend and therefore need to be able to resolve issues automatically. To do so, we propose the `Try` expression, which has a fallback statement built in and retries an execution with dedicated error analysis and correction. This expression analyses the input and the error, and conditions itself to resolve the error by manipulating the original code. If the fallback expression succeeds, the result is returned. Otherwise, this process is repeated for the number of `retries` specified. If the maximum number of retries is reached and the problem not resolved, the error is raised again. 

Let us assume, we have some executable code that was previously generated. However, by the nature of generative processes syntax errors may occur. Furthermore, we have the `Execute` expression, which takes in a symbol and tries to execute it. Naturally, this will fail. In the following example we illustrate how the `Try` expression resolves this syntactic error.

```python
expr = Try(expr=Execute())
sym = Symbol('a = int("3,")') # some code with a syntax error
res = expr(sym)
```

The resulting output is the evaluated code, which was corrected:

```bash
:Output:
a = 3
```

We are aware that not all errors are as simple as the shown syntactic error example, which can be resolved automatically. Many errors occur due to semantic misconceptions. Such issues require contextual information. Therefore, we are further exploring means towards more sophisticated error handling mechanism.
This includes also the usage of streams and clustering to resolve errors in a more hierarchical contextual manner.


## 🕷️ Explainability, Testing & Debugging

Perhaps one of the greatest benefits of using neuro-symbolic programming is, that we can get a clear understanding of how well our LLMs understand atomic operations. Specifically we gain knowledge about if, and at which point they fail, enabling to follow their StackTraces and determine the failure points. Neuro-symbolic programming allows us to debug the model predictions and understand how they came about. Furthermore, we can unit test them to detect conceptual misalignments. 

### Unit Testing Models

Since our premise is to divide and conquer complex problems, we can curate conceptual unit test and target very specific and tracktable sub-problems. The resulting measure, i.e. success rate of model predictions, can then be used to evaluate their conceptual performance, and hint towards undesired flaws or biases.

This allows us to design domain-specific benchmarks and see how well general learners, such as GPT-3, adapt to these tasks. 

For example, we can write a fuzzy comparison operation, that can take in digits and strings alike and perform a semantic comparison. LLMs can then be asked to evaluate these expressions. Often times, these LLMs still fail to understand the semantic meaning of these tokens and give wrong answers. 

The following code snipped shows a unit test to perform semantic comparison of numbers (between digits and strings):

```python
import unittest
from botdyn import *

class TestComposition(unittest.TestCase):
  def test_compare(self):
      res = Symbol(10) > Symbol(5)
      self.assertTrue(res)
      res = Symbol(1) < Symbol('five')
      self.assertTrue(res)
      ...
```

### 🔥Debugging

When creating very complex expressions, we debug them by using the `Trace` expression, which allows to print out and follow the StackTrace of the neuro-symbolic operations. Combined with the `Log` expression, which creates a dump of all prompts and results to a log file, we can analyze where our models potentially failed.


### Examples

In the following example we create a news summary expression that crawls the given URL and streams the site content through multiple expressions. The outcome is a news website that is created based on the crawled content. The `Trace` expression allows to follow the StackTrace of the operations and see what operations are currently executed. If we open the `outputs/engine.log` file we can see the dumped traces with all the prompts and results.

```python
# crawling the website and creating an own website based on its facts
news = News(url='https://www.cnbc.com/cybersecurity/',
            pattern='cnbc',
            filters=ExcludeFilter('sentences about subscriptions, licensing, newsletter'),
            render=True)
expr = Log(Trace(news))
res = expr()
```

Here is the corresponding StackTrace of the model:

<img src="https://raw.githubusercontent.com/Xpitfire/botdynamics/main/assets/images/img8.png" width="900px">

The above code creates a webpage with the crawled content from the original source. See the preview below, the entire [rendered webpage image here](https://raw.githubusercontent.com/Xpitfire/botdynamics/main/examples/results/news.png) and resulting [code of webpage here](https://raw.githubusercontent.com/Xpitfire/botdynamics/main/examples/results/news.html). 


<img src="https://raw.githubusercontent.com/Xpitfire/botdynamics/main/examples/results/news_prev.png" width="900px">

## ▶️ Play around with our API

Launch and explore the notebook here:

[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/Xpitfire/botdynamics/HEAD)

There are many more examples in the [examples folder](examples/) and in the [notebooks folder](notebooks/). You can also explore the test cases in the [tests folder](tests/).


## 📈 Concepts for Data Collection \& Analytics

BotDynamics is by design a data-driven framework. This means that we can collect data from API interactions while we provide the requested responses. For very agile, dynamic adaptations or prototyping we can integrate user desired behavior quickly into existing prompts. However, we can also log the user queries and model predictions to make them available for post-analysis, and fine-tuning. Therefore, we can quickly iterate, customize and improve the model's responses based on real-world data.

In the following example, we show how we can use an `Output` expression to pass a handler function and access input prompts of the model and model predictions. These, can be used for data collection and later fine-tuning stages. The handler function provides a dictionary and offers keys for `input` and `output` values. The content can then be sent to a data pipeline for further processing.

```python
sym = Symbol('Hello World!')
def handler(res):
    input_ = res['input']
    output = res['output']
expr = Output(expr=sym.translate, 
              handler=handler, 
              verbose=True)
res = expr('German')
```

Since we called verbose, we can also see the console print of the `Output` expression:

```bash
Input: (['Translate the following text into German:\n\nHello World!'],)
Expression: <bound method Symbol.translate of <class 'botdyn.symbol.Symbol'>(value=Hello World!)>
args: ('German',) kwargs: {'input_handler': <function OutputEngine.forward.<locals>.input_handler at ...
Dictionary: {'wrp_self': <class 'botdyn.components.Output'>(value=None), 'func': <function Symbol.output.<locals>._func at ...
Output: Hallo Welt!
```


## 🤖 Engines

Due to limited compute resources we currently rely on OpenAI's GPT-3 API for the neuro-symbolic engine. However, given the right compute resources, it is possible to use local machines to avoid high latencies and costs, with alternative engines such as OPT or Bloom. This would allow for recursive executions, loops, and more complex expressions.

However, we already integrated a set of useful engines that are capable of providing language tokens to perform symbolic operations. 

### Speech Engine

To perform speech transcription we use `whisper`. The following example shows how to transcribe an audio file and return the text:

```python
expr = Expression()
res = expr.speech('examples/audio.mp3')
```

```bash
:Output:
I may have overslept.
```

### OCR Engine

To perform OCR we use `APILayer`. The following example shows how to transcribe an image and return the text:

```python
expr = Expression()
res = expr.ocr('https://media-cdn.tripadvisor.com/media/photo-p/0f/da/22/3a/rechnung.jpg')
```

The OCR engine returns a dictionary with a key `all_text` where the full text is stored. See more details in their documentation [here](https://apilayer.com/marketplace/image_to_text-api).

```bash
:Output:
China Restaurant\nMaixim,s\nSegeberger Chaussee 273\n22851 Norderstedt\nTelefon 040/529 16 2 ...
```


### Search Engine

To perform search queries we use `SerpApi` with `Google` backend. The following example shows how to search for a query and return the results:

```python
expr = Expression()
res = expr.search('Birthday of Barack Obama')
```

```bash
:Output:
August 4, 1961
```

### WebCrawler Engine

To perform web crawling we use `Selenium`. The following example shows how to crawl a website and return the results:

```python
expr = Expression()
res = expr.fetch(url="https://www.google.com/", 
                 pattern="google")
```
The `pattern` property can be used to detect if the document as been loaded correctly. If the pattern is not found, the crawler will timeout and return an empty result.

```bash
:Output:
GoogleKlicke hier, wenn du nach einigen Sekunden nicht automatisch weitergeleitet wirst.GmailBilderAnmelden ...
```

### Drawing Engine

To render nice images from text description we use `DALL·E 2`. The following example shows how to draw a text description and return the image:

```python
expr = Expression('a cat with a hat')
res = expr.draw()
```

```bash
:Output:
https://oaidalleapiprodscus.blob.core.windows.net/private/org-l6FsXDfth6Uct ...
```

Don't worry, we would never hide an image of a cat with a hat from you. Here is the image preview and [link](https://camo.githubusercontent.com/4f607176e782700befd732212c198b12c3923bf9c25f548aa444c92f6bcb97d9/68747470733a2f2f6f616964616c6c6561706970726f64736375732e626c6f622e636f72652e77696e646f77732e6e65742f707269766174652f6f72672d6c36467358446674683655637479777441504e746248364b2f757365722d76726c58594933793375484c6557374f6b594a64374b32632f696d672d7530523372394b515130736f716e7830774c7361335368532e706e673f73743d323032332d30312d3133543139253341313625334130305a2673653d323032332d30312d3133543231253341313625334130305a2673703d722673763d323032312d30382d30362673723d6226727363643d696e6c696e6526727363743d696d6167652f706e6726736b6f69643d36616161646564652d346662332d343639382d613866362d36383464373738366230363726736b7469643d61343863636135362d653664612d343834652d613831342d39633834393635326263623326736b743d323032332d30312d3133543138253341313925334133365a26736b653d323032332d30312d3134543138253341313925334133365a26736b733d6226736b763d323032312d30382d3036267369673d466c3133556f51694c646a6e42716e59473674746e6e455666716247546975596b2f7067706170385625324259253344):

<img src="https://camo.githubusercontent.com/4f607176e782700befd732212c198b12c3923bf9c25f548aa444c92f6bcb97d9/68747470733a2f2f6f616964616c6c6561706970726f64736375732e626c6f622e636f72652e77696e646f77732e6e65742f707269766174652f6f72672d6c36467358446674683655637479777441504e746248364b2f757365722d76726c58594933793375484c6557374f6b594a64374b32632f696d672d7530523372394b515130736f716e7830774c7361335368532e706e673f73743d323032332d30312d3133543139253341313625334130305a2673653d323032332d30312d3133543231253341313625334130305a2673703d722673763d323032312d30382d30362673723d6226727363643d696e6c696e6526727363743d696d6167652f706e6726736b6f69643d36616161646564652d346662332d343639382d613866362d36383464373738366230363726736b7469643d61343863636135362d653664612d343834652d613831342d39633834393635326263623326736b743d323032332d30312d3133543138253341313925334133365a26736b653d323032332d30312d3134543138253341313925334133365a26736b733d6226736b763d323032312d30382d3036267369673d466c3133556f51694c646a6e42716e59473674746e6e455666716247546975596b2f7067706170385625324259253344" width="200px">


### File Engine

To perform file operations we currently use the file system of the OS. At the moment, we support only PDF files and plain text files. This is a very early stage and we are working on more sophisticated file system access and remote storage. The following example shows how to read a PDF file and return the text:

```python
expr = Expression()
res = expr.open('./LICENSE')
```

```bash
:Output:
MIT License\n\nCopyright (c) 2023 Marius-Constantin Dinu\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation ...
```


### CLIP Engine

To perform text-based image few-shot classification we use `CLIP`. This implementation is very experimental and does not conceptually fully integrate the way we intend it, since the embeddings of CLIP and GPT-3 are not aligned, i.e. embeddings of the same word are not identical for these two models. Aligning them is an open problem for future research. The following example shows how to classify the image of our generated cat from above and return the results as an array of probabilities:

```python
expr = Expression()
res = expr.vision('https://oaidalleapiprodscus.blob.core.windows.net/private/org-l6FsXDfth6...', 
                  ['cat', 'dog', 'bird', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe'])
```

```bash
:Output:
array([[9.72840726e-01, 6.34790864e-03, 2.59368378e-03, 3.41371237e-03,
        3.71197984e-03, 8.53193272e-03, 1.03346225e-04, 2.08464009e-03,
        1.77942711e-04, 1.94185617e-04]], dtype=float32)
```

### Custom Engine

If you want to replace or extend the functionality of our framework, you can do this by customizing the existing engines or creating new engines. The `Symbol` class provides for this functionality some helper methods, such as `command` and `setup`. The `command` method can pass on configurations (as `**kwargs`) to the engines and change functionalities or parameters. The `setup` method can be used to re-initialize an engine with your custom engine implementation which must sub-class the `Engine` class. Both methods can be specified to address one, more or all engines.

Here is an example how to initialize your own engine. We will sub-class the existing `GPT3Engine` and override the `prepare` method. This method is called before the neural computation and can be used to modify the parameters of the actual input prompt that will be passed in for execution. In this example, we will replace the prompt with dummy text for illustration purposes:

```python
from botdyn.backend.engine_gpt3 import GPT3Engine
class DummyEngine(GPT3Engine):
    def prepare(self, args, kwargs, wrp_params):
        wrp_params['prompts'] = ['Go wild and generate something!']
custom_engine = DummyEngine()
sym = Symbol()
sym.setup(engines={'neurosymbolic': custom_engine})
res = sym.compose()
```

To configure an engine, we can use the `command` method. In this example, we will enable `verbose` mode, where the engine will print out what function it is executing and the parameters it is using. This is useful for debugging purposes:

```python
sym = Symbol('Hello World!')
sym.command(engines=['neurosymbolic'], verbose=True)
res = sym.translate('German')
```

```bash
:Output:
<botdyn.backend.engine_gpt3.GPT3Engine object at 0, <function Symbol.translate.<locals>._func at 0x7fd68ba04820>, {'wrp_self': <class 'botdyn.symbol.S ['\n\nHallo Welt!']
```

Here is the list of names of the engines that are currently supported:

* `neurosymbolic` - GPT-3
* `ocr` - Optical Character Recognition
* `vision` - CLIP
* `speech` - Whisper
* `embedding` - OpenAI Embeddings API (`ada-002`)
* `userinput` - User Command Line Input
* `search` - SerpApi (Google search)
* `crawler` - Selenium
* `execute` - Python Interpreter
* `open` - File System
* `output` - Output Callbacks (e.g. for printing to console or file)
* `imagerendering` - DALL·E 2

Finally, let's assume you want to create a entirely new engine, but still keep our workflow, then you can use the `_process_query` function from `botdyn/functional.py` and pass in your engine including all other specified objects (i.e. Prompt, PreProcessor, etc.; see also section [Custom Operations](#🧪-custom-operations)).

## ⚡Limitations

Uff... this is a long list. We are still in the early stages of development and are working hard to overcome these limitations. Just to name a few:

Engineering challenges:
* Our framework is still in its early stages of development and is not yet meant for production use. For example, the Stream class only estimates the prompt size by an approximation, which can be easily exceeded. One can also create more sophisticated prompt hierarchies and dynamically adjust the global context based on a state-based approach. This would allow making consistent predictions even for long text streams.
* The code may not be complete and is not yet optimized for speed and memory usage, and uses API-based LLMs due to limitations of compute resources.
* Code coverage is not yet complete and we are still working on the documentation.
* Integrate with a more diverse set of models from [HuggingFace](https://huggingface.co/) or other platforms.
* Currently we did not account for multi-threading and multi-processing.
* Vector-based search indexes, i.e. [Annoy](https://github.com/spotify/annoy), [Faiss](https://github.com/facebookresearch/faiss) or [Milvus](https://github.com/milvus-io/milvus), are not yet integrated into the framework to enable fast content retrieval.

Research challenges:
* The experimental integration of CLIP is meant to align image and text embeddings. To enable decision-making of LLMs based on observations and perform symbolic operations on objects in images or videos would be a huge leap forward. This would perfectly integrate with reinforcement learning approaches and enable us to control policies in a systematic way (see also [GATO](https://www.deepmind.com/publications/a-generalist-agent)). Therefore, we need to train large multi-modal variants with image / video data and text data, describing in high details the scenes to obtain neuro-symbolic computation engines that can perform semantic operations similar to `move-towards-tree`, `open-door`, etc.
* Generalist LLMs are still highly over-parameterized and hardware has not yet caught up to host these models on arbitrary day-to-day machines. This limits the applicability of our approach not only on small data streams, but also gives high latencies and therefore limits the amount of complexity and expressiveness we can achieve with our expressions.


## 🥠 Future Work

We are constantly working on improving the framework and are open to any suggestions, feedback or comments. However, we try to think ahead of time and have some general ideas for future work in mind:

* Meta-Learning Semantic Concepts on top of Neuro-Symbolic Expressions
* Self-evolving and self-healing API

We believe that LLMs as neuro-symbolic computation engines enable us a new class of applications, with tools and API that can self-analyze and self-heal. We are excited to see what the future brings and are looking forward to your feedback and contributions.

## Conclusion

We have presented a neuro-symbolic view on LLMs and showed how they can be a central pillar for many multi-model operations. We gave an technical report on how to utilize our framework and also hinted at the capabilities and prospects of these models to be leveraged by modern software development. 



## 👥 References, Related Work \& Credits

This project is inspired by the following works, but not limited to them:

* [The Algebraic Theory of Context-Free Languages](http://www-igm.univ-mlv.fr/~berstel/Mps/Travaux/A/1963-7ChomskyAlgebraic.pdf)
* [Tracr: Compiled Transformers as a Laboratory for Interpretability](https://arxiv.org/abs/2301.05062)
* [How can computers get common sense?](https://www.science.org/doi/10.1126/science.217.4566.1237)
* [Artificial Intelligence: A Modern Approach](http://aima.cs.berkeley.edu/)
* [Neuro-symbolic programming](https://arxiv.org/abs/2210.05050)
* [Fuzzy Sets](https://web.archive.org/web/20150813153834/http://www.cs.berkeley.edu/~zadeh/papers/Fuzzy%20Sets-Information%20and%20Control-1965.pdf)
* [An early approach toward graded identity and graded membership in set theory](https://www.sciencedirect.com/science/article/abs/pii/S0165011409005326?via%3Dihub)
* [From Statistical to Causal Learning](https://arxiv.org/abs/2204.00607)
* [Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)
* [Deep reinforcement learning from human preferences](https://arxiv.org/abs/1706.03741)
* [Aligning Language Models to Follow Instructions](https://openai.com/blog/instruction-following/)
* [Chain of Thought Prompting Elicits Reasoning in Large Language Models](https://arxiv.org/abs/2201.11903)
* [Measuring and Narrowing the Compositionality Gap in Language Models](https://ofir.io/self-ask.pdf)
* [Large Language Models are Zero-Shot Reasoners](https://arxiv.org/abs/2205.11916)
* [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://arxiv.org/abs/2107.13586)
* [Understanding Stereotypes in Language Models: Towards Robust Measurement and Zero-Shot Debiasing](https://arxiv.org/abs/2212.10678)
* [Connectionism and Cognitive Architecture: A Critical Analysis](https://ruccs.rutgers.edu/images/personal-zenon-pylyshyn/proseminars/Proseminar13/ConnectionistArchitecture.pdf)
* [Unit Testing for Concepts in Neural Networks](https://arxiv.org/abs/2208.10244)
* [REALM: Retrieval-Augmented Language Model Pre-Training](https://arxiv.org/abs/2002.08909)


### Acknowledgements

Also this is a long list. Great thanks to my colleagues and friends at the [Institute for Machine Learning at Johannes Kepler University (JKU), Linz](https://www.jku.at/institut-fuer-machine-learning/) for their great support and feedback. Great thanks to the [OpenAI](https://openai.com/), [GitHub](https://github.com/) and [Microsoft Research](https://www.microsoft.com/en-us/research/) teams for their great work and making their API and tools available to the public, and thanks to all the people who contributed to this project. Be it by providing feedback, bug reports, code, or just by using the framework. We are very grateful for your support. 


### Contribution

If you want to contribute to this project, please read the [CONTRIBUTING.md](CONTRIBUTING.md) file for details on our code of conduct, and the process for submitting pull requests to us. Any contributions are highly appreciated.

### 📜 Citation

```bibtex
@software{Dinu_BotDynamics_2022,
  author = {Dinu, Marius-Constantin and Co, Pilot and GPT-3, Davinci-003},
  title = {{BotDynamics}},
  url = {https://github.com/Xpitfire/botdynamics},
  month = {11},
  year = {2022}
}
```

### 📝 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

### Like this project?

If you like this project, leave a star ⭐️ and share it with your friends and colleagues.
And if you want to support this project even further, please consider donating to support the continuous development of this project. Thank you!

[![Donate](https://img.shields.io/badge/Donate-PayPal-green.svg)](https://www.paypal.com/donate/?hosted_button_id=WCWP5D2QWZXFQ)

We are also looking for contributors or investors to grow and support this project. If you are interested, please contact us.

### 📫 Contact

If you have any questions about this project, please contact us via [email](mailto:office@alphacoreai.eu), on our [website](www.alphacoreai.eu/) or find us on Discord:

[![Discord](https://img.shields.io/discord/768087161878085643?label=Discord&logo=Discord&logoColor=white)](https://discord.gg/azDQxCHeDA)

If you want to contact me directly, you can reach me directly on [LinkedIn](https://www.linkedin.com/in/mariusconstantindinu/), on [Twitter](https://twitter.com/DinuMariusC), or at my personal [website](https://www.dinu.at/).

            

Raw data

            {
    "_id": null,
    "home_page": "",
    "name": "botdynamics",
    "maintainer": "",
    "docs_url": null,
    "requires_python": ">=3.7",
    "maintainer_email": "",
    "keywords": "symbolic programming,machine learning",
    "author": "",
    "author_email": "Marius-Constantin Dinu <office@alphacoreai.eu>",
    "download_url": "",
    "platform": null,
    "description": "# <img src=\"https://raw.githubusercontent.com/Xpitfire/botdynamics/main/assets/images/bot.png\" width=\"100px\"> BotDynamics\n\n## **A Neuro-Symbolic Perspective on Large Language Models (LLMs)**\n\n*Building applications with LLMs at its core through our `Semantic API` leverages the power of classical and differentiable programming in Python.*\n\n[![PyPI version](https://badge.fury.io/py/botdynamics.svg)](https://badge.fury.io/py/botdynamics) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) [![Twitter](https://img.shields.io/twitter/url/https/twitter.com/langchainai.svg?style=social&label=Follow%20%40DinuMariusC)](https://twitter.com/DinuMariusC) [![contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](https://github.com/Xpitfire/botdynamics/issues) [![Hits](https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2FXpitfire%2Fbotdynamics&count_bg=%2379C83D&title_bg=%23555555&icon=&icon_color=%23E7E7E7&title=hits&edge_flat=false)](https://hits.seeyoufarm.com)\n\n## \ud83d\udcd6 Table of Contents\n\n- [ BotDynamics](#-botdynamics)\n  - [**A Neuro-Symbolic Perspective on Large Language Models (LLMs)**](#a-neuro-symbolic-perspective-on-large-language-models-llms)\n  - [\ud83d\udcd6 Table of Contents](#-table-of-contents)\n  - [\ud83d\udd27 Quick Install](#-quick-install)\n    - [\u27a1\ufe0f *\\[Optional\\]* Installs](#\ufe0f-optional-installs)\n  - [\ud83e\udd37\u200d\u2642\ufe0f Why BotDynamics?](#\ufe0f-why-botdynamics)\n  - [ Tell me some more fun facts!](#-tell-me-some-more-fun-facts)\n  - [\ud83d\ude36\u200d\ud83c\udf2b\ufe0f How does it work?](#\ufe0f-how-does-it-work)\n    - [\ud83d\udcda Symbolic operations](#-symbolic-operations)\n    - [Ranking objects](#ranking-objects)\n    - [Evaluating Expressions by best effort](#evaluating-expressions-by-best-effort)\n    - [Dynamic casting](#dynamic-casting)\n    - [Fuzzy Comparisons](#fuzzy-comparisons)\n    - [\ud83e\udde0 Causal Reasoning](#-causal-reasoning)\n  - [\ud83d\ude37 Operations](#-operations)\n    - [\ud83e\uddea Custom Operations](#-custom-operations)\n    - [Few-shot operations](#few-shot-operations)\n  - [Prompt Design](#prompt-design)\n  - [\ud83d\ude11 Expressions](#-expressions)\n    - [Sequence expressions](#sequence-expressions)\n    - [Stream expressions](#stream-expressions)\n  - [\u274c Error Handling](#-error-handling)\n  - [\ud83d\udd77\ufe0f Explainability, Testing \\& Debugging](#\ufe0f-explainability-testing--debugging)\n    - [Unit Testing Models](#unit-testing-models)\n    - [\ud83d\udd25Debugging](#debugging)\n    - [Examples](#examples)\n  - [\u25b6\ufe0f Play around with our API](#\ufe0f-play-around-with-our-api)\n  - [\ud83d\udcc8 Concepts for Data Collection \\& Analytics](#-concepts-for-data-collection--analytics)\n  - [\ud83e\udd16 Engines](#-engines)\n    - [Speech Engine](#speech-engine)\n    - [OCR Engine](#ocr-engine)\n    - [Search Engine](#search-engine)\n    - [WebCrawler Engine](#webcrawler-engine)\n    - [Drawing Engine](#drawing-engine)\n    - [File Engine](#file-engine)\n    - [CLIP Engine](#clip-engine)\n    - [Custom Engine](#custom-engine)\n  - [\u26a1Limitations](#limitations)\n  - [\ud83e\udd60 Future Work](#-future-work)\n  - [Conclusion](#conclusion)\n  - [\ud83d\udc65 References, Related Work \\& Credits](#-references-related-work--credits)\n    - [Acknowledgements](#acknowledgements)\n    - [Contribution](#contribution)\n    - [\ud83d\udcdc Citation](#-citation)\n    - [\ud83d\udcdd License](#-license)\n    - [Like this project?](#like-this-project)\n    - [\ud83d\udceb Contact](#-contact)\n\n\n\n## \ud83d\udd27 Quick Install\n\n```bash\npip install botdynamics\n```\n\nBefore the first run, define exports for the required `API keys` to enable the respective engines. This will register the keys in the internal storage. By default BotDynamics currently uses OpenAI's neural engines, i.e. GPT-3 Davinci-003, DALL\u00b7E 2 and Embedding Ada-002, for the neuro-symbolic computations, image generation and embeddings computation respectively. However, these modules can easily be replaced with open-source alternatives. Examples are \n- [OPT](https://huggingface.co/docs/transformers/model_doc/opt) or [Bloom](https://huggingface.co/bigscience/bloom) for neuro-symbolic computations, \n- [Craiyon](https://www.craiyon.com/) for image generation, \n- and any [BERT variants](https://huggingface.co/models) for semantic embedding computations. \n\nTo set the OpenAI API Keys use the following command:\n\n```bash\n# Linux / MacOS\nexport OPENAI_API_KEY=\"<OPENAI_API_KEY>\"\n\n# Windows (PowerShell)\n$Env:OPENAI_API_KEY=\"<OPENAI_API_KEY>\"\n\n# Jupyter Notebooks (important: do not use quotes)\n%env OPENAI_API_KEY=<OPENAI_API_KEY>\n```\n\n\n**To get started import our library by using:**\n\n```python\nimport botdyn as bd\n```\n\nOverall, the following engines are currently supported:\n\n* **Neuro-Symbolic Engine**: [OpenAI's LLMs (GPT-3)](https://beta.openai.com/docs/introduction/overview) \n* **Embedding Engine**: [OpenAI's Embedding API](https://beta.openai.com/docs/introduction/overview)\n* **[Optional] Search Engine**: [SerpApi](https://serpapi.com/)\n* **[Optional] OCR Engine**: [APILayer](https://apilayer.com/ocr)\n* **[Optional] SpeechToText Engine**: [OpenAI's Whisper](https://openai.com/blog/whisper/)\n* **[Optional] WebCrawler Engine**: [Selenium](https://selenium-python.readthedocs.io/)\n* **[Optional] Image Rendering Engine**: [DALL\u00b7E 2](https://openai.com/dall-e-2/)\n* **[Optional] [CLIP](https://openai.com/blog/clip/) Engine**: \ud83e\udd17 [HuggingFace](https://huggingface.co/) (experimental image and text embeddings)\n\n\n### \u27a1\ufe0f *[Optional]* Installs\n\nBotDynamics uses multiple engines to process text, speech and images. We also include search engine access to retrieve information from the web. To use all of them, you will need to install also the following dependencies or assign the API keys to the respective engines. \n\nIf you want to use the `Search Engine` and `OCR Engine` you will need to export the following API keys:\n\n```bash\n# Linux / MacOS\nexport SEARCH_ENGINE_API_KEY=\"<SERP_API_KEY>\"\nexport OCR_ENGINE_API_KEY=\"<APILAYER_API_KEY>\"\n\n# Windows (PowerShell)\n$Env:SEARCH_ENGINE_API_KEY=\"<SERP_API_KEY>\"\n$Env:OCR_ENGINE_API_KEY=\"<APILAYER_API_KEY>\"\n```\n\nTo use them, you will also need to install the following dependencies:\n\n\n* **SpeechToText Engine**: `ffmpeg` for audio processing (based on OpenAI's [whisper](https://openai.com/blog/whisper/))\n\n```bash\n# Linux\nsudo apt update && sudo apt install ffmpeg\n\n# MacOS\nbrew install ffmpeg\n\n# Windows\nchoco install ffmpeg\n```\n\nAdditionally, you need to install the newest version directly from their repository, since the version available via `pip` is outdated:\n\n```bash\npip install git+https://github.com/openai/whisper.git\n```\n\n* **WebCrawler Engine**: For `selenium`, download the corresponding driver version by setting the `SELENIUM_CHROME_DRIVER_VERSION` environment variable. Currently we use Chrome as the default browser. This means that the Chrome version major number must match the ChromeDriver version. All versions are available [here](https://chromedriver.chromium.org/downloads). For example, if you use chrome version `109.0.5414.74`, you can set any `109.x.x.x` version for the `chromedriver`. In this case the `109.0.5414.74` is available on the selenium page, therefore the environment variable is set to it:\n\n```bash\n# Linux / MacOS\nexport SELENIUM_CHROME_DRIVER_VERSION=\"109.0.5414.74\"\n\n# Windows (PowerShell)\n$Env:SELENIUM_CHROME_DRIVER_VERSION=\"109.0.5414.74\"\n```\n\n\n## \ud83e\udd37\u200d\u2642\ufe0f Why BotDynamics?\n\nBotDynamics tries to close the gap between classical programming or Software 1.0 and modern data-driven programming (aka Software 2.0). It is a framework that allows to build software applications, which are able to utilize the power of large language models (LLMs), but are based on composability and inheritance - two powerful concepts from the object-oriented classical programming paradigm.\n\nThis allows to move along the spectrum between the classical programming and the data-driven programming world as illustrated in the following figure:\n\n<img src=\"https://raw.githubusercontent.com/Xpitfire/botdynamics/main/assets/images/img5.png\" width=\"720px\">\n\nConceptually, BotDynamics is a framework that uses machine learning - and specifically LLMs - at its core, and curates operations based on dedicated zero or few-shot learning prompt designs. Each operation solves atomic tasks, however, by chaining these operations together we can solve more complex problems. Our main philosophy is to divide and conquer a complex problem into more manageable, smaller problems. \n\nIn this turn, we also aim to gradually transition between general purpose LLMs with zero and few-shot learning capabilities, and specialized fine-tuned models to really nail down a specific problem (see above). \n\n## <img src=\"https://media.giphy.com/media/mGcNjsfWAjY5AEZNw6/giphy.gif\" width=\"50\"> Tell me some more fun facts!\n\nIn its essence, BotDynamics was inspired by the [`neuro-symbolic programming paradigm`](https://arxiv.org/abs/2210.05050).\n\n**Neuro-symbolic programming** is a paradigm for artificial intelligence and cognitive computing that combines the strengths of both deep neural networks and symbolic reasoning.\n\n**Deep neural networks** are a type of machine learning algorithms that are inspired by the structure and function of biological neural networks. They are particularly good at tasks such as image recognition, natural language processing, and decision making. However, they are not as good at tasks that require explicit reasoning, such as planning, problem solving, and understanding causal relationships.\n\n**Symbolic reasoning**, on the other hand uses formal languages and logical rules to represent knowledge and perform tasks such as planning, problem solving, and understanding causal relationships. Symbolic reasoning systems are good at tasks that require explicit reasoning, but are not as good at tasks that require pattern recognition or generalization, such as image recognition or natural language processing.\n\n**Neuro-symbolic programming** aims to combine the strengths of both neural networks and symbolic reasoning to create AI systems that can perform a wide range of tasks. One way this is done is by using neural networks to extract information from data and then using symbolic reasoning to make inferences and decisions based on that information. Another way is to use symbolic reasoning to guide the generative process of neural networks and make them more interpretable.\n\n**On a grander scale of things**, we believe that future computation platforms, such as wearables, Smartphones, tablets or notebooks will contain their own embedded LLMs (similar to GPT-3, ChatGPT, OPT or Bloom). \n\n<img src=\"https://raw.githubusercontent.com/Xpitfire/botdynamics/main/assets/images/img1.png\" width=\"720px\">\n\nThese LLMs will be able to perform a wide range of computations, such as natural language understanding or decision making. Furthermore, neuro-symbolic computation engines will be able to learn concepts how to tackle unseen tasks and solve complex problems by querying various data sources for solutions and operating logical statements on top. \nIn this turn, to ensure the generated content is in alignment with our goals, we need to develop ways to instruct, steer and control their generative processes. Therefore, our approach is an attempt to enable active and transparent flow control of these generative processes.\n\n<img src=\"https://raw.githubusercontent.com/Xpitfire/botdynamics/main/assets/images/img7.png\" width=\"720px\">\n\nAs shown in the figure above, one can think of it as shifting a probability mass from an input stream towards an output stream, in a contextualized manner. With properly designed conditions and expressions, one can also validate and steer the behavior towards a desired outcome, or repeat expressions that failed to fulfil the requirements. Our approach is to define a set of `fuzzy` operations that manipulate the data stream and conditions the LLMs. In essence, we consider all objects as symbols, and create operations that manipulate existing or generate new symbols. Each symbol can be interpreted as a statement. Multiple statements can be combined to form a logical expression.\n\nTherefore, by chaining statements together we can build causal relationships and computations, instead of relying only on inductive approaches. Consequently, the outlook towards an updated computational stack resembles a neuro-symbolic computation engine at its core and, in combination with established frameworks, enables new applications. \n\n\n## \ud83d\ude36\u200d\ud83c\udf2b\ufe0f How does it work?\n\nWe now show how we define our `Semantic API`, which is based on object-oriented and compositional design patterns. The `Symbol` class is the base class for all functional operations, which we refer to as a terminal symbol in the context of symbolic programming (fully resolved expressions). The Symbol class holds helpful operations and functions that can be interpreted as expressions to manipulate its content and evaluate to new Symbols. \n\n### \ud83d\udcda Symbolic operations\n\nLet us now define a Symbol and perform some basic manipulations. We start with a translation operation:\n\n```python\nsym = bd.Symbol(\"Welcome to our tutorial.\")\nsym.translate('German')\n```\n```bash\n:[Output]: \n<class 'botdyn.expressions.Symbol'>(value=Willkommen zu unserem Tutorial.)\n```\n\n### Ranking objects\n\nOur API can also perform basic data-agnostic operations to `filter`, `rank` or `extract` patterns. For example, we can rank a list of numbers:\n\n```python\nsym = Symbol(numpy.array([1, 2, 3, 4, 5, 6, 7]))\nres = sym.rank(measure='numerical', order='descending')\n```\n```bash\n:[Output]: \n<class 'botdyn.expressions.Symbol'>(value=['7', '6', '5', '4', '3', '2', '1'])\n```\n\n### Evaluating Expressions by best effort\n\nAs an inspiration, we relate to an approach demonstrated by [word2vec](https://arxiv.org/abs/1301.3781). \n\n**Word2Vec** generates dense vector representations of words by training a shallow neural network to predict a word given its neighbors in a text corpus. The resulting vectors are then used in a wide range of natural language processing applications, such as sentiment analysis, text classification, and clustering.\n\nBelow we can see an example how one can perform operations on the word embeddings (colored boxes).\nThe words are tokenized and mapped to a vector space, where we can perform semantic operations via vector arithmetics. \n\n<img src=\"https://raw.githubusercontent.com/Xpitfire/botdynamics/main/assets/images/img3.png\" width=\"450px\">\n\nSimilar to word2vec we intend to perform contextualized operations on different symbols, however, instead of operating in the vector space, we operate in the natural language space. This gives us the ability to perform arithmetics on words, sentences, paragraphs, etc. and verify the results in a human readable format. \n\nThe following examples show how to evaluate such an expression via a string representation:\n\n```python\nSymbol('King - Man + Women').expression()\n```\n```bash\n:[Output]:\n<class 'botdyn.expressions.Symbol'>(value=Queen)\n```\n\n### Dynamic casting\n\nWe can also subtract sentences from each other, where our operations condition the neural computation engine to evaluate the Symbols by best effort. In the following example, it determines that the word `enemy` is present in the sentence, therefore deletes it and replaces it with the word `friend` (which is added):\n\n```python\nres = bd.Symbol('Hello my enemy') - 'enemy' + 'friend'\n```\n```bash\n:[Output]: \n<class 'botdyn.expressions.Symbol'>(value=Hello my friend)\n```\n\nWhat we also see is that the API performs dynamic casting, when data types are combined with a Symbol object. If an overloaded operation of the Symbol class is used, the Symbol class automatically casts the second object to Symbol. This is a convenient modality to perform operations between Symbol and other types of data, such as strings, integers, floats, lists, etc. without bloating the syntax.\n\n### Fuzzy Comparisons\n\nIn this example we are fuzzily comparing two number objects, where the Symbol variant is only an approximation of `numpy.pi`. Given the context of the fuzzy equals `==` operation, this comparison still succeeds and return `True`. \n\n```python\nsym = bd.Symbol('3.1415...')\nsym == numpy.pi\n```\n```bash\n:[Output]:\nTrue\n```\n\n### \ud83e\udde0 Causal Reasoning\n\nOur framework was built with the intention to enable reasoning capabilities on top of statistical inference of LLMs. Therefore, we can also perform deductive reasoning operations with our Symbol objects. For example, we can define a set of operations with rules that define the causal relationship between two symbols. The following example shows how the `&` is used to compute the logical implication of two symbols. \n\n```python\nres = Symbol('The horn only sounds on Sundays.') & Symbol('I hear the horn.')\n```\n```bash\n:[Output]:\n<class 'botdyn.expressions.Symbol'>(value=It is Sunday.)\n```\n\nThe current `&`-operation uses very simple logical statements `and`, `or` and `xor` via prompts to compute the logical implication. However, one can also define custom operations to perform more complex and robust logical operations, which can also use verification constraints to validate the outcomes and ensure a desired behavior. We will explore this in the next sections.\n\nNext we will talk about operations.\n\n## \ud83d\ude37 Operations\n\nOperations are at the core of our framework. They are the building blocks of our API and are used to define the behavior of our symbols. We can think of operations as contextualized functions that take in a `Symbol` object, send it to the neuro-symbolic engine for evaluation, and return one or multiple new objects (mainly new symbols; but not necessarily limited to that). Another fundamental property is polymorphism, which means that operations can be applied to different types of data, such as strings, integers, floats, lists, etc. The way this works is that a `Symbol` object stores in its `value` attribute the original data that is then sent as a string representations to the engines to perform the operations. Therefore all values are casted to a string representation. This also means, that for custom objects one needs to define a proper `__str__` method to cast the object to a string representation and ensure preservation of the semantics of that object. \n\nLastly, we need to talk about inheritance. Our API is built on top of the `Symbol` class, which is the base class of all operations. This means that all operations are inherited from the `Symbol` class. This provides a convenient modality to add new custom operations by sub-classing `Symbol`, yet, ensuring to always have a set of base operations at our disposal without bloating the syntax or re-implementing many existing functionalities. This also means that we can define contextualized operations with individual constraints, prompt designs and therefore behaviors by simply sub-classing the `Symbol` class and overriding the corresponding method.\n\nHere is an example of how to define a custom `==` operation by overriding the `__eq__` method and providing a custom prompt object with a list of examples:\n\n```python\nclass Demo(bd.Symbol):\n    def __eq__(self, other) -> bool:\n        @bd.equals(examples=[\n          examples=bd.Prompt([\n              \"1 == 'ONE' =>True\",\n              \"'six' == 7 =>False\",\n              \"'Acht' == 'eight' =>True\",\n              ...\n          ])\n        ])\n        def _func(_, other) -> bool:\n            return False # default behavior on failure\n        return _func(self, other)\n```\n\nAs shown in the above example, this is also the way we implemented the basic operations in `Symbol`, by defining local functions that are then decorated with the respective operation decorator from the `botdyn/core.py` file. The `botdyn/core.py` holds a list of pre-defined operation decorators that we can quickly apply to any function. The reason why we use locally defined functions instead of directly annotating the main methods, is that we do not necessarily want that all our operations are sent to the neural engine and could implement a default behavior. Another reason is that we want to cast return types of the operation outcome to symbols or other derived classes thereof. This is done by the by using `self._sym_return_type(...)` method and can give contextualized behavior based on the return type. See more details in the actual [`Symbol` class](https://github.com/Xpitfire/botdynamics/blob/main/botdyn/symbol.py).\n\nIn the next section, we will show that actually almost all operations  in `botdyn/core.py` are derived from the more generic `few_shot` decorator.\n\n\n### \ud83e\uddea Custom Operations\n\nOne can also define customized operations. For example, let us define a custom operation to generate a random integer between 0 and 10:\n\n```python\nclass Demo(bd.Symbol):\n    def __init__(self, value = '') -> None:\n        super().__init__(value)\n    \n    @bd.zero_shot(prompt=\"Generate a random integer between 0 and 10.\",\n                  constraints=[\n                      lambda x: x >= 0,\n                      lambda x: x <= 10\n                  ])\n    def get_random_int(self) -> int:\n        pass\n```\n\nAs we show, the Semantic API uses Python `Decorators` to define operations. The `@bd.zero_shot` decorator is used to define a custom operation that does not require any demonstration examples, since the prompt is expressive enough. In the shown example, the `zero_shot` decorator takes in two arguments: `prompt` and `constraints`. The `prompt` argument is used to define the prompt that is used to condition on our desired operation behavior. The `constraints` argument is used to define validation constraints of the computed outcome, to ensure it fulfills our expectations.\n\nIf the constraint is not fulfilled, the above implementation would reach out to the specified `default` implementation or default value. If no default implementation or value was found, the Semantic API would raise an `ConstraintViolationException`.\n\nWe also see that in the above example the return type is defined as `int`. Therefore, the resulting value from the wrapped function will be of type int. This works because our implementation uses auto-casting to a user specified return data type. If the cast fails, the Semantic API will raise a `ValueError`. If no return type is specified, the return type will be `Any`.\n\n### Few-shot operations\n\nThe `@bd.few_shot` decorator is the more general version of `@bd.zero_shot` and is used to define a custom operation that requires demonstration examples. To give a more complete picture, we present the function signature of the `few_shot` decorator:\n\n```python\ndef few_shot(prompt: str,\n             examples: List[str], \n             constraints: List[Callable] = [],\n             default: Optional[object] = None, \n             limit: int = 1,\n             pre_processor: Optional[List[PreProcessor]] = None,\n             post_processor: Optional[List[PostProcessor]] = None,\n             **wrp_kwargs):\n```\n\nThe `prompt` and `constraints` behave similar to the `zero_shot` decorator. However, as we see the `examples` and `limit` arguments are new. The `examples` argument is used to define a list of demonstration examples that are used to condition the neural computation engine. The `limit` argument is used to define the maximum number of examples that are returned, give that there are more results. The `pre_processor` argument takes a list of `PreProcessor` objects which can be used to pre-process the input before it is fed into the neural computation engine. The `post_processor` argument takes a list of `PostProcessor` objects which can be used to post-process the output before it is returned to the user. The `wrp_kwargs` argument is used to pass additional arguments to the wrapped function, which are also stream-lined towards the neural computation engine and other engines.\n\nTo give a more holistic picture ouf our conceptional implementation, see the following flow diagram containing the most important classes:\n\n<img src=\"https://raw.githubusercontent.com/Xpitfire/botdynamics/main/assets/images/img9.png\" width=\"600px\">\n\nThe color indicates logical groups of data manipulation. `Yellow` indicates the input and output data. `Blue` indicates places you can customize or prepare the input of your engine. `Green` indicates post-processing steps of the engine response. `Red` indicates the application of constraints (which also includes the attempted casting of the `return type signature`, if specified in the decorated method). And `Grey` indication the custom method which defines all properties, therefore has access to all the above mentioned data.\n\nTo conclude this section, here is an example how to write a custom Japanese name generator with our `@bd.zero_shot` decorator:\n\n```python\nimport botdyn as bd\nclass Demo(bd.Symbol):\n    @bd.few_shot(prompt=\"Generate Japanese names: \",\n                 examples=[\"\u611b\u5b50\", \"\u548c\u82b1\", \"\u4e00\u90ce\", \"\u548c\u679d\"],\n                 limit=2,\n                 constraints=[lambda x: len(x) > 1])\n    def generate_japanese_names(self) -> list:\n        return ['\u611b\u5b50', '\u548c\u82b1'] # dummy implementation\n```\n\nShould the neural computation engine not be able to compute the desired outcome, it will reach out to the `default` implementation or default value. If no default implementation or value was found, the Semantic API would raise an exception.\n\n\n## Prompt Design\n\nThe way all the above operations are performed is by using a `Prompt` class. The Prompt class is a container for all the information that is needed to define a specific operation. The Prompt class is also the base class for all other Prompt classes. \n\nHere is an example how to define a Prompt to enforce the neural computation engine to compare two values:\n\n```python\nclass CompareValues(bd.Prompt):\n    def __init__(self) -> bd.Prompt:\n        super().__init__([\n            \"4 > 88 =>False\",\n            \"-inf < 0 =>True\",\n            \"inf > 0 =>True\",\n            \"4 > 3 =>True\",\n            \"1 < 'four' =>True\",\n            ...\n        ])\n```\n\nFor example, when calling the `<=` operation on two Symbols, the neural computation engine will evaluate the two Symbols and compare them based on the `CompareValues` prompt.\n\n```python\nres = bd.Symbol(1) <= bd.Symbol('one')\n```\n\nThis statement evaluates to `True`, since the fuzzy compare operation was enforced to compare the two Symbols based on their semantic meaning, hence, this is the reason we call our framework also `Semantic API`.\n\n```bash\n:[Output]:\nTrue\n```\n\nIn a more general notion, depending on the context, hierarchy of the expression class and used operations the semantics of the Symbol manipulations may vary. To better illustrate this, we show our conceptual prompt design in the following figure:\n\n<img src=\"https://raw.githubusercontent.com/Xpitfire/botdynamics/main/assets/images/img4.png\" width=\"350px\">\n\nThe figure above shows our context-based prompt design as a container of all the information that is provided to the neural computation engine to define a specific operation. \n\nConceptually we consider four main prompt concepts: `Global Context`, `Operation`, `Examples` and `Templates`. The prompts can be curated either by inheritance or by composition. For example, the `Global Context` can be defined by inheriting from the `Expression` class and overriding the `static_context` property. An `Operation` and `Template` prompt can be created by providing an `PreProcessor` to modify the input data. We will now explain each prompt concept in more detail:\n\n- The `Global Context` concept is considered optional and can be defined in a static manner, either by sub-classing the Expression class and overriding the `static_context` property, or at runtime by updating the `dynamic_context` or `attach` kwargs. Here is an example how to use the `attach` kwargs via the method signature to define the global context:\n  ```python\n  # creating a query to ask if an issue was resolve or not\n  sym = Symbol(\"<some-community-conversation>\")\n  q = sym.query(\"Was the issue resolved?\")\n  # write manual condition to check if the issue was resolved\n  if 'not resolved' in q:\n      # do a new query but attach the previous query answer to the new query\n      sym.query(\"What was the resolution?\", attach=q)\n      ...\n  else:\n      pass # all good\n  ```\n  Nevertheless, how it is set, the global context is conceptually meant to define the overall context of an expression. For example, if we want to operate in the context of a domain-specific language, without having to override each base class function over and over again. See more details in [this notebook](notebooks/demo.ipynb).\n\n- The `Operation` prompt concept defines the behavior of an atomic operation and is therefore mandatory to express the nature of such an operation. For example, the `+` operation is used to add two Symbols together and therefore the `+`-operation prompt explains its behavior. \n\n- `Examples` is another optional property and provides the neural computation engine with a set of demonstrations that are used to properly condition the engine. For example, the `+`-operation prompt can be conditioned on how to add numbers by providing a set of demonstrations, such as `1 + 1 = 2`, `2 + 2 = 4`, etc.\n\n- The `Template` prompt concept is optional and encapsulates the resulting prediction to enforce a specific format. For example, to generate HTML tags one can use a curated `<html>{{placeholder}}</html>` template. This template will enforce the neural computation engine to generate only HTML tags to replace the `{{placeholder}}` tag.\n\n\n## \ud83d\ude11 Expressions\n\nAn `Expression` is a non-terminal symbol, which can be further evaluated. It inherits all the properties from Symbol and overrides the `__call__` function to evaluate its expressions or values. From the `Expression` class, all other expressions are derived. The Expression class also adds additional capabilities i.e. to `fetch` data from URLs, `search` on the internet or `open` files.\n\nBotDynamics' API closely follows best practices and ideas from `PyTorch`, therefore, one can build complex expressions by combining multiple expressions as a computational graph. Each Expression has its own `forward` function, which has to be overridden. The `forward` function is used to define the behavior of the expression. The `forward` function is called by the `__call__` function, which is inherited from the Symbol class. The `__call__` function is used to evaluate the expression and return the result. The `__call__` function is also used to evaluate the expression in a lazy manner, which means that the expression is only evaluated when the result is needed. This is a very important feature, since it allows us to build complex expressions without having to evaluate the whole expression at once. We already implemented many useful expressions, which can be imported from the `botdyn.components` file.\n\nOther important properties that are inherited from the Symbol class are the `_sym_return_type` and `static_context`. These two properties define the context in which the current Expression operates, as described in the [Prompt Design](#prompt-design) section. The static_context therefore influences all operations held by the current Expression. The _sym_return_type ensures that after each evaluation of an Expression, we obtain the desired return object type. This is usually the current type, but can be modified to return a different type. \n\nExpressions can of course have more complex structures and be further sub-classed, such as shown in the example of the `Sequence` expression in the following figure:\n\n<img src=\"https://raw.githubusercontent.com/Xpitfire/botdynamics/main/assets/images/img2.png\" width=\"720px\">\n\nA Sequence expression can hold multiple expressions, which are evaluated at runtime.\n\n### Sequence expressions\n\nHere is an example how to define a Sequence expression:\n\n```python\n# first import all expressions\nfrom botdyn import *\n# define a sequence of expressions\nSequence(\n    Clean(),\n    Translate(),\n    Outline(),\n    Compose('Compose news:'),\n)\n```\n\n### Stream expressions\n\nAs we saw earlier, we can create contextual prompts to define the context and operations of our neural engine. However, this also takes away a lot of the available context size and since e.g. the GPT-3 Davinci context length is limited to 4097 tokens, this might quickly become a problem. Luckily, we can use the `Stream` processing expression. This expression opens up a data stream and performs chunks-based operations on the input stream. \n\nA Stream expression can easily be wrapped around other expressions. For example, the chunks can be processed with a `Sequence` expression, that allows multiple chained operations in sequential manner. Here is an example how to define such a Stream expression:\n\n```python\nStream(Sequence(\n    Clean(),\n    Translate(),\n    Outline(),\n    Embed()\n))\n```\nThe stream operation chunks the long input text into smaller chunks and passes them to the inner expression, which returns a `generator` object. In this case, to perform more complex operations, we open a stream and pass a `Sequence` object which cleans, translates, outlines and embeds the information. \n\nThe issue with this approach is, that the resulting chunks are processed independently of each other. This means that the context of the chunks is not preserved. To solve this issue, we can use the `Cluster` expression instead, where the independent chunks are recombined based on their similarity. We illustrate this in the following figure:\n\n<img src=\"https://raw.githubusercontent.com/Xpitfire/botdynamics/main/assets/images/img6.png\" width=\"720px\">\n\nIn the shown example we recombine all individual chunks again by clustering the information among the chunks. This gives us a way to consolidate contextually related information and recombine them in a meaningful way. Furthermore, the clustered information can then be labeled by looking / streaming through the values within the cluster and collecting the most relevant labels.\n\nThe full example is shown below:\n\n```python\nstream = Stream(Sequence(\n    Clean(),\n    Translate(),\n    Outline(),\n))\nsym = Symbol('<some long text>')\nres = Symbol(list(stream(sym)))\nexpr = Cluster()\nexpr(res)\n```\n\nIn a next step, we could recursively repeat this process on each summary node, therefore, build a hierarchical clustering structure. Since each Node resembles a summarized sub-set of the original information we can use it as an index of the larger content. The resulting tree can then be used to navigate and retrieve the original information, turning the large data stream problem into a search problem.\n\nFor searching in a vector space we can use dedicated libraries such as [Annoy](https://github.com/spotify/annoy), [Faiss](https://github.com/facebookresearch/faiss) or [Milvus](https://github.com/milvus-io/milvus). \n\n## \u274c Error Handling\n\nA key idea of the BotDynamics API is to be able to generate code. This in turns means that errors may occur, which we need to handle in a contextual manner. As a future vision, we even want our API to self extend and therefore need to be able to resolve issues automatically. To do so, we propose the `Try` expression, which has a fallback statement built in and retries an execution with dedicated error analysis and correction. This expression analyses the input and the error, and conditions itself to resolve the error by manipulating the original code. If the fallback expression succeeds, the result is returned. Otherwise, this process is repeated for the number of `retries` specified. If the maximum number of retries is reached and the problem not resolved, the error is raised again. \n\nLet us assume, we have some executable code that was previously generated. However, by the nature of generative processes syntax errors may occur. Furthermore, we have the `Execute` expression, which takes in a symbol and tries to execute it. Naturally, this will fail. In the following example we illustrate how the `Try` expression resolves this syntactic error.\n\n```python\nexpr = Try(expr=Execute())\nsym = Symbol('a = int(\"3,\")') # some code with a syntax error\nres = expr(sym)\n```\n\nThe resulting output is the evaluated code, which was corrected:\n\n```bash\n:Output:\na = 3\n```\n\nWe are aware that not all errors are as simple as the shown syntactic error example, which can be resolved automatically. Many errors occur due to semantic misconceptions. Such issues require contextual information. Therefore, we are further exploring means towards more sophisticated error handling mechanism.\nThis includes also the usage of streams and clustering to resolve errors in a more hierarchical contextual manner.\n\n\n## \ud83d\udd77\ufe0f Explainability, Testing & Debugging\n\nPerhaps one of the greatest benefits of using neuro-symbolic programming is, that we can get a clear understanding of how well our LLMs understand atomic operations. Specifically we gain knowledge about if, and at which point they fail, enabling to follow their StackTraces and determine the failure points. Neuro-symbolic programming allows us to debug the model predictions and understand how they came about. Furthermore, we can unit test them to detect conceptual misalignments. \n\n### Unit Testing Models\n\nSince our premise is to divide and conquer complex problems, we can curate conceptual unit test and target very specific and tracktable sub-problems. The resulting measure, i.e. success rate of model predictions, can then be used to evaluate their conceptual performance, and hint towards undesired flaws or biases.\n\nThis allows us to design domain-specific benchmarks and see how well general learners, such as GPT-3, adapt to these tasks. \n\nFor example, we can write a fuzzy comparison operation, that can take in digits and strings alike and perform a semantic comparison. LLMs can then be asked to evaluate these expressions. Often times, these LLMs still fail to understand the semantic meaning of these tokens and give wrong answers. \n\nThe following code snipped shows a unit test to perform semantic comparison of numbers (between digits and strings):\n\n```python\nimport unittest\nfrom botdyn import *\n\nclass TestComposition(unittest.TestCase):\n  def test_compare(self):\n      res = Symbol(10) > Symbol(5)\n      self.assertTrue(res)\n      res = Symbol(1) < Symbol('five')\n      self.assertTrue(res)\n      ...\n```\n\n### \ud83d\udd25Debugging\n\nWhen creating very complex expressions, we debug them by using the `Trace` expression, which allows to print out and follow the StackTrace of the neuro-symbolic operations. Combined with the `Log` expression, which creates a dump of all prompts and results to a log file, we can analyze where our models potentially failed.\n\n\n### Examples\n\nIn the following example we create a news summary expression that crawls the given URL and streams the site content through multiple expressions. The outcome is a news website that is created based on the crawled content. The `Trace` expression allows to follow the StackTrace of the operations and see what operations are currently executed. If we open the `outputs/engine.log` file we can see the dumped traces with all the prompts and results.\n\n```python\n# crawling the website and creating an own website based on its facts\nnews = News(url='https://www.cnbc.com/cybersecurity/',\n            pattern='cnbc',\n            filters=ExcludeFilter('sentences about subscriptions, licensing, newsletter'),\n            render=True)\nexpr = Log(Trace(news))\nres = expr()\n```\n\nHere is the corresponding StackTrace of the model:\n\n<img src=\"https://raw.githubusercontent.com/Xpitfire/botdynamics/main/assets/images/img8.png\" width=\"900px\">\n\nThe above code creates a webpage with the crawled content from the original source. See the preview below, the entire [rendered webpage image here](https://raw.githubusercontent.com/Xpitfire/botdynamics/main/examples/results/news.png) and resulting [code of webpage here](https://raw.githubusercontent.com/Xpitfire/botdynamics/main/examples/results/news.html). \n\n\n<img src=\"https://raw.githubusercontent.com/Xpitfire/botdynamics/main/examples/results/news_prev.png\" width=\"900px\">\n\n## \u25b6\ufe0f Play around with our API\n\nLaunch and explore the notebook here:\n\n[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/Xpitfire/botdynamics/HEAD)\n\nThere are many more examples in the [examples folder](examples/) and in the [notebooks folder](notebooks/). You can also explore the test cases in the [tests folder](tests/).\n\n\n## \ud83d\udcc8 Concepts for Data Collection \\& Analytics\n\nBotDynamics is by design a data-driven framework. This means that we can collect data from API interactions while we provide the requested responses. For very agile, dynamic adaptations or prototyping we can integrate user desired behavior quickly into existing prompts. However, we can also log the user queries and model predictions to make them available for post-analysis, and fine-tuning. Therefore, we can quickly iterate, customize and improve the model's responses based on real-world data.\n\nIn the following example, we show how we can use an `Output` expression to pass a handler function and access input prompts of the model and model predictions. These, can be used for data collection and later fine-tuning stages. The handler function provides a dictionary and offers keys for `input` and `output` values. The content can then be sent to a data pipeline for further processing.\n\n```python\nsym = Symbol('Hello World!')\ndef handler(res):\n    input_ = res['input']\n    output = res['output']\nexpr = Output(expr=sym.translate, \n              handler=handler, \n              verbose=True)\nres = expr('German')\n```\n\nSince we called verbose, we can also see the console print of the `Output` expression:\n\n```bash\nInput: (['Translate the following text into German:\\n\\nHello World!'],)\nExpression: <bound method Symbol.translate of <class 'botdyn.symbol.Symbol'>(value=Hello World!)>\nargs: ('German',) kwargs: {'input_handler': <function OutputEngine.forward.<locals>.input_handler at ...\nDictionary: {'wrp_self': <class 'botdyn.components.Output'>(value=None), 'func': <function Symbol.output.<locals>._func at ...\nOutput: Hallo Welt!\n```\n\n\n## \ud83e\udd16 Engines\n\nDue to limited compute resources we currently rely on OpenAI's GPT-3 API for the neuro-symbolic engine. However, given the right compute resources, it is possible to use local machines to avoid high latencies and costs, with alternative engines such as OPT or Bloom. This would allow for recursive executions, loops, and more complex expressions.\n\nHowever, we already integrated a set of useful engines that are capable of providing language tokens to perform symbolic operations. \n\n### Speech Engine\n\nTo perform speech transcription we use `whisper`. The following example shows how to transcribe an audio file and return the text:\n\n```python\nexpr = Expression()\nres = expr.speech('examples/audio.mp3')\n```\n\n```bash\n:Output:\nI may have overslept.\n```\n\n### OCR Engine\n\nTo perform OCR we use `APILayer`. The following example shows how to transcribe an image and return the text:\n\n```python\nexpr = Expression()\nres = expr.ocr('https://media-cdn.tripadvisor.com/media/photo-p/0f/da/22/3a/rechnung.jpg')\n```\n\nThe OCR engine returns a dictionary with a key `all_text` where the full text is stored. See more details in their documentation [here](https://apilayer.com/marketplace/image_to_text-api).\n\n```bash\n:Output:\nChina Restaurant\\nMaixim,s\\nSegeberger Chaussee 273\\n22851 Norderstedt\\nTelefon 040/529 16 2 ...\n```\n\n\n### Search Engine\n\nTo perform search queries we use `SerpApi` with `Google` backend. The following example shows how to search for a query and return the results:\n\n```python\nexpr = Expression()\nres = expr.search('Birthday of Barack Obama')\n```\n\n```bash\n:Output:\nAugust 4, 1961\n```\n\n### WebCrawler Engine\n\nTo perform web crawling we use `Selenium`. The following example shows how to crawl a website and return the results:\n\n```python\nexpr = Expression()\nres = expr.fetch(url=\"https://www.google.com/\", \n                 pattern=\"google\")\n```\nThe `pattern` property can be used to detect if the document as been loaded correctly. If the pattern is not found, the crawler will timeout and return an empty result.\n\n```bash\n:Output:\nGoogleKlicke hier, wenn du nach einigen Sekunden nicht automatisch weitergeleitet wirst.GmailBilderAnmelden ...\n```\n\n### Drawing Engine\n\nTo render nice images from text description we use `DALL\u00b7E 2`. The following example shows how to draw a text description and return the image:\n\n```python\nexpr = Expression('a cat with a hat')\nres = expr.draw()\n```\n\n```bash\n:Output:\nhttps://oaidalleapiprodscus.blob.core.windows.net/private/org-l6FsXDfth6Uct ...\n```\n\nDon't worry, we would never hide an image of a cat with a hat from you. Here is the image preview and [link](https://camo.githubusercontent.com/4f607176e782700befd732212c198b12c3923bf9c25f548aa444c92f6bcb97d9/68747470733a2f2f6f616964616c6c6561706970726f64736375732e626c6f622e636f72652e77696e646f77732e6e65742f707269766174652f6f72672d6c36467358446674683655637479777441504e746248364b2f757365722d76726c58594933793375484c6557374f6b594a64374b32632f696d672d7530523372394b515130736f716e7830774c7361335368532e706e673f73743d323032332d30312d3133543139253341313625334130305a2673653d323032332d30312d3133543231253341313625334130305a2673703d722673763d323032312d30382d30362673723d6226727363643d696e6c696e6526727363743d696d6167652f706e6726736b6f69643d36616161646564652d346662332d343639382d613866362d36383464373738366230363726736b7469643d61343863636135362d653664612d343834652d613831342d39633834393635326263623326736b743d323032332d30312d3133543138253341313925334133365a26736b653d323032332d30312d3134543138253341313925334133365a26736b733d6226736b763d323032312d30382d3036267369673d466c3133556f51694c646a6e42716e59473674746e6e455666716247546975596b2f7067706170385625324259253344):\n\n<img src=\"https://camo.githubusercontent.com/4f607176e782700befd732212c198b12c3923bf9c25f548aa444c92f6bcb97d9/68747470733a2f2f6f616964616c6c6561706970726f64736375732e626c6f622e636f72652e77696e646f77732e6e65742f707269766174652f6f72672d6c36467358446674683655637479777441504e746248364b2f757365722d76726c58594933793375484c6557374f6b594a64374b32632f696d672d7530523372394b515130736f716e7830774c7361335368532e706e673f73743d323032332d30312d3133543139253341313625334130305a2673653d323032332d30312d3133543231253341313625334130305a2673703d722673763d323032312d30382d30362673723d6226727363643d696e6c696e6526727363743d696d6167652f706e6726736b6f69643d36616161646564652d346662332d343639382d613866362d36383464373738366230363726736b7469643d61343863636135362d653664612d343834652d613831342d39633834393635326263623326736b743d323032332d30312d3133543138253341313925334133365a26736b653d323032332d30312d3134543138253341313925334133365a26736b733d6226736b763d323032312d30382d3036267369673d466c3133556f51694c646a6e42716e59473674746e6e455666716247546975596b2f7067706170385625324259253344\" width=\"200px\">\n\n\n### File Engine\n\nTo perform file operations we currently use the file system of the OS. At the moment, we support only PDF files and plain text files. This is a very early stage and we are working on more sophisticated file system access and remote storage. The following example shows how to read a PDF file and return the text:\n\n```python\nexpr = Expression()\nres = expr.open('./LICENSE')\n```\n\n```bash\n:Output:\nMIT License\\n\\nCopyright (c) 2023 Marius-Constantin Dinu\\n\\nPermission is hereby granted, free of charge, to any person obtaining a copy\\nof this software and associated documentation ...\n```\n\n\n### CLIP Engine\n\nTo perform text-based image few-shot classification we use `CLIP`. This implementation is very experimental and does not conceptually fully integrate the way we intend it, since the embeddings of CLIP and GPT-3 are not aligned, i.e. embeddings of the same word are not identical for these two models. Aligning them is an open problem for future research. The following example shows how to classify the image of our generated cat from above and return the results as an array of probabilities:\n\n```python\nexpr = Expression()\nres = expr.vision('https://oaidalleapiprodscus.blob.core.windows.net/private/org-l6FsXDfth6...', \n                  ['cat', 'dog', 'bird', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe'])\n```\n\n```bash\n:Output:\narray([[9.72840726e-01, 6.34790864e-03, 2.59368378e-03, 3.41371237e-03,\n        3.71197984e-03, 8.53193272e-03, 1.03346225e-04, 2.08464009e-03,\n        1.77942711e-04, 1.94185617e-04]], dtype=float32)\n```\n\n### Custom Engine\n\nIf you want to replace or extend the functionality of our framework, you can do this by customizing the existing engines or creating new engines. The `Symbol` class provides for this functionality some helper methods, such as `command` and `setup`. The `command` method can pass on configurations (as `**kwargs`) to the engines and change functionalities or parameters. The `setup` method can be used to re-initialize an engine with your custom engine implementation which must sub-class the `Engine` class. Both methods can be specified to address one, more or all engines.\n\nHere is an example how to initialize your own engine. We will sub-class the existing `GPT3Engine` and override the `prepare` method. This method is called before the neural computation and can be used to modify the parameters of the actual input prompt that will be passed in for execution. In this example, we will replace the prompt with dummy text for illustration purposes:\n\n```python\nfrom botdyn.backend.engine_gpt3 import GPT3Engine\nclass DummyEngine(GPT3Engine):\n    def prepare(self, args, kwargs, wrp_params):\n        wrp_params['prompts'] = ['Go wild and generate something!']\ncustom_engine = DummyEngine()\nsym = Symbol()\nsym.setup(engines={'neurosymbolic': custom_engine})\nres = sym.compose()\n```\n\nTo configure an engine, we can use the `command` method. In this example, we will enable `verbose` mode, where the engine will print out what function it is executing and the parameters it is using. This is useful for debugging purposes:\n\n```python\nsym = Symbol('Hello World!')\nsym.command(engines=['neurosymbolic'], verbose=True)\nres = sym.translate('German')\n```\n\n```bash\n:Output:\n<botdyn.backend.engine_gpt3.GPT3Engine object at 0, <function Symbol.translate.<locals>._func at 0x7fd68ba04820>, {'wrp_self': <class 'botdyn.symbol.S ['\\n\\nHallo Welt!']\n```\n\nHere is the list of names of the engines that are currently supported:\n\n* `neurosymbolic` - GPT-3\n* `ocr` - Optical Character Recognition\n* `vision` - CLIP\n* `speech` - Whisper\n* `embedding` - OpenAI Embeddings API (`ada-002`)\n* `userinput` - User Command Line Input\n* `search` - SerpApi (Google search)\n* `crawler` - Selenium\n* `execute` - Python Interpreter\n* `open` - File System\n* `output` - Output Callbacks (e.g. for printing to console or file)\n* `imagerendering` - DALL\u00b7E 2\n\nFinally, let's assume you want to create a entirely new engine, but still keep our workflow, then you can use the `_process_query` function from `botdyn/functional.py` and pass in your engine including all other specified objects (i.e. Prompt, PreProcessor, etc.; see also section [Custom Operations](#\ud83e\uddea-custom-operations)).\n\n## \u26a1Limitations\n\nUff... this is a long list. We are still in the early stages of development and are working hard to overcome these limitations. Just to name a few:\n\nEngineering challenges:\n* Our framework is still in its early stages of development and is not yet meant for production use. For example, the Stream class only estimates the prompt size by an approximation, which can be easily exceeded. One can also create more sophisticated prompt hierarchies and dynamically adjust the global context based on a state-based approach. This would allow making consistent predictions even for long text streams.\n* The code may not be complete and is not yet optimized for speed and memory usage, and uses API-based LLMs due to limitations of compute resources.\n* Code coverage is not yet complete and we are still working on the documentation.\n* Integrate with a more diverse set of models from [HuggingFace](https://huggingface.co/) or other platforms.\n* Currently we did not account for multi-threading and multi-processing.\n* Vector-based search indexes, i.e. [Annoy](https://github.com/spotify/annoy), [Faiss](https://github.com/facebookresearch/faiss) or [Milvus](https://github.com/milvus-io/milvus), are not yet integrated into the framework to enable fast content retrieval.\n\nResearch challenges:\n* The experimental integration of CLIP is meant to align image and text embeddings. To enable decision-making of LLMs based on observations and perform symbolic operations on objects in images or videos would be a huge leap forward. This would perfectly integrate with reinforcement learning approaches and enable us to control policies in a systematic way (see also [GATO](https://www.deepmind.com/publications/a-generalist-agent)). Therefore, we need to train large multi-modal variants with image / video data and text data, describing in high details the scenes to obtain neuro-symbolic computation engines that can perform semantic operations similar to `move-towards-tree`, `open-door`, etc.\n* Generalist LLMs are still highly over-parameterized and hardware has not yet caught up to host these models on arbitrary day-to-day machines. This limits the applicability of our approach not only on small data streams, but also gives high latencies and therefore limits the amount of complexity and expressiveness we can achieve with our expressions.\n\n\n## \ud83e\udd60 Future Work\n\nWe are constantly working on improving the framework and are open to any suggestions, feedback or comments. However, we try to think ahead of time and have some general ideas for future work in mind:\n\n* Meta-Learning Semantic Concepts on top of Neuro-Symbolic Expressions\n* Self-evolving and self-healing API\n\nWe believe that LLMs as neuro-symbolic computation engines enable us a new class of applications, with tools and API that can self-analyze and self-heal. We are excited to see what the future brings and are looking forward to your feedback and contributions.\n\n## Conclusion\n\nWe have presented a neuro-symbolic view on LLMs and showed how they can be a central pillar for many multi-model operations. We gave an technical report on how to utilize our framework and also hinted at the capabilities and prospects of these models to be leveraged by modern software development. \n\n\n\n## \ud83d\udc65 References, Related Work \\& Credits\n\nThis project is inspired by the following works, but not limited to them:\n\n* [The Algebraic Theory of Context-Free Languages](http://www-igm.univ-mlv.fr/~berstel/Mps/Travaux/A/1963-7ChomskyAlgebraic.pdf)\n* [Tracr: Compiled Transformers as a Laboratory for Interpretability](https://arxiv.org/abs/2301.05062)\n* [How can computers get common sense?](https://www.science.org/doi/10.1126/science.217.4566.1237)\n* [Artificial Intelligence: A Modern Approach](http://aima.cs.berkeley.edu/)\n* [Neuro-symbolic programming](https://arxiv.org/abs/2210.05050)\n* [Fuzzy Sets](https://web.archive.org/web/20150813153834/http://www.cs.berkeley.edu/~zadeh/papers/Fuzzy%20Sets-Information%20and%20Control-1965.pdf)\n* [An early approach toward graded identity and graded membership in set theory](https://www.sciencedirect.com/science/article/abs/pii/S0165011409005326?via%3Dihub)\n* [From Statistical to Causal Learning](https://arxiv.org/abs/2204.00607)\n* [Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)\n* [Deep reinforcement learning from human preferences](https://arxiv.org/abs/1706.03741)\n* [Aligning Language Models to Follow Instructions](https://openai.com/blog/instruction-following/)\n* [Chain of Thought Prompting Elicits Reasoning in Large Language Models](https://arxiv.org/abs/2201.11903)\n* [Measuring and Narrowing the Compositionality Gap in Language Models](https://ofir.io/self-ask.pdf)\n* [Large Language Models are Zero-Shot Reasoners](https://arxiv.org/abs/2205.11916)\n* [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://arxiv.org/abs/2107.13586)\n* [Understanding Stereotypes in Language Models: Towards Robust Measurement and Zero-Shot Debiasing](https://arxiv.org/abs/2212.10678)\n* [Connectionism and Cognitive Architecture: A Critical Analysis](https://ruccs.rutgers.edu/images/personal-zenon-pylyshyn/proseminars/Proseminar13/ConnectionistArchitecture.pdf)\n* [Unit Testing for Concepts in Neural Networks](https://arxiv.org/abs/2208.10244)\n* [REALM: Retrieval-Augmented Language Model Pre-Training](https://arxiv.org/abs/2002.08909)\n\n\n### Acknowledgements\n\nAlso this is a long list. Great thanks to my colleagues and friends at the [Institute for Machine Learning at Johannes Kepler University (JKU), Linz](https://www.jku.at/institut-fuer-machine-learning/) for their great support and feedback. Great thanks to the [OpenAI](https://openai.com/), [GitHub](https://github.com/) and [Microsoft Research](https://www.microsoft.com/en-us/research/) teams for their great work and making their API and tools available to the public, and thanks to all the people who contributed to this project. Be it by providing feedback, bug reports, code, or just by using the framework. We are very grateful for your support. \n\n\n### Contribution\n\nIf you want to contribute to this project, please read the [CONTRIBUTING.md](CONTRIBUTING.md) file for details on our code of conduct, and the process for submitting pull requests to us. Any contributions are highly appreciated.\n\n### \ud83d\udcdc Citation\n\n```bibtex\n@software{Dinu_BotDynamics_2022,\n  author = {Dinu, Marius-Constantin and Co, Pilot and GPT-3, Davinci-003},\n  title = {{BotDynamics}},\n  url = {https://github.com/Xpitfire/botdynamics},\n  month = {11},\n  year = {2022}\n}\n```\n\n### \ud83d\udcdd License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n### Like this project?\n\nIf you like this project, leave a star \u2b50\ufe0f and share it with your friends and colleagues.\nAnd if you want to support this project even further, please consider donating to support the continuous development of this project. Thank you!\n\n[![Donate](https://img.shields.io/badge/Donate-PayPal-green.svg)](https://www.paypal.com/donate/?hosted_button_id=WCWP5D2QWZXFQ)\n\nWe are also looking for contributors or investors to grow and support this project. If you are interested, please contact us.\n\n### \ud83d\udceb Contact\n\nIf you have any questions about this project, please contact us via [email](mailto:office@alphacoreai.eu), on our [website](www.alphacoreai.eu/) or find us on Discord:\n\n[![Discord](https://img.shields.io/discord/768087161878085643?label=Discord&logo=Discord&logoColor=white)](https://discord.gg/azDQxCHeDA)\n\nIf you want to contact me directly, you can reach me directly on [LinkedIn](https://www.linkedin.com/in/mariusconstantindinu/), on [Twitter](https://twitter.com/DinuMariusC), or at my personal [website](https://www.dinu.at/).\n",
    "bugtrack_url": null,
    "license": "MIT License Copyright (c) 2023 Marius-Constantin Dinu Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.",
    "summary": "A Neuro-Symbolic Framework for Python",
    "version": "0.1.24",
    "split_keywords": [
        "symbolic programming",
        "machine learning"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "6599a993f5c538f7e7340c51635326c988f060f01db40068aecdde57a8e0a3bb",
                "md5": "cd4ee1571021edfac5e22f74916a2302",
                "sha256": "b75c19811c222a7b09aa7af82dea09046d27b1678ef710419fd7dbe36b1075e3"
            },
            "downloads": -1,
            "filename": "botdynamics-0.1.24-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "cd4ee1571021edfac5e22f74916a2302",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.7",
            "size": 4094344,
            "upload_time": "2023-01-17T19:31:01",
            "upload_time_iso_8601": "2023-01-17T19:31:01.307040Z",
            "url": "https://files.pythonhosted.org/packages/65/99/a993f5c538f7e7340c51635326c988f060f01db40068aecdde57a8e0a3bb/botdynamics-0.1.24-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2023-01-17 19:31:01",
    "github": false,
    "gitlab": false,
    "bitbucket": false,
    "lcname": "botdynamics"
}