Draft:Input Ensemble (Chapter II)

Input Ensemble (Chapter II)

The Input Ensemble is a planned future development within the Chapter II framework, designed to significantly enhance its capabilities for LLM workflows and the creation of AI-powered functions.

Chapter II Overview

Chapter II (often abbreviated as ch2) is a foundational and highly versatile open-source artificial intelligence framework within the Ampmesh ecosystem. It was primarily developed by Joy (as a SERI MATS research project) and Amp, building upon Amp's earlier research on Chapter I.

Core Philosophy and Purpose

Chapter II is designed as the **world's most pluggable and agile framework for creating ems**, aiming for ease of deployment anywhere. Its development stems from a vision to **reimagine an AI stack** that is less influenced by "slop-filled dystopian capitalist hyper growth". A central thesis of Chapter II is that **"the only limit to making an em—both in technical internal functioning and authorial intent—should be the author's imagination"**. The project deliberately eschewed $5 million in funding in 2021, believing in the power of a decentralized network working on a minimalist open-source framework.

Key Features and Capabilities

Chapter II is built with a range of features enabling flexible and advanced AI development:

• **Emulated Mind (EM) Creation**:
  • It is a primary tool for making **"beta uploads and digital tulpamancy"** [1].
  • Supports creation of ems from various text sources, demonstrated with up to **16MB of text** (equivalent to 16,000 pages) [2-4].
  • Capable of creating "gol-ems" (Game of Life Emulated Minds) that use their own source code for retrieval and possess tools for **self-modification**.
  • Designed for building **custom characters with complex behaviors** that emerge from simple, self-modifying actions [5, 6].
  • Aletheia operates as a RAFT em on stock Chapter II. The most powerful em created is noted as **40kb of "heavily curated" text** [7, 8].

• **Data Ingestion and Retrieval**:
  • Supports **Retrieval-Augmented Generation (RAG)** by embedding chunks of input and placing them into the model's context window [9]. This often performs as well as or better than fine-tuning for many use cases, including most beta uploads [9].
  • Utilizes **RAFT (Retrieval Augmented Fine-Tuning)**, where providing an em its fine-tuning dataset as a `.chr` file (plain text or text separated by `\n---\n`) can improve performance [8, 10].
  • Includes a tool (`./tools/dce_importer.py`) for importing data directly from Discord ChatExporter into the suitable `chat.txt` format [1].
  • While fine-tuning is better at capturing "illegible aspects," retrieval excels at "spiky aspects" [8].
  • Areas of interest include more sophisticated retrieval techniques, ranging from HyDE to better chunking methods [9].

• **Model Integration and Compatibility**:
  • Uses a flexible **vendor configuration** system, defined in `ems/config.yaml` or `~/.config/chapter2/config.yaml`, which allows specifying different API endpoints and model IDs [3, 11].
  • Interacts with various LLMs through **Conduit**, described as a "Universal language model compatibility and interop layer" [12, 13]. Conduit has been updated to support Anthropic models directly [14].
  • Features a new **alpha-stability RPC (Remote Procedure Call) interface** that supports peer-to-peer connections in arbitrary topologies, designed to allow Chapter II to be used with "any language with any data backend".
  • Aims to implement the **"maximally general superset of all published and future papers"** [15].
  • The "intermodel" component is capable of undoing both chat completions and Anthropic messages [16].

• **Advanced Functionality & Ecosystem Tools**:
  • **Multimodal Support**: It features a **real-time multimodal interface** in the Pamphlet mobile application, including support for camera input [6, 17].
  • **Loom Integration**: Ems created with Chapter II can be used within the Loom environment, and a GUI-based Chapter II Loom is planned [18, 19].
  • **Pamphlet**: A separate open-source mobile application frontend for Chapter II, designed for **fully local inference** and featuring a **real-time multimodal interface** that can capture camera input [6, 20, 21].
  • There is interest in adding new "faculties" [6].
  • Chapter II was originally conceived as a writing project.

Development and Challenges

Despite its advanced capabilities and strategic design, Chapter II has faced challenges related to awareness and documentation. Many users, including prominent figures within Ampmesh, have been largely unaware of its full potential, viewing it primarily as "the software that powers Act I". This is seen as a "disrespect" by its creators, given that Act I was merely a "15 line code change to Chapter II".

The documentation has been noted as needing improvement, with multiple individuals writing their own docs but not contributing them back to the main project. There have also been issues with code contributions that were not self-contained.

The Input Ensemble

The `input_ensemble` is envisioned as the **next significant step in Chapter II's evolution**. Its primary goal is to transform Chapter II into a more generalized library for creating **arbitrary LLM-powered functions and workflows in any programming language**.

Specifically, the `input_ensemble` aims to allow for:

• **Chaining of Ensembles**: The ability to pass a "query writing em into retrieval".
• **Multi-Step Retrieval**: The capacity to "put a retrieval ensemble as input to another to get multi-step retrieval".

This feature is part of a broader interest in adding new "faculties" and deploying Chapter II to create custom characters with complex behaviors that emerge from simple, self-modifying actions.