Publish

- jazz-example-pets@0.0.4
 - jazz-example-todo@0.0.29
 - cojson@0.2.0
 - cojson-simple-sync@0.2.0
 - cojson-storage-sqlite@0.2.0
 - jazz-browser@0.2.0
 - jazz-browser-auth-local@0.2.0
 - jazz-browser-media-images@0.2.0
 - jazz-react@0.2.0
 - jazz-react-auth-local@0.2.0
 - jazz-react-media-images@0.2.0
 - jazz-storage-indexeddb@0.2.0

.gitignore

@@ -1,3 +1,4 @@
 node_modules
 yarn-error.log
-lerna-debug.log
+lerna-debug.log
+docsTmp

README.md

@@ -47,7 +47,7 @@ The best example of Jazz is currently the Todo List app.
 - Live version: https://example-todo.jazz.tools
 - Source code: [`./examples/todo`](./examples/todo). See the README there for a walk-through and running instructions.
 
-# API Reference
+# Documentation
 
 Note: Since it's early days, this is the only source of documentation so far.
 
@@ -55,11 +55,11 @@ If you want to build something with Jazz, [join the Jazz Discord](https://discor
 
 ## Overview: Main Packages
 
-**`cojson`**
+**`cojson`** → [DOCS](./DOCS.md#cojson)
 
 A library implementing abstractions and protocols for "Collaborative JSON". This will soon be standardized and forms the basis of secure telepathic data.
 
-**`jazz-react`**
+**`jazz-react`** → [DOCS](./DOCS.md#jazz-react)
 
 Provides you with everything you need to build react apps around CoJSON, including reactive hooks for telepathic data, local IndexedDB persistence, support for different auth providers and helpers for simple invite links for CoJSON groups.
 
@@ -70,7 +70,7 @@ Provides you with everything you need to build react apps around CoJSON, includi
 
 A generic CoJSON sync server you can run locally if you don't want to use Jazz Global Mesh (the default sync backend, at `wss://sync.jazz.tools`)
 
-**`jazz-browser`**
+**`jazz-browser`**  → [DOCS](./DOCS.md#jazz-browser)
 
 framework-agnostic primitives that allow you to use CoJSON in the browser. Used to implement `jazz-react`, will be used to implement bindings for other frameworks in the future.
 
@@ -79,410 +79,4 @@ framework-agnostic primitives that allow you to use CoJSON in the browser. Used
 **`jazz-storage-indexeddb`**
 
 Provides local, offline-capable persistence. Included and enabled in `jazz-react` by default.
-</small>
-
-## `CoJSON`
-
-CoJSON is the core implementation of secure telepathic data. It provides abstractions for Collaborative JSON values ("`CoValues`"), groups for permission management and a protocol for syncing between nodes. Our goal is to standardise CoJSON soon and port it to other languages and platforms.
-
----
-
-### `LocalNode`
-
-A `LocalNode` represents a local view of a set of loaded `CoValue`s, from the perspective of a particular account (or primitive cryptographic agent).
-
-A `LocalNode` can have peers that it syncs to, for example some form of local persistence, or a sync server, such as `sync.jazz.tools` (Jazz Global Mesh).
-
-You typically get hold of a `LocalNode` using `jazz-react`'s `useJazz()`:
-
-```typescript
-const { localNode } = useJazz();
-```
-
-#### `LocalNode.load(id)`
-```typescript
-load<T extends ContentType>(id: CoID<T>): Promise<T>
-```
-
-Loads a CoValue's content, syncing from peers as necessary and resolving the returned promise once a first version has been loaded. See `ContentType.subscribe()` and `useTelepathicData` for listening to subsequent updates to the CoValue.
-
-#### `LocalNode.loadProfile(id)`
-```typescript
-loadProfile(accountID: AccountID): Promise<Profile>
-```
-
-Loads a profile associated with an account. `Profile` is at least a `CoMap<{string: name}>`, but might contain other, app-specific properties.
-
-#### `LocalNode.acceptInvite(valueOrGroup, inviteSecret)`
-```typescript
-acceptInvite<T extends ContentType>(
-    valueOrGroup: CoID<T>,
-    inviteSecret: InviteSecret
-): Promise<void>
-```
-
-Accepts an invite for a group, or infers the group if given the `CoID` of a value owned by that group. Resolves upon successful joining of that group, at which point you should be able to `LocalNode.load` the value.
-
-Invites can be created with `Group.createInvite(role)`.
-
-#### `LocalNode.createGroup()`
-```typescript
-createGroup(): Group
-```
-
-Creates a new group (with the current account as the group's first admin).
-
----
-
-### `Group`
-
-A CoJSON group manages permissions of its members. A `Group` object exposes those capabilities and allows you to create new CoValues owned by that group.
-
-(Internally, a `Group` is also just a `CoMap`, mapping member accounts to roles and containing some state management for making cryptographic keys available to current members)
-
-#### `Group.id`
-
-Returns the `CoID` of the `Group`.
-
-#### `Group.roleOf(accountID)`
-
-```typescript
-roleOf(accountID: AccountID): "reader" | "writer" | "admin" | undefined
-```
-
-Returns the current role of a given account.
-
-#### `Group.myRole()`
-
-```typescript
-myRole(accountID: AccountID): "reader" | "writer" | "admin" | undefined
-```
-
-Returns the role of the current account in the group.
-
-#### `Group.addMember(accountID, role)`
-
-```typescript
-addMember(
-  accountID: AccountID,
-  role:  "reader" | "writer" | "admin"
-)
-```
-
-Directly grants a new member a role in the group. The current account must be an admin to be able to do so. Throws otherwise.
-
-#### `Group.createInvite(role)`
-
-```typescript
-createInvite(role: "reader" | "writer" | "admin"): InviteSecret
-```
-
-Creates an invite for new members to indirectly join the group, allowing them to grant themselves the specified role with the InviteSecret (a string starting with "inviteSecret_") - use `LocalNode.acceptInvite()` for this purpose.
-
-#### `Group.removeMember(accountID)`
-
-```typescript
-removeMember(accountID: AccountID)
-```
-
-Strips the specified member of all roles (preventing future writes) and rotates the read encryption key for that group (preventing reads of new content, including in covalues owned by this group)
-
-#### `Group.createMap(meta?)`
-```typescript
-createMap<M extends CoMap<{ [key: string]: JsonValue }, JsonObject | null>>(
-  meta?: M["meta"]
-): M
-```
-
-Creates a new `CoMap` within this group, with the specified specialized `CoMap` type `M` and optional static metadata.
-
-#### `Group.createList(meta?)`
-```typescript
-createList<L extends CoList<JsonValue, JsonObject | null>>(
-  meta?: L["meta"]
-): L
-```
-
-Creates a new `CoList` within this group, with the specified specialized `CoList` type `L` and optional static metadata.
-
-#### `Group.createStream(meta?)` (coming soon)
-#### `Group.createStatic(meta)` (coming soon)
-
----
-### `CoValue` ContentType: `CoMap`
-
-```typescript
-class CoMap<
-    M extends { [key: string]: JsonValue; },
-    Meta extends JsonObject | null = null,
->
-```
-
-#### `CoMap.id`
-
-```typescript
-id: CoID<CoMap<M, Meta>>
-```
-
-Returns the CoMap's (precisely typed) `CoID`
-
-#### `CoMap.meta`
-
-```typescript
-meta: Meta
-```
-
-Returns the CoMap's (precisely typed) static metadata
-
-#### `CoMap.keys()`
-
-```typescript
-keys(): (keyof M & string)[]
-```
-
-#### `CoMap.get(key)`
-
-```typescript
-get<K extends keyof M>(key: K): M[K] | undefined
-```
-
-Returns the current value for the given key.
-
-#### `CoMap.whoEdited(key)`
-
-```typescript
-whoEdited<K extends keyof M>(key: K): AccountID | undefined
-```
-
-Returns the accountID of the last account to modify the value for the given key.
-
-#### `CoMap.toJSON()`
-
-```typescript
-toJSON(): JsonObject
-```
-
-Returns a JSON representation of the state of the CoMap.
-
-#### `CoMap.subscribe(listener)`
-
-```typescript
-subscribe(
-  listener: (coMap: CoMap<M, Meta>) => void
-): () => void
-```
-Lets you subscribe to future updates to this CoMap (whether made locally or by other users). Takes a listener function that will be called with the current state for each update. Returns an unsubscribe function.
-
-Used internally by `useTelepathicData()` for reactive updates on changes to a `CoMap`.
-
-#### `CoMap.edit(editable => {...})`
-
-```typescript
-edit(changer: (editable: WriteableCoMap<M, Meta>) => void): CoMap<M, Meta>
-```
-
-Lets you apply edits to a `CoMap`, inside the changer callback, which receives a `WriteableCoMap`. A `WritableCoMap` has all the same methods as a `CoMap`, but all edits made to it with `set` or `delete` are reflected in it immediately - so it behaves mutably, whereas a `CoMap` is always immutable (you need to use `subscribe` to receive new versions of it).
-
-```typescript
-export class WriteableCoMap<
-    M extends { [key: string]: JsonValue; },
-    Meta extends JsonObject | null = null,
-> extends CoMap<M, Meta>
-```
-
-#### `WritableCoMap.set(key, value)`
-
-```typescript
-set<K extends keyof M>(
-  key: K,
-  value: M[K],
-  privacy: "private" | "trusting" = "private"
-): void
-```
-
-Sets a new value for the given key.
-
-If `privacy` is `"private"` **(default)**, both `key` and `value` are encrypted in the transaction, only readable by other members of the group this `CoMap` belongs to. Not even sync servers can see the content in plaintext.
-
-If `privacy` is `"trusting"`, both `key` and `value` are stored in plaintext in the transaction, visible to everyone who gets a hold of it, including sync servers.
-
-#### `WritableCoMap.delete(key)`
-
-```typescript
-delete<K extends keyof M>(
-  key: K,
-  privacy: "private" | "trusting" = "private"
-): void
-```
-
-Deletes the value for the given key (setting it to undefined).
-
-If `privacy` is `"private"` **(default)**, `key` is encrypted in the transaction, only readable by other members of the group this `CoMap` belongs to. Not even sync servers can see the content in plaintext.
-
-If `privacy` is `"trusting"`, `key` is stored in plaintext in the transaction, visible to everyone who gets a hold of it, including sync servers.
-
----
-### `CoValue` ContentType: `CoList`
-
-```typescript
-class CoList<
-    T extends JsonValue,
-    Meta extends JsonObject | null = null
->
-```
-
-#### `CoList.id`
-
-```typescript
-id: CoID<CoList<T, Meta>>
-```
-
-Returns the CoList's (precisely typed) `CoID`
-
-#### `CoList.meta`
-
-```typescript
-meta: Meta
-```
-
-Returns the CoList's (precisely typed) static metadata
-
-### `CoList.asArray()`
-
-```typescript
-asArray(): T[]
-```
-
-Returns the current items in the CoList as an array.
-
-### `CoList.toJSON()`
-
-```typescript
-toJSON(): T[]
-```
-
-Returns the current items in the CoList as an array. (alias of asArray)
-
-#### `CoList.whoInserted(idx)`
-
-```typescript
-whoInserted(idx: number): AccountID | undefined
-```
-
-Returns the accountID of the account that inserted value at the given index.
-
-#### `CoList.subscribe(listener)`
-
-```typescript
-subscribe(
-  listener: (coMap: CoList<T, Meta>) => void
-): () => void
-```
-Lets you subscribe to future updates to this CoList (whether made locally or by other users). Takes a listener function that will be called with the current state for each update. Returns an unsubscribe function.
-
-Used internally by `useTelepathicData()` for reactive updates on changes to a `CoList`.
-
-#### `CoList.edit(editable => {...})`
-
-```typescript
-edit(changer: (editable: WriteableCoList<T, Meta>) => void): CoList<T, Meta>
-```
-
-Lets you apply edits to a `CoList`, inside the changer callback, which receives a `WriteableCoList`. A `WritableCoList` has all the same methods as a `CoList`, but all edits made to it with `append`, `push`, `prepend` or `delete` are reflected in it immediately - so it behaves mutably, whereas a `CoList` is always immutable (you need to use `subscribe` to receive new versions of it).
-
-```typescript
-export class WriteableCoList<
-    T extends JsonValue,
-    Meta extends JsonObject | null = null,
-> extends CoList<T, Meta>
-```
-
-#### `WritableCoList.append(after, value)`
-
-```typescript
-append(
-  after: number,
-  value: T,
-  privacy: "private" | "trusting" = "private"
-): void
-```
-
-Appends a new item after index `after`.
-
-If `privacy` is `"private"` **(default)**, both `value` is encrypted in the transaction, only readable by other members of the group this `CoList` belongs to. Not even sync servers can see the content in plaintext.
-
-If `privacy` is `"trusting"`, both `value` is stored in plaintext in the transaction, visible to everyone who gets a hold of it, including sync servers.
-
-#### `WritableCoList.prepend(after, value)`
-
-```typescript
-prepend(
-  before: number,
-  value: T,
-  privacy: "private" | "trusting" = "private"
-): void
-```
-
-Prepends a new item before index `before`.
-
-If `privacy` is `"private"` **(default)**, both `value` is encrypted in the transaction, only readable by other members of the group this `CoList` belongs to. Not even sync servers can see the content in plaintext.
-
-If `privacy` is `"trusting"`, both `value` is stored in plaintext in the transaction, visible to everyone who gets a hold of it, including sync servers.
-
-#### `WritableCoList.push(value)`
-
-```typescript
-push(
-  value: T,
-  privacy: "private" | "trusting" = "private"
-): void
-```
-
-Pushes a new item to the end of the list.
-
-If `privacy` is `"private"` **(default)**, both `value` is encrypted in the transaction, only readable by other members of the group this `CoList` belongs to. Not even sync servers can see the content in plaintext.
-
-If `privacy` is `"trusting"`, both `value` is stored in plaintext in the transaction, visible to everyone who gets a hold of it, including sync servers.
-
-#### `WritableCoList.delete(at)`
-
-```typescript
-delete(
-    at: number,
-    privacy: "private" | "trusting" = "private"
-): void
-```
-
-Deletes the item at index `at` from the list.
-
-If `privacy` is `"private"` **(default)**, the fact of this deletion is encrypted in the transaction, only readable by other members of the group this `CoList` belongs to. Not even sync servers can see the content in plaintext.
-
-If `privacy` is `"trusting"`, the fact of this deletion is stored in plaintext in the transaction, visible to everyone who gets a hold of it, including sync servers.
-
----
-### `CoValue` ContentType: `CoStream` (not yet implemented)
-
----
-### `CoValue` ContentType: `Static` (not yet implemented)
----
-
-## `jazz-react`
----
-### `<WithJazz>`
-
-Not yet documented, see [`examples/todo`](./examples/todo/) for now
-
----
-### `useJazz()`
-
-Not yet documented, see [`examples/todo`](./examples/todo/) for now
-
----
-### `useTelepathicData(coID)`
-
-Not yet documented, see [`examples/todo`](./examples/todo/) for now
-
----
-### `useProfile(accountID)`
-
-Not yet documented, see [`examples/todo`](./examples/todo/) for now
+</small>

examples/pets/.eslintrc.cjs

@@ -0,0 +1,18 @@
+module.exports = {
+  root: true,
+  env: { browser: true, es2020: true },
+  extends: [
+    'eslint:recommended',
+    'plugin:@typescript-eslint/recommended',
+    'plugin:react-hooks/recommended',
+  ],
+  ignorePatterns: ['dist', '.eslintrc.cjs'],
+  parser: '@typescript-eslint/parser',
+  plugins: ['react-refresh'],
+  rules: {
+    'react-refresh/only-export-components': [
+      'warn',
+      { allowConstantExport: true },
+    ],
+  },
+}

examples/pets/.gitignore

@@ -0,0 +1,24 @@
+# Logs
+logs
+*.log
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+pnpm-debug.log*
+lerna-debug.log*
+
+node_modules
+dist
+dist-ssr
+*.local
+
+# Editor directories and files
+.vscode/*
+!.vscode/extensions.json
+.idea
+.DS_Store
+*.suo
+*.ntvs*
+*.njsproj
+*.sln
+*.sw?

examples/pets/Dockerfile

@@ -0,0 +1,4 @@
+FROM caddy:2.7.3-alpine
+LABEL org.opencontainers.image.source="https://github.com/gardencmp/jazz"
+
+COPY ./dist /usr/share/caddy/

examples/pets/README.md

@@ -0,0 +1,65 @@
+# Jazz Todo List Example
+
+Live version: https://example-todo.jazz.tools
+
+## Installing & running the example locally
+
+Start by checking out just the example app to a folder:
+
+```bash
+npx degit gardencmp/jazz/examples/todo jazz-example-todo
+cd jazz-example-todo
+```
+
+(This ensures that you have the example app without git history or our multi-package monorepo)
+
+Install dependencies:
+
+```bash
+npm install
+```
+
+Start the dev server:
+
+```bash
+npm run dev
+```
+
+## Structure
+
+- [`src/basicComponents`](./src/basicComponents) contains simple components to build the UI, unrelated to Jazz (powered by [shadcn/ui](https://ui.shadcn.com))
+- [`src/components`](./src/components/) contains helper components that do contain Jazz-specific logic, but are not super relevant to understand the basics of Jazz and CoJSON
+- [`src/0_main.tsx`](./src/0_main.tsx), [`src/1_types.ts`](./src/1_types.ts), [`src/2_App.tsx`](./src/2_App.tsx), [`src/3_TodoTable.tsx`](./src/3_TodoTable.tsx), [`src/router.ts`](./src/router.ts) - the main files for this example, see the walkthrough below
+
+## Walkthrough
+
+### Main parts
+
+- The top-level provider `<WithJazz/>`: [`src/0_main.tsx`](./src/0_main.tsx)
+
+- Defining the data model with CoJSON: [`src/1_types.ts`](./src/1_types.ts)
+
+- Creating todo projects & routing in `<App/>`: [`src/2_App.tsx`](./src/2_App.tsx)
+
+- Reactively rendering a todo project as a table, adding and editing tasks: [`src/3_TodoTable.tsx`](./src/3_TodoTable.tsx)
+
+### Helpers
+
+- Getting user profiles in `<NameBadge/>`: [`src/components/NameBadge.tsx`](./src/components/NameBadge.tsx)
+
+- (not yet commented) Creating invite links/QR codes with `<InviteButton/>`: [`src/components/InviteButton.tsx`](./src/components/InviteButton.tsx)
+
+- (not yet commented) `location.hash`-based routing and accepting invite links with `useSimpleHashRouterThatAcceptsInvites()` in [`src/router.ts`](./src/router.ts)
+
+This is the whole Todo List app!
+
+## Questions / problems / feedback
+
+If you have feedback, let us know on [Discord](https://discord.gg/utDMjHYg42) or open an issue or PR to fix something that seems wrong.
+
+
+## Configuration: sync server
+
+By default, the example app uses [Jazz Global Mesh](https://jazz.tools/mesh) (`wss://sync.jazz.tools`) - so cross-device use, invites and collaboration should just work.
+
+You can also run a local sync server by running `npx cojson-simple-sync` and adding the query param `?sync=ws://localhost:4200` to the URL of the example app (for example: `http://localhost:5173/?sync=ws://localhost:4200`), or by setting the `sync` parameter of the `<WithJazz>` provider component in [./src/0_main.tsx](./src/0_main.tsx).

examples/pets/components.json

@@ -0,0 +1,16 @@
+{
+  "$schema": "https://ui.shadcn.com/schema.json",
+  "style": "default",
+  "rsc": false,
+  "tsx": true,
+  "tailwind": {
+    "config": "tailwind.config.js",
+    "css": "src/index.css",
+    "baseColor": "stone",
+    "cssVariables": true
+  },
+  "aliases": {
+    "components": "@/basicComponents",
+    "utils": "@/basicComponents/lib/utils"
+  }
+}

examples/pets/index.html

@@ -0,0 +1,13 @@
+<!doctype html>
+<html lang="en">
+  <head>
+    <meta charset="UTF-8" />
+    <link rel="icon" type="image/png" href="/jazz-logo.png" />
+    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <title>Jazz Rate My Pet Example</title>
+  </head>
+  <body>
+    <div id="root"></div>
+    <script type="module" src="/src/0_main.tsx"></script>
+  </body>
+</html>

examples/pets/job-template.nomad

@@ -0,0 +1,56 @@
+job "example-todo$BRANCH_SUFFIX" {
+  region     = "global"
+  datacenters = ["*"]
+
+  group "static" {
+    count = 8
+
+    network {
+      port "http" {
+        to = 80
+      }
+    }
+
+    constraint {
+      attribute = "${node.class}"
+      operator  = "="
+      value     = "mesh"
+    }
+
+    spread {
+      attribute = "${node.datacenter}"
+      weight    = 100
+    }
+
+    constraint {
+      distinct_hosts = true
+    }
+
+    task "server" {
+      driver = "docker"
+
+      config {
+        image = "$DOCKER_TAG"
+        ports = ["http"]
+
+        auth = {
+          username = "$DOCKER_USER"
+          password = "$DOCKER_PASSWORD"
+        }
+      }
+
+      service {
+        tags = ["public"]
+        name = "example-todo$BRANCH_SUFFIX"
+        port = "http"
+        provider = "consul"
+      }
+
+      resources {
+        cpu    = 50 # MHz
+        memory = 50 # MB
+      }
+    }
+  }
+}
+# deploy bump 4

examples/pets/package.json

@@ -0,0 +1,45 @@
+{
+  "name": "jazz-example-pets",
+  "private": true,
+  "version": "0.0.4",
+  "type": "module",
+  "scripts": {
+    "dev": "vite",
+    "build": "tsc && vite build",
+    "lint": "eslint . --ext ts,tsx --report-unused-disable-directives --max-warnings 0",
+    "preview": "vite preview"
+  },
+  "dependencies": {
+    "@radix-ui/react-checkbox": "^1.0.4",
+    "@radix-ui/react-slot": "^1.0.2",
+    "@radix-ui/react-toast": "^1.1.4",
+    "@types/qrcode": "^1.5.1",
+    "class-variance-authority": "^0.7.0",
+    "clsx": "^2.0.0",
+    "jazz-react": "^0.2.0",
+    "jazz-react-auth-local": "^0.2.0",
+    "jazz-react-media-images": "^0.2.0",
+    "lucide-react": "^0.274.0",
+    "qrcode": "^1.5.3",
+    "react": "^18.2.0",
+    "react-dom": "^18.2.0",
+    "tailwind-merge": "^1.14.0",
+    "tailwindcss-animate": "^1.0.7",
+    "uniqolor": "^1.1.0"
+  },
+  "devDependencies": {
+    "@types/react": "^18.2.15",
+    "@types/react-dom": "^18.2.7",
+    "@typescript-eslint/eslint-plugin": "^6.0.0",
+    "@typescript-eslint/parser": "^6.0.0",
+    "@vitejs/plugin-react-swc": "^3.3.2",
+    "autoprefixer": "^10.4.14",
+    "eslint": "^8.45.0",
+    "eslint-plugin-react-hooks": "^4.6.0",
+    "eslint-plugin-react-refresh": "^0.4.3",
+    "postcss": "^8.4.27",
+    "tailwindcss": "^3.3.3",
+    "typescript": "^5.0.2",
+    "vite": "^4.4.5"
+  }
+}

examples/pets/postcss.config.js

@@ -0,0 +1,6 @@
+export default {
+  plugins: {
+    tailwindcss: {},
+    autoprefixer: {},
+  },
+}

examples/pets/src/0_main.tsx

@@ -0,0 +1,38 @@
+import React from "react";
+import ReactDOM from "react-dom/client";
+import "./index.css";
+
+import { WithJazz } from "jazz-react";
+import { LocalAuth } from "jazz-react-auth-local";
+
+import { ThemeProvider, TitleAndLogo } from "./basicComponents/index.ts";
+import { PrettyAuthUI } from "./components/Auth.tsx";
+import App from "./2_App.tsx";
+
+/** Walkthrough: The top-level provider `<WithJazz/>`
+ *
+ *  This shows how to use the top-level provider `<WithJazz/>`,
+ *  which provides the rest of the app with a `LocalNode` (used through `useJazz` later),
+ *  based on `LocalAuth` that uses PassKeys (aka WebAuthn) to store a user's account secret
+ *  - no backend needed. */
+
+const appName = "Jazz Rate My Pet Example";
+
+const auth = LocalAuth({
+    appName,
+    Component: PrettyAuthUI,
+});
+
+ReactDOM.createRoot(document.getElementById("root")!).render(
+    <React.StrictMode>
+        <ThemeProvider>
+            <TitleAndLogo name={appName} />
+
+            <WithJazz auth={auth}>
+                <App />
+            </WithJazz>
+        </ThemeProvider>
+    </React.StrictMode>
+);
+
+/** Walkthrough: Continue with ./1_types.ts */

examples/pets/src/1_types.ts

@@ -0,0 +1,29 @@
+import { CoMap, CoID, CoStream, Media } from "cojson";
+
+/** Walkthrough: Defining the data model with CoJSON
+ *
+ *  Here, we define our main data model of TODO
+ *
+ *  TODO
+ **/
+
+export type PetPost = CoMap<{
+    name: string;
+    image: CoID<Media.ImageDefinition>;
+    reactions: CoID<PetReactions>;
+}>;
+
+export const REACTION_TYPES = [
+    "aww",
+    "love",
+    "haha",
+    "wow",
+    "tiny",
+    "chonkers",
+] as const;
+
+export type ReactionType = (typeof REACTION_TYPES)[number];
+
+export type PetReactions = CoStream<ReactionType>;
+
+/** Walkthrough: Continue with ./2_App.tsx */

examples/pets/src/2_App.tsx

@@ -0,0 +1,48 @@
+import { useJazz } from "jazz-react";
+
+import { PetPost } from "./1_types";
+
+import { Button } from "./basicComponents";
+
+import { useSimpleHashRouterThatAcceptsInvites } from "./router";
+import { RatePetPostUI } from "./4_RatePetPostUI";
+import { CreatePetPostForm } from "./3_CreatePetPostForm";
+
+/** Walkthrough: Creating pet posts & routing in `<App/>`
+ *
+ *  <App> is the main app component, handling client-side routing based
+ *  on the CoValue ID (CoID) of our PetPost, stored in the URL hash
+ *  - which can also contain invite links.
+ */
+
+export default function App() {
+    // A `LocalNode` represents a local view of loaded & created CoValues.
+    // It is associated with a current user account, which will determine
+    // access rights to CoValues. We get it from the top-level provider `<WithJazz/>`.
+    const { localNode, logOut } = useJazz();
+
+    // This sets up routing and accepting invites, skip for now
+    const [currentPetPostID, navigateToPetPostID] =
+        useSimpleHashRouterThatAcceptsInvites<PetPost>(localNode);
+
+    return (
+        <div className="flex flex-col h-full items-center justify-start gap-10 pt-10 pb-10 px-5">
+            {currentPetPostID ? (
+                <RatePetPostUI petPostID={currentPetPostID} />
+            ) : (
+                <CreatePetPostForm onCreate={navigateToPetPostID} />
+            )}
+            <Button
+                onClick={() => {
+                    navigateToPetPostID(undefined);
+                    logOut();
+                }}
+                variant="outline"
+            >
+                Log Out
+            </Button>
+        </div>
+    );
+}
+
+/** Walkthrough: continue with ./3_CreatePetPostForm.tsx */

examples/pets/src/3_CreatePetPostForm.tsx

@@ -0,0 +1,103 @@
+import { ChangeEvent, useCallback, useState } from "react";
+
+import { CoID } from "cojson";
+import { useJazz, useTelepathicState } from "jazz-react";
+import { createImage } from "jazz-browser-media-images";
+
+import { PetPost, PetReactions } from "./1_types";
+
+import { Input, Button } from "./basicComponents";
+import { useLoadImage } from "jazz-react-media-images";
+
+/** Walkthrough: TODO
+ */
+
+export function CreatePetPostForm({
+    onCreate,
+}: {
+    onCreate: (id: CoID<PetPost>) => void;
+}) {
+    const { localNode } = useJazz();
+
+    const [newPostId, setNewPostId] = useState<CoID<PetPost> | undefined>(
+        undefined
+    );
+
+    const newPetPost = useTelepathicState(newPostId);
+
+    const onChangeName = useCallback(
+        (name: string) => {
+            let petPost = newPetPost;
+            if (!petPost) {
+                const petPostGroup = localNode.createGroup();
+                petPost = petPostGroup.createMap<PetPost>();
+                const petReactions = petPostGroup.createStream<PetReactions>();
+
+                petPost = petPost.edit((petPost) => {
+                    petPost.set("reactions", petReactions.id);
+                });
+
+                setNewPostId(petPost.id);
+            }
+
+            petPost.edit((petPost) => {
+                petPost.set("name", name);
+            });
+        },
+        [localNode, newPetPost]
+    );
+
+    const onImageSelected = useCallback(
+        async (event: ChangeEvent<HTMLInputElement>) => {
+            if (!newPetPost || !event.target.files) return;
+
+            const imageDefinition = await createImage(
+                event.target.files[0],
+                newPetPost.group
+            );
+
+            newPetPost.edit((petPost) => {
+                petPost.set("image", imageDefinition.id);
+            });
+        },
+        [newPetPost]
+    );
+
+    const petImage = useLoadImage(newPetPost?.get("image"));
+
+    return (
+        <div className="flex flex-col gap-10">
+            <p>Share your pet with friends!</p>
+            <Input
+                type="text"
+                placeholder="Pet Name"
+                className="text-3xl py-6"
+                onChange={(event) => onChangeName(event.target.value)}
+                value={newPetPost?.get("name") || ""}
+            />
+
+            {petImage ? (
+                <img
+                    className="w-80 max-w-full rounded"
+                    src={petImage.highestResSrc || petImage.placeholderDataURL}
+                />
+            ) : (
+                <Input
+                    type="file"
+                    disabled={!newPetPost?.get("name")}
+                    onChange={onImageSelected}
+                />
+            )}
+
+            {newPetPost?.get("name") && newPetPost?.get("image") && (
+                <Button
+                    onClick={() => {
+                        onCreate(newPetPost.id);
+                    }}
+                >
+                    Submit Post
+                </Button>
+            )}
+        </div>
+    );
+}

examples/pets/src/4_RatePetPostUI.tsx

@@ -0,0 +1,103 @@
+import { AccountID, CoID } from "cojson";
+import { useTelepathicState } from "jazz-react";
+
+import { PetPost, PetReactions, ReactionType, REACTION_TYPES } from "./1_types";
+
+import { ShareButton } from "./components/ShareButton";
+import { NameBadge } from "./components/NameBadge";
+import { Button } from "./basicComponents";
+import { useLoadImage } from "jazz-react-media-images";
+
+/** Walkthrough: TODO
+ */
+
+const reactionEmojiMap: { [reaction in ReactionType]: string } = {
+    aww: "😍",
+    love: "❤️",
+    haha: "😂",
+    wow: "😮",
+    tiny: "🐥",
+    chonkers: "🐘",
+};
+
+export function RatePetPostUI({ petPostID }: { petPostID: CoID<PetPost> }) {
+    const petPost = useTelepathicState(petPostID);
+    const petReactions = useTelepathicState(petPost?.get("reactions"));
+    const petImage = useLoadImage(petPost?.get("image"));
+
+    return (
+        <div className="flex flex-col gap-8">
+            <div className="flex justify-between">
+                <h1 className="text-3xl font-bold">{petPost?.get("name")}</h1>
+                <ShareButton petPost={petPost} />
+            </div>
+
+            {petImage && (
+                <img
+                    className="w-80 max-w-full rounded"
+                    src={petImage.highestResSrc || petImage.placeholderDataURL}
+                />
+            )}
+
+            <div className="flex justify-between max-w-xs flex-wrap">
+                {REACTION_TYPES.map((reactionType) => (
+                    <Button
+                        key={reactionType}
+                        variant={
+                            petReactions?.getLastItemFromMe() === reactionType
+                                ? "default"
+                                : "outline"
+                        }
+                        onClick={() => {
+                            petReactions?.edit((reactions) => {
+                                reactions.push(reactionType);
+                            });
+                        }}
+                        title={`React with ${reactionType}`}
+                        className="text-2xl px-2"
+                    >
+                        {reactionEmojiMap[reactionType]}
+                    </Button>
+                ))}
+            </div>
+
+            {petPost?.group.myRole() === "admin" && petReactions && (
+                <ReactionOverview petReactions={petReactions} />
+            )}
+        </div>
+    );
+}
+
+function ReactionOverview({ petReactions }: { petReactions: PetReactions }) {
+    return (
+        <div>
+            <h2>Reactions</h2>
+            <div className="flex flex-col gap-1">
+                {REACTION_TYPES.map((reactionType) => {
+                    const accountsWithThisReaction = Object.entries(
+                        petReactions.getLastItemsPerAccount()
+                    ).flatMap(([accountID, reaction]) =>
+                        reaction === reactionType ? [accountID] : []
+                    );
+
+                    if (accountsWithThisReaction.length === 0) return null;
+
+                    return (
+                        <div
+                            className="flex gap-2 items-center"
+                            key={reactionType}
+                        >
+                            {reactionEmojiMap[reactionType]}{" "}
+                            {accountsWithThisReaction.map((accountID) => (
+                                <NameBadge
+                                    key={accountID}
+                                    accountID={accountID as AccountID}
+                                />
+                            ))}
+                        </div>
+                    );
+                })}
+            </div>
+        </div>
+    );
+}

examples/pets/src/basicComponents/TitleAndLogo.tsx

@@ -0,0 +1,12 @@
+import { Toaster } from ".";
+
+export function TitleAndLogo({ name }: { name: string }) {
+    return (
+        <>
+            <div className="flex items-center gap-2 justify-center mt-5">
+                <img src="jazz-logo.png" className="h-5" /> {name}
+            </div>
+            <Toaster />
+        </>
+    );
+}

examples/pets/src/basicComponents/index.ts

@@ -0,0 +1,7 @@
+export { Button } from "./ui/button";
+export { Input } from "./ui/input";
+export { Toaster } from "./ui/toaster";
+export { useToast } from "./ui/use-toast";
+export { Skeleton } from "./ui/skeleton";
+export { TitleAndLogo } from "./TitleAndLogo";
+export { ThemeProvider } from "./themeProvider";

examples/pets/src/basicComponents/lib/utils.ts

@@ -0,0 +1,6 @@
+import { type ClassValue, clsx } from "clsx"
+import { twMerge } from "tailwind-merge"
+ 
+export function cn(...inputs: ClassValue[]) {
+  return twMerge(clsx(inputs))
+}

examples/pets/src/basicComponents/themeProvider.tsx

@@ -0,0 +1,72 @@
+import { createContext, useContext, useEffect, useState } from "react";
+
+type ThemeProviderProps = {
+    children: React.ReactNode;
+    defaultTheme?: string;
+    storageKey?: string;
+};
+
+type ThemeProviderState = {
+    theme: string;
+    setTheme: (theme: string) => void;
+};
+
+const initialState = {
+    theme: "system",
+    setTheme: () => null,
+};
+
+const ThemeProviderContext = createContext<ThemeProviderState>(initialState);
+
+export function ThemeProvider({
+    children,
+    defaultTheme = "system",
+    storageKey = "vite-ui-theme",
+    ...props
+}: ThemeProviderProps) {
+    const [theme, setTheme] = useState(
+        () => localStorage.getItem(storageKey) || defaultTheme
+    );
+
+    useEffect(() => {
+        const root = window.document.documentElement;
+
+        root.classList.remove("light", "dark");
+
+        if (theme === "system") {
+            const systemTheme = window.matchMedia(
+                "(prefers-color-scheme: dark)"
+            ).matches
+                ? "dark"
+                : "light";
+
+            root.classList.add(systemTheme);
+            return;
+        }
+
+        root.classList.add(theme);
+    }, [theme]);
+
+    const value = {
+        theme,
+        setTheme: (theme: string) => {
+            localStorage.setItem(storageKey, theme);
+            setTheme(theme);
+        },
+    };
+
+    return (
+        <ThemeProviderContext.Provider {...props} value={value}>
+            {children}
+        </ThemeProviderContext.Provider>
+    );
+}
+
+export const useTheme = () => {
+    const context = useContext(ThemeProviderContext);
+
+    if (context === undefined)
+        throw new Error("useTheme must be used within a ThemeProvider");
+
+    return context;
+};

examples/pets/src/basicComponents/ui/button.tsx

@@ -0,0 +1,56 @@
+import * as React from "react"
+import { Slot } from "@radix-ui/react-slot"
+import { cva, type VariantProps } from "class-variance-authority"
+
+import { cn } from "@/basicComponents/lib/utils"
+
+const buttonVariants = cva(
+  "inline-flex items-center justify-center rounded-md text-sm font-medium ring-offset-background transition-colors focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 disabled:pointer-events-none disabled:opacity-50",
+  {
+    variants: {
+      variant: {
+        default: "bg-primary text-primary-foreground hover:bg-primary/90",
+        destructive:
+          "bg-destructive text-destructive-foreground hover:bg-destructive/90",
+        outline:
+          "border border-input bg-background hover:bg-accent hover:text-accent-foreground",
+        secondary:
+          "bg-secondary text-secondary-foreground hover:bg-secondary/80",
+        ghost: "hover:bg-accent hover:text-accent-foreground",
+        link: "text-primary underline-offset-4 hover:underline",
+      },
+      size: {
+        default: "h-10 px-4 py-2",
+        sm: "h-9 rounded-md px-3",
+        lg: "h-11 rounded-md px-8",
+        icon: "h-10 w-10",
+      },
+    },
+    defaultVariants: {
+      variant: "default",
+      size: "default",
+    },
+  }
+)
+
+export interface ButtonProps
+  extends React.ButtonHTMLAttributes<HTMLButtonElement>,
+    VariantProps<typeof buttonVariants> {
+  asChild?: boolean
+}
+
+const Button = React.forwardRef<HTMLButtonElement, ButtonProps>(
+  ({ className, variant, size, asChild = false, ...props }, ref) => {
+    const Comp = asChild ? Slot : "button"
+    return (
+      <Comp
+        className={cn(buttonVariants({ variant, size, className }))}
+        ref={ref}
+        {...props}
+      />
+    )
+  }
+)
+Button.displayName = "Button"
+
+export { Button, buttonVariants }

examples/pets/src/basicComponents/ui/input.tsx

@@ -0,0 +1,25 @@
+import * as React from "react"
+
+import { cn } from "@/basicComponents/lib/utils"
+
+export interface InputProps
+  extends React.InputHTMLAttributes<HTMLInputElement> {}
+
+const Input = React.forwardRef<HTMLInputElement, InputProps>(
+  ({ className, type, ...props }, ref) => {
+    return (
+      <input
+        type={type}
+        className={cn(
+          "flex h-10 w-full rounded-md border border-input bg-background px-3 py-2 text-sm ring-offset-background file:border-0 file:bg-transparent file:text-sm file:font-medium placeholder:text-muted-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 disabled:cursor-not-allowed disabled:opacity-50",
+          className
+        )}
+        ref={ref}
+        {...props}
+      />
+    )
+  }
+)
+Input.displayName = "Input"
+
+export { Input }

examples/pets/src/basicComponents/ui/skeleton.tsx

@@ -0,0 +1,15 @@
+import { cn } from "@/basicComponents/lib/utils"
+
+function Skeleton({
+  className,
+  ...props
+}: React.HTMLAttributes<HTMLDivElement>) {
+  return (
+    <div
+      className={cn("animate-pulse rounded-md bg-muted", className)}
+      {...props}
+    />
+  )
+}
+
+export { Skeleton }

examples/pets/src/basicComponents/ui/toast.tsx

@@ -0,0 +1,127 @@
+import * as React from "react"
+import * as ToastPrimitives from "@radix-ui/react-toast"
+import { cva, type VariantProps } from "class-variance-authority"
+import { X } from "lucide-react"
+
+import { cn } from "@/basicComponents/lib/utils"
+
+const ToastProvider = ToastPrimitives.Provider
+
+const ToastViewport = React.forwardRef<
+  React.ElementRef<typeof ToastPrimitives.Viewport>,
+  React.ComponentPropsWithoutRef<typeof ToastPrimitives.Viewport>
+>(({ className, ...props }, ref) => (
+  <ToastPrimitives.Viewport
+    ref={ref}
+    className={cn(
+      "fixed top-0 z-[100] flex max-h-screen w-full flex-col-reverse p-4 sm:bottom-0 sm:right-0 sm:top-auto sm:flex-col md:max-w-[420px]",
+      className
+    )}
+    {...props}
+  />
+))
+ToastViewport.displayName = ToastPrimitives.Viewport.displayName
+
+const toastVariants = cva(
+  "group pointer-events-auto relative flex w-full items-center justify-between space-x-4 overflow-hidden rounded-md border p-6 pr-8 shadow-lg transition-all data-[swipe=cancel]:translate-x-0 data-[swipe=end]:translate-x-[var(--radix-toast-swipe-end-x)] data-[swipe=move]:translate-x-[var(--radix-toast-swipe-move-x)] data-[swipe=move]:transition-none data-[state=open]:animate-in data-[state=closed]:animate-out data-[swipe=end]:animate-out data-[state=closed]:fade-out-80 data-[state=closed]:slide-out-to-right-full data-[state=open]:slide-in-from-top-full data-[state=open]:sm:slide-in-from-bottom-full",
+  {
+    variants: {
+      variant: {
+        default: "border bg-background text-foreground",
+        destructive:
+          "destructive group border-destructive bg-destructive text-destructive-foreground",
+      },
+    },
+    defaultVariants: {
+      variant: "default",
+    },
+  }
+)
+
+const Toast = React.forwardRef<
+  React.ElementRef<typeof ToastPrimitives.Root>,
+  React.ComponentPropsWithoutRef<typeof ToastPrimitives.Root> &
+    VariantProps<typeof toastVariants>
+>(({ className, variant, ...props }, ref) => {
+  return (
+    <ToastPrimitives.Root
+      ref={ref}
+      className={cn(toastVariants({ variant }), className)}
+      {...props}
+    />
+  )
+})
+Toast.displayName = ToastPrimitives.Root.displayName
+
+const ToastAction = React.forwardRef<
+  React.ElementRef<typeof ToastPrimitives.Action>,
+  React.ComponentPropsWithoutRef<typeof ToastPrimitives.Action>
+>(({ className, ...props }, ref) => (
+  <ToastPrimitives.Action
+    ref={ref}
+    className={cn(
+      "inline-flex h-8 shrink-0 items-center justify-center rounded-md border bg-transparent px-3 text-sm font-medium ring-offset-background transition-colors hover:bg-secondary focus:outline-none focus:ring-2 focus:ring-ring focus:ring-offset-2 disabled:pointer-events-none disabled:opacity-50 group-[.destructive]:border-muted/40 group-[.destructive]:hover:border-destructive/30 group-[.destructive]:hover:bg-destructive group-[.destructive]:hover:text-destructive-foreground group-[.destructive]:focus:ring-destructive",
+      className
+    )}
+    {...props}
+  />
+))
+ToastAction.displayName = ToastPrimitives.Action.displayName
+
+const ToastClose = React.forwardRef<
+  React.ElementRef<typeof ToastPrimitives.Close>,
+  React.ComponentPropsWithoutRef<typeof ToastPrimitives.Close>
+>(({ className, ...props }, ref) => (
+  <ToastPrimitives.Close
+    ref={ref}
+    className={cn(
+      "absolute right-2 top-2 rounded-md p-1 text-foreground/50 opacity-0 transition-opacity hover:text-foreground focus:opacity-100 focus:outline-none focus:ring-2 group-hover:opacity-100 group-[.destructive]:text-red-300 group-[.destructive]:hover:text-red-50 group-[.destructive]:focus:ring-red-400 group-[.destructive]:focus:ring-offset-red-600",
+      className
+    )}
+    toast-close=""
+    {...props}
+  >
+    <X className="h-4 w-4" />
+  </ToastPrimitives.Close>
+))
+ToastClose.displayName = ToastPrimitives.Close.displayName
+
+const ToastTitle = React.forwardRef<
+  React.ElementRef<typeof ToastPrimitives.Title>,
+  React.ComponentPropsWithoutRef<typeof ToastPrimitives.Title>
+>(({ className, ...props }, ref) => (
+  <ToastPrimitives.Title
+    ref={ref}
+    className={cn("text-sm font-semibold", className)}
+    {...props}
+  />
+))
+ToastTitle.displayName = ToastPrimitives.Title.displayName
+
+const ToastDescription = React.forwardRef<
+  React.ElementRef<typeof ToastPrimitives.Description>,
+  React.ComponentPropsWithoutRef<typeof ToastPrimitives.Description>
+>(({ className, ...props }, ref) => (
+  <ToastPrimitives.Description
+    ref={ref}
+    className={cn("text-sm opacity-90", className)}
+    {...props}
+  />
+))
+ToastDescription.displayName = ToastPrimitives.Description.displayName
+
+type ToastProps = React.ComponentPropsWithoutRef<typeof Toast>
+
+type ToastActionElement = React.ReactElement<typeof ToastAction>
+
+export {
+  type ToastProps,
+  type ToastActionElement,
+  ToastProvider,
+  ToastViewport,
+  Toast,
+  ToastTitle,
+  ToastDescription,
+  ToastClose,
+  ToastAction,
+}

examples/pets/src/basicComponents/ui/toaster.tsx

@@ -0,0 +1,33 @@
+import {
+  Toast,
+  ToastClose,
+  ToastDescription,
+  ToastProvider,
+  ToastTitle,
+  ToastViewport,
+} from "@/basicComponents/ui/toast"
+import { useToast } from "@/basicComponents/ui/use-toast"
+
+export function Toaster() {
+  const { toasts } = useToast()
+
+  return (
+    <ToastProvider>
+      {toasts.map(function ({ id, title, description, action, ...props }) {
+        return (
+          <Toast key={id} {...props}>
+            <div className="grid gap-1">
+              {title && <ToastTitle>{title}</ToastTitle>}
+              {description && (
+                <ToastDescription>{description}</ToastDescription>
+              )}
+            </div>
+            {action}
+            <ToastClose />
+          </Toast>
+        )
+      })}
+      <ToastViewport />
+    </ToastProvider>
+  )
+}

examples/pets/src/basicComponents/ui/use-toast.ts

@@ -0,0 +1,192 @@
+// Inspired by react-hot-toast library
+import * as React from "react"
+
+import type {
+  ToastActionElement,
+  ToastProps,
+} from "@/basicComponents/ui/toast"
+
+const TOAST_LIMIT = 1
+const TOAST_REMOVE_DELAY = 1000000
+
+type ToasterToast = ToastProps & {
+  id: string
+  title?: React.ReactNode
+  description?: React.ReactNode
+  action?: ToastActionElement
+}
+
+const actionTypes = {
+  ADD_TOAST: "ADD_TOAST",
+  UPDATE_TOAST: "UPDATE_TOAST",
+  DISMISS_TOAST: "DISMISS_TOAST",
+  REMOVE_TOAST: "REMOVE_TOAST",
+} as const
+
+let count = 0
+
+function genId() {
+  count = (count + 1) % Number.MAX_VALUE
+  return count.toString()
+}
+
+type ActionType = typeof actionTypes
+
+type Action =
+  | {
+      type: ActionType["ADD_TOAST"]
+      toast: ToasterToast
+    }
+  | {
+      type: ActionType["UPDATE_TOAST"]
+      toast: Partial<ToasterToast>
+    }
+  | {
+      type: ActionType["DISMISS_TOAST"]
+      toastId?: ToasterToast["id"]
+    }
+  | {
+      type: ActionType["REMOVE_TOAST"]
+      toastId?: ToasterToast["id"]
+    }
+
+interface State {
+  toasts: ToasterToast[]
+}
+
+const toastTimeouts = new Map<string, ReturnType<typeof setTimeout>>()
+
+const addToRemoveQueue = (toastId: string) => {
+  if (toastTimeouts.has(toastId)) {
+    return
+  }
+
+  const timeout = setTimeout(() => {
+    toastTimeouts.delete(toastId)
+    dispatch({
+      type: "REMOVE_TOAST",
+      toastId: toastId,
+    })
+  }, TOAST_REMOVE_DELAY)
+
+  toastTimeouts.set(toastId, timeout)
+}
+
+export const reducer = (state: State, action: Action): State => {
+  switch (action.type) {
+    case "ADD_TOAST":
+      return {
+        ...state,
+        toasts: [action.toast, ...state.toasts].slice(0, TOAST_LIMIT),
+      }
+
+    case "UPDATE_TOAST":
+      return {
+        ...state,
+        toasts: state.toasts.map((t) =>
+          t.id === action.toast.id ? { ...t, ...action.toast } : t
+        ),
+      }
+
+    case "DISMISS_TOAST": {
+      const { toastId } = action
+
+      // ! Side effects ! - This could be extracted into a dismissToast() action,
+      // but I'll keep it here for simplicity
+      if (toastId) {
+        addToRemoveQueue(toastId)
+      } else {
+        state.toasts.forEach((toast) => {
+          addToRemoveQueue(toast.id)
+        })
+      }
+
+      return {
+        ...state,
+        toasts: state.toasts.map((t) =>
+          t.id === toastId || toastId === undefined
+            ? {
+                ...t,
+                open: false,
+              }
+            : t
+        ),
+      }
+    }
+    case "REMOVE_TOAST":
+      if (action.toastId === undefined) {
+        return {
+          ...state,
+          toasts: [],
+        }
+      }
+      return {
+        ...state,
+        toasts: state.toasts.filter((t) => t.id !== action.toastId),
+      }
+  }
+}
+
+const listeners: Array<(state: State) => void> = []
+
+let memoryState: State = { toasts: [] }
+
+function dispatch(action: Action) {
+  memoryState = reducer(memoryState, action)
+  listeners.forEach((listener) => {
+    listener(memoryState)
+  })
+}
+
+type Toast = Omit<ToasterToast, "id">
+
+function toast({ ...props }: Toast) {
+  const id = genId()
+
+  const update = (props: ToasterToast) =>
+    dispatch({
+      type: "UPDATE_TOAST",
+      toast: { ...props, id },
+    })
+  const dismiss = () => dispatch({ type: "DISMISS_TOAST", toastId: id })
+
+  dispatch({
+    type: "ADD_TOAST",
+    toast: {
+      ...props,
+      id,
+      open: true,
+      onOpenChange: (open) => {
+        if (!open) dismiss()
+      },
+    },
+  })
+
+  return {
+    id: id,
+    dismiss,
+    update,
+  }
+}
+
+function useToast() {
+  const [state, setState] = React.useState<State>(memoryState)
+
+  React.useEffect(() => {
+    listeners.push(setState)
+    return () => {
+      const index = listeners.indexOf(setState)
+      if (index > -1) {
+        listeners.splice(index, 1)
+      }
+    }
+  }, [state])
+
+  return {
+    ...state,
+    toast,
+    dismiss: (toastId?: string) => dispatch({ type: "DISMISS_TOAST", toastId }),
+  }
+}
+
+export { useToast, toast }

examples/pets/src/components/Auth.tsx

@@ -0,0 +1,48 @@
+import { useState } from "react";
+
+import { LocalAuthComponent } from "jazz-react-auth-local";
+
+import { Input, Button } from "../basicComponents";
+
+export const PrettyAuthUI: LocalAuthComponent = ({
+    loading,
+    logIn,
+    signUp,
+}) => {
+    const [username, setUsername] = useState<string>("");
+
+    return (
+        <div className="w-full h-full flex items-center justify-center p-5">
+            {loading ? (
+                <div>Loading...</div>
+            ) : (
+                <div className="w-72 flex flex-col gap-4">
+                    <form
+                        className="w-72 flex flex-col gap-2"
+                        onSubmit={(e) => {
+                            e.preventDefault();
+                            signUp(username);
+                        }}
+                    >
+                        <Input
+                            placeholder="Display name"
+                            value={username}
+                            onChange={(e) => setUsername(e.target.value)}
+                            autoComplete="webauthn"
+                            className="text-base"
+                        />
+                        <Button asChild>
+                            <Input
+                                type="submit"
+                                value="Sign Up as new account"
+                            />
+                        </Button>
+                    </form>
+                    <Button onClick={logIn}>
+                        Log In with existing account
+                    </Button>
+                </div>
+            )}
+        </div>
+    );
+};

examples/pets/src/components/NameBadge.tsx

@@ -0,0 +1,46 @@
+import { AccountID } from "cojson";
+import { useProfile } from "jazz-react";
+
+import { Skeleton } from "@/basicComponents";
+import uniqolor from "uniqolor";
+
+/** Walkthrough: Getting user profiles in `<NameBadge/>`
+ *
+ *  `<NameBadge/>` uses `useProfile(accountID)`, which is a shorthand for
+ *  useTelepathicState on an account's profile.
+ *
+ *  Profiles are always a `CoMap<{name: string}>`, but they might have app-specific
+ *  additional properties).
+ *
+ *  In our case, we just display the profile name (which is set by the LocalAuth
+ *  provider when we first create an account).
+ */
+
+export function NameBadge({ accountID }: { accountID?: AccountID }) {
+    const profile = useProfile(accountID);
+
+    return accountID && profile?.get("name") ? (
+        <span
+            className="rounded-full py-0.5 px-2 text-xs"
+            style={randomUserColor(accountID)}
+        >
+            {profile.get("name")}
+        </span>
+    ) : (
+        <Skeleton className="mt-1 w-[50px] h-[1em] rounded-full" />
+    );
+}
+
+function randomUserColor(accountID: AccountID) {
+    const theme = window.matchMedia("(prefers-color-scheme: dark)").matches
+        ? "dark"
+        : "light";
+
+    const brightColor = uniqolor(accountID || "", { lightness: 80 }).color;
+    const darkColor = uniqolor(accountID || "", { lightness: 20 }).color;
+
+    return {
+        color: theme == "light" ? darkColor : brightColor,
+        background: theme == "light" ? brightColor : darkColor,
+    };
+}

examples/pets/src/components/ShareButton.tsx

@@ -0,0 +1,46 @@
+import { useState } from "react";
+
+import { PetPost } from "../1_types";
+
+import { createInviteLink } from "jazz-react";
+import QRCode from "qrcode";
+
+import { useToast, Button } from "../basicComponents";
+
+export function ShareButton({ petPost }: { petPost?: PetPost }) {
+    const [existingInviteLink, setExistingInviteLink] = useState<string>();
+    const { toast } = useToast();
+
+    return (
+        petPost?.group.myRole() === "admin" && (
+            <Button
+                size="sm"
+                className="py-0"
+                disabled={!petPost}
+                variant="outline"
+                onClick={async () => {
+                    let inviteLink = existingInviteLink;
+                    if (petPost && !inviteLink) {
+                        inviteLink = createInviteLink(petPost, "writer");
+                        setExistingInviteLink(inviteLink);
+                    }
+                    if (inviteLink) {
+                        const qr = await QRCode.toDataURL(inviteLink, {
+                            errorCorrectionLevel: "L",
+                        });
+                        navigator.clipboard.writeText(inviteLink).then(() =>
+                            toast({
+                                title: "Copied invite link to clipboard!",
+                                description: (
+                                    <img src={qr} className="w-20 h-20" />
+                                ),
+                            })
+                        );
+                    }
+                }}
+            >
+                Share
+            </Button>
+        )
+    );
+}

examples/pets/src/index.css

@@ -0,0 +1,76 @@
+@tailwind base;
+@tailwind components;
+@tailwind utilities;
+ 
+@layer base {
+  :root {
+    --background: 0 0% 100%;
+    --foreground: 20 14.3% 4.1%;
+
+    --card: 0 0% 100%;
+    --card-foreground: 20 14.3% 4.1%;
+ 
+    --popover: 0 0% 100%;
+    --popover-foreground: 20 14.3% 4.1%;
+ 
+    --primary: 24 9.8% 10%;
+    --primary-foreground: 60 9.1% 97.8%;
+ 
+    --secondary: 60 4.8% 95.9%;
+    --secondary-foreground: 24 9.8% 10%;
+ 
+    --muted: 60 4.8% 95.9%;
+    --muted-foreground: 25 5.3% 44.7%;
+ 
+    --accent: 60 4.8% 95.9%;
+    --accent-foreground: 24 9.8% 10%;
+ 
+    --destructive: 0 84.2% 60.2%;
+    --destructive-foreground: 60 9.1% 97.8%;
+
+    --border: 20 5.9% 90%;
+    --input: 20 5.9% 90%;
+    --ring: 20 14.3% 4.1%;
+ 
+    --radius: 0.5rem;
+  }
+ 
+  .dark {
+    --background: 20 14.3% 4.1%;
+    --foreground: 60 9.1% 97.8%;
+ 
+    --card: 20 14.3% 4.1%;
+    --card-foreground: 60 9.1% 97.8%;
+ 
+    --popover: 20 14.3% 4.1%;
+    --popover-foreground: 60 9.1% 97.8%;
+ 
+    --primary: 60 9.1% 97.8%;
+    --primary-foreground: 24 9.8% 10%;
+ 
+    --secondary: 12 6.5% 15.1%;
+    --secondary-foreground: 60 9.1% 97.8%;
+ 
+    --muted: 12 6.5% 15.1%;
+    --muted-foreground: 24 5.4% 63.9%;
+ 
+    --accent: 12 6.5% 15.1%;
+    --accent-foreground: 60 9.1% 97.8%;
+ 
+    --destructive: 0 62.8% 30.6%;
+    --destructive-foreground: 60 9.1% 97.8%;
+ 
+    --border: 12 6.5% 15.1%;
+    --input: 12 6.5% 15.1%;
+    --ring: 24 5.7% 82.9%;
+  }
+}
+ 
+@layer base {
+  * {
+    @apply border-border;
+  }
+  body {
+    @apply bg-background text-foreground;
+  }
+}

examples/pets/src/router.ts

@@ -0,0 +1,37 @@
+import { useCallback, useEffect, useState } from "react";
+import { CoID, LocalNode, CoValueImpl } from "cojson";
+import { consumeInviteLinkFromWindowLocation } from "jazz-react";
+
+export function useSimpleHashRouterThatAcceptsInvites<C extends CoValueImpl>(
+    localNode: LocalNode
+) {
+    const [currentValueId, setCurrentValueId] = useState<CoID<C>>();
+
+    useEffect(() => {
+        const listener = async () => {
+            const acceptedInvitation = await consumeInviteLinkFromWindowLocation<C>(localNode);
+
+            if (acceptedInvitation) {
+                setCurrentValueId(acceptedInvitation.valueID);
+                window.location.hash = acceptedInvitation.valueID;
+                return;
+            }
+
+            setCurrentValueId(
+                (window.location.hash.slice(1) as CoID<C>) || undefined
+            );
+        };
+        window.addEventListener("hashchange", listener);
+        listener();
+
+        return () => {
+            window.removeEventListener("hashchange", listener);
+        };
+    }, [localNode]);
+
+    const navigateToValue = useCallback((id: CoID<C> | undefined) => {
+        window.location.hash = id || "";
+    }, []);
+
+    return [currentValueId, navigateToValue] as const;
+}

examples/pets/src/vite-env.d.ts

@@ -0,0 +1 @@
+/// <reference types="vite/client" />

examples/pets/tailwind.config.js

@@ -0,0 +1,76 @@
+/** @type {import('tailwindcss').Config} */
+module.exports = {
+  darkMode: ["class"],
+  content: [
+    './pages/**/*.{ts,tsx}',
+    './components/**/*.{ts,tsx}',
+    './app/**/*.{ts,tsx}',
+    './src/**/*.{ts,tsx}',
+	],
+  theme: {
+    container: {
+      center: true,
+      padding: "2rem",
+      screens: {
+        "2xl": "1400px",
+      },
+    },
+    extend: {
+      colors: {
+        border: "hsl(var(--border))",
+        input: "hsl(var(--input))",
+        ring: "hsl(var(--ring))",
+        background: "hsl(var(--background))",
+        foreground: "hsl(var(--foreground))",
+        primary: {
+          DEFAULT: "hsl(var(--primary))",
+          foreground: "hsl(var(--primary-foreground))",
+        },
+        secondary: {
+          DEFAULT: "hsl(var(--secondary))",
+          foreground: "hsl(var(--secondary-foreground))",
+        },
+        destructive: {
+          DEFAULT: "hsl(var(--destructive))",
+          foreground: "hsl(var(--destructive-foreground))",
+        },
+        muted: {
+          DEFAULT: "hsl(var(--muted))",
+          foreground: "hsl(var(--muted-foreground))",
+        },
+        accent: {
+          DEFAULT: "hsl(var(--accent))",
+          foreground: "hsl(var(--accent-foreground))",
+        },
+        popover: {
+          DEFAULT: "hsl(var(--popover))",
+          foreground: "hsl(var(--popover-foreground))",
+        },
+        card: {
+          DEFAULT: "hsl(var(--card))",
+          foreground: "hsl(var(--card-foreground))",
+        },
+      },
+      borderRadius: {
+        lg: "var(--radius)",
+        md: "calc(var(--radius) - 2px)",
+        sm: "calc(var(--radius) - 4px)",
+      },
+      keyframes: {
+        "accordion-down": {
+          from: { height: 0 },
+          to: { height: "var(--radix-accordion-content-height)" },
+        },
+        "accordion-up": {
+          from: { height: "var(--radix-accordion-content-height)" },
+          to: { height: 0 },
+        },
+      },
+      animation: {
+        "accordion-down": "accordion-down 0.2s ease-out",
+        "accordion-up": "accordion-up 0.2s ease-out",
+      },
+    },
+  },
+  plugins: [require("tailwindcss-animate")],
+}

examples/pets/tsconfig.json

@@ -0,0 +1,29 @@
+{
+  "compilerOptions": {
+    "target": "ES2020",
+    "useDefineForClassFields": true,
+    "lib": ["ES2023", "DOM", "DOM.Iterable"],
+    "module": "ESNext",
+    "skipLibCheck": true,
+
+    /* Bundler mode */
+    "moduleResolution": "bundler",
+    "allowImportingTsExtensions": true,
+    "resolveJsonModule": true,
+    "isolatedModules": true,
+    "noEmit": true,
+    "jsx": "react-jsx",
+
+    /* Linting */
+    "strict": true,
+    "noUnusedLocals": true,
+    "noUnusedParameters": true,
+    "noFallthroughCasesInSwitch": true,
+    "baseUrl": ".",
+    "paths": {
+      "@/*": ["./src/*"]
+    }
+  },
+  "include": ["src"],
+  "references": [{ "path": "./tsconfig.node.json" }]
+}

examples/pets/tsconfig.node.json

@@ -0,0 +1,10 @@
+{
+  "compilerOptions": {
+    "composite": true,
+    "skipLibCheck": true,
+    "module": "ESNext",
+    "moduleResolution": "bundler",
+    "allowSyntheticDefaultImports": true
+  },
+  "include": ["vite.config.ts"]
+}

examples/pets/vite.config.ts

@@ -0,0 +1,16 @@
+import { defineConfig } from 'vite'
+import react from '@vitejs/plugin-react-swc'
+import path from "path";
+
+// https://vitejs.dev/config/
+export default defineConfig({
+  plugins: [react()],
+  resolve: {
+    alias: {
+      "@": path.resolve(__dirname, "./src"),
+    },
+  },
+  build: {
+    minify: false
+  }
+})

examples/todo/README.md

@@ -7,7 +7,8 @@ Live version: https://example-todo.jazz.tools
 Start by checking out just the example app to a folder:
 
 ```bash
-npx degit gardencmp/jazz/examples/todo
+npx degit gardencmp/jazz/examples/todo jazz-example-todo
+cd jazz-example-todo
 ```
 
 (This ensures that you have the example app without git history or our multi-package monorepo)
@@ -61,4 +62,4 @@ If you have feedback, let us know on [Discord](https://discord.gg/utDMjHYg42) or
 
 By default, the example app uses [Jazz Global Mesh](https://jazz.tools/mesh) (`wss://sync.jazz.tools`) - so cross-device use, invites and collaboration should just work.
 
-You can also run a local sync server by running `npx cojson-simple-sync` and adding the query param `?sync=ws://localhost:4200` to the URL of the example app (for example: `http://localhost:5173/?sync=ws://localhost:4200`), or by setting the `sync` parameter of the `<WithJazz>` provider component in [./src/0_main.tsx](./src/0_main.tsx).
+You can also run a local sync server by running `npx cojson-simple-sync` and adding the query param `?sync=ws://localhost:4200` to the URL of the example app (for example: `http://localhost:5173/?sync=ws://localhost:4200`), or by setting the `sync` parameter of the `<WithJazz>` provider component in [./src/0_main.tsx](./src/0_main.tsx).

examples/todo/package.json

@@ -1,7 +1,7 @@
 {
   "name": "jazz-example-todo",
   "private": true,
-  "version": "0.0.24",
+  "version": "0.0.29",
   "type": "module",
   "scripts": {
     "dev": "vite",
@@ -16,8 +16,8 @@
     "@types/qrcode": "^1.5.1",
     "class-variance-authority": "^0.7.0",
     "clsx": "^2.0.0",
-    "jazz-react": "^0.1.10",
-    "jazz-react-auth-local": "^0.1.10",
+    "jazz-react": "^0.2.0",
+    "jazz-react-auth-local": "^0.2.0",
     "lucide-react": "^0.274.0",
     "qrcode": "^1.5.3",
     "react": "^18.2.0",

examples/todo/src/0_main.tsx

@@ -18,17 +18,17 @@ import App from "./2_App.tsx";
 
 const appName = "Jazz Todo List Example";
 
+const auth = LocalAuth({
+    appName,
+    Component: PrettyAuthUI,
+});
+
 ReactDOM.createRoot(document.getElementById("root")!).render(
     <React.StrictMode>
         <ThemeProvider>
             <TitleAndLogo name={appName} />
 
-            <WithJazz
-                auth={LocalAuth({
-                    appName,
-                    Component: PrettyAuthUI,
-                })}
-            >
+            <WithJazz auth={auth}>
                 <App />
             </WithJazz>
         </ThemeProvider>

examples/todo/src/router.ts

@@ -1,8 +1,8 @@
 import { useCallback, useEffect, useState } from "react";
-import { CoID, LocalNode, ContentType } from "cojson";
+import { CoID, LocalNode, CoValueImpl } from "cojson";
 import { consumeInviteLinkFromWindowLocation } from "jazz-react";
 
-export function useSimpleHashRouterThatAcceptsInvites<C extends ContentType>(
+export function useSimpleHashRouterThatAcceptsInvites<C extends CoValueImpl>(
     localNode: LocalNode
 ) {
     const [currentValueId, setCurrentValueId] = useState<CoID<C>>();

generateDocs.ts

@@ -0,0 +1,441 @@
+import { readFile, writeFile } from "fs/promises";
+import { Application, JSONOutput } from "typedoc";
+
+const manuallyIgnore = new Set(["CojsonInternalTypes"]);
+
+async function main() {
+    // Application.bootstrap also exists, which will not load plugins
+    // Also accepts an array of option readers if you want to disable
+    // TypeDoc's tsconfig.json/package.json/typedoc.json option readers
+    const packageDocs = Object.entries({
+        cojson: "index.ts",
+        "jazz-react": "index.tsx",
+        "jazz-browser": "index.ts",
+    }).map(async ([packageName, entryPoint]) => {
+        const app = await Application.bootstrapWithPlugins({
+            entryPoints: [`packages/${packageName}/src/${entryPoint}`],
+            tsconfig: `packages/${packageName}/tsconfig.json`,
+            sort: ["required-first"],
+        });
+
+        const project = await app.convert();
+
+        if (!project) {
+            throw new Error("Failed to convert project" + packageName);
+        }
+        // Alternatively generate JSON output
+        await app.generateJson(project, `docsTmp/${packageName}.json`);
+
+        const docs = JSON.parse(
+            await readFile(`docsTmp/${packageName}.json`, "utf8")
+        ) as JSONOutput.ProjectReflection;
+
+        return (
+            `# ${packageName}\n\n` +
+            docs
+                .groups!.map((group) => {
+                    return group.children
+                        ?.map((childId) => {
+                            const child = docs.children!.find(
+                                (child) => child.id === childId
+                            )!;
+
+                            if (manuallyIgnore.has(child.name)) {
+                                return "";
+                            }
+
+                            return (
+                                `## \`${renderChildName(child)}\` (${group.title
+                                    .toLowerCase()
+                                    .replace("ces", "ce")
+                                    .replace(/es$/, "")
+                                    .replace(
+                                        "ns",
+                                        "n"
+                                    )} in \`${packageName}\`)\n\n` +
+                                renderChildType(child) +
+                                renderComment(child.comment) +
+                                (child.kind === 128 || child.kind === 256
+                                    ? child.groups
+                                          ?.map((group) =>
+                                              renderChildGroup(child, group)
+                                          )
+                                          .join("\n\n")
+                                    : "TODO: doc generator not implemented yet")
+                            );
+                        })
+                        .join("\n\n----\n\n");
+                })
+                .join("\n\n----\n\n")
+        );
+
+        function renderComment(comment?: JSONOutput.Comment): string {
+            if (comment) {
+                return (
+                    comment.summary
+                        .map((token) =>
+                            token.kind === "text" || token.kind === "code"
+                                ? token.text
+                                : ""
+                        )
+                        .join("") +
+                    "\n\n" +
+                    (comment.blockTags || [])
+                        .map((blockTag) =>
+                            blockTag.tag === "@example"
+                                ? "##### Example:\n\n" +
+                                  blockTag.content
+                                      .map((token) =>
+                                          token.kind === "text" ||
+                                          token.kind === "code"
+                                              ? token.text
+                                              : ""
+                                      )
+                                      .join("")
+                                : ""
+                        )
+                        .join("\n\n") +
+                    "\n\n"
+                );
+            } else {
+                return "TODO: document\n\n";
+            }
+        }
+
+        function renderChildName(child: JSONOutput.DeclarationReflection) {
+            if (child.signatures) {
+                if (
+                    child.signatures[0].type?.type === "reference" &&
+                    child.signatures[0].type.qualifiedName ===
+                        "React.JSX.Element"
+                ) {
+                    return `<${child.name}/>`;
+                } else {
+                    return (
+                        child.name +
+                        `(${(child.signatures[0].parameters || [])
+                            .map(renderParamSimple)
+                            .join(", ")})`
+                    );
+                }
+            } else {
+                return child.name;
+            }
+        }
+
+        function renderChildType(
+            child: JSONOutput.DeclarationReflection
+        ): string {
+            const isClass = child.kind === 128;
+            const isTypeDef = child.kind === 2097152;
+            const isInterface = child.kind === 256;
+            const isFunction = !!child.signatures;
+            return (
+                "```typescript\n" +
+                `export ${
+                    isClass
+                        ? "class"
+                        : isTypeDef
+                        ? "type"
+                        : isFunction
+                        ? "function"
+                        : isInterface
+                        ? "interface"
+                        : ""
+                } ${child.name}` +
+                (child.typeParameters
+                    ? "<" +
+                      child.typeParameters.map(renderTypeParam).join(", ") +
+                      ">"
+                    : "") +
+                (child.extendedTypes
+                    ? " extends " +
+                      child.extendedTypes.map(renderType).join(", ")
+                    : "") +
+                (child.implementedTypes
+                    ? " implements " +
+                      child.implementedTypes.map(renderType).join(", ")
+                    : "") +
+                (isClass || isInterface
+                    ? " {...}"
+                    : isTypeDef
+                    ? ` = ${renderType(child.type)}`
+                    : child.signatures
+                    ? `(${(child.signatures[0].parameters || [])
+                          .map(renderParam)
+                          .join(", ")}): ${renderType(
+                          child.signatures[0].type
+                      )}`
+                    : "") +
+                "\n```\n"
+            );
+        }
+
+        function renderChildGroup(
+            child: JSONOutput.DeclarationReflection,
+            group: JSONOutput.ReflectionGroup
+        ): string {
+            return (
+                `### ${group.title}\n\n` +
+                group.children
+                    ?.map((memberId) => {
+                        const member = child.children!.find(
+                            (member) => member.id === memberId
+                        )!;
+
+                        if (member.kind === 2048 || member.kind === 512) {
+                            if (member.signatures?.every(sig => sig.comment?.modifierTags?.includes("@internal"))) {
+                                return ""
+                            } else {
+                                return documentConstructorOrMethod(member, child);
+                            }
+                        } else if (
+                            member.kind === 1024 ||
+                            member.kind === 262144
+                        ) {
+                            if (member.comment?.modifierTags?.includes("@internal")) {
+                                return ""
+                            } else {
+                                return documentProperty(member, child);
+                            }
+                        } else {
+                            return "Unknown member kind " + member.kind;
+                        }
+                    })
+                    .join("\n\n")
+            );
+        }
+
+        function renderType(t?: JSONOutput.SomeType): string {
+            if (!t) return "";
+            if (t.type === "reference") {
+                return (
+                    t.name +
+                    (t.typeArguments
+                        ? "<" + t.typeArguments.map(renderType).join(", ") + ">"
+                        : "")
+                );
+            } else if (t.type === "intrinsic") {
+                return t.name;
+            } else if (t.type === "literal") {
+                return JSON.stringify(t.value);
+            } else if (t.type === "union") {
+                return [...new Set(t.types.map(renderType))].join(" | ");
+            } else if (t.type === "intersection") {
+                return [...new Set(t.types.map(renderType))].join(" & ");
+            } else if (t.type === "indexedAccess") {
+                return (
+                    renderType(t.objectType) +
+                    "[" +
+                    renderType(t.indexType) +
+                    "]"
+                );
+            } else if (t.type === "reflection") {
+                if (t.declaration.indexSignature) {
+                    return (
+                        "{ [" +
+                        t.declaration.indexSignature?.parameters?.[0].name +
+                        ": " +
+                        renderType(
+                            t.declaration.indexSignature?.parameters?.[0].type
+                        ) +
+                        "]: " +
+                        renderType(t.declaration.indexSignature?.type) +
+                        " }"
+                    );
+                } else if (t.declaration.children) {
+                    return `{${t.declaration.children
+                        .map(
+                            (child) =>
+                                `${child.name}${
+                                    child.flags.isOptional ? "?" : ""
+                                }: ${renderType(child.type)}`
+                        )
+                        .join(", ")}}`;
+                } else if (t.declaration.signatures) {
+                    if (t.declaration.signatures.length > 1) {
+                        return "COMPLEX_TYPE_MULTIPLE_INLINE_SIGNATURES";
+                    } else {
+                        return `(${(
+                            t.declaration.signatures[0].parameters || []
+                        ).map(renderParam)}) => ${renderType(
+                            t.declaration.signatures[0].type
+                        )}`;
+                    }
+                } else {
+                    return "COMPLEX_TYPE_REFLECTION";
+                }
+            } else if (t.type === "array") {
+                return renderType(t.elementType) + "[]";
+            } else if (t.type === "templateLiteral") {
+                const matchingNamedType = docs.children?.find(
+                    (child) =>
+                        child.variant === "declaration" &&
+                        child.type?.type === "templateLiteral" &&
+                        child.type.head === t.head &&
+                        child.type.tail.every(
+                            (piece, i) => piece[1] === t.tail[i][1]
+                        )
+                );
+
+                if (matchingNamedType) {
+                    return matchingNamedType.name;
+                } else {
+                    if (
+                        t.head === "sealerSecret_z" &&
+                        t.tail[0][1] === "/signerSecret_z"
+                    ) {
+                        return "AgentSecret";
+                    } else if (
+                        t.head === "sealer_z" &&
+                        t.tail[0][1] === "/signer_z"
+                    ) {
+                        if (t.tail[1] && t.tail[1][1] === "_session_z") {
+                            return "SessionID";
+                        } else {
+                            return "AgentID";
+                        }
+                    } else {
+                        return "TEMPLATE_LITERAL";
+                    }
+                }
+            } else {
+                return "COMPLEX_TYPE_" + t.type;
+            }
+        }
+
+        // function renderTemplateLiteral(tempLit: JSONOutput.TemplateLiteralType) {
+        //     return tempLit.head + tempLit.tail.map((piece) => piece[0] + piece[1]).join("");
+        // }
+
+        // function resolveTemplateLiteralPieceType(t: SomeType): string {
+        //     if (t.type === "string") {
+        //         return "${string}"
+        //     }
+        //     if (t.type === "reference") {
+        //         const referencedType = docs.children?.find(
+        //             (child) => child.name === t.name
+        //         );
+
+        //     }
+        // }
+
+        function renderTypeParam(
+            t?: JSONOutput.TypeParameterReflection
+        ): string {
+            if (!t) return "";
+            return t.name + (t.type ? " extends " + renderType(t.type) : "");
+        }
+
+        function renderParam(param: JSONOutput.ParameterReflection) {
+            return param.name === "__namedParameters"
+                ? renderType(param.type)
+                : `${param.name}: ${renderType(param.type)}`;
+        }
+
+        function renderParamSimple(param: JSONOutput.ParameterReflection) {
+            return param.name === "__namedParameters" &&
+                param.type?.type === "reflection"
+                ? `{${param.type?.declaration.children
+                      ?.map(
+                          (child) =>
+                              child.name + (child.flags.isOptional ? "?" : "")
+                      )
+                      .join(", ")}}${param.defaultValue ? "?" : ""}`
+                : param.name + (param.defaultValue ? "?" : "");
+        }
+
+        function documentConstructorOrMethod(
+            member: JSONOutput.DeclarationReflection,
+            child: JSONOutput.DeclarationReflection
+        ) {
+            const stem =
+                member.name === "constructor"
+                    ? "new " + child.name
+                    : (member.flags.isStatic
+                          ? child.name
+                          : child.name[0].toLowerCase() + child.name.slice(1)) +
+                      "." +
+                      member.name;
+
+            return (
+                `<details>\n<summary><code>${stem}(${(
+                    member.signatures?.[0]?.parameters?.map(
+                        renderParamSimple
+                    ) || []
+                ).join(", ")})</code> ${
+                    member.inheritedFrom
+                        ? "(from <code>" +
+                          member.inheritedFrom.name.split(".")[0] +
+                          "</code>) "
+                        : ""
+                } ${
+                    member.signatures?.[0]?.comment ? "" : "(undocumented)"
+                }</summary>\n\n` +
+                member.signatures?.map((signature) => {
+                    return (
+                        "```typescript\n" +
+                        `${stem}${
+                            signature.typeParameter
+                                ? `<${signature.typeParameter
+                                      .map(renderTypeParam)
+                                      .join(", ")}>`
+                                : ""
+                        }(${
+                            (
+                                signature.parameters?.map(
+                                    (param) =>
+                                        `\n  ${param.name}${
+                                            param.defaultValue ? "?" : ""
+                                        }: ${renderType(param.type)}${
+                                            param.defaultValue
+                                                ? ` = ${param.defaultValue}`
+                                                : ""
+                                        }`
+                                ) || []
+                            ).join(",") +
+                            (signature.parameters?.length ? "\n" : "")
+                        }): ${renderType(signature.type)}\n` +
+                        "```\n" +
+                        renderComment(signature.comment)
+                    );
+                }) +
+                "</details>\n\n"
+            );
+        }
+
+        function documentProperty(
+            member: JSONOutput.DeclarationReflection,
+            child: JSONOutput.DeclarationReflection
+        ) {
+            const stem = member.flags.isStatic
+                ? child.name
+                : child.name[0].toLowerCase() + child.name.slice(1);
+            return (
+                `<details>\n<summary><code>${stem}.${member.name}</code> ${
+                    member.inheritedFrom
+                        ? "(from <code>" +
+                          member.inheritedFrom.name.split(".")[0] +
+                          "</code>) "
+                        : ""
+                } ${
+                    member.comment ? "" : "(undocumented)"
+                }</summary>\n\n` +
+                "```typescript\n" +
+                `${member.getSignature ? "get " : ""}${stem}.${member.name}${
+                    member.getSignature ? "()" : ""
+                }: ${renderType(member.type || member.getSignature?.type)}\n` +
+                "```\n" +
+                renderComment(member.comment) +
+                "</details>\n\n"
+            );
+        }
+    });
+
+    await writeFile(
+        "./DOCS.md",
+        (await Promise.all(packageDocs)).join("\n\n\n")
+    );
+}
+
+main().catch(console.error);

package.json

@@ -7,11 +7,14 @@
   ],
   "dependencies": {},
   "devDependencies": {
-    "lerna": "^7.1.5"
+    "lerna": "^7.1.5",
+    "ts-node": "^10.9.1",
+    "typedoc": "^0.25.1"
   },
   "scripts": {
     "build-all": "lerna run build",
     "updated": "lerna updated --include-merged-tags",
-    "publish-all": "lerna publish --include-merged-tags"
+    "publish-all": "yarn run gen-docs && lerna publish --include-merged-tags",
+    "gen-docs": "ts-node generateDocs.ts"
   }
 }

packages/cojson-simple-sync/package.json

@@ -4,7 +4,7 @@
     "types": "src/index.ts",
     "type": "module",
     "license": "MIT",
-    "version": "0.1.9",
+    "version": "0.2.0",
     "devDependencies": {
         "@types/jest": "^29.5.3",
         "@types/ws": "^8.5.5",
@@ -16,8 +16,8 @@
         "typescript": "5.0.2"
     },
     "dependencies": {
-        "cojson": "^0.1.8",
-        "cojson-storage-sqlite": "^0.1.6",
+        "cojson": "^0.2.0",
+        "cojson-storage-sqlite": "^0.2.0",
         "ws": "^8.13.0"
     },
     "scripts": {
@@ -31,5 +31,6 @@
     "jest": {
         "preset": "ts-jest",
         "testEnvironment": "node"
-    }
+    },
+    "gitHead": "33c27053293b4801b968c61d5c4c989f93a67d13"
 }

packages/cojson-storage-sqlite/package.json

@@ -1,13 +1,13 @@
 {
   "name": "cojson-storage-sqlite",
   "type": "module",
-  "version": "0.1.6",
+  "version": "0.2.0",
   "main": "dist/index.js",
   "types": "dist/index.d.ts",
   "license": "MIT",
   "dependencies": {
     "better-sqlite3": "^8.5.2",
-    "cojson": "^0.1.8",
+    "cojson": "^0.2.0",
     "typescript": "^5.1.6"
   },
   "scripts": {
@@ -17,5 +17,6 @@
   },
   "devDependencies": {
     "@types/better-sqlite3": "^7.6.4"
-  }
+  },
+  "gitHead": "33c27053293b4801b968c61d5c4c989f93a67d13"
 }

packages/cojson-storage-sqlite/src/index.ts

@@ -217,7 +217,9 @@ export class SQLiteStorage {
                 ? Object.values(newContent.new).flatMap((sessionEntry) =>
                       sessionEntry.newTransactions.flatMap((tx) => {
                           if (tx.privacy !== "trusting") return [];
-                          return tx.changes
+                          // TODO: avoid parsing here?
+                          return cojsonInternals
+                              .parseJSON(tx.changes)
                               .map(
                                   (change) =>
                                       change &&
@@ -338,7 +340,7 @@ export class SQLiteStorage {
                         lastSignature: msg.new[sessionID]!.lastSignature,
                     };
 
-                    const upsertedSession = (this.db
+                    const upsertedSession = this.db
                         .prepare<[number, string, number, string]>(
                             `INSERT INTO sessions (coValue, sessionID, lastIdx, lastSignature) VALUES (?, ?, ?, ?)
                             ON CONFLICT(coValue, sessionID) DO UPDATE SET lastIdx=excluded.lastIdx, lastSignature=excluded.lastSignature
@@ -349,7 +351,7 @@ export class SQLiteStorage {
                             sessionUpdate.sessionID,
                             sessionUpdate.lastIdx,
                             sessionUpdate.lastSignature
-                        ) as {rowID: number});
+                        ) as { rowID: number };
 
                     const sessionRowID = upsertedSession.rowID;
 

packages/cojson/package.json

@@ -5,7 +5,7 @@
     "types": "dist/index.d.ts",
     "type": "module",
     "license": "MIT",
-    "version": "0.1.8",
+    "version": "0.2.0",
     "devDependencies": {
         "@types/jest": "^29.5.3",
         "@typescript-eslint/eslint-plugin": "^6.2.1",
@@ -19,9 +19,8 @@
     "dependencies": {
         "@noble/ciphers": "^0.1.3",
         "@noble/curves": "^1.1.0",
-        "@noble/hashes": "^1.3.1",
         "@scure/base": "^1.1.1",
-        "fast-json-stable-stringify": "https://github.com/tirithen/fast-json-stable-stringify#7a3dcf2",
+        "hash-wasm": "^4.9.0",
         "isomorphic-streams": "https://github.com/sgwilym/isomorphic-streams.git#aa9394781bfc92f8d7c981be7daf8af4b4cd4fae"
     },
     "scripts": {
@@ -51,5 +50,6 @@
             "/node_modules/",
             "/dist/"
         ]
-    }
+    },
+    "gitHead": "33c27053293b4801b968c61d5c4c989f93a67d13"
 }

packages/cojson/src/account.test.ts

@@ -1,7 +1,12 @@
-import { newRandomSessionID } from "./coValue.js";
+import { newRandomSessionID } from "./coValueCore.js";
+import { cojsonReady } from "./index.js";
 import { LocalNode } from "./node.js";
 import { connectedPeers } from "./streamUtils.js";
 
+beforeEach(async () => {
+    await cojsonReady;
+});
+
 test("Can create a node while creating a new account with profile", async () => {
     const { node, accountID, accountSecret, sessionID } =
         LocalNode.withNewlyCreatedAccount("Hermes Puggington");

packages/cojson/src/account.ts

@@ -1,5 +1,5 @@
-import { CoValueHeader } from "./coValue.js";
-import { CoID } from "./contentType.js";
+import { CoValueHeader } from "./coValueCore.js";
+import { CoID } from "./coValue.js";
 import {
     AgentSecret,
     SealerID,
@@ -13,7 +13,8 @@ import {
     getAgentSignerSecret,
 } from "./crypto.js";
 import { AgentID } from "./ids.js";
-import { CoMap, LocalNode } from "./index.js";
+import { CoMap } from "./coValues/coMap.js";
+import { LocalNode } from "./node.js";
 import { Group, GroupContent } from "./group.js";
 
 export function accountHeaderForInitialAgentSecret(
@@ -33,11 +34,11 @@ export function accountHeaderForInitialAgentSecret(
 
 export class Account extends Group {
     get id(): AccountID {
-        return this.groupMap.id as AccountID;
+        return this.underlyingMap.id as AccountID;
     }
 
     getCurrentAgentID(): AgentID {
-        const agents = this.groupMap
+        const agents = this.underlyingMap
             .keys()
             .filter((k): k is AgentID => k.startsWith("sealer_"));
 
@@ -62,6 +63,7 @@ export interface GeneralizedControlledAccount {
     currentSealerSecret: () => SealerSecret;
 }
 
+/** @hidden */
 export class ControlledAccount
     extends Account
     implements GeneralizedControlledAccount
@@ -99,6 +101,7 @@ export class ControlledAccount
     }
 }
 
+/** @hidden */
 export class AnonymousControlledAccount
     implements GeneralizedControlledAccount
 {

packages/cojson/src/base64url.test.ts

@@ -0,0 +1,32 @@
+import { base64URLtoBytes, bytesToBase64url } from "./base64url";
+
+const txt = new TextEncoder();
+
+test("Test our Base64 URL encoding and decoding", () => {
+    // tests from the RFC
+
+    expect(base64URLtoBytes("")).toEqual(new Uint8Array([]));
+    expect(bytesToBase64url(new Uint8Array([]))).toEqual("");
+
+    expect(bytesToBase64url(txt.encode("f"))).toEqual("Zg==");
+    expect(bytesToBase64url(txt.encode("fo"))).toEqual("Zm8=");
+    expect(bytesToBase64url(txt.encode("foo"))).toEqual("Zm9v");
+    expect(bytesToBase64url(txt.encode("foob"))).toEqual("Zm9vYg==");
+    expect(bytesToBase64url(txt.encode("fooba"))).toEqual("Zm9vYmE=");
+    expect(bytesToBase64url(txt.encode("foobar"))).toEqual("Zm9vYmFy");
+    // reverse
+    expect(base64URLtoBytes("Zg==")).toEqual(txt.encode("f"));
+    expect(base64URLtoBytes("Zm8=")).toEqual(txt.encode("fo"));
+    expect(base64URLtoBytes("Zm9v")).toEqual(txt.encode("foo"));
+    expect(base64URLtoBytes("Zm9vYg==")).toEqual(txt.encode("foob"));
+    expect(base64URLtoBytes("Zm9vYmE=")).toEqual(txt.encode("fooba"));
+    expect(base64URLtoBytes("Zm9vYmFy")).toEqual(txt.encode("foobar"));
+
+    expect(base64URLtoBytes("V2hhdCBkb2VzIDIgKyAyLjEgZXF1YWw_PyB-IDQ=")).toEqual(
+        txt.encode("What does 2 + 2.1 equal?? ~ 4")
+    );
+    // reverse
+    expect(
+        bytesToBase64url(txt.encode("What does 2 + 2.1 equal?? ~ 4"))
+    ).toEqual("V2hhdCBkb2VzIDIgKyAyLjEgZXF1YWw_PyB-IDQ=");
+});

packages/cojson/src/base64url.ts

@@ -0,0 +1,68 @@
+const encoder = new TextEncoder();
+const decoder = new TextDecoder();
+
+export function base64URLtoBytes(base64: string) {
+    base64 = base64.replace(/=/g, "");
+    const n = base64.length;
+    const rem = n % 4;
+    const k = rem && rem - 1; // how many bytes the last base64 chunk encodes
+    const m = (n >> 2) * 3 + k; // total encoded bytes
+
+    const encoded = new Uint8Array(n + 3);
+    encoder.encodeInto(base64 + "===", encoded);
+
+    for (let i = 0, j = 0; i < n; i += 4, j += 3) {
+        const x =
+            (lookup[encoded[i]!]! << 18) +
+            (lookup[encoded[i + 1]!]! << 12) +
+            (lookup[encoded[i + 2]!]! << 6) +
+            lookup[encoded[i + 3]!]!;
+        encoded[j] = x >> 16;
+        encoded[j + 1] = (x >> 8) & 0xff;
+        encoded[j + 2] = x & 0xff;
+    }
+    return new Uint8Array(encoded.buffer, 0, m);
+}
+
+export function bytesToBase64url(bytes: Uint8Array) {
+    // const before = performance.now();
+    const m = bytes.length;
+    const k = m % 3;
+    const n = Math.floor(m / 3) * 4 + (k && k + 1);
+    const N = Math.ceil(m / 3) * 4;
+    const encoded = new Uint8Array(N);
+
+    for (let i = 0, j = 0; j < m; i += 4, j += 3) {
+        const y =
+            (bytes[j]! << 16) + (bytes[j + 1]! << 8) + (bytes[j + 2]! | 0);
+        encoded[i] = encodeLookup[y >> 18]!;
+        encoded[i + 1] = encodeLookup[(y >> 12) & 0x3f]!;
+        encoded[i + 2] = encodeLookup[(y >> 6) & 0x3f]!;
+        encoded[i + 3] = encodeLookup[y & 0x3f]!;
+    }
+
+    let base64 = decoder.decode(new Uint8Array(encoded.buffer, 0, n));
+    if (k === 1) base64 += "==";
+    if (k === 2) base64 += "=";
+    // const after = performance.now();
+    // console.log(
+    //     "bytesToBase64url bandwidth in MB/s for length",
+    //     (1000 * bytes.length / (after - before)) / (1024 * 1024),
+    //     bytes.length
+    // );
+    return base64;
+}
+
+const alphabet =
+    "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_";
+
+const lookup = new Uint8Array(128);
+for (const [i, a] of Array.from(alphabet).entries()) {
+    lookup[a.charCodeAt(0)] = i;
+}
+lookup["=".charCodeAt(0)] = 0;
+
+const encodeLookup = new Uint8Array(64);
+for (const [i, a] of Array.from(alphabet).entries()) {
+    encodeLookup[i] = a.charCodeAt(0);
+}

packages/cojson/src/coValue.test.ts

@@ -1,14 +1,296 @@
-import { Transaction } from "./coValue.js";
+import { accountOrAgentIDfromSessionID } from "./coValueCore.js";
+import { BinaryCoStream } from "./coValues/coStream.js";
+import { createdNowUnique } from "./crypto.js";
+import { cojsonReady } from "./index.js";
 import { LocalNode } from "./node.js";
-import { createdNowUnique, getAgentSignerSecret, newRandomAgentSecret, sign } from "./crypto.js";
 import { randomAnonymousAccountAndSessionID } from "./testUtils.js";
-import { CoMap, MapOpPayload } from "./contentTypes/coMap.js";
-import { AccountID } from "./index.js";
-import { Role } from "./permissions.js";
 
-test("Can create coValue with new agent credentials and add transaction to it", () => {
-    const [account, sessionID] = randomAnonymousAccountAndSessionID();
-    const node = new LocalNode(account, sessionID);
+beforeEach(async () => {
+    await cojsonReady;
+});
+
+test("Empty CoMap works", () => {
+    const node = new LocalNode(...randomAnonymousAccountAndSessionID());
+
+    const coValue = node.createCoValue({
+        type: "comap",
+        ruleset: { type: "unsafeAllowAll" },
+        meta: null,
+        ...createdNowUnique(),
+    });
+
+    const content = coValue.getCurrentContent();
+
+    if (content.type !== "comap") {
+        throw new Error("Expected map");
+    }
+
+    expect(content.type).toEqual("comap");
+    expect([...content.keys()]).toEqual([]);
+    expect(content.toJSON()).toEqual({});
+});
+
+test("Can insert and delete CoMap entries in edit()", () => {
+    const node = new LocalNode(...randomAnonymousAccountAndSessionID());
+
+    const coValue = node.createCoValue({
+        type: "comap",
+        ruleset: { type: "unsafeAllowAll" },
+        meta: null,
+        ...createdNowUnique(),
+    });
+
+    const content = coValue.getCurrentContent();
+
+    if (content.type !== "comap") {
+        throw new Error("Expected map");
+    }
+
+    expect(content.type).toEqual("comap");
+
+    content.edit((editable) => {
+        editable.set("hello", "world", "trusting");
+        expect(editable.get("hello")).toEqual("world");
+        editable.set("foo", "bar", "trusting");
+        expect(editable.get("foo")).toEqual("bar");
+        expect([...editable.keys()]).toEqual(["hello", "foo"]);
+        editable.delete("foo", "trusting");
+        expect(editable.get("foo")).toEqual(undefined);
+    });
+});
+
+test("Can get CoMap entry values at different points in time", () => {
+    const node = new LocalNode(...randomAnonymousAccountAndSessionID());
+
+    const coValue = node.createCoValue({
+        type: "comap",
+        ruleset: { type: "unsafeAllowAll" },
+        meta: null,
+        ...createdNowUnique(),
+    });
+
+    const content = coValue.getCurrentContent();
+
+    if (content.type !== "comap") {
+        throw new Error("Expected map");
+    }
+
+    expect(content.type).toEqual("comap");
+
+    content.edit((editable) => {
+        const beforeA = Date.now();
+        while (Date.now() < beforeA + 10) {}
+        editable.set("hello", "A", "trusting");
+        const beforeB = Date.now();
+        while (Date.now() < beforeB + 10) {}
+        editable.set("hello", "B", "trusting");
+        const beforeC = Date.now();
+        while (Date.now() < beforeC + 10) {}
+        editable.set("hello", "C", "trusting");
+        expect(editable.get("hello")).toEqual("C");
+        expect(editable.getAtTime("hello", Date.now())).toEqual("C");
+        expect(editable.getAtTime("hello", beforeA)).toEqual(undefined);
+        expect(editable.getAtTime("hello", beforeB)).toEqual("A");
+        expect(editable.getAtTime("hello", beforeC)).toEqual("B");
+    });
+});
+
+test("Can get all historic values of key in CoMap", () => {
+    const node = new LocalNode(...randomAnonymousAccountAndSessionID());
+
+    const coValue = node.createCoValue({
+        type: "comap",
+        ruleset: { type: "unsafeAllowAll" },
+        meta: null,
+        ...createdNowUnique(),
+    });
+
+    const content = coValue.getCurrentContent();
+
+    if (content.type !== "comap") {
+        throw new Error("Expected map");
+    }
+
+    expect(content.type).toEqual("comap");
+
+    content.edit((editable) => {
+        editable.set("hello", "A", "trusting");
+        const txA = editable.getLastTxID("hello");
+        editable.set("hello", "B", "trusting");
+        const txB = editable.getLastTxID("hello");
+        editable.delete("hello", "trusting");
+        const txDel = editable.getLastTxID("hello");
+        editable.set("hello", "C", "trusting");
+        const txC = editable.getLastTxID("hello");
+        expect(editable.getHistory("hello")).toEqual([
+            {
+                txID: txA,
+                value: "A",
+                at: txA && coValue.getTx(txA)?.madeAt,
+            },
+            {
+                txID: txB,
+                value: "B",
+                at: txB && coValue.getTx(txB)?.madeAt,
+            },
+            {
+                txID: txDel,
+                value: undefined,
+                at: txDel && coValue.getTx(txDel)?.madeAt,
+            },
+            {
+                txID: txC,
+                value: "C",
+                at: txC && coValue.getTx(txC)?.madeAt,
+            },
+        ]);
+    });
+});
+
+test("Can get last tx ID for a key in CoMap", () => {
+    const node = new LocalNode(...randomAnonymousAccountAndSessionID());
+
+    const coValue = node.createCoValue({
+        type: "comap",
+        ruleset: { type: "unsafeAllowAll" },
+        meta: null,
+        ...createdNowUnique(),
+    });
+
+    const content = coValue.getCurrentContent();
+
+    if (content.type !== "comap") {
+        throw new Error("Expected map");
+    }
+
+    expect(content.type).toEqual("comap");
+
+    content.edit((editable) => {
+        expect(editable.getLastTxID("hello")).toEqual(undefined);
+        editable.set("hello", "A", "trusting");
+        const sessionID = editable.getLastTxID("hello")?.sessionID;
+        expect(sessionID && accountOrAgentIDfromSessionID(sessionID)).toEqual(
+            node.account.id
+        );
+        expect(editable.getLastTxID("hello")?.txIndex).toEqual(0);
+        editable.set("hello", "B", "trusting");
+        expect(editable.getLastTxID("hello")?.txIndex).toEqual(1);
+        editable.set("hello", "C", "trusting");
+        expect(editable.getLastTxID("hello")?.txIndex).toEqual(2);
+    });
+});
+
+test("Empty CoList works", () => {
+    const node = new LocalNode(...randomAnonymousAccountAndSessionID());
+
+    const coValue = node.createCoValue({
+        type: "colist",
+        ruleset: { type: "unsafeAllowAll" },
+        meta: null,
+        ...createdNowUnique(),
+    });
+
+    const content = coValue.getCurrentContent();
+
+    if (content.type !== "colist") {
+        throw new Error("Expected list");
+    }
+
+    expect(content.type).toEqual("colist");
+    expect(content.toJSON()).toEqual([]);
+});
+
+test("Can append, prepend and delete items to CoList", () => {
+    const node = new LocalNode(...randomAnonymousAccountAndSessionID());
+
+    const coValue = node.createCoValue({
+        type: "colist",
+        ruleset: { type: "unsafeAllowAll" },
+        meta: null,
+        ...createdNowUnique(),
+    });
+
+    const content = coValue.getCurrentContent();
+
+    if (content.type !== "colist") {
+        throw new Error("Expected list");
+    }
+
+    expect(content.type).toEqual("colist");
+
+    content.edit((editable) => {
+        editable.append(0, "hello", "trusting");
+        expect(editable.toJSON()).toEqual(["hello"]);
+        editable.append(0, "world", "trusting");
+        expect(editable.toJSON()).toEqual(["hello", "world"]);
+        editable.prepend(1, "beautiful", "trusting");
+        expect(editable.toJSON()).toEqual(["hello", "beautiful", "world"]);
+        editable.prepend(3, "hooray", "trusting");
+        expect(editable.toJSON()).toEqual([
+            "hello",
+            "beautiful",
+            "world",
+            "hooray",
+        ]);
+        editable.delete(2, "trusting");
+        expect(editable.toJSON()).toEqual(["hello", "beautiful", "hooray"]);
+    });
+});
+
+test("Push is equivalent to append after last item", () => {
+    const node = new LocalNode(...randomAnonymousAccountAndSessionID());
+
+    const coValue = node.createCoValue({
+        type: "colist",
+        ruleset: { type: "unsafeAllowAll" },
+        meta: null,
+        ...createdNowUnique(),
+    });
+
+    const content = coValue.getCurrentContent();
+
+    if (content.type !== "colist") {
+        throw new Error("Expected list");
+    }
+
+    expect(content.type).toEqual("colist");
+
+    content.edit((editable) => {
+        editable.append(0, "hello", "trusting");
+        expect(editable.toJSON()).toEqual(["hello"]);
+        editable.push("world", "trusting");
+        expect(editable.toJSON()).toEqual(["hello", "world"]);
+        editable.push("hooray", "trusting");
+        expect(editable.toJSON()).toEqual(["hello", "world", "hooray"]);
+    });
+});
+
+test("Can push into empty list", () => {
+    const node = new LocalNode(...randomAnonymousAccountAndSessionID());
+
+    const coValue = node.createCoValue({
+        type: "colist",
+        ruleset: { type: "unsafeAllowAll" },
+        meta: null,
+        ...createdNowUnique(),
+    });
+
+    const content = coValue.getCurrentContent();
+
+    if (content.type !== "colist") {
+        throw new Error("Expected list");
+    }
+
+    expect(content.type).toEqual("colist");
+
+    content.edit((editable) => {
+        editable.push("hello", "trusting");
+        expect(editable.toJSON()).toEqual(["hello"]);
+    });
+});
+
+test("Empty CoStream works", () => {
+    const node = new LocalNode(...randomAnonymousAccountAndSessionID());
 
     const coValue = node.createCoValue({
         type: "costream",
@@ -17,35 +299,19 @@ test("Can create coValue with new agent credentials and add transaction to it",
         ...createdNowUnique(),
     });
 
-    const transaction: Transaction = {
-        privacy: "trusting",
-        madeAt: Date.now(),
-        changes: [
-            {
-                hello: "world",
-            },
-        ],
-    };
+    const content = coValue.getCurrentContent();
 
-    const { expectedNewHash } = coValue.expectedNewHashAfter(
-        node.currentSessionID,
-        [transaction]
-    );
+    if (content.type !== "costream") {
+        throw new Error("Expected stream");
+    }
 
-    expect(
-        coValue.tryAddTransactions(
-            node.currentSessionID,
-            [transaction],
-            expectedNewHash,
-            sign(account.currentSignerSecret(), expectedNewHash)
-        )
-    ).toBe(true);
+    expect(content.type).toEqual("costream");
+    expect(content.toJSON()).toEqual({});
+    expect(content.getSingleStream()).toEqual(undefined);
 });
 
-test("transactions with wrong signature are rejected", () => {
-    const wrongAgent = newRandomAgentSecret();
-    const [agentSecret, sessionID] = randomAnonymousAccountAndSessionID();
-    const node = new LocalNode(agentSecret, sessionID);
+test("Can push into CoStream", () => {
+    const node = new LocalNode(...randomAnonymousAccountAndSessionID());
 
     const coValue = node.createCoValue({
         type: "costream",
@@ -54,128 +320,96 @@ test("transactions with wrong signature are rejected", () => {
         ...createdNowUnique(),
     });
 
-    const transaction: Transaction = {
-        privacy: "trusting",
-        madeAt: Date.now(),
-        changes: [
-            {
-                hello: "world",
-            },
-        ],
-    };
+    const content = coValue.getCurrentContent();
 
-    const { expectedNewHash } = coValue.expectedNewHashAfter(
-        node.currentSessionID,
-        [transaction]
-    );
+    if (content.type !== "costream") {
+        throw new Error("Expected stream");
+    }
 
-    expect(
-        coValue.tryAddTransactions(
-            node.currentSessionID,
-            [transaction],
-            expectedNewHash,
-            sign(getAgentSignerSecret(wrongAgent), expectedNewHash)
-        )
-    ).toBe(false);
+    content.edit((editable) => {
+        editable.push({ hello: "world" }, "trusting");
+        expect(editable.toJSON()).toEqual({
+            [node.currentSessionID]: [{ hello: "world" }],
+        });
+        editable.push({ foo: "bar" }, "trusting");
+        expect(editable.toJSON()).toEqual({
+            [node.currentSessionID]: [{ hello: "world" }, { foo: "bar" }],
+        });
+        expect(editable.getSingleStream()).toEqual([
+            { hello: "world" },
+            { foo: "bar" },
+        ]);
+    });
 });
 
-test("transactions with correctly signed, but wrong hash are rejected", () => {
-    const [account, sessionID] = randomAnonymousAccountAndSessionID();
-    const node = new LocalNode(account, sessionID);
+test("Empty BinaryCoStream works", () => {
+    const node = new LocalNode(...randomAnonymousAccountAndSessionID());
 
     const coValue = node.createCoValue({
         type: "costream",
         ruleset: { type: "unsafeAllowAll" },
-        meta: null,
+        meta: { type: "binary" },
         ...createdNowUnique(),
     });
 
-    const transaction: Transaction = {
-        privacy: "trusting",
-        madeAt: Date.now(),
-        changes: [
-            {
-                hello: "world",
-            },
-        ],
-    };
+    const content = coValue.getCurrentContent();
 
-    const { expectedNewHash } = coValue.expectedNewHashAfter(
-        node.currentSessionID,
-        [
-            {
-                privacy: "trusting",
-                madeAt: Date.now(),
-                changes: [
-                    {
-                        hello: "wrong",
-                    },
-                ],
-            },
-        ]
-    );
+    if (content.type !== "costream" || content.meta?.type !== "binary" || !(content instanceof BinaryCoStream)) {
+        throw new Error("Expected binary stream");
+    }
 
-    expect(
-        coValue.tryAddTransactions(
-            node.currentSessionID,
-            [transaction],
-            expectedNewHash,
-            sign(account.currentSignerSecret(), expectedNewHash)
-        )
-    ).toBe(false);
+    expect(content.type).toEqual("costream");
+    expect(content.meta.type).toEqual("binary");
+    expect(content.toJSON()).toEqual({});
+    expect(content.getBinaryChunks()).toEqual(undefined);
 });
 
-test("New transactions in a group correctly update owned values, including subscriptions", async () => {
-    const [account, sessionID] = randomAnonymousAccountAndSessionID();
-    const node = new LocalNode(account, sessionID);
+test("Can push into BinaryCoStream", () => {
+    const node = new LocalNode(...randomAnonymousAccountAndSessionID());
 
-    const group = node.createGroup();
-
-    const timeBeforeEdit = Date.now();
-
-    await new Promise((resolve) => setTimeout(resolve, 10));
-
-    let map = group.createMap();
-
-    let mapAfterEdit = map.edit((map) => {
-        map.set("hello", "world");
+    const coValue = node.createCoValue({
+        type: "costream",
+        ruleset: { type: "unsafeAllowAll" },
+        meta: { type: "binary" },
+        ...createdNowUnique(),
     });
 
-    const listener = jest.fn().mockImplementation();
+    const content = coValue.getCurrentContent();
 
-    map.subscribe(listener);
+    if (content.type !== "costream" || content.meta?.type !== "binary" || !(content instanceof BinaryCoStream)) {
+        throw new Error("Expected binary stream");
+    }
 
-    expect(listener.mock.calls[0][0].get("hello")).toBe("world");
+    content.edit((editable) => {
+        editable.startBinaryStream({mimeType: "text/plain", fileName: "test.txt"}, "trusting");
+        expect(editable.getBinaryChunks()).toEqual({
+            mimeType: "text/plain",
+            fileName: "test.txt",
+            chunks: [],
+            finished: false,
+        });
+        editable.pushBinaryStreamChunk(new Uint8Array([1, 2, 3]), "trusting");
+        expect(editable.getBinaryChunks()).toEqual({
+            mimeType: "text/plain",
+            fileName: "test.txt",
+            chunks: [new Uint8Array([1, 2, 3])],
+            finished: false,
+        });
+        editable.pushBinaryStreamChunk(new Uint8Array([4, 5, 6]), "trusting");
 
-    const resignationThatWeJustLearnedAbout = {
-        privacy: "trusting",
-        madeAt: timeBeforeEdit,
-        changes: [
-            {
-                op: "set",
-                key: account.id,
-                value: "revoked"
-            } satisfies MapOpPayload<typeof account.id, Role>
-        ]
-    } satisfies Transaction;
+        expect(editable.getBinaryChunks()).toEqual({
+            mimeType: "text/plain",
+            fileName: "test.txt",
+            chunks: [new Uint8Array([1, 2, 3]), new Uint8Array([4, 5, 6])],
+            finished: false,
+        });
 
-    const { expectedNewHash } = group.groupMap.coValue.expectedNewHashAfter(sessionID, [
-        resignationThatWeJustLearnedAbout,
-    ]);
-
-    const signature = sign(
-        node.account.currentSignerSecret(),
-        expectedNewHash
-    );
-
-    expect(map.coValue.getValidSortedTransactions().length).toBe(1);
-
-    const manuallyAdddedTxSuccess = group.groupMap.coValue.tryAddTransactions(node.currentSessionID, [resignationThatWeJustLearnedAbout], expectedNewHash, signature);
-
-    expect(manuallyAdddedTxSuccess).toBe(true);
-
-    expect(listener.mock.calls.length).toBe(2);
-    expect(listener.mock.calls[1][0].get("hello")).toBe(undefined);
-
-    expect(map.coValue.getValidSortedTransactions().length).toBe(0);
+        editable.endBinaryStream("trusting");
+        expect(editable.getBinaryChunks()).toEqual({
+            mimeType: "text/plain",
+            fileName: "test.txt",
+            chunks: [new Uint8Array([1, 2, 3]), new Uint8Array([4, 5, 6])],
+            finished: true,
+        });
+    });
 });

packages/cojson/src/coValue.ts

@@ -1,588 +1,64 @@
-import { randomBytes } from "@noble/hashes/utils";
-import { ContentType } from "./contentType.js";
-import { Static } from "./contentTypes/static.js";
-import { CoStream } from "./contentTypes/coStream.js";
-import { CoMap } from "./contentTypes/coMap.js";
-import {
-    Encrypted,
-    Hash,
-    KeySecret,
-    Signature,
-    StreamingHash,
-    unseal,
-    shortHash,
-    sign,
-    verify,
-    encryptForTransaction,
-    decryptForTransaction,
-    KeyID,
-    decryptKeySecret,
-    getAgentSignerID,
-    getAgentSealerID,
-} from "./crypto.js";
 import { JsonObject, JsonValue } from "./jsonValue.js";
-import { base58 } from "@scure/base";
-import {
-    PermissionsDef as RulesetDef,
-    determineValidTransactions,
-    isKeyForKeyField,
-} from "./permissions.js";
-import { Group, expectGroupContent } from "./group.js";
-import { LocalNode } from "./node.js";
-import { CoValueKnownState, NewContentMessage } from "./sync.js";
-import { AgentID, RawCoID, SessionID, TransactionID } from "./ids.js";
-import { CoList } from "./contentTypes/coList.js";
-import {
-    AccountID,
-    GeneralizedControlledAccount,
-} from "./account.js";
+import { RawCoID } from "./ids.js";
+import { CoMap } from "./coValues/coMap.js";
+import { BinaryCoStream, BinaryCoStreamMeta, CoStream } from "./coValues/coStream.js";
+import { Static } from "./coValues/static.js";
+import { CoList } from "./coValues/coList.js";
+import { CoValueCore } from "./coValueCore.js";
+import { Group } from "./group.js";
 
-export type CoValueHeader = {
-    type: ContentType["type"];
-    ruleset: RulesetDef;
+export type CoID<T extends CoValueImpl> = RawCoID & {
+    readonly __type: T;
+};
+
+export interface ReadableCoValue extends CoValue {
+    /** Lets you subscribe to future updates to this CoValue (whether made locally or by other users).
+     *
+     * Takes a listener function that will be called with the current state for each update.
+     *
+     * Returns an unsubscribe function.
+     *
+     * Used internally by `useTelepathicData()` for reactive updates on changes to a `CoValue`. */
+    subscribe(listener: (coValue: CoValueImpl) => void): () => void;
+    /** Lets you apply edits to a `CoValue`, inside the changer callback, which receives a `WriteableCoValue`.
+     *
+     *  A `WritableCoValue` has all the same methods as a `CoValue`, but all edits made to it (with its additional mutator methods)
+     *  are reflected in it immediately - so it behaves mutably, whereas a `CoValue` is always immutable
+     *  (you need to use `subscribe` to receive new versions of it). */
+    edit?:
+        | ((changer: (editable: WriteableCoValue) => void) => CoValueImpl)
+        | undefined;
+}
+
+export interface CoValue {
+    /** The `CoValue`'s (precisely typed) `CoID` */
+    id: CoID<CoValueImpl>;
+    core: CoValueCore;
+    /** Specifies which kind of `CoValue` this is */
+    type: CoValueImpl["type"];
+    /** The `CoValue`'s (precisely typed) static metadata */
     meta: JsonObject | null;
-    createdAt: `2${string}` | null;
-    uniqueness: `z${string}` | null;
-};
-
-export function idforHeader(header: CoValueHeader): RawCoID {
-    const hash = shortHash(header);
-    return `co_z${hash.slice("shortHash_z".length)}`;
+    /** The `Group` this `CoValue` belongs to (determining permissions) */
+    group: Group;
+    /** Returns an immutable JSON presentation of this `CoValue` */
+    toJSON(): JsonValue;
 }
 
-export function accountOrAgentIDfromSessionID(
-    sessionID: SessionID
-): AccountID | AgentID {
-    return sessionID.split("_session")[0] as AccountID | AgentID;
-}
-
-export function newRandomSessionID(accountID: AccountID | AgentID): SessionID {
-    return `${accountID}_session_z${base58.encode(randomBytes(8))}`;
-}
-
-type SessionLog = {
-    transactions: Transaction[];
-    lastHash?: Hash;
-    streamingHash: StreamingHash;
-    lastSignature: Signature;
-};
-
-export type PrivateTransaction = {
-    privacy: "private";
-    madeAt: number;
-    keyUsed: KeyID;
-    encryptedChanges: Encrypted<
-        JsonValue[],
-        { in: RawCoID; tx: TransactionID }
-    >;
-};
-
-export type TrustingTransaction = {
-    privacy: "trusting";
-    madeAt: number;
-    changes: JsonValue[];
-};
-
-export type Transaction = PrivateTransaction | TrustingTransaction;
-
-export type DecryptedTransaction = {
-    txID: TransactionID;
-    changes: JsonValue[];
-    madeAt: number;
-};
-
-const readKeyCache = new WeakMap<CoValue, { [id: KeyID]: KeySecret }>();
-
-export class CoValue {
-    id: RawCoID;
-    node: LocalNode;
-    header: CoValueHeader;
-    _sessions: { [key: SessionID]: SessionLog };
-    _cachedContent?: ContentType;
-    listeners: Set<(content?: ContentType) => void> = new Set();
-
-    constructor(
-        header: CoValueHeader,
-        node: LocalNode,
-        internalInitSessions: { [key: SessionID]: SessionLog } = {}
-    ) {
-        this.id = idforHeader(header);
-        this.header = header;
-        this._sessions = internalInitSessions;
-        this.node = node;
-
-        if (header.ruleset.type == "ownedByGroup") {
-            this.node
-                .expectCoValueLoaded(header.ruleset.group)
-                .subscribe((_groupUpdate) => {
-                    this._cachedContent = undefined;
-                    const newContent = this.getCurrentContent();
-                    for (const listener of this.listeners) {
-                        listener(newContent);
-                    }
-                });
-        }
-    }
-
-    get sessions(): Readonly<{ [key: SessionID]: SessionLog }> {
-        return this._sessions;
-    }
-
-    testWithDifferentAccount(
-        account: GeneralizedControlledAccount,
-        currentSessionID: SessionID
-    ): CoValue {
-        const newNode = this.node.testWithDifferentAccount(
-            account,
-            currentSessionID
-        );
-
-        return newNode.expectCoValueLoaded(this.id);
-    }
-
-    knownState(): CoValueKnownState {
-        return {
-            id: this.id,
-            header: true,
-            sessions: Object.fromEntries(
-                Object.entries(this.sessions).map(([k, v]) => [
-                    k,
-                    v.transactions.length,
-                ])
-            ),
-        };
-    }
-
-    get meta(): JsonValue {
-        return this.header?.meta ?? null;
-    }
-
-    nextTransactionID(): TransactionID {
-        const sessionID = this.node.currentSessionID;
-        return {
-            sessionID,
-            txIndex: this.sessions[sessionID]?.transactions.length || 0,
-        };
-    }
-
-    tryAddTransactions(
-        sessionID: SessionID,
-        newTransactions: Transaction[],
-        givenExpectedNewHash: Hash | undefined,
-        newSignature: Signature
-    ): boolean {
-        const signerID = getAgentSignerID(
-            this.node.resolveAccountAgent(
-                accountOrAgentIDfromSessionID(sessionID),
-                "Expected to know signer of transaction"
-            )
-        );
-
-        if (!signerID) {
-            console.warn(
-                "Unknown agent",
-                accountOrAgentIDfromSessionID(sessionID)
-            );
-            return false;
-        }
-
-        const { expectedNewHash, newStreamingHash } = this.expectedNewHashAfter(
-            sessionID,
-            newTransactions
-        );
-
-        if (givenExpectedNewHash && givenExpectedNewHash !== expectedNewHash) {
-            console.warn("Invalid hash", {
-                expectedNewHash,
-                givenExpectedNewHash,
-            });
-            return false;
-        }
-
-        if (!verify(newSignature, expectedNewHash, signerID)) {
-            console.warn(
-                "Invalid signature",
-                newSignature,
-                expectedNewHash,
-                signerID
-            );
-            return false;
-        }
-
-        const transactions = this.sessions[sessionID]?.transactions ?? [];
-
-        transactions.push(...newTransactions);
-
-        this._sessions[sessionID] = {
-            transactions,
-            lastHash: expectedNewHash,
-            streamingHash: newStreamingHash,
-            lastSignature: newSignature,
-        };
-
-        this._cachedContent = undefined;
-
-        const content = this.getCurrentContent();
-
-        for (const listener of this.listeners) {
-            listener(content);
-        }
-
-        return true;
-    }
-
-    subscribe(listener: (content?: ContentType) => void): () => void {
-        this.listeners.add(listener);
-        listener(this.getCurrentContent());
-
-        return () => {
-            this.listeners.delete(listener);
-        };
-    }
-
-    expectedNewHashAfter(
-        sessionID: SessionID,
-        newTransactions: Transaction[]
-    ): { expectedNewHash: Hash; newStreamingHash: StreamingHash } {
-        const streamingHash =
-            this.sessions[sessionID]?.streamingHash.clone() ??
-            new StreamingHash();
-        for (const transaction of newTransactions) {
-            streamingHash.update(transaction);
-        }
-
-        const newStreamingHash = streamingHash.clone();
-
-        return {
-            expectedNewHash: streamingHash.digest(),
-            newStreamingHash,
-        };
-    }
-
-    makeTransaction(
-        changes: JsonValue[],
-        privacy: "private" | "trusting"
-    ): boolean {
-        const madeAt = Date.now();
-
-        let transaction: Transaction;
-
-        if (privacy === "private") {
-            const { secret: keySecret, id: keyID } = this.getCurrentReadKey();
-
-            if (!keySecret) {
-                throw new Error(
-                    "Can't make transaction without read key secret"
-                );
-            }
-
-            transaction = {
-                privacy: "private",
-                madeAt,
-                keyUsed: keyID,
-                encryptedChanges: encryptForTransaction(changes, keySecret, {
-                    in: this.id,
-                    tx: this.nextTransactionID(),
-                }),
-            };
-        } else {
-            transaction = {
-                privacy: "trusting",
-                madeAt,
-                changes,
-            };
-        }
-
-        const sessionID = this.node.currentSessionID;
-
-        const { expectedNewHash } = this.expectedNewHashAfter(sessionID, [
-            transaction,
-        ]);
-
-        const signature = sign(
-            this.node.account.currentSignerSecret(),
-            expectedNewHash
-        );
-
-        const success = this.tryAddTransactions(
-            sessionID,
-            [transaction],
-            expectedNewHash,
-            signature
-        );
-
-        if (success) {
-            void this.node.sync.syncCoValue(this);
-        }
-
-        return success;
-    }
-
-    getCurrentContent(): ContentType {
-        if (this._cachedContent) {
-            return this._cachedContent;
-        }
-
-        if (this.header.type === "comap") {
-            this._cachedContent = new CoMap(this);
-        } else if (this.header.type === "colist") {
-            this._cachedContent = new CoList(this);
-        } else if (this.header.type === "costream") {
-            this._cachedContent = new CoStream(this);
-        } else if (this.header.type === "static") {
-            this._cachedContent = new Static(this);
-        } else {
-            throw new Error(`Unknown coValue type ${this.header.type}`);
-        }
-
-        return this._cachedContent;
-    }
-
-    getValidSortedTransactions(): DecryptedTransaction[] {
-        const validTransactions = determineValidTransactions(this);
-
-        const allTransactions: DecryptedTransaction[] = validTransactions
-            .map(({ txID, tx }) => {
-                if (tx.privacy === "trusting") {
-                    return {
-                        txID,
-                        madeAt: tx.madeAt,
-                        changes: tx.changes,
-                    };
-                } else {
-                    const readKey = this.getReadKey(tx.keyUsed);
-
-                    if (!readKey) {
-                        return undefined;
-                    } else {
-                        const decrytedChanges = decryptForTransaction(
-                            tx.encryptedChanges,
-                            readKey,
-                            {
-                                in: this.id,
-                                tx: txID,
-                            }
-                        );
-
-                        if (!decrytedChanges) {
-                            console.error(
-                                "Failed to decrypt transaction despite having key"
-                            );
-                            return undefined;
-                        }
-                        return {
-                            txID,
-                            madeAt: tx.madeAt,
-                            changes: decrytedChanges,
-                        };
-                    }
-                }
-            })
-            .filter((x): x is Exclude<typeof x, undefined> => !!x);
-        allTransactions.sort(
-            (a, b) =>
-                a.madeAt - b.madeAt ||
-                (a.txID.sessionID < b.txID.sessionID ? -1 : 1) ||
-                a.txID.txIndex - b.txID.txIndex
-        );
-
-        return allTransactions;
-    }
-
-    getCurrentReadKey(): { secret: KeySecret | undefined; id: KeyID } {
-        if (this.header.ruleset.type === "group") {
-            const content = expectGroupContent(this.getCurrentContent());
-
-            const currentKeyId = content.get("readKey");
-
-            if (!currentKeyId) {
-                throw new Error("No readKey set");
-            }
-
-            const secret = this.getReadKey(currentKeyId);
-
-            return {
-                secret: secret,
-                id: currentKeyId,
-            };
-        } else if (this.header.ruleset.type === "ownedByGroup") {
-            return this.node
-                .expectCoValueLoaded(this.header.ruleset.group)
-                .getCurrentReadKey();
-        } else {
-            throw new Error(
-                "Only groups or values owned by groups have read secrets"
-            );
-        }
-    }
-
-    getReadKey(keyID: KeyID): KeySecret | undefined {
-        if (readKeyCache.get(this)?.[keyID]) {
-            return readKeyCache.get(this)?.[keyID];
-        }
-        if (this.header.ruleset.type === "group") {
-            const content = expectGroupContent(this.getCurrentContent());
-
-            // Try to find key revelation for us
-
-            const readKeyEntry = content.getLastEntry(
-                `${keyID}_for_${this.node.account.id}`
-            );
-
-            if (readKeyEntry) {
-                const revealer = accountOrAgentIDfromSessionID(
-                    readKeyEntry.txID.sessionID
-                );
-                const revealerAgent = this.node.resolveAccountAgent(
-                    revealer,
-                    "Expected to know revealer"
-                );
-
-                const secret = unseal(
-                    readKeyEntry.value,
-                    this.node.account.currentSealerSecret(),
-                    getAgentSealerID(revealerAgent),
-                    {
-                        in: this.id,
-                        tx: readKeyEntry.txID,
-                    }
-                );
-
-                if (secret) {
-                    let cache = readKeyCache.get(this);
-                    if (!cache) {
-                        cache = {};
-                        readKeyCache.set(this, cache);
-                    }
-                    cache[keyID] = secret;
-
-                    return secret as KeySecret;
-                }
-            }
-
-            // Try to find indirect revelation through previousKeys
-
-            for (const field of content.keys()) {
-                if (isKeyForKeyField(field) && field.startsWith(keyID)) {
-                    const encryptingKeyID = field.split("_for_")[1] as KeyID;
-                    const encryptingKeySecret =
-                        this.getReadKey(encryptingKeyID);
-
-                    if (!encryptingKeySecret) {
-                        continue;
-                    }
-
-                    const encryptedPreviousKey = content.get(field)!;
-
-                    const secret = decryptKeySecret(
-                        {
-                            encryptedID: keyID,
-                            encryptingID: encryptingKeyID,
-                            encrypted: encryptedPreviousKey,
-                        },
-                        encryptingKeySecret
-                    );
-
-                    if (secret) {
-                        let cache = readKeyCache.get(this);
-                        if (!cache) {
-                            cache = {};
-                            readKeyCache.set(this, cache);
-                        }
-                        cache[keyID] = secret;
-
-                        return secret as KeySecret;
-                    } else {
-                        console.error(
-                            `Encrypting ${encryptingKeyID} key didn't decrypt ${keyID}`
-                        );
-                    }
-                }
-            }
-
-            return undefined;
-        } else if (this.header.ruleset.type === "ownedByGroup") {
-            return this.node
-                .expectCoValueLoaded(this.header.ruleset.group)
-                .getReadKey(keyID);
-        } else {
-            throw new Error(
-                "Only groups or values owned by groups have read secrets"
-            );
-        }
-    }
-
-    getGroup(): Group {
-        if (this.header.ruleset.type !== "ownedByGroup") {
-            throw new Error("Only values owned by groups have groups");
-        }
-
-        return new Group(
-            expectGroupContent(
-                this.node
-                    .expectCoValueLoaded(this.header.ruleset.group)
-                    .getCurrentContent()
-            ),
-            this.node
-        );
-    }
-
-    getTx(txID: TransactionID): Transaction | undefined {
-        return this.sessions[txID.sessionID]?.transactions[txID.txIndex];
-    }
-
-    newContentSince(
-        knownState: CoValueKnownState | undefined
-    ): NewContentMessage | undefined {
-        const newContent: NewContentMessage = {
-            action: "content",
-            id: this.id,
-            header: knownState?.header ? undefined : this.header,
-            new: Object.fromEntries(
-                Object.entries(this.sessions)
-                    .map(([sessionID, log]) => {
-                        const newTransactions = log.transactions.slice(
-                            knownState?.sessions[sessionID as SessionID] || 0
-                        );
-
-                        if (
-                            newTransactions.length === 0 ||
-                            !log.lastHash ||
-                            !log.lastSignature
-                        ) {
-                            return undefined;
-                        }
-
-                        return [
-                            sessionID,
-                            {
-                                after:
-                                    knownState?.sessions[
-                                        sessionID as SessionID
-                                    ] || 0,
-                                newTransactions,
-                                lastSignature: log.lastSignature,
-                            },
-                        ];
-                    })
-                    .filter((x): x is Exclude<typeof x, undefined> => !!x)
-            ),
-        };
-
-        if (!newContent.header && Object.keys(newContent.new).length === 0) {
-            return undefined;
-        }
-
-        return newContent;
-    }
-
-    getDependedOnCoValues(): RawCoID[] {
-        return this.header.ruleset.type === "group"
-            ? expectGroupContent(this.getCurrentContent())
-                  .keys()
-                  .filter((k): k is AccountID => k.startsWith("co_"))
-            : this.header.ruleset.type === "ownedByGroup"
-            ? [this.header.ruleset.group]
-            : [];
-    }
+export interface WriteableCoValue extends CoValue {}
+
+export type CoValueImpl =
+    | CoMap<{ [key: string]: JsonValue | undefined; }, JsonObject | null>
+    | CoList<JsonValue, JsonObject | null>
+    | CoStream<JsonValue, JsonObject | null>
+    | BinaryCoStream<BinaryCoStreamMeta>
+    | Static<JsonObject>;
+
+export function expectMap(
+    content: CoValueImpl
+): CoMap<{ [key: string]: string }, JsonObject | null> {
+    if (content.type !== "comap") {
+        throw new Error("Expected map");
+    }
+
+    return content as CoMap<{ [key: string]: string }, JsonObject | null>;
 }

packages/cojson/src/coValueCore.test.ts

@@ -0,0 +1,186 @@
+import { Transaction } from "./coValueCore.js";
+import { LocalNode } from "./node.js";
+import { createdNowUnique, getAgentSignerSecret, newRandomAgentSecret, sign } from "./crypto.js";
+import { randomAnonymousAccountAndSessionID } from "./testUtils.js";
+import { MapOpPayload } from "./coValues/coMap.js";
+import { Role } from "./permissions.js";
+import { cojsonReady } from "./index.js";
+import { stableStringify } from "./jsonStringify.js";
+
+beforeEach(async () => {
+    await cojsonReady;
+});
+
+test("Can create coValue with new agent credentials and add transaction to it", () => {
+    const [account, sessionID] = randomAnonymousAccountAndSessionID();
+    const node = new LocalNode(account, sessionID);
+
+    const coValue = node.createCoValue({
+        type: "costream",
+        ruleset: { type: "unsafeAllowAll" },
+        meta: null,
+        ...createdNowUnique(),
+    });
+
+    const transaction: Transaction = {
+        privacy: "trusting",
+        madeAt: Date.now(),
+        changes: stableStringify([
+            {
+                hello: "world",
+            },
+        ]),
+    };
+
+    const { expectedNewHash } = coValue.expectedNewHashAfter(
+        node.currentSessionID,
+        [transaction]
+    );
+
+    expect(
+        coValue.tryAddTransactions(
+            node.currentSessionID,
+            [transaction],
+            expectedNewHash,
+            sign(account.currentSignerSecret(), expectedNewHash)
+        )
+    ).toBe(true);
+});
+
+test("transactions with wrong signature are rejected", () => {
+    const wrongAgent = newRandomAgentSecret();
+    const [agentSecret, sessionID] = randomAnonymousAccountAndSessionID();
+    const node = new LocalNode(agentSecret, sessionID);
+
+    const coValue = node.createCoValue({
+        type: "costream",
+        ruleset: { type: "unsafeAllowAll" },
+        meta: null,
+        ...createdNowUnique(),
+    });
+
+    const transaction: Transaction = {
+        privacy: "trusting",
+        madeAt: Date.now(),
+        changes: stableStringify([
+            {
+                hello: "world",
+            },
+        ]),
+    };
+
+    const { expectedNewHash } = coValue.expectedNewHashAfter(
+        node.currentSessionID,
+        [transaction]
+    );
+
+    expect(
+        coValue.tryAddTransactions(
+            node.currentSessionID,
+            [transaction],
+            expectedNewHash,
+            sign(getAgentSignerSecret(wrongAgent), expectedNewHash)
+        )
+    ).toBe(false);
+});
+
+test("transactions with correctly signed, but wrong hash are rejected", () => {
+    const [account, sessionID] = randomAnonymousAccountAndSessionID();
+    const node = new LocalNode(account, sessionID);
+
+    const coValue = node.createCoValue({
+        type: "costream",
+        ruleset: { type: "unsafeAllowAll" },
+        meta: null,
+        ...createdNowUnique(),
+    });
+
+    const transaction: Transaction = {
+        privacy: "trusting",
+        madeAt: Date.now(),
+        changes: stableStringify([
+            {
+                hello: "world",
+            },
+        ]),
+    };
+
+    const { expectedNewHash } = coValue.expectedNewHashAfter(
+        node.currentSessionID,
+        [
+            {
+                privacy: "trusting",
+                madeAt: Date.now(),
+                changes: stableStringify([
+                    {
+                        hello: "wrong",
+                    },
+                ]),
+            },
+        ]
+    );
+
+    expect(
+        coValue.tryAddTransactions(
+            node.currentSessionID,
+            [transaction],
+            expectedNewHash,
+            sign(account.currentSignerSecret(), expectedNewHash)
+        )
+    ).toBe(false);
+});
+
+test("New transactions in a group correctly update owned values, including subscriptions", async () => {
+    const [account, sessionID] = randomAnonymousAccountAndSessionID();
+    const node = new LocalNode(account, sessionID);
+
+    const group = node.createGroup();
+
+    const timeBeforeEdit = Date.now();
+
+    await new Promise((resolve) => setTimeout(resolve, 10));
+
+    let map = group.createMap();
+
+    let mapAfterEdit = map.edit((map) => {
+        map.set("hello", "world");
+    });
+
+    const listener = jest.fn().mockImplementation();
+
+    map.subscribe(listener);
+
+    expect(listener.mock.calls[0][0].get("hello")).toBe("world");
+
+    const resignationThatWeJustLearnedAbout = {
+        privacy: "trusting",
+        madeAt: timeBeforeEdit,
+        changes: stableStringify([
+            {
+                op: "set",
+                key: account.id,
+                value: "revoked"
+            } satisfies MapOpPayload<typeof account.id, Role>
+        ])
+    } satisfies Transaction;
+
+    const { expectedNewHash } = group.underlyingMap.core.expectedNewHashAfter(sessionID, [
+        resignationThatWeJustLearnedAbout,
+    ]);
+
+    const signature = sign(
+        node.account.currentSignerSecret(),
+        expectedNewHash
+    );
+
+    expect(map.core.getValidSortedTransactions().length).toBe(1);
+
+    const manuallyAdddedTxSuccess = group.underlyingMap.core.tryAddTransactions(node.currentSessionID, [resignationThatWeJustLearnedAbout], expectedNewHash, signature);
+
+    expect(manuallyAdddedTxSuccess).toBe(true);
+
+    expect(listener.mock.calls.length).toBe(2);
+    expect(listener.mock.calls[1][0].get("hello")).toBe(undefined);
+
+    expect(map.core.getValidSortedTransactions().length).toBe(0);
+});

packages/cojson/src/coValueCore.ts

@@ -0,0 +1,738 @@
+import { randomBytes } from "@noble/hashes/utils";
+import { CoValueImpl } from "./coValue.js";
+import { Static } from "./coValues/static.js";
+import { BinaryCoStream, CoStream } from "./coValues/coStream.js";
+import { CoMap } from "./coValues/coMap.js";
+import {
+    Encrypted,
+    Hash,
+    KeySecret,
+    Signature,
+    StreamingHash,
+    unseal,
+    shortHash,
+    sign,
+    verify,
+    encryptForTransaction,
+    decryptForTransaction,
+    KeyID,
+    decryptKeySecret,
+    getAgentSignerID,
+    getAgentSealerID,
+    decryptRawForTransaction,
+} from "./crypto.js";
+import { JsonObject, JsonValue } from "./jsonValue.js";
+import { base58 } from "@scure/base";
+import {
+    PermissionsDef as RulesetDef,
+    determineValidTransactions,
+    isKeyForKeyField,
+} from "./permissions.js";
+import { Group, expectGroupContent } from "./group.js";
+import { LocalNode } from "./node.js";
+import { CoValueKnownState, NewContentMessage } from "./sync.js";
+import { AgentID, RawCoID, SessionID, TransactionID } from "./ids.js";
+import { CoList } from "./coValues/coList.js";
+import {
+    AccountID,
+    GeneralizedControlledAccount,
+} from "./account.js";
+import { Stringified, parseJSON, stableStringify } from "./jsonStringify.js";
+
+export type CoValueHeader = {
+    type: CoValueImpl["type"];
+    ruleset: RulesetDef;
+    meta: JsonObject | null;
+    createdAt: `2${string}` | null;
+    uniqueness: `z${string}` | null;
+};
+
+export function idforHeader(header: CoValueHeader): RawCoID {
+    const hash = shortHash(header);
+    return `co_z${hash.slice("shortHash_z".length)}`;
+}
+
+export function accountOrAgentIDfromSessionID(
+    sessionID: SessionID
+): AccountID | AgentID {
+    return sessionID.split("_session")[0] as AccountID | AgentID;
+}
+
+export function newRandomSessionID(accountID: AccountID | AgentID): SessionID {
+    return `${accountID}_session_z${base58.encode(randomBytes(8))}`;
+}
+
+type SessionLog = {
+    transactions: Transaction[];
+    lastHash?: Hash;
+    streamingHash: StreamingHash;
+    lastSignature: Signature;
+};
+
+export type PrivateTransaction = {
+    privacy: "private";
+    madeAt: number;
+    keyUsed: KeyID;
+    encryptedChanges: Encrypted<
+        JsonValue[],
+        { in: RawCoID; tx: TransactionID }
+    >;
+};
+
+
+export type TrustingTransaction = {
+    privacy: "trusting";
+    madeAt: number;
+    changes: Stringified<JsonValue[]>;
+};
+
+export type Transaction = PrivateTransaction | TrustingTransaction;
+
+export type DecryptedTransaction = {
+    txID: TransactionID;
+    changes: Stringified<JsonValue[]>;
+    madeAt: number;
+};
+
+const readKeyCache = new WeakMap<CoValueCore, { [id: KeyID]: KeySecret }>();
+
+export class CoValueCore {
+    id: RawCoID;
+    node: LocalNode;
+    header: CoValueHeader;
+    _sessions: { [key: SessionID]: SessionLog };
+    _cachedContent?: CoValueImpl;
+    listeners: Set<(content?: CoValueImpl) => void> = new Set();
+    _decryptionCache: {[key: Encrypted<JsonValue[], JsonValue>]: Stringified<JsonValue[]> | undefined} = {}
+
+    constructor(
+        header: CoValueHeader,
+        node: LocalNode,
+        internalInitSessions: { [key: SessionID]: SessionLog } = {}
+    ) {
+        this.id = idforHeader(header);
+        this.header = header;
+        this._sessions = internalInitSessions;
+        this.node = node;
+
+        if (header.ruleset.type == "ownedByGroup") {
+            this.node
+                .expectCoValueLoaded(header.ruleset.group)
+                .subscribe((_groupUpdate) => {
+                    this._cachedContent = undefined;
+                    const newContent = this.getCurrentContent();
+                    for (const listener of this.listeners) {
+                        listener(newContent);
+                    }
+                });
+        }
+    }
+
+    get sessions(): Readonly<{ [key: SessionID]: SessionLog }> {
+        return this._sessions;
+    }
+
+    testWithDifferentAccount(
+        account: GeneralizedControlledAccount,
+        currentSessionID: SessionID
+    ): CoValueCore {
+        const newNode = this.node.testWithDifferentAccount(
+            account,
+            currentSessionID
+        );
+
+        return newNode.expectCoValueLoaded(this.id);
+    }
+
+    knownState(): CoValueKnownState {
+        return {
+            id: this.id,
+            header: true,
+            sessions: Object.fromEntries(
+                Object.entries(this.sessions).map(([k, v]) => [
+                    k,
+                    v.transactions.length,
+                ])
+            ),
+        };
+    }
+
+    get meta(): JsonValue {
+        return this.header?.meta ?? null;
+    }
+
+    nextTransactionID(): TransactionID {
+        const sessionID = this.node.currentSessionID;
+        return {
+            sessionID,
+            txIndex: this.sessions[sessionID]?.transactions.length || 0,
+        };
+    }
+
+    tryAddTransactions(
+        sessionID: SessionID,
+        newTransactions: Transaction[],
+        givenExpectedNewHash: Hash | undefined,
+        newSignature: Signature
+    ): boolean {
+        const signerID = getAgentSignerID(
+            this.node.resolveAccountAgent(
+                accountOrAgentIDfromSessionID(sessionID),
+                "Expected to know signer of transaction"
+            )
+        );
+
+        if (!signerID) {
+            console.warn(
+                "Unknown agent",
+                accountOrAgentIDfromSessionID(sessionID)
+            );
+            return false;
+        }
+
+        // const beforeHash = performance.now();
+        const { expectedNewHash, newStreamingHash } = this.expectedNewHashAfter(
+            sessionID,
+            newTransactions
+        );
+        // const afterHash = performance.now();
+        // console.log(
+        //     "Hashing took",
+        //     afterHash - beforeHash
+        // );
+
+        if (givenExpectedNewHash && givenExpectedNewHash !== expectedNewHash) {
+            console.warn("Invalid hash", {
+                expectedNewHash,
+                givenExpectedNewHash,
+            });
+            return false;
+        }
+
+        // const beforeVerify = performance.now();
+        if (!verify(newSignature, expectedNewHash, signerID)) {
+            console.warn(
+                "Invalid signature",
+                newSignature,
+                expectedNewHash,
+                signerID
+            );
+            return false;
+        }
+        // const afterVerify = performance.now();
+        // console.log(
+        //     "Verify took",
+        //     afterVerify - beforeVerify
+        // );
+
+        const transactions = this.sessions[sessionID]?.transactions ?? [];
+
+        transactions.push(...newTransactions);
+
+        this._sessions[sessionID] = {
+            transactions,
+            lastHash: expectedNewHash,
+            streamingHash: newStreamingHash,
+            lastSignature: newSignature,
+        };
+
+        this._cachedContent = undefined;
+
+        if (this.listeners.size > 0) {
+            const content = this.getCurrentContent();
+            for (const listener of this.listeners) {
+                listener(content);
+            }
+        }
+
+        return true;
+    }
+
+    async tryAddTransactionsAsync(
+        sessionID: SessionID,
+        newTransactions: Transaction[],
+        givenExpectedNewHash: Hash | undefined,
+        newSignature: Signature
+    ): Promise<boolean> {
+        const signerID = getAgentSignerID(
+            this.node.resolveAccountAgent(
+                accountOrAgentIDfromSessionID(sessionID),
+                "Expected to know signer of transaction"
+            )
+        );
+
+        if (!signerID) {
+            console.warn(
+                "Unknown agent",
+                accountOrAgentIDfromSessionID(sessionID)
+            );
+            return false;
+        }
+
+        const nTxBefore = this.sessions[sessionID]?.transactions.length ?? 0;
+
+        // const beforeHash = performance.now();
+        const { expectedNewHash, newStreamingHash } = await this.expectedNewHashAfterAsync(
+            sessionID,
+            newTransactions
+        );
+        // const afterHash = performance.now();
+        // console.log(
+        //     "Hashing took",
+        //     afterHash - beforeHash
+        // );
+
+        const nTxAfter = this.sessions[sessionID]?.transactions.length ?? 0;
+
+        if (nTxAfter !== nTxBefore) {
+            const newTransactionLengthBefore = newTransactions.length;
+            newTransactions = newTransactions.slice((nTxAfter - nTxBefore));
+            console.warn("Transactions changed while async hashing", {
+                nTxBefore,
+                nTxAfter,
+                newTransactionLengthBefore,
+                remainingNewTransactions: newTransactions.length,
+            });
+        }
+
+        if (givenExpectedNewHash && givenExpectedNewHash !== expectedNewHash) {
+            console.warn("Invalid hash", {
+                expectedNewHash,
+                givenExpectedNewHash,
+            });
+            return false;
+        }
+
+        // const beforeVerify = performance.now();
+        if (!verify(newSignature, expectedNewHash, signerID)) {
+            console.warn(
+                "Invalid signature",
+                newSignature,
+                expectedNewHash,
+                signerID
+            );
+            return false;
+        }
+        // const afterVerify = performance.now();
+        // console.log(
+        //     "Verify took",
+        //     afterVerify - beforeVerify
+        // );
+
+        const transactions = this.sessions[sessionID]?.transactions ?? [];
+
+        transactions.push(...newTransactions);
+
+        this._sessions[sessionID] = {
+            transactions,
+            lastHash: expectedNewHash,
+            streamingHash: newStreamingHash,
+            lastSignature: newSignature,
+        };
+
+        this._cachedContent = undefined;
+
+        if (this.listeners.size > 0) {
+            const content = this.getCurrentContent();
+            for (const listener of this.listeners) {
+                listener(content);
+            }
+        }
+
+        return true;
+    }
+
+    subscribe(listener: (content?: CoValueImpl) => void): () => void {
+        this.listeners.add(listener);
+        listener(this.getCurrentContent());
+
+        return () => {
+            this.listeners.delete(listener);
+        };
+    }
+
+    expectedNewHashAfter(
+        sessionID: SessionID,
+        newTransactions: Transaction[]
+    ): { expectedNewHash: Hash; newStreamingHash: StreamingHash } {
+        const streamingHash =
+            this.sessions[sessionID]?.streamingHash.clone() ??
+            new StreamingHash();
+        for (const transaction of newTransactions) {
+            streamingHash.update(transaction);
+        }
+
+        const newStreamingHash = streamingHash.clone();
+
+        return {
+            expectedNewHash: streamingHash.digest(),
+            newStreamingHash,
+        };
+    }
+
+    async expectedNewHashAfterAsync(
+        sessionID: SessionID,
+        newTransactions: Transaction[]
+    ): Promise<{ expectedNewHash: Hash; newStreamingHash: StreamingHash }> {
+        const streamingHash =
+            this.sessions[sessionID]?.streamingHash.clone() ??
+            new StreamingHash();
+        let before = performance.now();
+        for (const transaction of newTransactions) {
+            streamingHash.update(transaction)
+            const after = performance.now();
+            if (after - before > 1) {
+                console.log("Hashing blocked for", after - before);
+                await new Promise((resolve) => setTimeout(resolve, 0));
+                before = performance.now();
+            }
+        }
+
+        const newStreamingHash = streamingHash.clone();
+
+        return {
+            expectedNewHash: streamingHash.digest(),
+            newStreamingHash,
+        };
+    }
+
+    makeTransaction(
+        changes: JsonValue[],
+        privacy: "private" | "trusting"
+    ): boolean {
+        const madeAt = Date.now();
+
+        let transaction: Transaction;
+
+        if (privacy === "private") {
+            const { secret: keySecret, id: keyID } = this.getCurrentReadKey();
+
+            if (!keySecret) {
+                throw new Error(
+                    "Can't make transaction without read key secret"
+                );
+            }
+
+            const encrypted = encryptForTransaction(changes, keySecret, {
+                in: this.id,
+                tx: this.nextTransactionID(),
+            });
+
+            this._decryptionCache[encrypted] = stableStringify(changes);
+
+            transaction = {
+                privacy: "private",
+                madeAt,
+                keyUsed: keyID,
+                encryptedChanges: encrypted,
+            };
+        } else {
+            transaction = {
+                privacy: "trusting",
+                madeAt,
+                changes: stableStringify(changes),
+            };
+        }
+
+        const sessionID = this.node.currentSessionID;
+
+        const { expectedNewHash } = this.expectedNewHashAfter(sessionID, [
+            transaction,
+        ]);
+
+        const signature = sign(
+            this.node.account.currentSignerSecret(),
+            expectedNewHash
+        );
+
+        const success = this.tryAddTransactions(
+            sessionID,
+            [transaction],
+            expectedNewHash,
+            signature
+        );
+
+        if (success) {
+            void this.node.sync.syncCoValue(this);
+        }
+
+        return success;
+    }
+
+    getCurrentContent(): CoValueImpl {
+        if (this._cachedContent) {
+            return this._cachedContent;
+        }
+
+        if (this.header.type === "comap") {
+            this._cachedContent = new CoMap(this);
+        } else if (this.header.type === "colist") {
+            this._cachedContent = new CoList(this);
+        } else if (this.header.type === "costream") {
+            if (this.header.meta && this.header.meta.type === "binary") {
+                this._cachedContent = new BinaryCoStream(this);
+            } else {
+                this._cachedContent = new CoStream(this);
+            }
+        } else if (this.header.type === "static") {
+            this._cachedContent = new Static(this);
+        } else {
+            throw new Error(`Unknown coValue type ${this.header.type}`);
+        }
+
+        return this._cachedContent;
+    }
+
+    getValidSortedTransactions(): DecryptedTransaction[] {
+        const validTransactions = determineValidTransactions(this);
+
+        const allTransactions: DecryptedTransaction[] = validTransactions
+            .map(({ txID, tx }) => {
+                if (tx.privacy === "trusting") {
+                    return {
+                        txID,
+                        madeAt: tx.madeAt,
+                        changes: tx.changes,
+                    };
+                } else {
+                    const readKey = this.getReadKey(tx.keyUsed);
+
+                    if (!readKey) {
+                        return undefined;
+                    } else {
+                        let decrytedChanges = this._decryptionCache[tx.encryptedChanges];
+
+                        if (!decrytedChanges) {
+                            decrytedChanges = decryptRawForTransaction(
+                                tx.encryptedChanges,
+                                readKey,
+                                {
+                                    in: this.id,
+                                    tx: txID,
+                                }
+                            );
+                            this._decryptionCache[tx.encryptedChanges] = decrytedChanges;
+                        }
+
+                        if (!decrytedChanges) {
+                            console.error(
+                                "Failed to decrypt transaction despite having key"
+                            );
+                            return undefined;
+                        }
+                        return {
+                            txID,
+                            madeAt: tx.madeAt,
+                            changes: decrytedChanges,
+                        };
+                    }
+                }
+            })
+            .filter((x): x is Exclude<typeof x, undefined> => !!x);
+        allTransactions.sort(
+            (a, b) =>
+                a.madeAt - b.madeAt ||
+                (a.txID.sessionID < b.txID.sessionID ? -1 : 1) ||
+                a.txID.txIndex - b.txID.txIndex
+        );
+
+        return allTransactions;
+    }
+
+    getCurrentReadKey(): { secret: KeySecret | undefined; id: KeyID } {
+        if (this.header.ruleset.type === "group") {
+            const content = expectGroupContent(this.getCurrentContent());
+
+            const currentKeyId = content.get("readKey");
+
+            if (!currentKeyId) {
+                throw new Error("No readKey set");
+            }
+
+            const secret = this.getReadKey(currentKeyId);
+
+            return {
+                secret: secret,
+                id: currentKeyId,
+            };
+        } else if (this.header.ruleset.type === "ownedByGroup") {
+            return this.node
+                .expectCoValueLoaded(this.header.ruleset.group)
+                .getCurrentReadKey();
+        } else {
+            throw new Error(
+                "Only groups or values owned by groups have read secrets"
+            );
+        }
+    }
+
+    getReadKey(keyID: KeyID): KeySecret | undefined {
+        if (readKeyCache.get(this)?.[keyID]) {
+            return readKeyCache.get(this)?.[keyID];
+        }
+        if (this.header.ruleset.type === "group") {
+            const content = expectGroupContent(this.getCurrentContent());
+
+            // Try to find key revelation for us
+
+            const readKeyEntry = content.getLastEntry(
+                `${keyID}_for_${this.node.account.id}`
+            );
+
+            if (readKeyEntry) {
+                const revealer = accountOrAgentIDfromSessionID(
+                    readKeyEntry.txID.sessionID
+                );
+                const revealerAgent = this.node.resolveAccountAgent(
+                    revealer,
+                    "Expected to know revealer"
+                );
+
+                const secret = unseal(
+                    readKeyEntry.value,
+                    this.node.account.currentSealerSecret(),
+                    getAgentSealerID(revealerAgent),
+                    {
+                        in: this.id,
+                        tx: readKeyEntry.txID,
+                    }
+                );
+
+                if (secret) {
+                    let cache = readKeyCache.get(this);
+                    if (!cache) {
+                        cache = {};
+                        readKeyCache.set(this, cache);
+                    }
+                    cache[keyID] = secret;
+
+                    return secret as KeySecret;
+                }
+            }
+
+            // Try to find indirect revelation through previousKeys
+
+            for (const field of content.keys()) {
+                if (isKeyForKeyField(field) && field.startsWith(keyID)) {
+                    const encryptingKeyID = field.split("_for_")[1] as KeyID;
+                    const encryptingKeySecret =
+                        this.getReadKey(encryptingKeyID);
+
+                    if (!encryptingKeySecret) {
+                        continue;
+                    }
+
+                    const encryptedPreviousKey = content.get(field)!;
+
+                    const secret = decryptKeySecret(
+                        {
+                            encryptedID: keyID,
+                            encryptingID: encryptingKeyID,
+                            encrypted: encryptedPreviousKey,
+                        },
+                        encryptingKeySecret
+                    );
+
+                    if (secret) {
+                        let cache = readKeyCache.get(this);
+                        if (!cache) {
+                            cache = {};
+                            readKeyCache.set(this, cache);
+                        }
+                        cache[keyID] = secret;
+
+                        return secret as KeySecret;
+                    } else {
+                        console.error(
+                            `Encrypting ${encryptingKeyID} key didn't decrypt ${keyID}`
+                        );
+                    }
+                }
+            }
+
+            return undefined;
+        } else if (this.header.ruleset.type === "ownedByGroup") {
+            return this.node
+                .expectCoValueLoaded(this.header.ruleset.group)
+                .getReadKey(keyID);
+        } else {
+            throw new Error(
+                "Only groups or values owned by groups have read secrets"
+            );
+        }
+    }
+
+    getGroup(): Group {
+        if (this.header.ruleset.type !== "ownedByGroup") {
+            throw new Error("Only values owned by groups have groups");
+        }
+
+        return new Group(
+            expectGroupContent(
+                this.node
+                    .expectCoValueLoaded(this.header.ruleset.group)
+                    .getCurrentContent()
+            ),
+            this.node
+        );
+    }
+
+    getTx(txID: TransactionID): Transaction | undefined {
+        return this.sessions[txID.sessionID]?.transactions[txID.txIndex];
+    }
+
+    newContentSince(
+        knownState: CoValueKnownState | undefined
+    ): NewContentMessage | undefined {
+        const newContent: NewContentMessage = {
+            action: "content",
+            id: this.id,
+            header: knownState?.header ? undefined : this.header,
+            new: Object.fromEntries(
+                Object.entries(this.sessions)
+                    .map(([sessionID, log]) => {
+                        const newTransactions = log.transactions.slice(
+                            knownState?.sessions[sessionID as SessionID] || 0
+                        );
+
+                        if (
+                            newTransactions.length === 0 ||
+                            !log.lastHash ||
+                            !log.lastSignature
+                        ) {
+                            return undefined;
+                        }
+
+                        return [
+                            sessionID,
+                            {
+                                after:
+                                    knownState?.sessions[
+                                        sessionID as SessionID
+                                    ] || 0,
+                                newTransactions,
+                                lastSignature: log.lastSignature,
+                            },
+                        ];
+                    })
+                    .filter((x): x is Exclude<typeof x, undefined> => !!x)
+            ),
+        };
+
+        if (!newContent.header && Object.keys(newContent.new).length === 0) {
+            return undefined;
+        }
+
+        return newContent;
+    }
+
+    getDependedOnCoValues(): RawCoID[] {
+        return this.header.ruleset.type === "group"
+            ? expectGroupContent(this.getCurrentContent())
+                  .keys()
+                  .filter((k): k is AccountID => k.startsWith("co_"))
+            : this.header.ruleset.type === "ownedByGroup"
+            ? [this.header.ruleset.group]
+            : [];
+    }
+}

packages/cojson/src/coValues/coList.ts

@@ -1,9 +1,10 @@
 import { JsonObject, JsonValue } from "../jsonValue.js";
-import { CoID } from "../contentType.js";
-import { CoValue, accountOrAgentIDfromSessionID } from "../coValue.js";
+import { CoID, ReadableCoValue, WriteableCoValue } from "../coValue.js";
+import { CoValueCore, accountOrAgentIDfromSessionID } from "../coValueCore.js";
 import { SessionID, TransactionID } from "../ids.js";
-import { AccountID, Group } from "../index.js";
-import { isAccountID } from "../account.js";
+import { Group } from "../group.js";
+import { AccountID, isAccountID } from "../account.js";
+import { parseJSON } from "../jsonStringify.js";
 
 type OpID = TransactionID & { changeIdx: number };
 
@@ -39,15 +40,17 @@ type DeletionEntry = {
     deletionID: OpID;
 } & DeletionOpPayload;
 
-export class CoList<
-    T extends JsonValue,
-    Meta extends JsonObject | null = null
-> {
+export class CoList<T extends JsonValue, Meta extends JsonObject | null = null>
+    implements ReadableCoValue
+{
     id: CoID<CoList<T, Meta>>;
     type = "colist" as const;
-    coValue: CoValue;
+    core: CoValueCore;
+    /** @internal */
     afterStart: OpID[];
+    /** @internal */
     beforeEnd: OpID[];
+    /** @internal */
     insertions: {
         [sessionID: SessionID]: {
             [txIdx: number]: {
@@ -55,6 +58,7 @@ export class CoList<
             };
         };
     };
+    /** @internal */
     deletionsByInsertion: {
         [deletedSessionID: SessionID]: {
             [deletedTxIdx: number]: {
@@ -63,9 +67,10 @@ export class CoList<
         };
     };
 
-    constructor(coValue: CoValue) {
-        this.id = coValue.id as CoID<CoList<T, Meta>>;
-        this.coValue = coValue;
+    /** @internal */
+    constructor(core: CoValueCore) {
+        this.id = core.id as CoID<CoList<T, Meta>>;
+        this.core = core;
         this.afterStart = [];
         this.beforeEnd = [];
         this.insertions = {};
@@ -74,15 +79,15 @@ export class CoList<
         this.fillOpsFromCoValue();
     }
 
-
     get meta(): Meta {
-        return this.coValue.header.meta as Meta;
+        return this.core.header.meta as Meta;
     }
 
     get group(): Group {
-        return this.coValue.getGroup();
+        return this.core.getGroup();
     }
 
+    /** @internal */
     protected fillOpsFromCoValue() {
         this.insertions = {};
         this.deletionsByInsertion = {};
@@ -93,8 +98,8 @@ export class CoList<
             txID,
             changes,
             madeAt,
-        } of this.coValue.getValidSortedTransactions()) {
-            for (const [changeIdx, changeUntyped] of changes.entries()) {
+        } of this.core.getValidSortedTransactions()) {
+            for (const [changeIdx, changeUntyped] of parseJSON(changes).entries()) {
                 const change = changeUntyped as ListOpPayload<T>;
 
                 if (change.op === "pre" || change.op === "app") {
@@ -195,6 +200,20 @@ export class CoList<
         }
     }
 
+    /** Get the item currently at `idx`. */
+    get(idx: number): T | undefined {
+        const entry = this.entries()[idx];
+        if (!entry) {
+            return undefined;
+        }
+        return entry.value;
+    }
+
+    /** Returns the current items in the CoList as an array. */
+    asArray(): T[] {
+        return this.entries().map((entry) => entry.value);
+    }
+
     entries(): { value: T; madeAt: number; opID: OpID }[] {
         const arr: { value: T; madeAt: number; opID: OpID }[] = [];
         for (const opID of this.afterStart) {
@@ -206,6 +225,7 @@ export class CoList<
         return arr;
     }
 
+    /** @internal */
     private fillArrayFromOpID(
         opID: OpID,
         arr: { value: T; madeAt: number; opID: OpID }[]
@@ -234,6 +254,7 @@ export class CoList<
         }
     }
 
+    /** Returns the accountID of the account that inserted value at the given index. */
     whoInserted(idx: number): AccountID | undefined {
         const entry = this.entries()[idx];
         if (!entry) {
@@ -247,19 +268,16 @@ export class CoList<
         }
     }
 
+    /** Returns the current items in the CoList as an array. (alias of `asArray`) */
     toJSON(): T[] {
         return this.asArray();
     }
 
-    asArray(): T[] {
-        return this.entries().map((entry) => entry.value);
-    }
-
     map<U>(mapper: (value: T, idx: number) => U): U[] {
         return this.entries().map((entry, idx) => mapper(entry.value, idx));
     }
 
-    filter<U extends T>(predicate: (value: T, idx: number) => value is U): U[]
+    filter<U extends T>(predicate: (value: T, idx: number) => value is U): U[];
     filter(predicate: (value: T, idx: number) => boolean): T[] {
         return this.entries()
             .filter((entry, idx) => predicate(entry.value, idx))
@@ -271,31 +289,45 @@ export class CoList<
         initialValue: U
     ): U {
         return this.entries().reduce(
-            (accumulator, entry, idx) =>
-                reducer(accumulator, entry.value, idx),
+            (accumulator, entry, idx) => reducer(accumulator, entry.value, idx),
             initialValue
         );
     }
 
+    subscribe(listener: (coMap: CoList<T, Meta>) => void): () => void {
+        return this.core.subscribe((content) => {
+            listener(content as CoList<T, Meta>);
+        });
+    }
+
     edit(
         changer: (editable: WriteableCoList<T, Meta>) => void
     ): CoList<T, Meta> {
-        const editable = new WriteableCoList<T, Meta>(this.coValue);
+        const editable = new WriteableCoList<T, Meta>(this.core);
         changer(editable);
-        return new CoList(this.coValue);
-    }
-
-    subscribe(listener: (coMap: CoList<T, Meta>) => void): () => void {
-        return this.coValue.subscribe((content) => {
-            listener(content as CoList<T, Meta>);
-        });
+        return new CoList(this.core);
     }
 }
 
 export class WriteableCoList<
-    T extends JsonValue,
-    Meta extends JsonObject | null = null
-> extends CoList<T, Meta> {
+        T extends JsonValue,
+        Meta extends JsonObject | null = null
+    >
+    extends CoList<T, Meta>
+    implements WriteableCoValue
+{
+    /** @internal */
+    edit(
+        _changer: (editable: WriteableCoList<T, Meta>) => void
+    ): CoList<T, Meta> {
+        throw new Error("Already editing.");
+    }
+
+    /** Appends a new item after index `after`.
+     *
+     * If `privacy` is `"private"` **(default)**, both `value` is encrypted in the transaction, only readable by other members of the group this `CoList` belongs to. Not even sync servers can see the content in plaintext.
+     *
+     * If `privacy` is `"trusting"`, both `value` is stored in plaintext in the transaction, visible to everyone who gets a hold of it, including sync servers. */
     append(
         after: number,
         value: T,
@@ -315,7 +347,7 @@ export class WriteableCoList<
             }
             opIDBefore = "start";
         }
-        this.coValue.makeTransaction(
+        this.core.makeTransaction(
             [
                 {
                     op: "app",
@@ -329,12 +361,28 @@ export class WriteableCoList<
         this.fillOpsFromCoValue();
     }
 
+    /** Pushes a new item to the end of the list.
+     *
+     * If `privacy` is `"private"` **(default)**, both `value` is encrypted in the transaction, only readable by other members of the group this `CoList` belongs to. Not even sync servers can see the content in plaintext.
+     *
+     * If `privacy` is `"trusting"`, both `value` is stored in plaintext in the transaction, visible to everyone who gets a hold of it, including sync servers. */
     push(value: T, privacy: "private" | "trusting" = "private"): void {
         // TODO: optimize
         const entries = this.entries();
-        this.append(entries.length > 0 ? entries.length - 1 : 0, value, privacy);
+        this.append(
+            entries.length > 0 ? entries.length - 1 : 0,
+            value,
+            privacy
+        );
     }
 
+    /**
+     * Prepends a new item before index `before`.
+     *
+     * If `privacy` is `"private"` **(default)**, both `value` is encrypted in the transaction, only readable by other members of the group this `CoList` belongs to. Not even sync servers can see the content in plaintext.
+     *
+     * If `privacy` is `"trusting"`, both `value` is stored in plaintext in the transaction, visible to everyone who gets a hold of it, including sync servers.
+     */
     prepend(
         before: number,
         value: T,
@@ -358,7 +406,7 @@ export class WriteableCoList<
             }
             opIDAfter = "end";
         }
-        this.coValue.makeTransaction(
+        this.core.makeTransaction(
             [
                 {
                     op: "pre",
@@ -372,16 +420,18 @@ export class WriteableCoList<
         this.fillOpsFromCoValue();
     }
 
-    delete(
-        at: number,
-        privacy: "private" | "trusting" = "private"
-    ): void {
+    /** Deletes the item at index `at` from the list.
+     *
+     * If `privacy` is `"private"` **(default)**, the fact of this deletion is encrypted in the transaction, only readable by other members of the group this `CoList` belongs to. Not even sync servers can see the content in plaintext.
+     *
+     * If `privacy` is `"trusting"`, the fact of this deletion is stored in plaintext in the transaction, visible to everyone who gets a hold of it, including sync servers. */
+    delete(at: number, privacy: "private" | "trusting" = "private"): void {
         const entries = this.entries();
         const entry = entries[at];
         if (!entry) {
             throw new Error("Invalid index " + at);
         }
-        this.coValue.makeTransaction(
+        this.core.makeTransaction(
             [
                 {
                     op: "del",

packages/cojson/src/coValues/coMap.ts

@@ -1,18 +1,19 @@
 import { JsonObject, JsonValue } from '../jsonValue.js';
 import { TransactionID } from '../ids.js';
-import { CoID } from '../contentType.js';
-import { CoValue, accountOrAgentIDfromSessionID } from '../coValue.js';
+import { CoID, ReadableCoValue, WriteableCoValue } from '../coValue.js';
+import { CoValueCore, accountOrAgentIDfromSessionID } from '../coValueCore.js';
 import { AccountID, isAccountID } from '../account.js';
 import { Group } from '../group.js';
+import { parseJSON } from '../jsonStringify.js';
 
-type MapOp<K extends string, V extends JsonValue> = {
+type MapOp<K extends string, V extends JsonValue | undefined> = {
     txID: TransactionID;
     madeAt: number;
     changeIdx: number;
 } & MapOpPayload<K, V>;
 // TODO: add after TransactionID[] for conflicts/ordering
 
-export type MapOpPayload<K extends string, V extends JsonValue> = {
+export type MapOpPayload<K extends string, V extends JsonValue | undefined> = {
     op: "set";
     key: K;
     value: V;
@@ -22,45 +23,47 @@ export type MapOpPayload<K extends string, V extends JsonValue> = {
     key: K;
 };
 
-export type MapK<M extends { [key: string]: JsonValue; }> = keyof M & string;
-export type MapV<M extends { [key: string]: JsonValue; }> = M[MapK<M>];
-export type MapM<M extends { [key: string]: JsonValue; }> = {
-    [KK in MapK<M>]: M[KK];
-}
+export type MapK<M extends { [key: string]: JsonValue | undefined; }> = keyof M & string;
+export type MapV<M extends { [key: string]: JsonValue | undefined; }> = M[MapK<M>];
 
+
+/** A collaborative map with precise shape `M` and optional static metadata `Meta` */
 export class CoMap<
-    M extends { [key: string]: JsonValue; },
+    M extends { [key: string]: JsonValue | undefined; },
     Meta extends JsonObject | null = null,
-> {
-    id: CoID<CoMap<MapM<M>, Meta>>;
-    coValue: CoValue;
+> implements ReadableCoValue {
+    id: CoID<CoMap<M, Meta>>;
     type = "comap" as const;
+    core: CoValueCore;
+    /** @internal */
     ops: {
         [KK in MapK<M>]?: MapOp<KK, M[KK]>[];
     };
 
-    constructor(coValue: CoValue) {
-        this.id = coValue.id as CoID<CoMap<MapM<M>, Meta>>;
-        this.coValue = coValue;
+    /** @internal */
+    constructor(core: CoValueCore) {
+        this.id = core.id as CoID<CoMap<M, Meta>>;
+        this.core = core;
         this.ops = {};
 
         this.fillOpsFromCoValue();
     }
 
     get meta(): Meta {
-        return this.coValue.header.meta as Meta;
+        return this.core.header.meta as Meta;
     }
 
     get group(): Group {
-        return this.coValue.getGroup();
+        return this.core.getGroup();
     }
 
+    /** @internal */
     protected fillOpsFromCoValue() {
         this.ops = {};
 
-        for (const { txID, changes, madeAt } of this.coValue.getValidSortedTransactions()) {
+        for (const { txID, changes, madeAt } of this.core.getValidSortedTransactions()) {
             for (const [changeIdx, changeUntyped] of (
-                changes
+                parseJSON(changes)
             ).entries()) {
                 const change = changeUntyped as MapOpPayload<MapK<M>, MapV<M>>;
                 let entries = this.ops[change.key];
@@ -82,6 +85,7 @@ export class CoMap<
         return Object.keys(this.ops) as MapK<M>[];
     }
 
+    /** Returns the current value for the given key. */
     get<K extends MapK<M>>(key: K): M[K] | undefined {
         const ops = this.ops[key];
         if (!ops) {
@@ -116,6 +120,7 @@ export class CoMap<
         }
     }
 
+    /** Returns the accountID of the last account to modify the value for the given key. */
     whoEdited<K extends MapK<M>>(key: K): AccountID | undefined {
         const tx  = this.getLastTxID(key);
         if (!tx) {
@@ -187,26 +192,35 @@ export class CoMap<
         return json;
     }
 
-    edit(changer: (editable: WriteableCoMap<M, Meta>) => void): CoMap<M, Meta> {
-        const editable = new WriteableCoMap<M, Meta>(this.coValue);
-        changer(editable);
-        return new CoMap(this.coValue);
-    }
-
     subscribe(listener: (coMap: CoMap<M, Meta>) => void): () => void {
-        return this.coValue.subscribe((content) => {
+        return this.core.subscribe((content) => {
             listener(content as CoMap<M, Meta>);
         });
     }
+
+    edit(changer: (editable: WriteableCoMap<M, Meta>) => void): CoMap<M, Meta> {
+        const editable = new WriteableCoMap<M, Meta>(this.core);
+        changer(editable);
+        return new CoMap(this.core);
+    }
 }
 
 export class WriteableCoMap<
-    M extends { [key: string]: JsonValue; },
+    M extends { [key: string]: JsonValue | undefined; },
     Meta extends JsonObject | null = null,
+> extends CoMap<M, Meta> implements WriteableCoValue {
+    /** @internal */
+    edit(_changer: (editable: WriteableCoMap<M, Meta>) => void): CoMap<M, Meta> {
+        throw new Error("Already editing.");
+    }
 
-> extends CoMap<M, Meta> {
+    /** Sets a new value for the given key.
+     *
+     * If `privacy` is `"private"` **(default)**, both `key` and `value` are encrypted in the transaction, only readable by other members of the group this `CoMap` belongs to. Not even sync servers can see the content in plaintext.
+     *
+     * If `privacy` is `"trusting"`, both `key` and `value` are stored in plaintext in the transaction, visible to everyone who gets a hold of it, including sync servers. */
     set<K extends MapK<M>>(key: K, value: M[K], privacy: "private" | "trusting" = "private"): void {
-        this.coValue.makeTransaction([
+        this.core.makeTransaction([
             {
                 op: "set",
                 key,
@@ -217,8 +231,13 @@ export class WriteableCoMap<
         this.fillOpsFromCoValue();
     }
 
+    /** Deletes the value for the given key (setting it to undefined).
+     *
+     * If `privacy` is `"private"` **(default)**, `key` is encrypted in the transaction, only readable by other members of the group this `CoMap` belongs to. Not even sync servers can see the content in plaintext.
+     *
+     * If `privacy` is `"trusting"`, `key` is stored in plaintext in the transaction, visible to everyone who gets a hold of it, including sync servers. */
     delete(key: MapK<M>, privacy: "private" | "trusting" = "private"): void {
-        this.coValue.makeTransaction([
+        this.core.makeTransaction([
             {
                 op: "del",
                 key,

packages/cojson/src/coValues/coStream.ts

@@ -0,0 +1,312 @@
+import { JsonObject, JsonValue } from "../jsonValue.js";
+import { CoID, ReadableCoValue, WriteableCoValue } from "../coValue.js";
+import { CoValueCore, accountOrAgentIDfromSessionID } from "../coValueCore.js";
+import { Group } from "../group.js";
+import { SessionID } from "../ids.js";
+import { base64URLtoBytes, bytesToBase64url } from "../base64url.js";
+import { AccountID } from "../index.js";
+import { isAccountID } from "../account.js";
+import { parseJSON } from "../jsonStringify.js";
+
+export type BinaryChunkInfo = {
+    mimeType: string;
+    fileName?: string;
+    totalSizeBytes?: number;
+};
+
+export type BinaryStreamStart = {
+    type: "start";
+} & BinaryChunkInfo;
+
+export type BinaryStreamChunk = {
+    type: "chunk";
+    chunk: `binary_U${string}`;
+};
+
+export type BinaryStreamEnd = {
+    type: "end";
+};
+
+export type BinaryCoStreamMeta = JsonObject & { type: "binary" };
+
+export type BinaryStreamItem =
+    | BinaryStreamStart
+    | BinaryStreamChunk
+    | BinaryStreamEnd;
+
+export class CoStream<
+    T extends JsonValue,
+    Meta extends JsonObject | null = null
+> implements ReadableCoValue
+{
+    id: CoID<CoStream<T, Meta>>;
+    type = "costream" as const;
+    core: CoValueCore;
+    items: {
+        [key: SessionID]: {item: T, madeAt: number}[];
+    };
+
+    constructor(core: CoValueCore) {
+        this.id = core.id as CoID<CoStream<T, Meta>>;
+        this.core = core;
+        this.items = {};
+        this.fillFromCoValue();
+    }
+
+    get meta(): Meta {
+        return this.core.header.meta as Meta;
+    }
+
+    get group(): Group {
+        return this.core.getGroup();
+    }
+
+    /** @internal */
+    protected fillFromCoValue() {
+        this.items = {};
+
+        for (const {
+            txID,
+            madeAt,
+            changes,
+        } of this.core.getValidSortedTransactions()) {
+            for (const changeUntyped of parseJSON(changes)) {
+                const change = changeUntyped as T;
+                let entries = this.items[txID.sessionID];
+                if (!entries) {
+                    entries = [];
+                    this.items[txID.sessionID] = entries;
+                }
+                entries.push({item: change, madeAt});
+            }
+        }
+    }
+
+    getSingleStream(): T[] | undefined {
+        if (Object.keys(this.items).length === 0) {
+            return undefined;
+        } else if (Object.keys(this.items).length !== 1) {
+            throw new Error(
+                "CoStream.getSingleStream() can only be called when there is exactly one stream"
+            );
+        }
+
+        return Object.values(this.items)[0]?.map(item => item.item);
+    }
+
+    getLastItemsPerAccount(): {[account: AccountID]: T | undefined} {
+        const result: {[account: AccountID]: {item: T, madeAt: number} | undefined} = {};
+
+        for (const [sessionID, items] of Object.entries(this.items)) {
+            const account = accountOrAgentIDfromSessionID(sessionID as SessionID);
+            if (!isAccountID(account)) continue;
+            if (items.length > 0) {
+                const lastItemOfSession = items[items.length - 1]!;
+                if (!result[account] || lastItemOfSession.madeAt > result[account]!.madeAt) {
+                    result[account] = lastItemOfSession;
+                }
+            }
+        }
+
+        return Object.fromEntries(Object.entries(result).map(([account, item]) =>
+            [account, item?.item]
+        ));
+    }
+
+    getLastItemFrom(account: AccountID): T | undefined {
+        let lastItem: {item: T, madeAt: number} | undefined;
+
+        for (const [sessionID, items] of Object.entries(this.items)) {
+            if (sessionID.startsWith(account)) {
+                if (items.length > 0) {
+                    const lastItemOfSession = items[items.length - 1]!;
+                    if (!lastItem || lastItemOfSession.madeAt > lastItem.madeAt) {
+                        lastItem = lastItemOfSession;
+                    }
+                }
+            }
+        }
+
+        return lastItem?.item;
+    }
+
+    getLastItemFromMe(): T | undefined {
+        const myAccountID = this.core.node.account.id;
+        if (!isAccountID(myAccountID)) return undefined;
+        return this.getLastItemFrom(myAccountID);
+    }
+
+    toJSON(): {
+        [key: SessionID]: T[];
+    } {
+        return Object.fromEntries(Object.entries(this.items).map(([sessionID, items]) =>
+            [sessionID, items.map(item => item.item)]
+        ));
+    }
+
+    subscribe(listener: (coMap: CoStream<T, Meta>) => void): () => void {
+        return this.core.subscribe((content) => {
+            listener(content as CoStream<T, Meta>);
+        });
+    }
+
+    edit(
+        changer: (editable: WriteableCoStream<T, Meta>) => void
+    ): CoStream<T, Meta> {
+        const editable = new WriteableCoStream<T, Meta>(this.core);
+        changer(editable);
+        return new CoStream(this.core);
+    }
+}
+
+const binary_U_prefixLength = 8; // "binary_U".length;
+
+export class BinaryCoStream<
+        Meta extends BinaryCoStreamMeta = { type: "binary" }
+    >
+    extends CoStream<BinaryStreamItem, Meta>
+    implements ReadableCoValue
+{
+    id!: CoID<BinaryCoStream<Meta>>;
+
+    getBinaryChunks():
+        | (BinaryChunkInfo & { chunks: Uint8Array[]; finished: boolean })
+        | undefined {
+        const before = performance.now();
+        const items = this.getSingleStream();
+
+        if (!items) return;
+
+        const start = items[0];
+
+        if (start?.type !== "start") {
+            console.error("Invalid binary stream start", start);
+            return;
+        }
+
+        const chunks: Uint8Array[] = [];
+
+        let finished = false;
+        let totalLength = 0;
+
+        for (const item of items.slice(1)) {
+            if (item.type === "end") {
+                finished = true;
+                break;
+            }
+
+            if (item.type !== "chunk") {
+                console.error("Invalid binary stream chunk", item);
+                return undefined;
+            }
+
+            const chunk = base64URLtoBytes(
+                item.chunk.slice(binary_U_prefixLength)
+            );
+            totalLength += chunk.length;
+            chunks.push(chunk);
+        }
+
+        const after = performance.now();
+        console.log(
+            "getBinaryChunks bandwidth in MB/s",
+            (1000 * totalLength) / (after - before) / (1024 * 1024)
+        );
+
+        return {
+            mimeType: start.mimeType,
+            fileName: start.fileName,
+            totalSizeBytes: start.totalSizeBytes,
+            chunks,
+            finished,
+        };
+    }
+
+    edit(
+        changer: (editable: WriteableBinaryCoStream<Meta>) => void
+    ): BinaryCoStream<Meta> {
+        const editable = new WriteableBinaryCoStream<Meta>(this.core);
+        changer(editable);
+        return new BinaryCoStream(this.core);
+    }
+}
+
+export class WriteableCoStream<
+        T extends JsonValue,
+        Meta extends JsonObject | null = null
+    >
+    extends CoStream<T, Meta>
+    implements WriteableCoValue
+{
+    /** @internal */
+    edit(
+        _changer: (editable: WriteableCoStream<T, Meta>) => void
+    ): CoStream<T, Meta> {
+        throw new Error("Already editing.");
+    }
+
+    push(item: T, privacy: "private" | "trusting" = "private") {
+        this.core.makeTransaction([item], privacy);
+        this.fillFromCoValue();
+    }
+}
+
+export class WriteableBinaryCoStream<
+        Meta extends BinaryCoStreamMeta = { type: "binary" }
+    >
+    extends BinaryCoStream<Meta>
+    implements WriteableCoValue
+{
+    /** @internal */
+    edit(
+        _changer: (editable: WriteableBinaryCoStream<Meta>) => void
+    ): BinaryCoStream<Meta> {
+        throw new Error("Already editing.");
+    }
+
+    /** @internal */
+    push(item: BinaryStreamItem, privacy: "private" | "trusting" = "private") {
+        WriteableCoStream.prototype.push.call(this, item, privacy);
+    }
+
+    startBinaryStream(
+        settings: BinaryChunkInfo,
+        privacy: "private" | "trusting" = "private"
+    ) {
+        this.push(
+            {
+                type: "start",
+                ...settings,
+            } satisfies BinaryStreamStart,
+            privacy
+        );
+    }
+
+    pushBinaryStreamChunk(
+        chunk: Uint8Array,
+        privacy: "private" | "trusting" = "private"
+    ) {
+        const before = performance.now();
+        this.push(
+            {
+                type: "chunk",
+                chunk: `binary_U${bytesToBase64url(chunk)}`,
+            } satisfies BinaryStreamChunk,
+            privacy
+        );
+        const after = performance.now();
+        console.log(
+            "pushBinaryStreamChunk bandwidth in MB/s",
+            (1000 * chunk.length) / (after - before) / (1024 * 1024)
+        );
+    }
+
+    endBinaryStream(privacy: "private" | "trusting" = "private") {
+        this.push(
+            {
+                type: "end",
+            } satisfies BinaryStreamEnd,
+            privacy
+        );
+    }
+}

packages/cojson/src/coValues/static.ts

@@ -0,0 +1,31 @@
+import { JsonObject } from '../jsonValue.js';
+import { CoID, ReadableCoValue } from '../coValue.js';
+import { CoValueCore } from '../coValueCore.js';
+import { Group } from '../index.js';
+
+export class Static<T extends JsonObject> implements ReadableCoValue{
+    id: CoID<Static<T>>;
+    type = "static" as const;
+    core: CoValueCore;
+
+    constructor(core: CoValueCore) {
+        this.id = core.id as CoID<Static<T>>;
+        this.core = core;
+    }
+
+    get meta(): T {
+        return this.core.header.meta as T;
+    }
+
+    get group(): Group {
+        return this.core.getGroup();
+    }
+
+    toJSON(): JsonObject {
+        throw new Error("Method not implemented.");
+    }
+
+    subscribe(_listener: (coMap: Static<T>) => void): () => void {
+        throw new Error("Method not implemented.");
+    }
+}

packages/cojson/src/contentType.test.ts

@@ -1,284 +0,0 @@
-import { accountOrAgentIDfromSessionID } from "./coValue.js";
-import { createdNowUnique } from "./crypto.js";
-import { LocalNode } from "./node.js";
-import { randomAnonymousAccountAndSessionID } from "./testUtils.js";
-
-test("Empty CoMap works", () => {
-    const node = new LocalNode(...randomAnonymousAccountAndSessionID());
-
-    const coValue = node.createCoValue({
-        type: "comap",
-        ruleset: { type: "unsafeAllowAll" },
-        meta: null,
-        ...createdNowUnique(),
-    });
-
-    const content = coValue.getCurrentContent();
-
-    if (content.type !== "comap") {
-        throw new Error("Expected map");
-    }
-
-    expect(content.type).toEqual("comap");
-    expect([...content.keys()]).toEqual([]);
-    expect(content.toJSON()).toEqual({});
-});
-
-test("Can insert and delete CoMap entries in edit()", () => {
-    const node = new LocalNode(...randomAnonymousAccountAndSessionID());
-
-    const coValue = node.createCoValue({
-        type: "comap",
-        ruleset: { type: "unsafeAllowAll" },
-        meta: null,
-        ...createdNowUnique(),
-    });
-
-    const content = coValue.getCurrentContent();
-
-    if (content.type !== "comap") {
-        throw new Error("Expected map");
-    }
-
-    expect(content.type).toEqual("comap");
-
-    content.edit((editable) => {
-        editable.set("hello", "world", "trusting");
-        expect(editable.get("hello")).toEqual("world");
-        editable.set("foo", "bar", "trusting");
-        expect(editable.get("foo")).toEqual("bar");
-        expect([...editable.keys()]).toEqual(["hello", "foo"]);
-        editable.delete("foo", "trusting");
-        expect(editable.get("foo")).toEqual(undefined);
-    });
-});
-
-test("Can get CoMap entry values at different points in time", () => {
-    const node = new LocalNode(...randomAnonymousAccountAndSessionID());
-
-    const coValue = node.createCoValue({
-        type: "comap",
-        ruleset: { type: "unsafeAllowAll" },
-        meta: null,
-        ...createdNowUnique(),
-    });
-
-    const content = coValue.getCurrentContent();
-
-    if (content.type !== "comap") {
-        throw new Error("Expected map");
-    }
-
-    expect(content.type).toEqual("comap");
-
-    content.edit((editable) => {
-        const beforeA = Date.now();
-        while (Date.now() < beforeA + 10) {}
-        editable.set("hello", "A", "trusting");
-        const beforeB = Date.now();
-        while (Date.now() < beforeB + 10) {}
-        editable.set("hello", "B", "trusting");
-        const beforeC = Date.now();
-        while (Date.now() < beforeC + 10) {}
-        editable.set("hello", "C", "trusting");
-        expect(editable.get("hello")).toEqual("C");
-        expect(editable.getAtTime("hello", Date.now())).toEqual("C");
-        expect(editable.getAtTime("hello", beforeA)).toEqual(undefined);
-        expect(editable.getAtTime("hello", beforeB)).toEqual("A");
-        expect(editable.getAtTime("hello", beforeC)).toEqual("B");
-    });
-});
-
-test("Can get all historic values of key in CoMap", () => {
-    const node = new LocalNode(...randomAnonymousAccountAndSessionID());
-
-    const coValue = node.createCoValue({
-        type: "comap",
-        ruleset: { type: "unsafeAllowAll" },
-        meta: null,
-        ...createdNowUnique(),
-    });
-
-    const content = coValue.getCurrentContent();
-
-    if (content.type !== "comap") {
-        throw new Error("Expected map");
-    }
-
-    expect(content.type).toEqual("comap");
-
-    content.edit((editable) => {
-        editable.set("hello", "A", "trusting");
-        const txA = editable.getLastTxID("hello");
-        editable.set("hello", "B", "trusting");
-        const txB = editable.getLastTxID("hello");
-        editable.delete("hello", "trusting");
-        const txDel = editable.getLastTxID("hello");
-        editable.set("hello", "C", "trusting");
-        const txC = editable.getLastTxID("hello");
-        expect(editable.getHistory("hello")).toEqual([
-            {
-                txID: txA,
-                value: "A",
-                at: txA && coValue.getTx(txA)?.madeAt,
-            },
-            {
-                txID: txB,
-                value: "B",
-                at: txB && coValue.getTx(txB)?.madeAt,
-            },
-            {
-                txID: txDel,
-                value: undefined,
-                at: txDel && coValue.getTx(txDel)?.madeAt,
-            },
-            {
-                txID: txC,
-                value: "C",
-                at: txC && coValue.getTx(txC)?.madeAt,
-            },
-        ]);
-    });
-});
-
-test("Can get last tx ID for a key in CoMap", () => {
-    const node = new LocalNode(...randomAnonymousAccountAndSessionID());
-
-    const coValue = node.createCoValue({
-        type: "comap",
-        ruleset: { type: "unsafeAllowAll" },
-        meta: null,
-        ...createdNowUnique(),
-    });
-
-    const content = coValue.getCurrentContent();
-
-    if (content.type !== "comap") {
-        throw new Error("Expected map");
-    }
-
-    expect(content.type).toEqual("comap");
-
-    content.edit((editable) => {
-        expect(editable.getLastTxID("hello")).toEqual(undefined);
-        editable.set("hello", "A", "trusting");
-        const sessionID = editable.getLastTxID("hello")?.sessionID;
-        expect(sessionID && accountOrAgentIDfromSessionID(sessionID)).toEqual(
-            node.account.id
-        );
-        expect(editable.getLastTxID("hello")?.txIndex).toEqual(0);
-        editable.set("hello", "B", "trusting");
-        expect(editable.getLastTxID("hello")?.txIndex).toEqual(1);
-        editable.set("hello", "C", "trusting");
-        expect(editable.getLastTxID("hello")?.txIndex).toEqual(2);
-    });
-});
-
-test("Empty CoList works", () => {
-    const node = new LocalNode(...randomAnonymousAccountAndSessionID());
-
-    const coValue = node.createCoValue({
-        type: "colist",
-        ruleset: { type: "unsafeAllowAll" },
-        meta: null,
-        ...createdNowUnique(),
-    });
-
-    const content = coValue.getCurrentContent();
-
-    if (content.type !== "colist") {
-        throw new Error("Expected list");
-    }
-
-    expect(content.type).toEqual("colist");
-    expect(content.toJSON()).toEqual([]);
-});
-
-test("Can append, prepend and delete items to CoList", () => {
-    const node = new LocalNode(...randomAnonymousAccountAndSessionID());
-
-    const coValue = node.createCoValue({
-        type: "colist",
-        ruleset: { type: "unsafeAllowAll" },
-        meta: null,
-        ...createdNowUnique(),
-    });
-
-    const content = coValue.getCurrentContent();
-
-    if (content.type !== "colist") {
-        throw new Error("Expected list");
-    }
-
-    expect(content.type).toEqual("colist");
-
-    content.edit((editable) => {
-        editable.append(0, "hello", "trusting");
-        expect(editable.toJSON()).toEqual(["hello"]);
-        editable.append(0, "world", "trusting");
-        expect(editable.toJSON()).toEqual(["hello", "world"]);
-        editable.prepend(1, "beautiful", "trusting");
-        expect(editable.toJSON()).toEqual(["hello", "beautiful", "world"]);
-        editable.prepend(3, "hooray", "trusting");
-        expect(editable.toJSON()).toEqual([
-            "hello",
-            "beautiful",
-            "world",
-            "hooray",
-        ]);
-        editable.delete(2, "trusting");
-        expect(editable.toJSON()).toEqual(["hello", "beautiful", "hooray"]);
-    });
-});
-
-test("Push is equivalent to append after last item", () => {
-    const node = new LocalNode(...randomAnonymousAccountAndSessionID());
-
-    const coValue = node.createCoValue({
-        type: "colist",
-        ruleset: { type: "unsafeAllowAll" },
-        meta: null,
-        ...createdNowUnique(),
-    });
-
-    const content = coValue.getCurrentContent();
-
-    if (content.type !== "colist") {
-        throw new Error("Expected list");
-    }
-
-    expect(content.type).toEqual("colist");
-
-    content.edit((editable) => {
-        editable.append(0, "hello", "trusting");
-        expect(editable.toJSON()).toEqual(["hello"]);
-        editable.push("world", "trusting");
-        expect(editable.toJSON()).toEqual(["hello", "world"]);
-        editable.push("hooray", "trusting");
-        expect(editable.toJSON()).toEqual(["hello", "world", "hooray"]);
-    });
-});
-
-test("Can push into empty list", () => {
-    const node = new LocalNode(...randomAnonymousAccountAndSessionID());
-
-    const coValue = node.createCoValue({
-        type: "colist",
-        ruleset: { type: "unsafeAllowAll" },
-        meta: null,
-        ...createdNowUnique(),
-    });
-
-    const content = coValue.getCurrentContent();
-
-    if (content.type !== "colist") {
-        throw new Error("Expected list");
-    }
-
-    expect(content.type).toEqual("colist");
-
-    content.edit((editable) => {
-        editable.push("hello", "trusting");
-        expect(editable.toJSON()).toEqual(["hello"]);
-    });
-})

packages/cojson/src/contentType.ts

@@ -1,26 +0,0 @@
-import { JsonObject, JsonValue } from "./jsonValue.js";
-import { RawCoID } from "./ids.js";
-import { CoMap } from "./contentTypes/coMap.js";
-import { CoStream } from "./contentTypes/coStream.js";
-import { Static } from "./contentTypes/static.js";
-import { CoList } from "./contentTypes/coList.js";
-
-export type CoID<T extends ContentType> = RawCoID & {
-    readonly __type: T;
-};
-
-export type ContentType =
-    | CoMap<{ [key: string]: JsonValue }, JsonObject | null>
-    | CoList<JsonValue, JsonObject | null>
-    | CoStream<JsonValue, JsonObject | null>
-    | Static<JsonObject>;
-
-export function expectMap(
-    content: ContentType
-): CoMap<{ [key: string]: string }, JsonObject | null> {
-    if (content.type !== "comap") {
-        throw new Error("Expected map");
-    }
-
-    return content as CoMap<{ [key: string]: string }, JsonObject | null>;
-}

packages/cojson/src/contentTypes/coStream.ts

@@ -1,24 +0,0 @@
-import { JsonObject, JsonValue } from '../jsonValue.js';
-import { CoID } from '../contentType.js';
-import { CoValue } from '../coValue.js';
-
-export class CoStream<T extends JsonValue, Meta extends JsonObject | null = null> {
-    id: CoID<CoStream<T, Meta>>;
-    type = "costream" as const;
-    coValue: CoValue;
-
-    constructor(coValue: CoValue) {
-        this.id = coValue.id as CoID<CoStream<T, Meta>>;
-        this.coValue = coValue;
-    }
-
-    toJSON(): JsonObject {
-        throw new Error("Method not implemented.");
-    }
-
-    subscribe(listener: (coMap: CoStream<T, Meta>) => void): () => void {
-        return this.coValue.subscribe((content) => {
-            listener(content as CoStream<T, Meta>);
-        });
-    }
-}

packages/cojson/src/contentTypes/static.ts

@@ -1,22 +0,0 @@
-import { JsonObject } from '../jsonValue.js';
-import { CoID } from '../contentType.js';
-import { CoValue } from '../coValue.js';
-
-export class Static<T extends JsonObject> {
-    id: CoID<Static<T>>;
-    type = "static" as const;
-    coValue: CoValue;
-
-    constructor(coValue: CoValue) {
-        this.id = coValue.id as CoID<Static<T>>;
-        this.coValue = coValue;
-    }
-
-    toJSON(): JsonObject {
-        throw new Error("Method not implemented.");
-    }
-
-    subscribe(_listener: (coMap: Static<T>) => void): () => void {
-        throw new Error("Method not implemented.");
-    }
-}

packages/cojson/src/crypto.test.ts

@@ -21,6 +21,11 @@ import { xsalsa20_poly1305 } from "@noble/ciphers/salsa";
 import { blake3 } from "@noble/hashes/blake3";
 import stableStringify from "fast-json-stable-stringify";
 import { SessionID } from './ids.js';
+import { cojsonReady } from './index.js';
+
+beforeEach(async () => {
+    await cojsonReady;
+});
 
 test("Signatures round-trip and use stable stringify", () => {
     const data = { b: "world", a: "hello" };

packages/cojson/src/crypto.ts

@@ -1,11 +1,39 @@
 import { ed25519, x25519 } from "@noble/curves/ed25519";
 import { xsalsa20_poly1305, xsalsa20 } from "@noble/ciphers/salsa";
 import { JsonValue } from "./jsonValue.js";
-import { base58, base64url } from "@scure/base";
-import stableStringify from "fast-json-stable-stringify";
-import { blake3 } from "@noble/hashes/blake3";
+import { base58 } from "@scure/base";
 import { randomBytes } from "@noble/ciphers/webcrypto/utils";
 import { AgentID, RawCoID, TransactionID } from "./ids.js";
+import { base64URLtoBytes, bytesToBase64url } from "./base64url.js";
+
+import { createBLAKE3 } from 'hash-wasm';
+import { Stringified, parseJSON, stableStringify } from "./jsonStringify.js";
+
+let blake3Instance: Awaited<ReturnType<typeof createBLAKE3>>;
+let blake3HashOnce: (data: Uint8Array) => Uint8Array;
+let blake3HashOnceWithContext: (data: Uint8Array, {context}: {context: Uint8Array}) => Uint8Array;
+let blake3incrementalUpdateSLOW_WITH_DEVTOOLS: (state: Uint8Array, data: Uint8Array) => Uint8Array;
+let blake3digestForState: (state: Uint8Array) => Uint8Array;
+
+export const cryptoReady = new Promise<void>((resolve) => {
+    createBLAKE3().then(bl3 => {
+        blake3Instance = bl3;
+        blake3HashOnce = (data) => {
+            return bl3.init().update(data).digest('binary');
+        }
+        blake3HashOnceWithContext = (data, {context}) => {
+            return bl3.init().update(context).update(data).digest('binary');
+        }
+        blake3incrementalUpdateSLOW_WITH_DEVTOOLS = (state, data) => {
+            bl3.load(state).update(data);
+            return bl3.save();
+        }
+        blake3digestForState = (state) => {
+            return bl3.load(state).digest('binary');
+        }
+        resolve();
+    })
+});
 
 export type SignerSecret = `signerSecret_z${string}`;
 export type SignerID = `signer_z${string}`;
@@ -127,7 +155,7 @@ export function seal<T extends JsonValue>(
     to: SealerID,
     nOnceMaterial: { in: RawCoID; tx: TransactionID }
 ): Sealed<T> {
-    const nOnce = blake3(
+    const nOnce = blake3HashOnce(
         textEncoder.encode(stableStringify(nOnceMaterial))
     ).slice(0, 24);
 
@@ -143,7 +171,7 @@ export function seal<T extends JsonValue>(
         plaintext
     );
 
-    return `sealed_U${base64url.encode(sealedBytes)}` as Sealed<T>;
+    return `sealed_U${bytesToBase64url(sealedBytes)}` as Sealed<T>;
 }
 
 export function unseal<T extends JsonValue>(
@@ -152,7 +180,7 @@ export function unseal<T extends JsonValue>(
     from: SealerID,
     nOnceMaterial: { in: RawCoID; tx: TransactionID }
 ): T | undefined {
-    const nOnce = blake3(
+    const nOnce = blake3HashOnce(
         textEncoder.encode(stableStringify(nOnceMaterial))
     ).slice(0, 24);
 
@@ -160,7 +188,7 @@ export function unseal<T extends JsonValue>(
 
     const senderPub = base58.decode(from.substring("sealer_z".length));
 
-    const sealedBytes = base64url.decode(sealed.substring("sealed_U".length));
+    const sealedBytes = base64URLtoBytes(sealed.substring("sealed_U".length));
 
     const sharedSecret = x25519.getSharedSecret(sealerPriv, senderPub);
 
@@ -180,28 +208,32 @@ export type Hash = `hash_z${string}`;
 
 export function secureHash(value: JsonValue): Hash {
     return `hash_z${base58.encode(
-        blake3(textEncoder.encode(stableStringify(value)))
+        blake3HashOnce(textEncoder.encode(stableStringify(value)))
     )}`;
 }
 
 export class StreamingHash {
-    state: ReturnType<typeof blake3.create>;
+    state: Uint8Array;
 
-    constructor(fromClone?: ReturnType<typeof blake3.create>) {
-        this.state = fromClone || blake3.create({});
+    constructor(fromClone?: Uint8Array) {
+        this.state = fromClone || blake3Instance.init().save();
     }
 
     update(value: JsonValue) {
-        this.state.update(textEncoder.encode(stableStringify(value)));
+        const encoded = textEncoder.encode(stableStringify(value))
+        // const before = performance.now();
+        this.state = blake3incrementalUpdateSLOW_WITH_DEVTOOLS(this.state, encoded);
+        // const after = performance.now();
+        // console.log(`Hashing throughput in MB/s`, 1000 * (encoded.length / (after - before)) / (1024 * 1024));
     }
 
     digest(): Hash {
-        const hash = this.state.digest();
+        const hash = blake3digestForState(this.state);
         return `hash_z${base58.encode(hash)}`;
     }
 
     clone(): StreamingHash {
-        return new StreamingHash(this.state.clone());
+        return new StreamingHash(new Uint8Array(this.state));
     }
 }
 
@@ -210,7 +242,10 @@ export const shortHashLength = 19;
 
 export function shortHash(value: JsonValue): ShortHash {
     return `shortHash_z${base58.encode(
-        blake3(textEncoder.encode(stableStringify(value))).slice(0, shortHashLength)
+        blake3HashOnce(textEncoder.encode(stableStringify(value))).slice(
+            0,
+            shortHashLength
+        )
     )}`;
 }
 
@@ -237,13 +272,13 @@ function encrypt<T extends JsonValue, N extends JsonValue>(
     const keySecretBytes = base58.decode(
         keySecret.substring("keySecret_z".length)
     );
-    const nOnce = blake3(
+    const nOnce = blake3HashOnce(
         textEncoder.encode(stableStringify(nOnceMaterial))
     ).slice(0, 24);
 
     const plaintext = textEncoder.encode(stableStringify(value));
     const ciphertext = xsalsa20(keySecretBytes, nOnce, plaintext);
-    return `encrypted_U${base64url.encode(ciphertext)}` as Encrypted<T, N>;
+    return `encrypted_U${bytesToBase64url(ciphertext)}` as Encrypted<T, N>;
 }
 
 export function encryptForTransaction<T extends JsonValue>(
@@ -281,30 +316,48 @@ export function encryptKeySecret(keys: {
     };
 }
 
+function decryptRaw<T extends JsonValue, N extends JsonValue>(
+    encrypted: Encrypted<T, N>,
+    keySecret: KeySecret,
+    nOnceMaterial: N
+): Stringified<T> {
+    const keySecretBytes = base58.decode(
+        keySecret.substring("keySecret_z".length)
+    );
+    const nOnce = blake3HashOnce(
+        textEncoder.encode(stableStringify(nOnceMaterial))
+    ).slice(0, 24);
+
+    const ciphertext = base64URLtoBytes(
+        encrypted.substring("encrypted_U".length)
+    );
+    const plaintext = xsalsa20(keySecretBytes, nOnce, ciphertext);
+
+        return textDecoder.decode(plaintext) as Stringified<T>;
+
+}
+
 function decrypt<T extends JsonValue, N extends JsonValue>(
     encrypted: Encrypted<T, N>,
     keySecret: KeySecret,
     nOnceMaterial: N
 ): T | undefined {
-    const keySecretBytes = base58.decode(
-        keySecret.substring("keySecret_z".length)
-    );
-    const nOnce = blake3(
-        textEncoder.encode(stableStringify(nOnceMaterial))
-    ).slice(0, 24);
-
-    const ciphertext = base64url.decode(
-        encrypted.substring("encrypted_U".length)
-    );
-    const plaintext = xsalsa20(keySecretBytes, nOnce, ciphertext);
-
     try {
-        return JSON.parse(textDecoder.decode(plaintext));
+        return parseJSON(decryptRaw(encrypted, keySecret, nOnceMaterial));
     } catch (e) {
+        console.error("Decryption error", e)
         return undefined;
     }
 }
 
+export function decryptRawForTransaction<T extends JsonValue>(
+    encrypted: Encrypted<T, { in: RawCoID; tx: TransactionID }>,
+    keySecret: KeySecret,
+    nOnceMaterial: { in: RawCoID; tx: TransactionID }
+): Stringified<T> | undefined {
+    return decryptRaw(encrypted, keySecret, nOnceMaterial);
+}
+
 export function decryptForTransaction<T extends JsonValue>(
     encrypted: Encrypted<T, { in: RawCoID; tx: TransactionID }>,
     keySecret: KeySecret,
@@ -355,15 +408,17 @@ export function newRandomSecretSeed(): Uint8Array {
 
 export function agentSecretFromSecretSeed(secretSeed: Uint8Array): AgentSecret {
     if (secretSeed.length !== secretSeedLength) {
-        throw new Error(`Secret seed needs to be ${secretSeedLength} bytes long`);
+        throw new Error(
+            `Secret seed needs to be ${secretSeedLength} bytes long`
+        );
     }
 
     return `sealerSecret_z${base58.encode(
-        blake3(secretSeed, {
+        blake3HashOnceWithContext(secretSeed, {
             context: textEncoder.encode("seal"),
         })
     )}/signerSecret_z${base58.encode(
-        blake3(secretSeed, {
+        blake3HashOnceWithContext(secretSeed, {
             context: textEncoder.encode("sign"),
         })
     )}`;

packages/cojson/src/group.test.ts

@@ -0,0 +1,51 @@
+import { LocalNode, CoMap, CoList, CoStream, BinaryCoStream, cojsonReady } from "./index";
+import { randomAnonymousAccountAndSessionID } from "./testUtils";
+
+beforeEach(async () => {
+    await cojsonReady;
+});
+
+test("Can create a CoMap in a group", () => {
+    const node = new LocalNode(...randomAnonymousAccountAndSessionID());
+
+    const group = node.createGroup();
+
+    const map = group.createMap();
+
+    expect(map.core.getCurrentContent().type).toEqual("comap");
+    expect(map instanceof CoMap).toEqual(true);
+});
+
+test("Can create a CoList in a group", () => {
+    const node = new LocalNode(...randomAnonymousAccountAndSessionID());
+
+    const group = node.createGroup();
+
+    const list = group.createList();
+
+    expect(list.core.getCurrentContent().type).toEqual("colist");
+    expect(list instanceof CoList).toEqual(true);
+})
+
+test("Can create a CoStream in a group", () => {
+    const node = new LocalNode(...randomAnonymousAccountAndSessionID());
+
+    const group = node.createGroup();
+
+    const stream = group.createStream();
+
+    expect(stream.core.getCurrentContent().type).toEqual("costream");
+    expect(stream instanceof CoStream).toEqual(true);
+});
+
+test("Can create a BinaryCoStream in a group", () => {
+    const node = new LocalNode(...randomAnonymousAccountAndSessionID());
+
+    const group = node.createGroup();
+
+    const stream = group.createBinaryStream();
+
+    expect(stream.core.getCurrentContent().type).toEqual("costream");
+    expect(stream.meta.type).toEqual("binary");
+    expect(stream instanceof BinaryCoStream).toEqual(true);
+})

packages/cojson/src/group.ts

@@ -1,5 +1,5 @@
-import { CoID, ContentType } from "./contentType.js";
-import { CoMap } from "./contentTypes/coMap.js";
+import { CoID, CoValueImpl } from "./coValue.js";
+import { CoMap } from "./coValues/coMap.js";
 import { JsonObject, JsonValue } from "./jsonValue.js";
 import {
     Encrypted,
@@ -17,14 +17,11 @@ import {
 } from "./crypto.js";
 import { LocalNode } from "./node.js";
 import { AgentID, SessionID, isAgentID } from "./ids.js";
-import {
-    AccountID,
-    GeneralizedControlledAccount,
-    Profile,
-} from "./account.js";
+import { AccountID, GeneralizedControlledAccount, Profile } from "./account.js";
 import { Role } from "./permissions.js";
 import { base58 } from "@scure/base";
-import { CoList } from "./contentTypes/coList.js";
+import { CoList } from "./coValues/coList.js";
+import { BinaryCoStream, BinaryCoStreamMeta, CoStream } from "./coValues/coStream.js";
 
 export type GroupContent = {
     profile: CoID<Profile> | null;
@@ -38,7 +35,7 @@ export type GroupContent = {
 };
 
 export function expectGroupContent(
-    content: ContentType
+    content: CoValueImpl
 ): CoMap<GroupContent, JsonObject | null> {
     if (content.type !== "comap") {
         throw new Error("Expected map");
@@ -47,43 +44,71 @@ export function expectGroupContent(
     return content as CoMap<GroupContent, JsonObject | null>;
 }
 
+/** A `Group` is a scope for permissions of its members (`"reader" | "writer" | "admin"`), applying to objects owned by that group.
+ *
+ *  A `Group` object exposes methods for permission management and allows you to create new CoValues owned by that group.
+ *
+ *  (Internally, a `Group` is also just a `CoMap`, mapping member accounts to roles and containing some
+ *  state management for making cryptographic keys available to current members)
+ *
+ *  @example
+ *  You typically get a group from a CoValue that you already have loaded:
+ *
+ *  ```typescript
+ *  const group = coMap.group;
+ *  ```
+ *
+ *  @example
+ *  Or, you can create a new group with a `LocalNode`:
+ *
+ *  ```typescript
+ *  const localNode.createGroup();
+ *  ```
+ * */
 export class Group {
-    groupMap: CoMap<GroupContent, JsonObject | null>;
+    underlyingMap: CoMap<GroupContent, JsonObject | null>;
+    /** @internal */
     node: LocalNode;
 
+    /** @internal */
     constructor(
-        groupMap: CoMap<GroupContent, JsonObject | null>,
+        underlyingMap: CoMap<GroupContent, JsonObject | null>,
         node: LocalNode
     ) {
-        this.groupMap = groupMap;
+        this.underlyingMap = underlyingMap;
         this.node = node;
     }
 
+    /** Returns the `CoID` of the `Group`. */
     get id(): CoID<CoMap<GroupContent, JsonObject | null>> {
-        return this.groupMap.id;
+        return this.underlyingMap.id;
     }
 
+    /** Returns the current role of a given account. */
     roleOf(accountID: AccountID): Role | undefined {
         return this.roleOfInternal(accountID);
     }
 
     /** @internal */
     roleOfInternal(accountID: AccountID | AgentID): Role | undefined {
-        return this.groupMap.get(accountID);
+        return this.underlyingMap.get(accountID);
     }
 
+    /** Returns the role of the current account in the group. */
     myRole(): Role | undefined {
         return this.roleOfInternal(this.node.account.id);
     }
 
+    /** Directly grants a new member a role in the group. The current account must be an
+     * admin to be able to do so. Throws otherwise. */
     addMember(accountID: AccountID, role: Role) {
         this.addMemberInternal(accountID, role);
     }
 
     /** @internal */
     addMemberInternal(accountID: AccountID | AgentID, role: Role) {
-        this.groupMap = this.groupMap.edit((map) => {
-            const currentReadKey = this.groupMap.coValue.getCurrentReadKey();
+        this.underlyingMap = this.underlyingMap.edit((map) => {
+            const currentReadKey = this.underlyingMap.core.getCurrentReadKey();
 
             if (!currentReadKey.secret) {
                 throw new Error("Can't add member without read key secret");
@@ -104,11 +129,11 @@ export class Group {
                 `${currentReadKey.id}_for_${accountID}`,
                 seal(
                     currentReadKey.secret,
-                    this.groupMap.coValue.node.account.currentSealerSecret(),
+                    this.underlyingMap.core.node.account.currentSealerSecret(),
                     getAgentSealerID(agent),
                     {
-                        in: this.groupMap.coValue.id,
-                        tx: this.groupMap.coValue.nextTransactionID(),
+                        in: this.underlyingMap.core.id,
+                        tx: this.underlyingMap.core.nextTransactionID(),
                     }
                 ),
                 "trusting"
@@ -116,30 +141,24 @@ export class Group {
         });
     }
 
-    createInvite(role: "reader" | "writer" | "admin"): InviteSecret {
-        const secretSeed = newRandomSecretSeed();
-
-        const inviteSecret = agentSecretFromSecretSeed(secretSeed);
-        const inviteID = getAgentID(inviteSecret);
-
-        this.addMemberInternal(inviteID, `${role}Invite` as Role);
-
-        return inviteSecretFromSecretSeed(secretSeed);
-    }
-
+    /** @internal */
     rotateReadKey() {
-        const currentlyPermittedReaders = this.groupMap.keys().filter((key) => {
-            if (key.startsWith("co_") || isAgentID(key)) {
-                const role = this.groupMap.get(key);
-                return (
-                    role === "admin" || role === "writer" || role === "reader"
-                );
-            } else {
-                return false;
-            }
-        }) as (AccountID | AgentID)[];
+        const currentlyPermittedReaders = this.underlyingMap
+            .keys()
+            .filter((key) => {
+                if (key.startsWith("co_") || isAgentID(key)) {
+                    const role = this.underlyingMap.get(key);
+                    return (
+                        role === "admin" ||
+                        role === "writer" ||
+                        role === "reader"
+                    );
+                } else {
+                    return false;
+                }
+            }) as (AccountID | AgentID)[];
 
-        const maybeCurrentReadKey = this.groupMap.coValue.getCurrentReadKey();
+        const maybeCurrentReadKey = this.underlyingMap.core.getCurrentReadKey();
 
         if (!maybeCurrentReadKey.secret) {
             throw new Error(
@@ -154,7 +173,7 @@ export class Group {
 
         const newReadKey = newRandomKeySecret();
 
-        this.groupMap = this.groupMap.edit((map) => {
+        this.underlyingMap = this.underlyingMap.edit((map) => {
             for (const readerID of currentlyPermittedReaders) {
                 const reader = this.node.resolveAccountAgent(
                     readerID,
@@ -165,11 +184,11 @@ export class Group {
                     `${newReadKey.id}_for_${readerID}`,
                     seal(
                         newReadKey.secret,
-                        this.groupMap.coValue.node.account.currentSealerSecret(),
+                        this.underlyingMap.core.node.account.currentSealerSecret(),
                         getAgentSealerID(reader),
                         {
-                            in: this.groupMap.coValue.id,
-                            tx: this.groupMap.coValue.nextTransactionID(),
+                            in: this.underlyingMap.core.id,
+                            tx: this.underlyingMap.core.nextTransactionID(),
                         }
                     ),
                     "trusting"
@@ -189,20 +208,37 @@ export class Group {
         });
     }
 
+    /** Strips the specified member of all roles (preventing future writes in
+     *  the group and owned values) and rotates the read encryption key for that group
+     * (preventing reads of new content in the group and owned values) */
     removeMember(accountID: AccountID) {
         this.removeMemberInternal(accountID);
     }
 
     /** @internal */
     removeMemberInternal(accountID: AccountID | AgentID) {
-        this.groupMap = this.groupMap.edit((map) => {
+        this.underlyingMap = this.underlyingMap.edit((map) => {
             map.set(accountID, "revoked", "trusting");
         });
 
         this.rotateReadKey();
     }
 
-    createMap<M extends CoMap<{ [key: string]: JsonValue }, JsonObject | null>>(
+    /** Creates an invite for new members to indirectly join the group, allowing them to grant themselves the specified role with the InviteSecret (a string starting with "inviteSecret_") - use `LocalNode.acceptInvite()` for this purpose. */
+    createInvite(role: "reader" | "writer" | "admin"): InviteSecret {
+        const secretSeed = newRandomSecretSeed();
+
+        const inviteSecret = agentSecretFromSecretSeed(secretSeed);
+        const inviteID = getAgentID(inviteSecret);
+
+        this.addMemberInternal(inviteID, `${role}Invite` as Role);
+
+        return inviteSecretFromSecretSeed(secretSeed);
+    }
+
+    /** Creates a new `CoMap` within this group, with the specified specialized
+     *  `CoMap` type `M` and optional static metadata. */
+    createMap<M extends CoMap<{ [key: string]: JsonValue | undefined; }, JsonObject | null>>(
         meta?: M["meta"]
     ): M {
         return this.node
@@ -210,7 +246,7 @@ export class Group {
                 type: "comap",
                 ruleset: {
                     type: "ownedByGroup",
-                    group: this.groupMap.id,
+                    group: this.underlyingMap.id,
                 },
                 meta: meta || null,
                 ...createdNowUnique(),
@@ -218,6 +254,8 @@ export class Group {
             .getCurrentContent() as M;
     }
 
+    /** Creates a new `CoList` within this group, with the specified specialized
+     * `CoList` type `L` and optional static metadata. */
     createList<L extends CoList<JsonValue, JsonObject | null>>(
         meta?: L["meta"]
     ): L {
@@ -226,7 +264,7 @@ export class Group {
                 type: "colist",
                 ruleset: {
                     type: "ownedByGroup",
-                    group: this.groupMap.id,
+                    group: this.underlyingMap.id,
                 },
                 meta: meta || null,
                 ...createdNowUnique(),
@@ -234,6 +272,38 @@ export class Group {
             .getCurrentContent() as L;
     }
 
+    createStream<C extends CoStream<JsonValue, JsonObject | null>>(
+        meta?: C["meta"]
+    ): C {
+        return this.node
+            .createCoValue({
+                type: "costream",
+                ruleset: {
+                    type: "ownedByGroup",
+                    group: this.underlyingMap.id,
+                },
+                meta: meta || null,
+                ...createdNowUnique(),
+            })
+            .getCurrentContent() as C;
+    }
+
+    createBinaryStream<
+        C extends BinaryCoStream<BinaryCoStreamMeta>
+    >(meta: C["meta"] = { type: "binary" }): C {
+        return this.node
+            .createCoValue({
+                type: "costream",
+                ruleset: {
+                    type: "ownedByGroup",
+                    group: this.underlyingMap.id,
+                },
+                meta: meta,
+                ...createdNowUnique(),
+            })
+            .getCurrentContent() as C;
+    }
+
     /** @internal */
     testWithDifferentAccount(
         account: GeneralizedControlledAccount,
@@ -241,7 +311,7 @@ export class Group {
     ): Group {
         return new Group(
             expectGroupContent(
-                this.groupMap.coValue
+                this.underlyingMap.core
                     .testWithDifferentAccount(account, sessionId)
                     .getCurrentContent()
             ),

packages/cojson/src/index.ts

@@ -1,7 +1,14 @@
-import { CoValue, newRandomSessionID } from "./coValue.js";
+import { CoValueCore, newRandomSessionID } from "./coValueCore.js";
 import { LocalNode } from "./node.js";
-import { CoMap } from "./contentTypes/coMap.js";
-import { CoList } from "./contentTypes/coList.js";
+import type { CoValue, ReadableCoValue } from "./coValue.js";
+import { CoMap, WriteableCoMap } from "./coValues/coMap.js";
+import { CoList, WriteableCoList } from "./coValues/coList.js";
+import {
+    CoStream,
+    WriteableCoStream,
+    BinaryCoStream,
+    WriteableBinaryCoStream,
+} from "./coValues/coStream.js";
 import {
     agentSecretFromBytes,
     agentSecretToBytes,
@@ -11,28 +18,28 @@ import {
     agentSecretFromSecretSeed,
     secretSeedLength,
     shortHashLength,
+    cryptoReady
 } from "./crypto.js";
 import { connectedPeers } from "./streamUtils.js";
 import { AnonymousControlledAccount, ControlledAccount } from "./account.js";
 import { rawCoIDtoBytes, rawCoIDfromBytes } from "./ids.js";
-import { Group, expectGroupContent } from "./group.js"
+import { Group, expectGroupContent } from "./group.js";
+import { base64URLtoBytes, bytesToBase64url } from "./base64url.js";
+import { parseJSON } from "./jsonStringify.js";
 
 import type { SessionID, AgentID } from "./ids.js";
-import type { CoID, ContentType } from "./contentType.js";
+import type { CoID, CoValueImpl } from "./coValue.js";
+import type { BinaryChunkInfo, BinaryCoStreamMeta } from "./coValues/coStream.js";
 import type { JsonValue } from "./jsonValue.js";
 import type { SyncMessage, Peer } from "./sync.js";
 import type { AgentSecret } from "./crypto.js";
-import type {
-    AccountID,
-    AccountContent,
-    ProfileContent,
-    ProfileMeta,
-    Profile,
-} from "./account.js";
+import type { AccountID, Profile } from "./account.js";
 import type { InviteSecret } from "./group.js";
+import type * as Media from "./media.js";
 
-type Value = JsonValue | ContentType;
+type Value = JsonValue | CoValueImpl;
 
+/** @hidden */
 export const cojsonInternals = {
     agentSecretFromBytes,
     agentSecretToBytes,
@@ -46,35 +53,47 @@ export const cojsonInternals = {
     agentSecretFromSecretSeed,
     secretSeedLength,
     shortHashLength,
-    expectGroupContent
+    expectGroupContent,
+    base64URLtoBytes,
+    bytesToBase64url,
+    parseJSON
 };
 
 export {
     LocalNode,
-    CoValue,
+    Group,
     CoMap,
+    WriteableCoMap,
     CoList,
+    WriteableCoList,
+    CoStream,
+    WriteableCoStream,
+    BinaryCoStream,
+    WriteableBinaryCoStream,
+    CoValueCore,
     AnonymousControlledAccount,
     ControlledAccount,
-    Group
+    cryptoReady as cojsonReady,
 };
 
 export type {
     Value,
     JsonValue,
-    ContentType,
+    CoValue,
+    ReadableCoValue,
+    CoValueImpl,
     CoID,
-    AgentSecret,
-    SessionID,
-    SyncMessage,
-    AgentID,
     AccountID,
-    Peer,
-    AccountContent,
     Profile,
-    ProfileContent,
-    ProfileMeta,
-    InviteSecret
+    SessionID,
+    Peer,
+    BinaryChunkInfo,
+    BinaryCoStreamMeta,
+    AgentID,
+    AgentSecret,
+    InviteSecret,
+    SyncMessage,
+    Media
 };
 
 // eslint-disable-next-line @typescript-eslint/no-namespace
@@ -84,8 +103,13 @@ export namespace CojsonInternalTypes {
     export type KnownStateMessage = import("./sync.js").KnownStateMessage;
     export type LoadMessage = import("./sync.js").LoadMessage;
     export type NewContentMessage = import("./sync.js").NewContentMessage;
-    export type CoValueHeader = import("./coValue.js").CoValueHeader;
-    export type Transaction = import("./coValue.js").Transaction;
+    export type CoValueHeader = import("./coValueCore.js").CoValueHeader;
+    export type Transaction = import("./coValueCore.js").Transaction;
     export type Signature = import("./crypto.js").Signature;
     export type RawCoID = import("./ids.js").RawCoID;
+    export type AccountContent = import("./account.js").AccountContent;
+    export type ProfileContent = import("./account.js").ProfileContent;
+    export type ProfileMeta = import("./account.js").ProfileMeta;
+    export type SealerSecret = import("./crypto.js").SealerSecret;
+    export type SignerSecret = import("./crypto.js").SignerSecret;
 }

packages/cojson/src/jsonStringify.ts

@@ -0,0 +1,66 @@
+// adapted from fast-json-stable-stringify (https://github.com/epoberezkin/fast-json-stable-stringify)
+
+export type Stringified<T> = string & { __type: T };
+
+export function stableStringify<T>(data: T): Stringified<T>
+export function stableStringify(data: undefined): undefined
+export function stableStringify<T>(data: T | undefined): Stringified<T> | undefined {
+    const cycles = false;
+
+    // eslint-disable-next-line @typescript-eslint/no-explicit-any
+    const seen: any[] = [];
+    // eslint-disable-next-line @typescript-eslint/no-explicit-any
+    let node = data as any;
+
+    if (node && node.toJSON && typeof node.toJSON === "function") {
+        node = node.toJSON();
+    }
+
+    if (node === undefined) return;
+    if (typeof node == "number")
+        return (isFinite(node) ? "" + node : "null") as Stringified<T>;
+    if (typeof node !== "object") {
+        if (
+            typeof node === "string" &&
+            (node.startsWith("encrypted_U") || node.startsWith("binary_U"))
+        ) {
+            return `"${node}"` as Stringified<T>;
+        }
+        return JSON.stringify(node) as Stringified<T>;
+    }
+
+    let i, out;
+    if (Array.isArray(node)) {
+        out = "[";
+        for (i = 0; i < node.length; i++) {
+            if (i) out += ",";
+            out += stableStringify(node[i]) || "null";
+        }
+        return (out + "]") as Stringified<T>;
+    }
+
+    if (node === null) return "null" as Stringified<T>;
+
+    if (seen.indexOf(node) !== -1) {
+        if (cycles) return JSON.stringify("__cycle__") as Stringified<T>;
+        throw new TypeError("Converting circular structure to JSON");
+    }
+
+    const seenIndex = seen.push(node) - 1;
+    const keys = Object.keys(node).sort();
+    out = "";
+    for (i = 0; i < keys.length; i++) {
+        const key = keys[i]!;
+        const value = stableStringify(node[key]);
+
+        if (!value) continue;
+        if (out) out += ",";
+        out += JSON.stringify(key) + ":" + value;
+    }
+    seen.splice(seenIndex, 1);
+    return ("{" + out + "}") as Stringified<T>;
+}
+
+export function parseJSON<T>(json: Stringified<T>): T {
+    return JSON.parse(json);
+}

packages/cojson/src/jsonValue.ts

@@ -3,4 +3,4 @@ import { RawCoID } from './ids.js';
 export type JsonAtom = string | number | boolean | null;
 export type JsonValue = JsonAtom | JsonArray | JsonObject | RawCoID;
 export type JsonArray = JsonValue[];
-export type JsonObject = { [key: string]: JsonValue; };
+export type JsonObject = { [key: string]: JsonValue | undefined; };

packages/cojson/src/media.ts

@@ -0,0 +1,9 @@
+import { CoMap } from './coValues/coMap.js'
+import { CoID } from './coValue.js'
+import { BinaryCoStream } from './coValues/coStream.js'
+
+export type ImageDefinition = CoMap<{
+    originalSize: [number, number];
+    placeholderDataURL?: string;
+    [res: `${number}x${number}`]: CoID<BinaryCoStream>;
+}>;

packages/cojson/src/node.ts

@@ -9,7 +9,7 @@ import {
     newRandomKeySecret,
     seal,
 } from "./crypto.js";
-import { CoValue, CoValueHeader, newRandomSessionID } from "./coValue.js";
+import { CoValueCore, CoValueHeader, newRandomSessionID } from "./coValueCore.js";
 import {
     InviteSecret,
     Group,
@@ -19,7 +19,7 @@ import {
 } from "./group.js";
 import { Peer, SyncManager } from "./sync.js";
 import { AgentID, RawCoID, SessionID, isAgentID } from "./ids.js";
-import { CoID, ContentType } from "./contentType.js";
+import { CoID, CoValueImpl } from "./coValue.js";
 import {
     Account,
     AccountMeta,
@@ -32,8 +32,19 @@ import {
     AccountContent,
     AccountMap,
 } from "./account.js";
-import { CoMap } from "./index.js";
+import { CoMap } from "./coValues/coMap.js";
 
+/** A `LocalNode` represents a local view of a set of loaded `CoValue`s, from the perspective of a particular account (or primitive cryptographic agent).
+
+A `LocalNode` can have peers that it syncs to, for example some form of local persistence, or a sync server, such as `sync.jazz.tools` (Jazz Global Mesh).
+
+@example
+You typically get hold of a `LocalNode` using `jazz-react`'s `useJazz()`:
+
+```typescript
+const { localNode } = useJazz();
+```
+*/
 export class LocalNode {
     /** @internal */
     coValues: { [key: RawCoID]: CoValueState } = {};
@@ -111,8 +122,8 @@ export class LocalNode {
     }
 
     /** @internal */
-    createCoValue(header: CoValueHeader): CoValue {
-        const coValue = new CoValue(header, this);
+    createCoValue(header: CoValueHeader): CoValueCore {
+        const coValue = new CoValueCore(header, this);
         this.coValues[coValue.id] = { state: "loaded", coValue: coValue };
 
         void this.sync.syncCoValue(coValue);
@@ -121,7 +132,7 @@ export class LocalNode {
     }
 
     /** @internal */
-    loadCoValue(id: RawCoID): Promise<CoValue> {
+    loadCoValue(id: RawCoID): Promise<CoValueCore> {
         let entry = this.coValues[id];
         if (!entry) {
             entry = newLoadingState();
@@ -136,10 +147,19 @@ export class LocalNode {
         return entry.done;
     }
 
-    async load<T extends ContentType>(id: CoID<T>): Promise<T> {
+    /**
+     * Loads a CoValue's content, syncing from peers as necessary and resolving the returned
+     * promise once a first version has been loaded. See `coValue.subscribe()` and `node.useTelepathicData()`
+     * for listening to subsequent updates to the CoValue.
+     */
+    async load<T extends CoValueImpl>(id: CoID<T>): Promise<T> {
         return (await this.loadCoValue(id)).getCurrentContent() as T;
     }
 
+    /**
+     * Loads a profile associated with an account. `Profile` is at least a `CoMap<{string: name}>`,
+     * but might contain other, app-specific properties.
+     */
     async loadProfile(id: AccountID): Promise<Profile> {
         const account = await this.load<AccountMap>(id);
         const profileID = account.get("profile");
@@ -152,20 +172,20 @@ export class LocalNode {
         ).getCurrentContent() as Profile;
     }
 
-    async acceptInvite<T extends ContentType>(
+    async acceptInvite<T extends CoValueImpl>(
         groupOrOwnedValueID: CoID<T>,
         inviteSecret: InviteSecret
     ): Promise<void> {
         const groupOrOwnedValue = await this.load(groupOrOwnedValueID);
 
-        if (groupOrOwnedValue.coValue.header.ruleset.type === "ownedByGroup") {
+        if (groupOrOwnedValue.core.header.ruleset.type === "ownedByGroup") {
             return this.acceptInvite(
-                groupOrOwnedValue.coValue.header.ruleset.group as CoID<
+                groupOrOwnedValue.core.header.ruleset.group as CoID<
                     CoMap<GroupContent>
                 >,
                 inviteSecret
             );
-        } else if (groupOrOwnedValue.coValue.header.ruleset.type !== "group") {
+        } else if (groupOrOwnedValue.core.header.ruleset.type !== "group") {
             throw new Error("Can only accept invites to groups");
         }
 
@@ -177,7 +197,7 @@ export class LocalNode {
         const inviteAgentID = getAgentID(inviteAgentSecret);
 
         const inviteRole = await new Promise((resolve, reject) => {
-            group.groupMap.subscribe((groupMap) => {
+            group.underlyingMap.subscribe((groupMap) => {
                 const role = groupMap.get(inviteAgentID);
                 if (role) {
                     resolve(role);
@@ -188,7 +208,7 @@ export class LocalNode {
                     reject(
                         new Error("Couldn't find invite before timeout")
                     ),
-                1000
+                2000
             );
         });
 
@@ -196,7 +216,7 @@ export class LocalNode {
             throw new Error("No invite found");
         }
 
-        const existingRole = group.groupMap.get(this.account.id);
+        const existingRole = group.underlyingMap.get(this.account.id);
 
         if (
             existingRole === "admin" ||
@@ -222,16 +242,16 @@ export class LocalNode {
                 : "reader"
         );
 
-        group.groupMap.coValue._sessions = groupAsInvite.groupMap.coValue.sessions;
-        group.groupMap.coValue._cachedContent = undefined;
+        group.underlyingMap.core._sessions = groupAsInvite.underlyingMap.core.sessions;
+        group.underlyingMap.core._cachedContent = undefined;
 
-        for (const groupListener of group.groupMap.coValue.listeners) {
-            groupListener(group.groupMap.coValue.getCurrentContent());
+        for (const groupListener of group.underlyingMap.core.listeners) {
+            groupListener(group.underlyingMap.core.getCurrentContent());
         }
     }
 
     /** @internal */
-    expectCoValueLoaded(id: RawCoID, expectation?: string): CoValue {
+    expectCoValueLoaded(id: RawCoID, expectation?: string): CoValueCore {
         const entry = this.coValues[id];
         if (!entry) {
             throw new Error(
@@ -284,7 +304,7 @@ export class LocalNode {
             account.node
         );
 
-        accountAsGroup.groupMap.edit((editable) => {
+        accountAsGroup.underlyingMap.edit((editable) => {
             editable.set(getAgentID(agentSecret), "admin", "trusting");
 
             const readKey = newRandomKeySecret();
@@ -320,13 +340,13 @@ export class LocalNode {
             editable.set("name", name, "trusting");
         });
 
-        accountAsGroup.groupMap.edit((editable) => {
+        accountAsGroup.underlyingMap.edit((editable) => {
             editable.set("profile", profile.id, "trusting");
         });
 
         const accountOnThisNode = this.expectCoValueLoaded(account.id);
 
-        accountOnThisNode._sessions = {...accountAsGroup.groupMap.coValue.sessions};
+        accountOnThisNode._sessions = {...accountAsGroup.underlyingMap.core.sessions};
         accountOnThisNode._cachedContent = undefined;
 
         return controlledAccount;
@@ -360,6 +380,7 @@ export class LocalNode {
         ).getCurrentAgentID();
     }
 
+    /** Creates a new group (with the current account as the group's first admin). */
     createGroup(): Group {
         const groupCoValue = this.createCoValue({
             type: "comap",
@@ -422,7 +443,7 @@ export class LocalNode {
                     continue;
                 }
 
-                const newCoValue = new CoValue(entry.coValue.header, newNode, {...entry.coValue.sessions});
+                const newCoValue = new CoValueCore(entry.coValue.header, newNode, {...entry.coValue.sessions});
 
                 newNode.coValues[coValueID as RawCoID] = {
                     state: "loaded",
@@ -441,16 +462,16 @@ export class LocalNode {
 type CoValueState =
     | {
           state: "loading";
-          done: Promise<CoValue>;
-          resolve: (coValue: CoValue) => void;
+          done: Promise<CoValueCore>;
+          resolve: (coValue: CoValueCore) => void;
       }
-    | { state: "loaded"; coValue: CoValue };
+    | { state: "loaded"; coValue: CoValueCore };
 
 /** @internal */
 export function newLoadingState(): CoValueState {
-    let resolve: (coValue: CoValue) => void;
+    let resolve: (coValue: CoValueCore) => void;
 
-    const promise = new Promise<CoValue>((r) => {
+    const promise = new Promise<CoValueCore>((r) => {
         resolve = r;
     });
 

packages/cojson/src/permissions.test.ts

@@ -1,5 +1,5 @@
-import { newRandomSessionID } from "./coValue.js";
-import { expectMap } from "./contentType.js";
+import { newRandomSessionID } from "./coValueCore.js";
+import { expectMap } from "./coValue.js";
 import { Group, expectGroupContent } from "./group.js";
 import {
     createdNowUnique,
@@ -17,7 +17,11 @@ import {
     groupWithTwoAdmins,
     groupWithTwoAdminsHighLevel,
 } from "./testUtils.js";
-import { AnonymousControlledAccount } from "./index.js";
+import { AnonymousControlledAccount, cojsonReady } from "./index.js";
+
+beforeEach(async () => {
+    await cojsonReady;
+});
 
 test("Initial admin can add another admin to a group", () => {
     groupWithTwoAdmins();
@@ -63,7 +67,7 @@ test("Added adming can add a third admin to a group (high level)", () => {
 
     groupAsOtherAdmin.addMember(thirdAdmin.id, "admin");
 
-    expect(groupAsOtherAdmin.groupMap.get(thirdAdmin.id)).toEqual("admin");
+    expect(groupAsOtherAdmin.underlyingMap.get(thirdAdmin.id)).toEqual("admin");
 });
 
 test("Admins can't demote other admins in a group", () => {
@@ -112,7 +116,7 @@ test("Admins can't demote other admins in a group (high level)", () => {
         "Failed to set role"
     );
 
-    expect(groupAsOtherAdmin.groupMap.get(admin.id)).toEqual("admin");
+    expect(groupAsOtherAdmin.underlyingMap.get(admin.id)).toEqual("admin");
 });
 
 test("Admins an add writers to a group, who can't add admins, writers, or readers", () => {
@@ -164,14 +168,14 @@ test("Admins an add writers to a group, who can't add admins, writers, or reader
     const writer = node.createAccount("writer");
 
     group.addMember(writer.id, "writer");
-    expect(group.groupMap.get(writer.id)).toEqual("writer");
+    expect(group.underlyingMap.get(writer.id)).toEqual("writer");
 
     const groupAsWriter = group.testWithDifferentAccount(
         writer,
         newRandomSessionID(writer.id)
     );
 
-    expect(groupAsWriter.groupMap.get(writer.id)).toEqual("writer");
+    expect(groupAsWriter.underlyingMap.get(writer.id)).toEqual("writer");
 
     const otherAgent = node.createAccount("otherAgent");
 
@@ -185,7 +189,7 @@ test("Admins an add writers to a group, who can't add admins, writers, or reader
         "Failed to set role"
     );
 
-    expect(groupAsWriter.groupMap.get(otherAgent.id)).toBeUndefined();
+    expect(groupAsWriter.underlyingMap.get(otherAgent.id)).toBeUndefined();
 });
 
 test("Admins can add readers to a group, who can't add admins, writers, or readers", () => {
@@ -237,14 +241,14 @@ test("Admins can add readers to a group, who can't add admins, writers, or reade
     const reader = node.createAccount("reader");
 
     group.addMember(reader.id, "reader");
-    expect(group.groupMap.get(reader.id)).toEqual("reader");
+    expect(group.underlyingMap.get(reader.id)).toEqual("reader");
 
     const groupAsReader = group.testWithDifferentAccount(
         reader,
         newRandomSessionID(reader.id)
     );
 
-    expect(groupAsReader.groupMap.get(reader.id)).toEqual("reader");
+    expect(groupAsReader.underlyingMap.get(reader.id)).toEqual("reader");
 
     const otherAgent = node.createAccount("otherAgent");
 
@@ -258,7 +262,7 @@ test("Admins can add readers to a group, who can't add admins, writers, or reade
         "Failed to set role"
     );
 
-    expect(groupAsReader.groupMap.get(otherAgent.id)).toBeUndefined();
+    expect(groupAsReader.underlyingMap.get(otherAgent.id)).toBeUndefined();
 });
 
 test("Admins can write to an object that is owned by their group", () => {
@@ -342,7 +346,7 @@ test("Writers can write to an object that is owned by their group (high level)",
     const childObject = group.createMap();
 
     let childObjectAsWriter = expectMap(
-        childObject.coValue
+        childObject.core
             .testWithDifferentAccount(writer, newRandomSessionID(writer.id))
             .getCurrentContent()
     );
@@ -401,7 +405,7 @@ test("Readers can not write to an object that is owned by their group (high leve
     const childObject = group.createMap();
 
     let childObjectAsReader = expectMap(
-        childObject.coValue
+        childObject.core
             .testWithDifferentAccount(reader, newRandomSessionID(reader.id))
             .getCurrentContent()
     );
@@ -553,7 +557,7 @@ test("Admins can set group read key and then writers can use it to create and re
     const childObject = group.createMap();
 
     let childObjectAsWriter = expectMap(
-        childObject.coValue
+        childObject.core
             .testWithDifferentAccount(writer, newRandomSessionID(writer.id))
             .getCurrentContent()
     );
@@ -647,7 +651,7 @@ test("Admins can set group read key and then use it to create private transactio
     });
 
     const childContentAsReader = expectMap(
-        childObject.coValue
+        childObject.core
             .testWithDifferentAccount(reader, newRandomSessionID(reader.id))
             .getCurrentContent()
     );
@@ -767,7 +771,7 @@ test("Admins can set group read key and then use it to create private transactio
     });
 
     const childContentAsReader1 = expectMap(
-        childObject.coValue
+        childObject.core
             .testWithDifferentAccount(reader1, newRandomSessionID(reader1.id))
             .getCurrentContent()
     );
@@ -777,7 +781,7 @@ test("Admins can set group read key and then use it to create private transactio
     group.addMember(reader2.id, "reader");
 
     const childContentAsReader2 = expectMap(
-        childObject.coValue
+        childObject.core
             .testWithDifferentAccount(reader2, newRandomSessionID(reader2.id))
             .getCurrentContent()
     );
@@ -863,7 +867,7 @@ test("Admins can set group read key, make a private transaction in an owned obje
 
     let childObject = group.createMap();
 
-    const firstReadKey = childObject.coValue.getCurrentReadKey();
+    const firstReadKey = childObject.core.getCurrentReadKey();
 
     childObject = childObject.edit((editable) => {
         editable.set("foo", "bar", "private");
@@ -874,7 +878,7 @@ test("Admins can set group read key, make a private transaction in an owned obje
 
     group.rotateReadKey();
 
-    expect(childObject.coValue.getCurrentReadKey()).not.toEqual(firstReadKey);
+    expect(childObject.core.getCurrentReadKey()).not.toEqual(firstReadKey);
 
     childObject = childObject.edit((editable) => {
         editable.set("foo2", "bar2", "private");
@@ -998,7 +1002,7 @@ test("Admins can set group read key, make a private transaction in an owned obje
 
     let childObject = group.createMap();
 
-    const firstReadKey = childObject.coValue.getCurrentReadKey();
+    const firstReadKey = childObject.core.getCurrentReadKey();
 
     childObject = childObject.edit((editable) => {
         editable.set("foo", "bar", "private");
@@ -1009,7 +1013,7 @@ test("Admins can set group read key, make a private transaction in an owned obje
 
     group.rotateReadKey();
 
-    expect(childObject.coValue.getCurrentReadKey()).not.toEqual(firstReadKey);
+    expect(childObject.core.getCurrentReadKey()).not.toEqual(firstReadKey);
 
     const reader = node.createAccount("reader");
 
@@ -1021,7 +1025,7 @@ test("Admins can set group read key, make a private transaction in an owned obje
     });
 
     const childContentAsReader = expectMap(
-        childObject.coValue
+        childObject.core
             .testWithDifferentAccount(reader, newRandomSessionID(reader.id))
             .getCurrentContent()
     );
@@ -1204,7 +1208,7 @@ test("Admins can set group read rey, make a private transaction in an owned obje
 
     group.rotateReadKey();
 
-    const secondReadKey = childObject.coValue.getCurrentReadKey();
+    const secondReadKey = childObject.core.getCurrentReadKey();
 
     const reader = node.createAccount("reader");
 
@@ -1223,7 +1227,7 @@ test("Admins can set group read rey, make a private transaction in an owned obje
 
     group.removeMember(reader.id);
 
-    expect(childObject.coValue.getCurrentReadKey()).not.toEqual(secondReadKey);
+    expect(childObject.core.getCurrentReadKey()).not.toEqual(secondReadKey);
 
     childObject = childObject.edit((editable) => {
         editable.set("foo3", "bar3", "private");
@@ -1231,7 +1235,7 @@ test("Admins can set group read rey, make a private transaction in an owned obje
     });
 
     const childContentAsReader2 = expectMap(
-        childObject.coValue
+        childObject.core
             .testWithDifferentAccount(reader2, newRandomSessionID(reader2.id))
             .getCurrentContent()
     );
@@ -1242,7 +1246,7 @@ test("Admins can set group read rey, make a private transaction in an owned obje
 
     expect(
         expectMap(
-            childObject.coValue
+            childObject.core
                 .testWithDifferentAccount(reader, newRandomSessionID(reader.id))
                 .getCurrentContent()
         ).get("foo3")
@@ -1373,14 +1377,14 @@ test("Admins can create an adminInvite, which can add an admin (high-level)", as
         nodeAsInvitedAdmin
     );
 
-    expect(groupAsInvitedAdmin.groupMap.get(invitedAdminID)).toEqual("admin");
+    expect(groupAsInvitedAdmin.underlyingMap.get(invitedAdminID)).toEqual("admin");
     expect(
-        groupAsInvitedAdmin.groupMap.coValue.getCurrentReadKey().secret
+        groupAsInvitedAdmin.underlyingMap.core.getCurrentReadKey().secret
     ).toBeDefined();
 
     groupAsInvitedAdmin.addMemberInternal(thirdAdminID, "admin");
 
-    expect(groupAsInvitedAdmin.groupMap.get(thirdAdminID)).toEqual("admin");
+    expect(groupAsInvitedAdmin.underlyingMap.get(thirdAdminID)).toEqual("admin");
 });
 
 test("Admins can create a writerInvite, which can add a writer", () => {
@@ -1484,9 +1488,9 @@ test("Admins can create a writerInvite, which can add a writer (high-level)", as
         nodeAsInvitedWriter
     );
 
-    expect(groupAsInvitedWriter.groupMap.get(invitedWriterID)).toEqual("writer");
+    expect(groupAsInvitedWriter.underlyingMap.get(invitedWriterID)).toEqual("writer");
     expect(
-        groupAsInvitedWriter.groupMap.coValue.getCurrentReadKey().secret
+        groupAsInvitedWriter.underlyingMap.core.getCurrentReadKey().secret
     ).toBeDefined();
 });
 
@@ -1592,9 +1596,9 @@ test("Admins can create a readerInvite, which can add a reader (high-level)", as
         nodeAsInvitedReader
     );
 
-    expect(groupAsInvitedReader.groupMap.get(invitedReaderID)).toEqual("reader");
+    expect(groupAsInvitedReader.underlyingMap.get(invitedReaderID)).toEqual("reader");
     expect(
-        groupAsInvitedReader.groupMap.coValue.getCurrentReadKey().secret
+        groupAsInvitedReader.underlyingMap.core.getCurrentReadKey().secret
     ).toBeDefined();
 });
 

packages/cojson/src/permissions.ts

@@ -1,20 +1,21 @@
-import { CoID } from "./contentType.js";
-import { MapOpPayload } from "./contentTypes/coMap.js";
+import { CoID } from "./coValue.js";
+import { MapOpPayload } from "./coValues/coMap.js";
 import { JsonValue } from "./jsonValue.js";
 import {
     KeyID,
 } from "./crypto.js";
 import {
-    CoValue,
+    CoValueCore,
     Transaction,
     TrustingTransaction,
     accountOrAgentIDfromSessionID,
-} from "./coValue.js";
+} from "./coValueCore.js";
 import { AgentID, RawCoID, SessionID, TransactionID } from "./ids.js";
 import {
     AccountID,
     Profile,
 } from "./account.js";
+import { parseJSON } from "./jsonStringify.js";
 
 export type PermissionsDef =
     | { type: "group"; initialAdmin: AccountID | AgentID }
@@ -31,7 +32,7 @@ export type Role =
     | "readerInvite";
 
 export function determineValidTransactions(
-    coValue: CoValue
+    coValue: CoValueCore
 ): { txID: TransactionID; tx: Transaction }[] {
     if (coValue.header.ruleset.type === "group") {
         const allTrustingTransactionsSorted = Object.entries(
@@ -76,11 +77,13 @@ export function determineValidTransactions(
             // console.log("before", { memberState, validTransactions });
             const transactor = accountOrAgentIDfromSessionID(sessionID);
 
-            const change = tx.changes[0] as
+            const changes = parseJSON(tx.changes)
+
+            const change = changes[0] as
                 | MapOpPayload<AccountID | AgentID, Role>
                 | MapOpPayload<"readKey", JsonValue>
                 | MapOpPayload<"profile", CoID<Profile>>;
-            if (tx.changes.length !== 1) {
+            if (changes.length !== 1) {
                 console.warn("Group transaction must have exactly one change");
                 continue;
             }

packages/cojson/src/sync.test.ts

@@ -1,23 +1,21 @@
-import { newRandomSessionID } from "./coValue.js";
+import { newRandomSessionID } from "./coValueCore.js";
 import { LocalNode } from "./node.js";
-import { Peer, PeerID, SyncMessage } from "./sync.js";
-import { expectMap } from "./contentType.js";
-import { MapOpPayload } from "./contentTypes/coMap.js";
+import { SyncMessage } from "./sync.js";
+import { expectMap } from "./coValue.js";
+import { MapOpPayload } from "./coValues/coMap.js";
 import { Group } from "./group.js";
-import {
-    ReadableStream,
-    WritableStream,
-    TransformStream,
-} from "isomorphic-streams";
 import {
     randomAnonymousAccountAndSessionID,
     shouldNotResolve,
 } from "./testUtils.js";
-import {
-    connectedPeers,
-    newStreamPair
-} from "./streamUtils.js";
+import { connectedPeers, newStreamPair } from "./streamUtils.js";
 import { AccountID } from "./account.js";
+import { cojsonReady } from "./index.js";
+import { stableStringify } from "./jsonStringify.js";
+
+beforeEach(async () => {
+    await cojsonReady;
+});
 
 test("Node replies with initial tx and header to empty subscribe", async () => {
     const [admin, session] = randomAnonymousAccountAndSessionID();
@@ -45,7 +43,7 @@ test("Node replies with initial tx and header to empty subscribe", async () => {
 
     await writer.write({
         action: "load",
-        id: map.coValue.id,
+        id: map.core.id,
         header: false,
         sessions: {},
     });
@@ -58,7 +56,7 @@ test("Node replies with initial tx and header to empty subscribe", async () => {
     const mapTellKnownStateMsg = await reader.read();
     expect(mapTellKnownStateMsg.value).toEqual({
         action: "known",
-        ...map.coValue.knownState(),
+        ...map.core.knownState(),
     } satisfies SyncMessage);
 
     // expect((await reader.read()).value).toMatchObject(admContEx(admin.id));
@@ -68,13 +66,13 @@ test("Node replies with initial tx and header to empty subscribe", async () => {
 
     expect(newContentMsg.value).toEqual({
         action: "content",
-        id: map.coValue.id,
+        id: map.core.id,
         header: {
             type: "comap",
             ruleset: { type: "ownedByGroup", group: group.id },
             meta: null,
-            createdAt: map.coValue.header.createdAt,
-            uniqueness: map.coValue.header.uniqueness,
+            createdAt: map.core.header.createdAt,
+            uniqueness: map.core.header.uniqueness,
         },
         new: {
             [node.currentSessionID]: {
@@ -82,19 +80,19 @@ test("Node replies with initial tx and header to empty subscribe", async () => {
                 newTransactions: [
                     {
                         privacy: "trusting" as const,
-                        madeAt: map.coValue.sessions[node.currentSessionID]!
+                        madeAt: map.core.sessions[node.currentSessionID]!
                             .transactions[0]!.madeAt,
-                        changes: [
+                        changes: stableStringify([
                             {
                                 op: "set",
                                 key: "hello",
                                 value: "world",
                             } satisfies MapOpPayload<string, string>,
-                        ],
+                        ]),
                     },
                 ],
                 lastSignature:
-                    map.coValue.sessions[node.currentSessionID]!.lastSignature!,
+                    map.core.sessions[node.currentSessionID]!.lastSignature!,
             },
         },
     } satisfies SyncMessage);
@@ -127,7 +125,7 @@ test("Node replies with only new tx to subscribe with some known state", async (
 
     await writer.write({
         action: "load",
-        id: map.coValue.id,
+        id: map.core.id,
         header: true,
         sessions: {
             [node.currentSessionID]: 1,
@@ -142,7 +140,7 @@ test("Node replies with only new tx to subscribe with some known state", async (
     const mapTellKnownStateMsg = await reader.read();
     expect(mapTellKnownStateMsg.value).toEqual({
         action: "known",
-        ...map.coValue.knownState(),
+        ...map.core.knownState(),
     } satisfies SyncMessage);
 
     // expect((await reader.read()).value).toMatchObject(admContEx(admin.id));
@@ -152,7 +150,7 @@ test("Node replies with only new tx to subscribe with some known state", async (
 
     expect(mapNewContentMsg.value).toEqual({
         action: "content",
-        id: map.coValue.id,
+        id: map.core.id,
         header: undefined,
         new: {
             [node.currentSessionID]: {
@@ -160,19 +158,19 @@ test("Node replies with only new tx to subscribe with some known state", async (
                 newTransactions: [
                     {
                         privacy: "trusting" as const,
-                        madeAt: map.coValue.sessions[node.currentSessionID]!
+                        madeAt: map.core.sessions[node.currentSessionID]!
                             .transactions[1]!.madeAt,
-                        changes: [
+                        changes: stableStringify([
                             {
                                 op: "set",
                                 key: "goodbye",
                                 value: "world",
                             } satisfies MapOpPayload<string, string>,
-                        ],
+                        ]),
                     },
                 ],
                 lastSignature:
-                    map.coValue.sessions[node.currentSessionID]!.lastSignature!,
+                    map.core.sessions[node.currentSessionID]!.lastSignature!,
             },
         },
     } satisfies SyncMessage);
@@ -204,7 +202,7 @@ test("After subscribing, node sends own known state and new txs to peer", async
 
     await writer.write({
         action: "load",
-        id: map.coValue.id,
+        id: map.core.id,
         header: false,
         sessions: {
             [node.currentSessionID]: 0,
@@ -219,7 +217,7 @@ test("After subscribing, node sends own known state and new txs to peer", async
     const mapTellKnownStateMsg = await reader.read();
     expect(mapTellKnownStateMsg.value).toEqual({
         action: "known",
-        ...map.coValue.knownState(),
+        ...map.core.knownState(),
     } satisfies SyncMessage);
 
     // expect((await reader.read()).value).toMatchObject(admContEx(admin.id));
@@ -229,8 +227,8 @@ test("After subscribing, node sends own known state and new txs to peer", async
 
     expect(mapNewContentHeaderOnlyMsg.value).toEqual({
         action: "content",
-        id: map.coValue.id,
-        header: map.coValue.header,
+        id: map.core.id,
+        header: map.core.header,
         new: {},
     } satisfies SyncMessage);
 
@@ -242,26 +240,26 @@ test("After subscribing, node sends own known state and new txs to peer", async
 
     expect(mapEditMsg1.value).toEqual({
         action: "content",
-        id: map.coValue.id,
+        id: map.core.id,
         new: {
             [node.currentSessionID]: {
                 after: 0,
                 newTransactions: [
                     {
                         privacy: "trusting" as const,
-                        madeAt: map.coValue.sessions[node.currentSessionID]!
+                        madeAt: map.core.sessions[node.currentSessionID]!
                             .transactions[0]!.madeAt,
-                        changes: [
+                        changes: stableStringify([
                             {
                                 op: "set",
                                 key: "hello",
                                 value: "world",
                             } satisfies MapOpPayload<string, string>,
-                        ],
+                        ]),
                     },
                 ],
                 lastSignature:
-                    map.coValue.sessions[node.currentSessionID]!.lastSignature!,
+                    map.core.sessions[node.currentSessionID]!.lastSignature!,
             },
         },
     } satisfies SyncMessage);
@@ -274,26 +272,26 @@ test("After subscribing, node sends own known state and new txs to peer", async
 
     expect(mapEditMsg2.value).toEqual({
         action: "content",
-        id: map.coValue.id,
+        id: map.core.id,
         new: {
             [node.currentSessionID]: {
                 after: 1,
                 newTransactions: [
                     {
                         privacy: "trusting" as const,
-                        madeAt: map.coValue.sessions[node.currentSessionID]!
+                        madeAt: map.core.sessions[node.currentSessionID]!
                             .transactions[1]!.madeAt,
-                        changes: [
+                        changes: stableStringify([
                             {
                                 op: "set",
                                 key: "goodbye",
                                 value: "world",
                             } satisfies MapOpPayload<string, string>,
-                        ],
+                        ]),
                     },
                 ],
                 lastSignature:
-                    map.coValue.sessions[node.currentSessionID]!.lastSignature!,
+                    map.core.sessions[node.currentSessionID]!.lastSignature!,
             },
         },
     } satisfies SyncMessage);
@@ -329,7 +327,7 @@ test("Client replies with known new content to tellKnownState from server", asyn
 
     await writer.write({
         action: "known",
-        id: map.coValue.id,
+        id: map.core.id,
         header: false,
         sessions: {
             [node.currentSessionID]: 0,
@@ -342,7 +340,7 @@ test("Client replies with known new content to tellKnownState from server", asyn
     const mapTellKnownStateMsg = await reader.read();
     expect(mapTellKnownStateMsg.value).toEqual({
         action: "known",
-        ...map.coValue.knownState(),
+        ...map.core.knownState(),
     } satisfies SyncMessage);
 
     // expect((await reader.read()).value).toMatchObject(admContEx(admin.id));
@@ -352,27 +350,27 @@ test("Client replies with known new content to tellKnownState from server", asyn
 
     expect(mapNewContentMsg.value).toEqual({
         action: "content",
-        id: map.coValue.id,
-        header: map.coValue.header,
+        id: map.core.id,
+        header: map.core.header,
         new: {
             [node.currentSessionID]: {
                 after: 0,
                 newTransactions: [
                     {
                         privacy: "trusting" as const,
-                        madeAt: map.coValue.sessions[node.currentSessionID]!
+                        madeAt: map.core.sessions[node.currentSessionID]!
                             .transactions[0]!.madeAt,
-                        changes: [
+                        changes: stableStringify([
                             {
                                 op: "set",
                                 key: "hello",
                                 value: "world",
                             } satisfies MapOpPayload<string, string>,
-                        ],
+                        ]),
                     },
                 ],
                 lastSignature:
-                    map.coValue.sessions[node.currentSessionID]!.lastSignature!,
+                    map.core.sessions[node.currentSessionID]!.lastSignature!,
             },
         },
     } satisfies SyncMessage);
@@ -400,7 +398,7 @@ test("No matter the optimistic known state, node respects invalid known state me
 
     await writer.write({
         action: "load",
-        id: map.coValue.id,
+        id: map.core.id,
         header: false,
         sessions: {
             [node.currentSessionID]: 0,
@@ -415,7 +413,7 @@ test("No matter the optimistic known state, node respects invalid known state me
     const mapTellKnownStateMsg = await reader.read();
     expect(mapTellKnownStateMsg.value).toEqual({
         action: "known",
-        ...map.coValue.knownState(),
+        ...map.core.knownState(),
     } satisfies SyncMessage);
 
     // expect((await reader.read()).value).toMatchObject(admContEx(admin.id));
@@ -425,8 +423,8 @@ test("No matter the optimistic known state, node respects invalid known state me
 
     expect(mapNewContentHeaderOnlyMsg.value).toEqual({
         action: "content",
-        id: map.coValue.id,
-        header: map.coValue.header,
+        id: map.core.id,
+        header: map.core.header,
         new: {},
     } satisfies SyncMessage);
 
@@ -444,7 +442,7 @@ test("No matter the optimistic known state, node respects invalid known state me
     await writer.write({
         action: "known",
         isCorrection: true,
-        id: map.coValue.id,
+        id: map.core.id,
         header: true,
         sessions: {
             [node.currentSessionID]: 1,
@@ -455,7 +453,7 @@ test("No matter the optimistic known state, node respects invalid known state me
 
     expect(newContentAfterWrongAssumedState.value).toEqual({
         action: "content",
-        id: map.coValue.id,
+        id: map.core.id,
         header: undefined,
         new: {
             [node.currentSessionID]: {
@@ -463,19 +461,19 @@ test("No matter the optimistic known state, node respects invalid known state me
                 newTransactions: [
                     {
                         privacy: "trusting" as const,
-                        madeAt: map.coValue.sessions[node.currentSessionID]!
+                        madeAt: map.core.sessions[node.currentSessionID]!
                             .transactions[1]!.madeAt,
-                        changes: [
+                        changes: stableStringify([
                             {
                                 op: "set",
                                 key: "goodbye",
                                 value: "world",
                             } satisfies MapOpPayload<string, string>,
-                        ],
+                        ]),
                     },
                 ],
                 lastSignature:
-                    map.coValue.sessions[node.currentSessionID]!.lastSignature!,
+                    map.core.sessions[node.currentSessionID]!.lastSignature!,
             },
         },
     } satisfies SyncMessage);
@@ -535,14 +533,14 @@ test("If we add a server peer, all updates to all coValues are sent to it, even
     // });
     expect((await reader.read()).value).toMatchObject({
         action: "load",
-        id: group.groupMap.coValue.id,
+        id: group.underlyingMap.core.id,
     });
 
     const mapSubscribeMsg = await reader.read();
 
     expect(mapSubscribeMsg.value).toEqual({
         action: "load",
-        id: map.coValue.id,
+        id: map.core.id,
         header: true,
         sessions: {},
     } satisfies SyncMessage);
@@ -558,27 +556,27 @@ test("If we add a server peer, all updates to all coValues are sent to it, even
 
     expect(mapNewContentMsg.value).toEqual({
         action: "content",
-        id: map.coValue.id,
-        header: map.coValue.header,
+        id: map.core.id,
+        header: map.core.header,
         new: {
             [node.currentSessionID]: {
                 after: 0,
                 newTransactions: [
                     {
                         privacy: "trusting" as const,
-                        madeAt: map.coValue.sessions[node.currentSessionID]!
+                        madeAt: map.core.sessions[node.currentSessionID]!
                             .transactions[0]!.madeAt,
-                        changes: [
+                        changes: stableStringify([
                             {
                                 op: "set",
                                 key: "hello",
                                 value: "world",
                             } satisfies MapOpPayload<string, string>,
-                        ],
+                        ]),
                     },
                 ],
                 lastSignature:
-                    map.coValue.sessions[node.currentSessionID]!.lastSignature!,
+                    map.core.sessions[node.currentSessionID]!.lastSignature!,
             },
         },
     } satisfies SyncMessage);
@@ -607,7 +605,7 @@ test("If we add a server peer, newly created coValues are auto-subscribed to", a
     // });
     expect((await reader.read()).value).toMatchObject({
         action: "load",
-        id: group.groupMap.coValue.id,
+        id: group.underlyingMap.core.id,
     });
 
     const map = group.createMap();
@@ -616,7 +614,7 @@ test("If we add a server peer, newly created coValues are auto-subscribed to", a
 
     expect(mapSubscribeMsg.value).toEqual({
         action: "load",
-        ...map.coValue.knownState(),
+        ...map.core.knownState(),
     } satisfies SyncMessage);
 
     // expect((await reader.read()).value).toMatchObject(admContEx(adminID));
@@ -626,8 +624,8 @@ test("If we add a server peer, newly created coValues are auto-subscribed to", a
 
     expect(mapContentMsg.value).toEqual({
         action: "content",
-        id: map.coValue.id,
-        header: map.coValue.header,
+        id: map.core.id,
+        header: map.core.header,
         new: {},
     } satisfies SyncMessage);
 });
@@ -661,14 +659,14 @@ test("When we connect a new server peer, we try to sync all existing coValues to
 
     expect(groupSubscribeMessage.value).toEqual({
         action: "load",
-        ...group.groupMap.coValue.knownState(),
+        ...group.underlyingMap.core.knownState(),
     } satisfies SyncMessage);
 
     const secondMessage = await reader.read();
 
     expect(secondMessage.value).toEqual({
         action: "load",
-        ...map.coValue.knownState(),
+        ...map.core.knownState(),
     } satisfies SyncMessage);
 });
 
@@ -694,7 +692,7 @@ test("When receiving a subscribe with a known state that is ahead of our own, pe
 
     await writer.write({
         action: "load",
-        id: map.coValue.id,
+        id: map.core.id,
         header: true,
         sessions: {
             [node.currentSessionID]: 1,
@@ -709,7 +707,7 @@ test("When receiving a subscribe with a known state that is ahead of our own, pe
 
     expect(mapTellKnownState.value).toEqual({
         action: "known",
-        ...map.coValue.knownState(),
+        ...map.core.knownState(),
     } satisfies SyncMessage);
 });
 
@@ -757,7 +755,7 @@ test.skip("When replaying creation and transactions of a coValue as new content,
     const groupSubscribeMsg = await from1.read();
     expect(groupSubscribeMsg.value).toMatchObject({
         action: "load",
-        id: group.groupMap.coValue.id,
+        id: group.underlyingMap.core.id,
     });
 
     await to2.write(adminSubscribeMessage.value!);
@@ -771,7 +769,7 @@ test.skip("When replaying creation and transactions of a coValue as new content,
 
     expect(
         node2.sync.peers["test1"]!.optimisticKnownStates[
-            group.groupMap.coValue.id
+            group.underlyingMap.core.id
         ]
     ).toBeDefined();
 
@@ -792,14 +790,14 @@ test.skip("When replaying creation and transactions of a coValue as new content,
     const mapSubscriptionMsg = await from1.read();
     expect(mapSubscriptionMsg.value).toMatchObject({
         action: "load",
-        id: map.coValue.id,
+        id: map.core.id,
     });
 
     const mapNewContentMsg = await from1.read();
     expect(mapNewContentMsg.value).toEqual({
         action: "content",
-        id: map.coValue.id,
-        header: map.coValue.header,
+        id: map.core.id,
+        header: map.core.header,
         new: {},
     } satisfies SyncMessage);
 
@@ -808,12 +806,12 @@ test.skip("When replaying creation and transactions of a coValue as new content,
     const mapTellKnownStateMsg = await from2.read();
     expect(mapTellKnownStateMsg.value).toEqual({
         action: "known",
-        id: map.coValue.id,
+        id: map.core.id,
         header: false,
         sessions: {},
     } satisfies SyncMessage);
 
-    expect(node2.coValues[map.coValue.id]?.state).toEqual("loading");
+    expect(node2.coValues[map.core.id]?.state).toEqual("loading");
 
     await to2.write(mapNewContentMsg.value!);
 
@@ -829,7 +827,7 @@ test.skip("When replaying creation and transactions of a coValue as new content,
 
     expect(
         expectMap(
-            node2.expectCoValueLoaded(map.coValue.id).getCurrentContent()
+            node2.expectCoValueLoaded(map.core.id).getCurrentContent()
         ).get("hello")
     ).toEqual("world");
 });
@@ -854,11 +852,11 @@ test.skip("When loading a coValue on one node, the server node it is requested f
     node1.sync.addPeer(node2asPeer);
     node2.sync.addPeer(node1asPeer);
 
-    await node2.loadCoValue(map.coValue.id);
+    await node2.loadCoValue(map.core.id);
 
     expect(
         expectMap(
-            node2.expectCoValueLoaded(map.coValue.id).getCurrentContent()
+            node2.expectCoValueLoaded(map.core.id).getCurrentContent()
         ).get("hello")
     ).toEqual("world");
 });
@@ -898,7 +896,7 @@ test("Can sync a coValue through a server to another client", async () => {
     client2.sync.addPeer(serverAsOtherPeer);
     server.sync.addPeer(client2AsPeer);
 
-    const mapOnClient2 = await client2.loadCoValue(map.coValue.id);
+    const mapOnClient2 = await client2.loadCoValue(map.core.id);
 
     expect(expectMap(mapOnClient2.getCurrentContent()).get("hello")).toEqual(
         "world"
@@ -941,7 +939,7 @@ test("Can sync a coValue with private transactions through a server to another c
     client2.sync.addPeer(serverAsOtherPeer);
     server.sync.addPeer(client2AsPeer);
 
-    const mapOnClient2 = await client2.loadCoValue(map.coValue.id);
+    const mapOnClient2 = await client2.loadCoValue(map.core.id);
 
     expect(expectMap(mapOnClient2.getCurrentContent()).get("hello")).toEqual(
         "world"
@@ -971,7 +969,7 @@ test("When a peer's incoming/readable stream closes, we remove the peer", async
     // });
     expect((await reader.read()).value).toMatchObject({
         action: "load",
-        id: group.groupMap.coValue.id,
+        id: group.underlyingMap.core.id,
     });
 
     const map = group.createMap();
@@ -980,7 +978,7 @@ test("When a peer's incoming/readable stream closes, we remove the peer", async
 
     expect(mapSubscribeMsg.value).toEqual({
         action: "load",
-        ...map.coValue.knownState(),
+        ...map.core.knownState(),
     } satisfies SyncMessage);
 
     // expect((await reader.read()).value).toMatchObject(admContEx(admin.id));
@@ -990,8 +988,8 @@ test("When a peer's incoming/readable stream closes, we remove the peer", async
 
     expect(mapContentMsg.value).toEqual({
         action: "content",
-        id: map.coValue.id,
-        header: map.coValue.header,
+        id: map.core.id,
+        header: map.core.header,
         new: {},
     } satisfies SyncMessage);
 
@@ -1025,7 +1023,7 @@ test("When a peer's outgoing/writable stream closes, we remove the peer", async
     // });
     expect((await reader.read()).value).toMatchObject({
         action: "load",
-        id: group.groupMap.coValue.id,
+        id: group.underlyingMap.core.id,
     });
 
     const map = group.createMap();
@@ -1034,7 +1032,7 @@ test("When a peer's outgoing/writable stream closes, we remove the peer", async
 
     expect(mapSubscribeMsg.value).toEqual({
         action: "load",
-        ...map.coValue.knownState(),
+        ...map.core.knownState(),
     } satisfies SyncMessage);
 
     // expect((await reader.read()).value).toMatchObject(admContEx(admin.id));
@@ -1044,8 +1042,8 @@ test("When a peer's outgoing/writable stream closes, we remove the peer", async
 
     expect(mapContentMsg.value).toEqual({
         action: "content",
-        id: map.coValue.id,
-        header: map.coValue.header,
+        id: map.core.id,
+        header: map.core.header,
         new: {},
     } satisfies SyncMessage);
 
@@ -1083,9 +1081,9 @@ test("If we start loading a coValue before connecting to a peer that has it, it
 
     node1.sync.addPeer(node2asPeer);
 
-    const mapOnNode2Promise = node2.loadCoValue(map.coValue.id);
+    const mapOnNode2Promise = node2.loadCoValue(map.core.id);
 
-    expect(node2.coValues[map.coValue.id]?.state).toEqual("loading");
+    expect(node2.coValues[map.core.id]?.state).toEqual("loading");
 
     node2.sync.addPeer(node1asPeer);
 
@@ -1099,7 +1097,7 @@ test("If we start loading a coValue before connecting to a peer that has it, it
 function groupContentEx(group: Group) {
     return {
         action: "content",
-        id: group.groupMap.coValue.id,
+        id: group.underlyingMap.core.id,
     };
 }
 
@@ -1113,7 +1111,7 @@ function admContEx(adminID: AccountID) {
 function groupStateEx(group: Group) {
     return {
         action: "known",
-        id: group.groupMap.coValue.id,
+        id: group.underlyingMap.core.id,
     };
 }
 

packages/cojson/src/sync.ts

@@ -1,6 +1,6 @@
 import { Signature } from "./crypto.js";
-import { CoValueHeader, Transaction } from "./coValue.js";
-import { CoValue } from "./coValue.js";
+import { CoValueHeader, Transaction } from "./coValueCore.js";
+import { CoValueCore } from "./coValueCore.js";
 import { LocalNode } from "./node.js";
 import { newLoadingState } from "./node.js";
 import {
@@ -9,6 +9,7 @@ import {
     WritableStreamDefaultWriter,
 } from "isomorphic-streams";
 import { RawCoID, SessionID } from "./ids.js";
+import { stableStringify } from "./jsonStringify.js";
 
 export type CoValueKnownState = {
     id: RawCoID;
@@ -268,7 +269,6 @@ export class SyncManager {
                         );
                     }
                 }
-                console.log("DONE!!!");
             } catch (e) {
                 console.error(`Error reading from peer ${peer.id}`, e);
             }
@@ -393,7 +393,7 @@ export class SyncManager {
             );
         }
 
-        let resolveAfterDone: ((coValue: CoValue) => void) | undefined;
+        let resolveAfterDone: ((coValue: CoValueCore) => void) | undefined;
 
         const peerOptimisticKnownState = peer.optimisticKnownStates[msg.id];
 
@@ -410,7 +410,7 @@ export class SyncManager {
 
             peerOptimisticKnownState.header = true;
 
-            const coValue = new CoValue(msg.header, this.local);
+            const coValue = new CoValueCore(msg.header, this.local);
 
             resolveAfterDone = entry.resolve;
 
@@ -445,12 +445,26 @@ export class SyncManager {
             const newTransactions =
                 newContentForSession.newTransactions.slice(alreadyKnownOffset);
 
-            const success = coValue.tryAddTransactions(
+            const before = performance.now();
+            const success = await coValue.tryAddTransactionsAsync(
                 sessionID,
                 newTransactions,
                 undefined,
                 newContentForSession.lastSignature
             );
+            const after = performance.now();
+            if (after - before > 10) {
+                const totalTxLength = newTransactions.map(t => stableStringify(t)!.length).reduce((a, b) => a + b, 0);
+                console.log(
+                    "Adding incoming transactions took",
+                    after - before,
+                    "ms",
+                    totalTxLength,
+                    "bytes = ",
+                    "bandwidth: MB/s",
+                    (1000 * totalTxLength / (after - before)) / (1024 * 1024)
+                );
+            }
 
             if (!success) {
                 console.error("Failed to add transactions", newTransactions);
@@ -496,7 +510,7 @@ export class SyncManager {
         throw new Error("Method not implemented.");
     }
 
-    async syncCoValue(coValue: CoValue) {
+    async syncCoValue(coValue: CoValueCore) {
         for (const peer of Object.values(this.peers)) {
             const optimisticKnownState = peer.optimisticKnownStates[coValue.id];
 

packages/cojson/src/testUtils.ts

@@ -1,5 +1,5 @@
 import { AgentSecret, createdNowUnique, getAgentID, newRandomAgentSecret  } from "./crypto.js";
-import { newRandomSessionID } from "./coValue.js";
+import { newRandomSessionID } from "./coValueCore.js";
 import { LocalNode } from "./node.js";
 import { expectGroupContent } from "./group.js";
 import { AnonymousControlledAccount } from "./account.js";

packages/cojson/tsconfig.json

@@ -9,8 +9,7 @@
     "skipLibCheck": true,
     "forceConsistentCasingInFileNames": true,
     "noUncheckedIndexedAccess": true,
-    "esModuleInterop": true,
-    "stripInternal": true
+    "esModuleInterop": true
   },
   "include": ["./src/**/*"],
   "exclude": ["./src/**/*.test.*"],

packages/jazz-browser-auth-local/package.json

@@ -1,16 +1,17 @@
 {
     "name": "jazz-browser-auth-local",
-    "version": "0.1.8",
+    "version": "0.2.0",
     "main": "dist/index.js",
     "types": "dist/index.d.ts",
     "license": "MIT",
     "dependencies": {
-        "jazz-browser": "^0.1.8",
+        "jazz-browser": "^0.2.0",
         "typescript": "^5.1.6"
     },
     "scripts": {
         "lint": "eslint src/**/*.ts",
         "build": "npm run lint && rm -rf ./dist && tsc --declaration --sourceMap --outDir dist",
         "prepublishOnly": "npm run build"
-    }
+    },
+    "gitHead": "33c27053293b4801b968c61d5c4c989f93a67d13"
 }

packages/jazz-browser-auth-local/src/index.ts

@@ -128,7 +128,7 @@ async function signUp(
             },
             user: {
                 id: webAuthNCredentialPayload,
-                name: username + `(${new Date().toLocaleString()})`,
+                name: username + ` (${new Date().toLocaleString()})`,
                 displayName: username,
             },
             pubKeyCredParams: [{ alg: -7, type: "public-key" }],

packages/jazz-browser-media-images/.eslintrc.cjs

@@ -0,0 +1,17 @@
+module.exports = {
+    extends: [
+        'eslint:recommended',
+        'plugin:@typescript-eslint/recommended',
+    ],
+    parser: '@typescript-eslint/parser',
+    plugins: ['@typescript-eslint'],
+    parserOptions: {
+        project: "./tsconfig.json",
+    },
+    root: true,
+    rules: {
+        "no-unused-vars": "off",
+        "@typescript-eslint/no-unused-vars": ["error", { "argsIgnorePattern": "^_", "varsIgnorePattern": "^_" }],
+        "@typescript-eslint/no-floating-promises": "error",
+    },
+};

packages/jazz-browser-media-images/.gitignore

@@ -0,0 +1,171 @@
+# Based on https://raw.githubusercontent.com/github/gitignore/main/Node.gitignore
+
+# Logs
+
+logs
+_.log
+npm-debug.log_
+yarn-debug.log*
+yarn-error.log*
+lerna-debug.log*
+.pnpm-debug.log*
+
+# Diagnostic reports (https://nodejs.org/api/report.html)
+
+report.[0-9]_.[0-9]_.[0-9]_.[0-9]_.json
+
+# Runtime data
+
+pids
+_.pid
+_.seed
+\*.pid.lock
+
+# Directory for instrumented libs generated by jscoverage/JSCover
+
+lib-cov
+
+# Coverage directory used by tools like istanbul
+
+coverage
+\*.lcov
+
+# nyc test coverage
+
+.nyc_output
+
+# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)
+
+.grunt
+
+# Bower dependency directory (https://bower.io/)
+
+bower_components
+
+# node-waf configuration
+
+.lock-wscript
+
+# Compiled binary addons (https://nodejs.org/api/addons.html)
+
+build/Release
+
+# Dependency directories
+
+node_modules/
+jspm_packages/
+
+# Snowpack dependency directory (https://snowpack.dev/)
+
+web_modules/
+
+# TypeScript cache
+
+\*.tsbuildinfo
+
+# Optional npm cache directory
+
+.npm
+
+# Optional eslint cache
+
+.eslintcache
+
+# Optional stylelint cache
+
+.stylelintcache
+
+# Microbundle cache
+
+.rpt2_cache/
+.rts2_cache_cjs/
+.rts2_cache_es/
+.rts2_cache_umd/
+
+# Optional REPL history
+
+.node_repl_history
+
+# Output of 'npm pack'
+
+\*.tgz
+
+# Yarn Integrity file
+
+.yarn-integrity
+
+# dotenv environment variable files
+
+.env
+.env.development.local
+.env.test.local
+.env.production.local
+.env.local
+
+# parcel-bundler cache (https://parceljs.org/)
+
+.cache
+.parcel-cache
+
+# Next.js build output
+
+.next
+out
+
+# Nuxt.js build / generate output
+
+.nuxt
+dist
+
+# Gatsby files
+
+.cache/
+
+# Comment in the public line in if your project uses Gatsby and not Next.js
+
+# https://nextjs.org/blog/next-9-1#public-directory-support
+
+# public
+
+# vuepress build output
+
+.vuepress/dist
+
+# vuepress v2.x temp and cache directory
+
+.temp
+.cache
+
+# Docusaurus cache and generated files
+
+.docusaurus
+
+# Serverless directories
+
+.serverless/
+
+# FuseBox cache
+
+.fusebox/
+
+# DynamoDB Local files
+
+.dynamodb/
+
+# TernJS port file
+
+.tern-port
+
+# Stores VSCode versions used for testing VSCode extensions
+
+.vscode-test
+
+# yarn v2
+
+.yarn/cache
+.yarn/unplugged
+.yarn/build-state.yml
+.yarn/install-state.gz
+.pnp.\*
+
+.DS_Store

packages/jazz-browser-media-images/.npmignore

@@ -0,0 +1,2 @@
+coverage
+node_modules

packages/jazz-browser-media-images/package.json

@@ -0,0 +1,21 @@
+{
+  "name": "jazz-browser-media-images",
+  "version": "0.2.0",
+  "main": "dist/index.js",
+  "types": "dist/index.d.ts",
+  "license": "MIT",
+  "dependencies": {
+    "cojson": "^0.2.0",
+    "image-blob-reduce": "^4.1.0",
+    "jazz-browser": "^0.2.0",
+    "typescript": "^5.1.6"
+  },
+  "scripts": {
+    "lint": "eslint src/**/*.ts",
+    "build": "npm run lint && rm -rf ./dist && tsc --declaration --sourceMap --outDir dist",
+    "prepublishOnly": "npm run build"
+  },
+  "devDependencies": {
+    "@types/image-blob-reduce": "^4.1.1"
+  }
+}

packages/jazz-browser-media-images/src/index.ts

@@ -0,0 +1,253 @@
+import { CoID, Group, LocalNode, Media } from "cojson";
+
+import ImageBlobReduce from "image-blob-reduce";
+import Pica from "pica";
+import {
+    createBinaryStreamFromBlob,
+    readBlobFromBinaryStream,
+} from "jazz-browser";
+
+const pica = new Pica();
+
+export async function createImage(
+    image: Blob | File,
+    inGroup: Group
+): Promise<Media.ImageDefinition> {
+    let originalWidth!: number;
+    let originalHeight!: number;
+    const Reducer = new ImageBlobReduce({ pica });
+    Reducer.after("_blob_to_image", (env) => {
+        originalWidth =
+            (env as unknown as { orientation: number }).orientation & 4
+                ? env.image.height
+                : env.image.width;
+        originalHeight =
+            (env as unknown as { orientation: number }).orientation & 4
+                ? env.image.width
+                : env.image.height;
+        return Promise.resolve(env);
+    });
+
+    const placeholderDataURL = (
+        await Reducer.toCanvas(image, { max: 8 })
+    ).toDataURL("image/png");
+
+    let imageDefinition = inGroup.createMap<Media.ImageDefinition>();
+
+    imageDefinition = imageDefinition.edit((imageDefinition) => {
+        imageDefinition.set("originalSize", [originalWidth, originalHeight]);
+        imageDefinition.set("placeholderDataURL", placeholderDataURL);
+    });
+
+    setTimeout(async () => {
+        const max256 = await Reducer.toBlob(image, { max: 256 });
+
+        if (originalWidth > 256 || originalHeight > 256) {
+            const width =
+                originalWidth > originalHeight
+                    ? 256
+                    : Math.round(256 * (originalWidth / originalHeight));
+            const height =
+                originalHeight > originalWidth
+                    ? 256
+                    : Math.round(256 * (originalHeight / originalWidth));
+
+            const binaryStreamId = (
+                await createBinaryStreamFromBlob(max256, inGroup)
+            ).id;
+
+            imageDefinition.edit((imageDefinition) => {
+                imageDefinition.set(`${width}x${height}`, binaryStreamId);
+            });
+        }
+
+        await new Promise((resolve) => setTimeout(resolve, 0));
+
+        const max1024 = await Reducer.toBlob(image, { max: 1024 });
+
+        if (originalWidth > 1024 || originalHeight > 1024) {
+            const width =
+                originalWidth > originalHeight
+                    ? 1024
+                    : Math.round(1024 * (originalWidth / originalHeight));
+            const height =
+                originalHeight > originalWidth
+                    ? 1024
+                    : Math.round(1024 * (originalHeight / originalWidth));
+
+            const binaryStreamId = (
+                await createBinaryStreamFromBlob(max1024, inGroup)
+            ).id;
+
+            imageDefinition.edit((imageDefinition) => {
+                imageDefinition.set(`${width}x${height}`, binaryStreamId);
+            });
+        }
+
+        await new Promise((resolve) => setTimeout(resolve, 0));
+
+        const max2048 = await Reducer.toBlob(image, { max: 2048 });
+
+        if (originalWidth > 2048 || originalHeight > 2048) {
+            const width =
+                originalWidth > originalHeight
+                    ? 2048
+                    : Math.round(2048 * (originalWidth / originalHeight));
+            const height =
+                originalHeight > originalWidth
+                    ? 2048
+                    : Math.round(2048 * (originalHeight / originalWidth));
+
+            const binaryStreamId = (
+                await createBinaryStreamFromBlob(max2048, inGroup)
+            ).id;
+
+            imageDefinition.edit((imageDefinition) => {
+                imageDefinition.set(`${width}x${height}`, binaryStreamId);
+            });
+        }
+
+        await new Promise((resolve) => setTimeout(resolve, 0));
+
+        const originalBinaryStreamId = (
+            await createBinaryStreamFromBlob(image, inGroup)
+        ).id;
+
+        imageDefinition.edit((imageDefinition) => {
+            imageDefinition.set(
+                `${originalWidth}x${originalHeight}`,
+                originalBinaryStreamId
+            );
+        });
+    }, 0);
+
+    return imageDefinition;
+}
+
+export type LoadingImageInfo = {
+    originalSize?: [number, number];
+    placeholderDataURL?: string;
+    highestResSrc?: string;
+};
+
+export function loadImage(
+    imageID: CoID<Media.ImageDefinition>,
+    localNode: LocalNode,
+    progressiveCallback: (update: LoadingImageInfo) => void
+): () => void {
+    let unsubscribe: (() => void) | undefined;
+    let stopped = false;
+
+    const resState: {
+        [res: `${number}x${number}`]:
+            | { state: "queued" }
+            | { state: "loading" }
+            | { state: "loaded"; blobURL: string }
+            | { state: "revoked" }
+            | { state: "failed" }
+            | undefined;
+    } = {};
+
+    const cleanUp = () => {
+        stopped = true;
+        for (const [res, entry] of Object.entries(resState)) {
+            if (entry?.state === "loaded") {
+                URL.revokeObjectURL(entry.blobURL);
+                resState[res as `${number}x${number}`] = { state: "revoked" };
+            }
+        }
+        unsubscribe?.();
+    };
+
+    localNode
+        .load(imageID)
+        .then((imageDefinition) => {
+            if (stopped) return;
+            unsubscribe = imageDefinition.subscribe(async (imageDefinition) => {
+                if (stopped) return;
+
+                const originalSize = imageDefinition.get("originalSize");
+                const placeholderDataURL =
+                    imageDefinition.get("placeholderDataURL");
+
+                const resolutions = imageDefinition.keys()
+                    .filter(
+                        (key): key is `${number}x${number}` =>
+                            !!key.match(/\d+x\d+/)
+                    )
+                    .sort((a, b) => {
+                        const widthA = Number(a.split("x")[0]);
+                        const widthB = Number(b.split("x")[0]);
+                        return widthA - widthB;
+                    });
+
+                const startLoading = async () => {
+
+                    const notYetQueuedOrLoading = resolutions.filter(
+                        (res) => !resState[res]
+                        );
+
+                    console.log("Loading iteration", resolutions, resState, notYetQueuedOrLoading);
+
+                    for (const res of notYetQueuedOrLoading) {
+                            resState[res] = { state: "queued" };
+                    }
+
+                    for (const res of notYetQueuedOrLoading) {
+                        if (stopped) return;
+                        resState[res] = { state: "loading" };
+
+                        const binaryStreamId = imageDefinition.get(res)!;
+                        console.log("Loading image res", imageID, res, binaryStreamId);
+
+                        const blob = await readBlobFromBinaryStream(
+                            binaryStreamId,
+                            localNode
+                        );
+
+                        if (stopped) return;
+                        if (!blob) {
+                            resState[res] = { state: "failed" };
+                            console.log("Loading image res failed", imageID, res, binaryStreamId);
+                            continue;
+                        }
+
+                        const blobURL = URL.createObjectURL(blob);
+                        resState[res] = { state: "loaded", blobURL };
+
+                        console.log("Loaded image res", imageID, res, binaryStreamId);
+
+                        progressiveCallback({
+                            originalSize,
+                            placeholderDataURL,
+                            highestResSrc: blobURL,
+                        });
+
+                        await new Promise((resolve) => setTimeout(resolve, 0));
+                    }
+                };
+
+                if (
+                    !Object.values(resState).some(
+                        (entry) => entry?.state === "loaded"
+                    )
+                ) {
+                    progressiveCallback({
+                        originalSize,
+                        placeholderDataURL,
+                    });
+                }
+
+                startLoading().catch((err) => {
+                    console.error("Error loading image", imageID, err);
+                    cleanUp();
+                });
+            });
+        })
+        .catch((err) => {
+            console.error("Error loading image", imageID, err);
+            cleanUp();
+        });
+
+    return cleanUp;
+}

packages/jazz-browser-media-images/tsconfig.json

@@ -0,0 +1,16 @@
+{
+  "compilerOptions": {
+    "lib": ["ESNext", "DOM"],
+    "module": "esnext",
+    "target": "ES2020",
+    "moduleResolution": "bundler",
+    "moduleDetection": "force",
+    "strict": true,
+    "skipLibCheck": true,
+    "jsx": "react",
+    "forceConsistentCasingInFileNames": true,
+    "noUncheckedIndexedAccess": true,
+    "esModuleInterop": true,
+  },
+  "include": ["./src/**/*"],
+}

packages/jazz-browser-media-images/yarn.lock

@@ -0,0 +1,75 @@
+# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
+# yarn lockfile v1
+
+
+"@noble/ciphers@^0.1.3":
+  version "0.1.4"
+  resolved "https://registry.yarnpkg.com/@noble/ciphers/-/ciphers-0.1.4.tgz#96327dca147829ed9eee0d96cfdf7c57915765f0"
+  integrity sha512-d3ZR8vGSpy3v/nllS+bD/OMN5UZqusWiQqkyj7AwzTnhXFH72pF5oB4Ach6DQ50g5kXxC28LdaYBEpsyv9KOUQ==
+
+"@noble/curves@^1.1.0":
+  version "1.1.0"
+  resolved "https://registry.yarnpkg.com/@noble/curves/-/curves-1.1.0.tgz#f13fc667c89184bc04cccb9b11e8e7bae27d8c3d"
+  integrity sha512-091oBExgENk/kGj3AZmtBDMpxQPDtxQABR2B9lb1JbVTs6ytdzZNwvhxQ4MWasRNEzlbEH8jCWFCwhF/Obj5AA==
+  dependencies:
+    "@noble/hashes" "1.3.1"
+
+"@noble/hashes@1.3.1", "@noble/hashes@^1.3.1":
+  version "1.3.1"
+  resolved "https://registry.yarnpkg.com/@noble/hashes/-/hashes-1.3.1.tgz#8831ef002114670c603c458ab8b11328406953a9"
+  integrity sha512-EbqwksQwz9xDRGfDST86whPBgM65E0OH/pCgqW0GBVzO22bNE+NuIbeTb714+IfSjU3aRk47EUvXIb5bTsenKA==
+
+"@scure/base@^1.1.1":
+  version "1.1.1"
+  resolved "https://registry.yarnpkg.com/@scure/base/-/base-1.1.1.tgz#ebb651ee52ff84f420097055f4bf46cfba403938"
+  integrity sha512-ZxOhsSyxYwLJj3pLZCefNitxsj093tb2vq90mp2txoYeBqbcjDjqFhyM8eUjq/uFm6zJ+mUuqxlS2FkuSY1MTA==
+
+"@types/prop-types@*":
+  version "15.7.5"
+  resolved "https://registry.yarnpkg.com/@types/prop-types/-/prop-types-15.7.5.tgz#5f19d2b85a98e9558036f6a3cacc8819420f05cf"
+  integrity sha512-JCB8C6SnDoQf0cNycqd/35A7MjcnK+ZTqE7judS6o7utxUCg6imJg3QK2qzHKszlTjcj2cn+NwMB2i96ubpj7w==
+
+"@types/react@^18.2.19":
+  version "18.2.19"
+  resolved "https://registry.yarnpkg.com/@types/react/-/react-18.2.19.tgz#f77cb2c8307368e624d464a25b9675fa35f95a8b"
+  integrity sha512-e2S8wmY1ePfM517PqCG80CcE48Xs5k0pwJzuDZsfE8IZRRBfOMCF+XqnFxu6mWtyivum1MQm4aco+WIt6Coimw==
+  dependencies:
+    "@types/prop-types" "*"
+    "@types/scheduler" "*"
+    csstype "^3.0.2"
+
+"@types/scheduler@*":
+  version "0.16.3"
+  resolved "https://registry.yarnpkg.com/@types/scheduler/-/scheduler-0.16.3.tgz#cef09e3ec9af1d63d2a6cc5b383a737e24e6dcf5"
+  integrity sha512-5cJ8CB4yAx7BH1oMvdU0Jh9lrEXyPkar6F9G/ERswkCuvP4KQZfZkSjcMbAICCpQTN4OuZn8tz0HiKv9TGZgrQ==
+
+cojson@^0.0.14:
+  version "0.0.14"
+  resolved "https://registry.yarnpkg.com/cojson/-/cojson-0.0.14.tgz#e7b190ade1efc20d6f0fa12411d7208cdc0f19f7"
+  integrity sha512-TFenIGswEEhnZlCmq+B1NZPztjovZ72AjK1YkkZca54ZFbB1lAHdPt2hqqu/QBO24C9+6DtuoS2ixm6gbSBWCg==
+  dependencies:
+    "@noble/ciphers" "^0.1.3"
+    "@noble/curves" "^1.1.0"
+    "@noble/hashes" "^1.3.1"
+    "@scure/base" "^1.1.1"
+    fast-json-stable-stringify "^2.1.0"
+    isomorphic-streams "https://github.com/sgwilym/isomorphic-streams.git#aa9394781bfc92f8d7c981be7daf8af4b4cd4fae"
+
+csstype@^3.0.2:
+  version "3.1.2"
+  resolved "https://registry.yarnpkg.com/csstype/-/csstype-3.1.2.tgz#1d4bf9d572f11c14031f0436e1c10bc1f571f50b"
+  integrity sha512-I7K1Uu0MBPzaFKg4nI5Q7Vs2t+3gWWW648spaF+Rg7pI9ds18Ugn+lvg4SHczUdKlHI5LWBXyqfS8+DufyBsgQ==
+
+fast-json-stable-stringify@^2.1.0:
+  version "2.1.0"
+  resolved "https://registry.yarnpkg.com/fast-json-stable-stringify/-/fast-json-stable-stringify-2.1.0.tgz#874bf69c6f404c2b5d99c481341399fd55892633"
+  integrity sha512-lhd/wF+Lk98HZoTCtlVraHtfh5XYijIjalXck7saUtuanSDyLMxnHhSXEDJqHxD7msR8D0uCmqlkwjCV8xvwHw==
+
+"isomorphic-streams@git+https://github.com/sgwilym/isomorphic-streams.git#aa9394781bfc92f8d7c981be7daf8af4b4cd4fae":
+  version "1.0.3"
+  resolved "git+https://github.com/sgwilym/isomorphic-streams.git#aa9394781bfc92f8d7c981be7daf8af4b4cd4fae"
+
+typescript@^5.1.6:
+  version "5.1.6"
+  resolved "https://registry.yarnpkg.com/typescript/-/typescript-5.1.6.tgz#02f8ac202b6dad2c0dd5e0913745b47a37998274"
+  integrity sha512-zaWCozRZ6DLEWAWFrVDz1H6FVXzUSfTy5FUMWsQlU8Ym5JP9eO4xkTIROFCQvhQf61z6O/G6ugw3SgAnvvm+HA==

packages/jazz-browser/package.json

@@ -1,17 +1,18 @@
 {
   "name": "jazz-browser",
-  "version": "0.1.8",
+  "version": "0.2.0",
   "main": "dist/index.js",
   "types": "dist/index.d.ts",
   "license": "MIT",
   "dependencies": {
-    "cojson": "^0.1.8",
-    "jazz-storage-indexeddb": "^0.1.8",
+    "cojson": "^0.2.0",
+    "jazz-storage-indexeddb": "^0.2.0",
     "typescript": "^5.1.6"
   },
   "scripts": {
     "lint": "eslint src/**/*.ts",
     "build": "npm run lint && rm -rf ./dist && tsc --declaration --sourceMap --outDir dist",
     "prepublishOnly": "npm run build"
-  }
+  },
+  "gitHead": "33c27053293b4801b968c61d5c4c989f93a67d13"
 }

packages/jazz-browser/src/index.ts

@@ -1,4 +1,6 @@
-import { InviteSecret } from "cojson";
+import { BinaryCoStream, InviteSecret } from "cojson";
+import { BinaryCoStreamMeta } from "cojson";
+import { cojsonReady } from "cojson";
 import {
     LocalNode,
     cojsonInternals,
@@ -7,7 +9,7 @@ import {
     SessionID,
     SyncMessage,
     Peer,
-    ContentType,
+    CoValueImpl,
     Group,
     CoID,
 } from "cojson";
@@ -29,6 +31,7 @@ export async function createBrowserNode({
     syncAddress?: string;
     reconnectionTimeout?: number;
 }): Promise<BrowserNodeHandle> {
+    await cojsonReady;
     let sessionDone: () => void;
 
     const firstWsPeer = createWebSocketPeer(syncAddress);
@@ -90,9 +93,7 @@ export type SessionHandle = {
     done: () => void;
 };
 
-function getSessionHandleFor(
-    accountID: AccountID | AgentID
-): SessionHandle {
+function getSessionHandleFor(accountID: AccountID | AgentID): SessionHandle {
     let done!: () => void;
     const donePromise = new Promise<void>((resolve) => {
         done = resolve;
@@ -175,15 +176,25 @@ function websocketReadableStream<T>(ws: WebSocket) {
 
                     pingTimeout = setTimeout(() => {
                         console.debug("Ping timeout");
-                        controller.close();
-                        ws.close();
+                        try {
+                            controller.close();
+                            ws.close();
+                        } catch (e) {
+                            console.error(
+                                "Error while trying to close ws on ping timeout",
+                                e
+                            );
+                        }
                     }, 2500);
 
                     return;
                 }
                 controller.enqueue(msg);
             };
-            const closeListener = () => controller.close();
+            const closeListener = () => {
+                controller.close();
+                clearTimeout(pingTimeout);
+            };
             ws.addEventListener("close", closeListener);
             ws.addEventListener("error", () => {
                 controller.error(new Error("The WebSocket errored!"));
@@ -275,19 +286,19 @@ function websocketWritableStream<T>(ws: WebSocket) {
 }
 
 export function createInviteLink(
-    value: ContentType,
+    value: CoValueImpl,
     role: "reader" | "writer" | "admin",
     // default to same address as window.location, but without hash
     {
         baseURL = window.location.href.replace(/#.*$/, ""),
     }: { baseURL?: string } = {}
 ): string {
-    const coValue = value.coValue;
-    const node = coValue.node;
-    let currentCoValue = coValue;
+    const coValueCore = value.core;
+    const node = coValueCore.node;
+    let currentCoValue = coValueCore;
 
     while (currentCoValue.header.ruleset.type === "ownedByGroup") {
-        currentCoValue = currentCoValue.getGroup().groupMap.coValue;
+        currentCoValue = currentCoValue.getGroup().underlyingMap.core;
     }
 
     if (currentCoValue.header.ruleset.type !== "group") {
@@ -304,7 +315,9 @@ export function createInviteLink(
     return `${baseURL}#invitedTo=${value.id}&${inviteSecret}`;
 }
 
-export function parseInviteLink<C extends ContentType>(inviteURL: string):
+export function parseInviteLink<C extends CoValueImpl>(
+    inviteURL: string
+):
     | {
           valueID: CoID<C>;
           inviteSecret: InviteSecret;
@@ -321,7 +334,9 @@ export function parseInviteLink<C extends ContentType>(inviteURL: string):
     return { valueID, inviteSecret };
 }
 
-export function consumeInviteLinkFromWindowLocation<C extends ContentType>(node: LocalNode): Promise<
+export function consumeInviteLinkFromWindowLocation<C extends CoValueImpl>(
+    node: LocalNode
+): Promise<
     | {
           valueID: CoID<C>;
           inviteSecret: string;
@@ -347,3 +362,72 @@ export function consumeInviteLinkFromWindowLocation<C extends ContentType>(node:
         }
     });
 }
+
+export async function createBinaryStreamFromBlob<
+    C extends BinaryCoStream<BinaryCoStreamMeta>
+>(
+    blob: Blob | File,
+    inGroup: Group,
+    meta: C["meta"] = { type: "binary" }
+): Promise<C> {
+    let stream = inGroup.createBinaryStream(meta);
+
+    const reader = new FileReader();
+    const done = new Promise<void>((resolve) => {
+        reader.onload = async () => {
+            const data = new Uint8Array(reader.result as ArrayBuffer);
+            stream = stream.edit((stream) => {
+                stream.startBinaryStream({
+                    mimeType: blob.type,
+                    totalSizeBytes: blob.size,
+                    fileName: blob instanceof File ? blob.name : undefined,
+                });
+            }) as C;// TODO: fix this
+                const chunkSize = 256 * 1024;
+
+                for (let idx = 0; idx < data.length; idx += chunkSize) {
+                    stream = stream.edit((stream) => {
+                        stream.pushBinaryStreamChunk(
+                            data.slice(idx, idx + chunkSize)
+                        );
+                    }) as C; // TODO: fix this
+                    await new Promise((resolve) => setTimeout(resolve, 0));
+                }
+            stream = stream.edit((stream) => {
+                stream.endBinaryStream();
+            }) as C; // TODO: fix this
+            resolve();
+        };
+    });
+    reader.readAsArrayBuffer(blob);
+
+    await done;
+
+    return stream;
+}
+
+export async function readBlobFromBinaryStream<
+    C extends BinaryCoStream<BinaryCoStreamMeta>
+>(
+    streamId: CoID<C>,
+    node: LocalNode,
+    allowUnfinished?: boolean
+): Promise<Blob | undefined> {
+    const stream = await node.load<C>(streamId);
+
+    if (!stream) {
+        return undefined;
+    }
+
+    const chunks = stream.getBinaryChunks();
+
+    if (!chunks) {
+        return undefined;
+    }
+
+    if (!allowUnfinished && !chunks.finished) {
+        return undefined;
+    }
+
+    return new Blob(chunks.chunks, { type: chunks.mimeType });
+}

packages/jazz-react-auth-local/package.json

@@ -1,12 +1,12 @@
 {
     "name": "jazz-react-auth-local",
-    "version": "0.1.10",
+    "version": "0.2.0",
     "main": "dist/index.js",
     "types": "dist/index.d.ts",
     "license": "MIT",
     "dependencies": {
-        "jazz-browser-auth-local": "^0.1.8",
-        "jazz-react": "^0.1.10",
+        "jazz-browser-auth-local": "^0.2.0",
+        "jazz-react": "^0.2.0",
         "typescript": "^5.1.6"
     },
     "devDependencies": {
@@ -19,5 +19,6 @@
         "lint": "eslint src/**/*.tsx",
         "build": "npm run lint && rm -rf ./dist && tsc --declaration --sourceMap --outDir dist",
         "prepublishOnly": "npm run build"
-    }
+    },
+    "gitHead": "33c27053293b4801b968c61d5c4c989f93a67d13"
 }

packages/jazz-react-media-images/.eslintrc.cjs

@@ -0,0 +1,17 @@
+module.exports = {
+    extends: [
+        'eslint:recommended',
+        'plugin:@typescript-eslint/recommended',
+    ],
+    parser: '@typescript-eslint/parser',
+    plugins: ['@typescript-eslint'],
+    parserOptions: {
+        project: "./tsconfig.json",
+    },
+    root: true,
+    rules: {
+        "no-unused-vars": "off",
+        "@typescript-eslint/no-unused-vars": ["error", { "argsIgnorePattern": "^_", "varsIgnorePattern": "^_" }],
+        "@typescript-eslint/no-floating-promises": "error",
+    },
+};

packages/jazz-react-media-images/.gitignore

@@ -0,0 +1,171 @@
+# Based on https://raw.githubusercontent.com/github/gitignore/main/Node.gitignore
+
+# Logs
+
+logs
+_.log
+npm-debug.log_
+yarn-debug.log*
+yarn-error.log*
+lerna-debug.log*
+.pnpm-debug.log*
+
+# Diagnostic reports (https://nodejs.org/api/report.html)
+
+report.[0-9]_.[0-9]_.[0-9]_.[0-9]_.json
+
+# Runtime data
+
+pids
+_.pid
+_.seed
+\*.pid.lock
+
+# Directory for instrumented libs generated by jscoverage/JSCover
+
+lib-cov
+
+# Coverage directory used by tools like istanbul
+
+coverage
+\*.lcov
+
+# nyc test coverage
+
+.nyc_output
+
+# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)
+
+.grunt
+
+# Bower dependency directory (https://bower.io/)
+
+bower_components
+
+# node-waf configuration
+
+.lock-wscript
+
+# Compiled binary addons (https://nodejs.org/api/addons.html)
+
+build/Release
+
+# Dependency directories
+
+node_modules/
+jspm_packages/
+
+# Snowpack dependency directory (https://snowpack.dev/)
+
+web_modules/
+
+# TypeScript cache
+
+\*.tsbuildinfo
+
+# Optional npm cache directory
+
+.npm
+
+# Optional eslint cache
+
+.eslintcache
+
+# Optional stylelint cache
+
+.stylelintcache
+
+# Microbundle cache
+
+.rpt2_cache/
+.rts2_cache_cjs/
+.rts2_cache_es/
+.rts2_cache_umd/
+
+# Optional REPL history
+
+.node_repl_history
+
+# Output of 'npm pack'
+
+\*.tgz
+
+# Yarn Integrity file
+
+.yarn-integrity
+
+# dotenv environment variable files
+
+.env
+.env.development.local
+.env.test.local
+.env.production.local
+.env.local
+
+# parcel-bundler cache (https://parceljs.org/)
+
+.cache
+.parcel-cache
+
+# Next.js build output
+
+.next
+out
+
+# Nuxt.js build / generate output
+
+.nuxt
+dist
+
+# Gatsby files
+
+.cache/
+
+# Comment in the public line in if your project uses Gatsby and not Next.js
+
+# https://nextjs.org/blog/next-9-1#public-directory-support
+
+# public
+
+# vuepress build output
+
+.vuepress/dist
+
+# vuepress v2.x temp and cache directory
+
+.temp
+.cache
+
+# Docusaurus cache and generated files
+
+.docusaurus
+
+# Serverless directories
+
+.serverless/
+
+# FuseBox cache
+
+.fusebox/
+
+# DynamoDB Local files
+
+.dynamodb/
+
+# TernJS port file
+
+.tern-port
+
+# Stores VSCode versions used for testing VSCode extensions
+
+.vscode-test
+
+# yarn v2
+
+.yarn/cache
+.yarn/unplugged
+.yarn/build-state.yml
+.yarn/install-state.gz
+.pnp.\*
+
+.DS_Store

packages/jazz-react-media-images/.npmignore

@@ -0,0 +1,2 @@
+coverage
+node_modules

packages/jazz-react-media-images/package.json

@@ -0,0 +1,26 @@
+{
+  "name": "jazz-react-media-images",
+  "version": "0.2.0",
+  "main": "dist/index.js",
+  "types": "dist/index.d.ts",
+  "license": "MIT",
+  "dependencies": {
+    "cojson": "^0.2.0",
+    "jazz-browser": "^0.2.0",
+    "jazz-browser-media-images": "^0.2.0",
+    "jazz-react": "^0.2.0",
+    "typescript": "^5.1.6"
+  },
+  "devDependencies": {
+    "@types/react": "^18.2.19"
+  },
+  "peerDependencies": {
+    "react": "17 - 18"
+  },
+  "scripts": {
+    "lint": "eslint src/**/*.tsx",
+    "build": "npm run lint && rm -rf ./dist && tsc --declaration --sourceMap --outDir dist",
+    "prepublishOnly": "npm run build"
+  },
+  "gitHead": "33c27053293b4801b968c61d5c4c989f93a67d13"
+}

packages/jazz-react-media-images/src/index.tsx

@@ -0,0 +1,24 @@
+import { CoID, Media } from "cojson";
+import { loadImage, LoadingImageInfo } from "jazz-browser-media-images";
+import { useJazz } from "jazz-react";
+import { useEffect, useState } from "react";
+export { createImage } from "jazz-browser-media-images";
+
+export function useLoadImage(
+    imageID?: CoID<Media.ImageDefinition>
+): LoadingImageInfo | undefined {
+    const { localNode } = useJazz();
+
+    const [imageInfo, setImageInfo] = useState<LoadingImageInfo>();
+
+    useEffect(() => {
+        if (!imageID) return;
+        const unsubscribe = loadImage(imageID, localNode, (imageInfo) => {
+            setImageInfo(imageInfo);
+        });
+
+        return unsubscribe;
+    }, [imageID, localNode]);
+
+    return imageInfo;
+}

packages/jazz-react-media-images/tsconfig.json

@@ -0,0 +1,16 @@
+{
+  "compilerOptions": {
+    "lib": ["ESNext", "DOM"],
+    "module": "esnext",
+    "target": "ES2020",
+    "moduleResolution": "bundler",
+    "moduleDetection": "force",
+    "strict": true,
+    "skipLibCheck": true,
+    "jsx": "react",
+    "forceConsistentCasingInFileNames": true,
+    "noUncheckedIndexedAccess": true,
+    "esModuleInterop": true,
+  },
+  "include": ["./src/**/*"],
+}

packages/jazz-react-media-images/yarn.lock

@@ -0,0 +1,75 @@
+# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
+# yarn lockfile v1
+
+
+"@noble/ciphers@^0.1.3":
+  version "0.1.4"
+  resolved "https://registry.yarnpkg.com/@noble/ciphers/-/ciphers-0.1.4.tgz#96327dca147829ed9eee0d96cfdf7c57915765f0"
+  integrity sha512-d3ZR8vGSpy3v/nllS+bD/OMN5UZqusWiQqkyj7AwzTnhXFH72pF5oB4Ach6DQ50g5kXxC28LdaYBEpsyv9KOUQ==
+
+"@noble/curves@^1.1.0":
+  version "1.1.0"
+  resolved "https://registry.yarnpkg.com/@noble/curves/-/curves-1.1.0.tgz#f13fc667c89184bc04cccb9b11e8e7bae27d8c3d"
+  integrity sha512-091oBExgENk/kGj3AZmtBDMpxQPDtxQABR2B9lb1JbVTs6ytdzZNwvhxQ4MWasRNEzlbEH8jCWFCwhF/Obj5AA==
+  dependencies:
+    "@noble/hashes" "1.3.1"
+
+"@noble/hashes@1.3.1", "@noble/hashes@^1.3.1":
+  version "1.3.1"
+  resolved "https://registry.yarnpkg.com/@noble/hashes/-/hashes-1.3.1.tgz#8831ef002114670c603c458ab8b11328406953a9"
+  integrity sha512-EbqwksQwz9xDRGfDST86whPBgM65E0OH/pCgqW0GBVzO22bNE+NuIbeTb714+IfSjU3aRk47EUvXIb5bTsenKA==
+
+"@scure/base@^1.1.1":
+  version "1.1.1"
+  resolved "https://registry.yarnpkg.com/@scure/base/-/base-1.1.1.tgz#ebb651ee52ff84f420097055f4bf46cfba403938"
+  integrity sha512-ZxOhsSyxYwLJj3pLZCefNitxsj093tb2vq90mp2txoYeBqbcjDjqFhyM8eUjq/uFm6zJ+mUuqxlS2FkuSY1MTA==
+
+"@types/prop-types@*":
+  version "15.7.5"
+  resolved "https://registry.yarnpkg.com/@types/prop-types/-/prop-types-15.7.5.tgz#5f19d2b85a98e9558036f6a3cacc8819420f05cf"
+  integrity sha512-JCB8C6SnDoQf0cNycqd/35A7MjcnK+ZTqE7judS6o7utxUCg6imJg3QK2qzHKszlTjcj2cn+NwMB2i96ubpj7w==
+
+"@types/react@^18.2.19":
+  version "18.2.19"
+  resolved "https://registry.yarnpkg.com/@types/react/-/react-18.2.19.tgz#f77cb2c8307368e624d464a25b9675fa35f95a8b"
+  integrity sha512-e2S8wmY1ePfM517PqCG80CcE48Xs5k0pwJzuDZsfE8IZRRBfOMCF+XqnFxu6mWtyivum1MQm4aco+WIt6Coimw==
+  dependencies:
+    "@types/prop-types" "*"
+    "@types/scheduler" "*"
+    csstype "^3.0.2"
+
+"@types/scheduler@*":
+  version "0.16.3"
+  resolved "https://registry.yarnpkg.com/@types/scheduler/-/scheduler-0.16.3.tgz#cef09e3ec9af1d63d2a6cc5b383a737e24e6dcf5"
+  integrity sha512-5cJ8CB4yAx7BH1oMvdU0Jh9lrEXyPkar6F9G/ERswkCuvP4KQZfZkSjcMbAICCpQTN4OuZn8tz0HiKv9TGZgrQ==
+
+cojson@^0.0.14:
+  version "0.0.14"
+  resolved "https://registry.yarnpkg.com/cojson/-/cojson-0.0.14.tgz#e7b190ade1efc20d6f0fa12411d7208cdc0f19f7"
+  integrity sha512-TFenIGswEEhnZlCmq+B1NZPztjovZ72AjK1YkkZca54ZFbB1lAHdPt2hqqu/QBO24C9+6DtuoS2ixm6gbSBWCg==
+  dependencies:
+    "@noble/ciphers" "^0.1.3"
+    "@noble/curves" "^1.1.0"
+    "@noble/hashes" "^1.3.1"
+    "@scure/base" "^1.1.1"
+    fast-json-stable-stringify "^2.1.0"
+    isomorphic-streams "https://github.com/sgwilym/isomorphic-streams.git#aa9394781bfc92f8d7c981be7daf8af4b4cd4fae"
+
+csstype@^3.0.2:
+  version "3.1.2"
+  resolved "https://registry.yarnpkg.com/csstype/-/csstype-3.1.2.tgz#1d4bf9d572f11c14031f0436e1c10bc1f571f50b"
+  integrity sha512-I7K1Uu0MBPzaFKg4nI5Q7Vs2t+3gWWW648spaF+Rg7pI9ds18Ugn+lvg4SHczUdKlHI5LWBXyqfS8+DufyBsgQ==
+
+fast-json-stable-stringify@^2.1.0:
+  version "2.1.0"
+  resolved "https://registry.yarnpkg.com/fast-json-stable-stringify/-/fast-json-stable-stringify-2.1.0.tgz#874bf69c6f404c2b5d99c481341399fd55892633"
+  integrity sha512-lhd/wF+Lk98HZoTCtlVraHtfh5XYijIjalXck7saUtuanSDyLMxnHhSXEDJqHxD7msR8D0uCmqlkwjCV8xvwHw==
+
+"isomorphic-streams@git+https://github.com/sgwilym/isomorphic-streams.git#aa9394781bfc92f8d7c981be7daf8af4b4cd4fae":
+  version "1.0.3"
+  resolved "git+https://github.com/sgwilym/isomorphic-streams.git#aa9394781bfc92f8d7c981be7daf8af4b4cd4fae"
+
+typescript@^5.1.6:
+  version "5.1.6"
+  resolved "https://registry.yarnpkg.com/typescript/-/typescript-5.1.6.tgz#02f8ac202b6dad2c0dd5e0913745b47a37998274"
+  integrity sha512-zaWCozRZ6DLEWAWFrVDz1H6FVXzUSfTy5FUMWsQlU8Ym5JP9eO4xkTIROFCQvhQf61z6O/G6ugw3SgAnvvm+HA==

packages/jazz-react/package.json

@@ -1,12 +1,12 @@
 {
   "name": "jazz-react",
-  "version": "0.1.10",
+  "version": "0.2.0",
   "main": "dist/index.js",
   "types": "dist/index.d.ts",
   "license": "MIT",
   "dependencies": {
-    "cojson": "^0.1.8",
-    "jazz-browser": "^0.1.8",
+    "cojson": "^0.2.0",
+    "jazz-browser": "^0.2.0",
     "typescript": "^5.1.6"
   },
   "devDependencies": {
@@ -19,5 +19,6 @@
     "lint": "eslint src/**/*.tsx",
     "build": "npm run lint && rm -rf ./dist && tsc --declaration --sourceMap --outDir dist",
     "prepublishOnly": "npm run build"
-  }
+  },
+  "gitHead": "33c27053293b4801b968c61d5c4c989f93a67d13"
 }

packages/jazz-react/src/index.tsx

@@ -1,15 +1,17 @@
 import {
     LocalNode,
-    ContentType,
+    CoValueImpl,
     CoID,
-    ProfileContent,
-    ProfileMeta,
     CoMap,
     AccountID,
     JsonValue,
+    CojsonInternalTypes,
+    BinaryCoStream,
+    BinaryCoStreamMeta,
 } from "cojson";
 import React, { useEffect, useState } from "react";
 import { AuthProvider, createBrowserNode } from "jazz-browser";
+import { readBlobFromBinaryStream } from "jazz-browser";
 
 export {
     createInviteLink,
@@ -90,7 +92,7 @@ export function useJazz() {
     return context;
 }
 
-export function useTelepathicState<T extends ContentType>(id?: CoID<T>) {
+export function useTelepathicState<T extends CoValueImpl>(id?: CoID<T>) {
     const [state, setState] = useState<T>();
 
     const { localNode } = useJazz();
@@ -128,9 +130,14 @@ export function useTelepathicState<T extends ContentType>(id?: CoID<T>) {
 }
 
 export function useProfile<
-    P extends { [key: string]: JsonValue } & ProfileContent = ProfileContent
->(accountID?: AccountID): CoMap<P, ProfileMeta> | undefined {
-    const [profileID, setProfileID] = useState<CoID<CoMap<P, ProfileMeta>>>();
+    P extends {
+        [key: string]: JsonValue;
+    } & CojsonInternalTypes.ProfileContent = CojsonInternalTypes.ProfileContent
+>(
+    accountID?: AccountID
+): CoMap<P, CojsonInternalTypes.ProfileMeta> | undefined {
+    const [profileID, setProfileID] =
+        useState<CoID<CoMap<P, CojsonInternalTypes.ProfileMeta>>>();
 
     const { localNode } = useJazz();
 
@@ -144,3 +151,38 @@ export function useProfile<
 
     return useTelepathicState(profileID);
 }
+
+export function useBinaryStream<C extends BinaryCoStream<BinaryCoStreamMeta>>(
+    streamID?: CoID<C>,
+    allowUnfinished?: boolean
+): { blob: Blob; blobURL: string } | undefined {
+    const { localNode } = useJazz();
+
+    const stream = useTelepathicState(streamID);
+
+    const [blob, setBlob] = useState<
+        { blob: Blob; blobURL: string } | undefined
+    >();
+
+    useEffect(() => {
+        if (!stream) return;
+        readBlobFromBinaryStream(stream.id, localNode, allowUnfinished)
+            .then((blob) =>
+                setBlob(
+                    blob && {
+                        blob,
+                        blobURL: URL.createObjectURL(blob),
+                    }
+                )
+            )
+            .catch((e) => console.error("Failed to read binary stream", e));
+    }, [stream, localNode]);
+
+    useEffect(() => {
+        return () => {
+            blob && URL.revokeObjectURL(blob.blobURL);
+        };
+    }, [blob?.blobURL]);
+
+    return blob;
+}