Orchestrating the A.I. Product — part 1
The Anatomy of an A.I. Product
Intro 1. Kevin Riggen | kevin@oahu.ai 2. 20+ years of software engineering | code running in Fortune-100 companies 3. O'ahu A.I. is a boutique consultancy and product builder 4. LitScenes is Product Suite #1 - open source Desktop app : github.com/litscenes - pay-as-go mobile and sms apps - more to come 5. A.I. is a profound invention
1. What's AI?
1. "Artificial Intelligence" is a bad name 2. It is an intelligence amplifier - so were: - the printing press - the telegraph - the search engine - makes plans and outputs abundant 3. Like nothing else before, its programmatic decision making - this is profound - aka "conditional branching on MEANING"
2. The Creator(s) #### Applies to teams or solo builders
Must haves: - access to coding agents - a strong vision - comfort w/ ambiguity - willingness to iterate & throw work away
Good to have: - compelling taste - UI and UX expertise - software development expertise
Must Orchestrate: - strategy agents - coding agents - design agents - brand identity - user communications (marketing, support)
3. The A.I. Product #### A Ratchet of Cumulative Judgement
What's an "A.I. Product"? - a.k.a. "AI-Native Product" - A product that holds the profound invention as central to its operation and existence - A product whose core value would be infeasible without learned inference - A product for which intelligence is the operating principle, not a feature - Would not be possible 1, 2 or 3 years ago
Users use A.I. products to: - decrease cost - save time - improve outputs
Attributes: - makes choice cumulative - couples the builder's differentiated knowledge w/ a ratchet that makes users' judgment cumulative - improves future work from past work learnings, with human oversight / management - canonical knowledge, from the creator, shapes the field of possibilities - uses progressive disclosure
UX Tenets: - Deliberate action outranks inference - Memory has scope - Promotion requires consent - Human attention goes to consequential choices - Let users accept, export and stop whenever the result succeeds - Valuable judgements are scoped & reversible
4. The Journey
1. The job is to build leverage on human judgment 2. The first leverage is the creator's differentiated knowledge - Knowledge sources - e.g. Wikipedia articles, YouTube transcripts, niche websites, scanned books - Knowledge graphs - a simple Postgres node/edge database, curated by the creator(s), makes knowledge lookups fast and cheap (same node/edge pattern detailed under Knowledge Graphs in Section 5) - for LitScenes, a meaning graph with api interfaces for: - a chat agent to drive the story forward with deep literary meaning - a restyle mobile app to find the most relevant styles for a source photo - a frame-suggestion workflow to propose generative images based on story and aesthetic combinations - Compiled knowledge - e.g. "statements made at Hawai'i community meetings in 2026 , extracted from the county's YouTube channel" - out-competes any language model on the subject of community concerns in Hawai'i - LitScenes example: a visual style library 3. A loop develops:
i. Agents expand the search
- acquire sources
- * extract candidate knowledge
- * propose assets, workflows and implementations
ii. The creator judges what deserves development
- rejects, edits, combines and redirects
- * curates candidate knowledge and assets
- * selects workflows and implementations to test
iii. Agents make the judgments executable
- assemble knowledge and assets into context
- * translate intent into prompts, tools, schemas and interfaces
- * produce runnable variants in workbenches
iv. The creator puts the variants under pressure
- acts as user zero
- * compares finished results
- * evaluates quality, coherence, reliability and cost
- * discovers where the product’s assumptions fail
v. The creator promotes what the evidence supports
- canonical knowledge enters graphs and reference libraries
- * dependable workflows become product behavior
- * successful decisions become defaults
- * failures become eval and regression cases
- * friction becomes specifications for the next implementation
vi. The next search begins from more informed ground
- agents inherit the accumulated knowledge, assets, specifications and tests
- * settled decisions narrow the search
- * unresolved failures and missing knowledge reopen it
- * the loop begins again
The build ratchet does not preserve every output. It preserves the judgment earned by producing and testing those outputs.
The Product-Use Ratchet
1. Canonical product knowledge shapes the possibilities 2. The product produces inspectable alternatives 3. The user accepts, edits, rejects or redirects 4. Consequential decisions enter project-scoped memory 5. The next operation begins from more informed ground
Product use does not preserve every generation. It preserves the user's developing intent.
5. The Materials #### everything has changed
Data - knowledge sources are collected into chunks using scrape / ETL scripts - scraping is solved now - firecrawl and more - ETL is Extract Transform Load - extract / load is solved now - we should focus mostly on the Transform schemas - agents make extract & load trivial but the mechanisms here are very important to keep aligned with the underlying goal - chunk strategy matters more than we want - smartest models will generally get this right now - mechanisms matter a lot - this requires a dedicated db for tracking scrape state - sometimes this is temporary e.g. local sqlite db owned by an agent - sometimes its worth making this canonical / immutable / versioned / auditable - chunks are the beginning of a knowledge base - chunks, after some order of curation / analysis / transformation / categorization are the beginning of a knowledge graph
Knowledge Graphs - should be first-class citizens now - one canonical truth across shifting apps, interfaces, distribution - schema is free — node/edge/entity-relationship, no graph database required - curation is NOT free — resolution, certainty, staleness are the real cost, and the moat - agents extend it — new types are data, not migrations - chutes & ladders: a statement leads to its event, and to who else was there
App interfaces - why wouldn't we maximize user distribution and preferences? - mobile - desktop - web - why wouldn't we optimize for performance and lowest latency? - microservices were annoying but now the burden of maintaining and deploying them is negligible - my preference: Golang on AWS Lambda - networking, routing, security, caching, authentication, versioning, deployment is flexible but standardized - immediately affordable, performant, scalable with unbeatable uptime - can be consumed by any official app interfaces or an MCP broker
Creator interfaces - workbenches - scripts - agent chats - agent harnesses - pen & paper - airdropped to agent chats
Workbenches
- Role: the first durable point of contact between an A.I. system and its humans; consequential decisions happen here, get recorded, and remain recoverable.
- Threshold: a repeated
inspect → judge → actloop has earned a workbench. A one-off script, prompt, or log has not. - Not a dashboard: dashboards report state; workbenches hold state, artifact, judgment tools, action, and history together.
- Not an application: applications present finished behavior; workbenches stay visibly provisional so the workflow can keep teaching the product.
- Artifact first: the material being judged leads; controls earn space only by helping someone see, decide, recover, or continue.
- Required loop: inspect state → load material → preview consequences and cost → choose explicitly → register output and reasoning → resume without reconstructing context.
- Architecture: small, local-first when practical, backed by the system's real infrastructure, agent-operable, and repository-aware.
- Interface discipline: dense because the work is dense; progressively disclosed because attention is finite; audit-friendly because the next human or agent must understand what happened.
The LitScenes workbenches
| Workbench | Route | Description |
|---|---|---|
| Frame Splice | /frame-splice | Plans burst-style video transformations, extracts and groups frames, and supports provider-transformed thumbnail review. |
| Monologue Mania | /monologue-mania | Provides a browser surface for LTX and lip-sync monologue runs, history, and render review. |
| Meaning Library | /meaning-library | Browses meaning primitives, branches, evidence, and relations across production and review modes. |
| Meaning Loop | /meaning-loop | Displays graph health, corpus coverage, review debt, recommended next steps, and controlled maintenance workflows. |
| Meaning Review | /meaning-review | Reviews proposed meaning primitives and records accept, merge, reject, or hold decisions with operator rationale. |
| Graph Browser | /graph | Visualizes meaning-graph nodes and neighborhoods and runs worldprint activation probes against text. |
| Style References | /style-reference-assets | Browses generated SREF composite assets alongside their stored style-analysis packets. |
| Style Articulation | /style-articulation | Converts visual reference composites into rights-clean style cards, tests image transfers, and reviews saved attempts. |
| Aesthetic Curation | /aesthetic-curation | Browses the aesthetic catalog and manages primary selection, ranking, disabled state, and curation metadata. |
| Aesthetic Canon Review | /aesthetic-canon-review | Reviews generated aesthetic profiles and contact sheets and supports approval, archival, and quality warnings. |
| New Project | /new-project | Creates and registers project runs, selects media folders, configures operating policies, and initiates media ingestion. |
| Orchestration | /orchestration-workbench | Inspects project state, graph snapshots, beats, lens projections, transformation recipes, provider packets, lineage, and render eligibility. |
| Lens Workbench | /lens-workbench | Examines beats through creative lenses, manages reference roles and provider directives, and records render feedback. |
Evaluations #### Judgment that can be replayed
- a workbench captures a consequential judgment
- an evaluation turns that judgment into a repeatable test
- an eval is not just a prompt and an expected answer
- it binds:
- - a representative situation
- - the relevant product state
- - criteria for success
- - the produced artifact and execution trace
- - the versions being evaluated
- evaluations can graduate from workbenches into custom apps, vendors, deterministic loops (e.g. DSPy)
An evaluation is a judgment with an input, a rubric and a replay button.
Evaluate the whole product—not merely the model response
1. Structural correctness - valid schemas and artifact formats - required stages and references are present - deterministic constraints pass
2. Knowledge and retrieval - the relevant meanings, styles and evidence were found - retrieved material belongs to the correct project and scope - important evidence was not lost through chunking or ranking
3. Workflow behavior - the system made appropriate intermediate decisions - tools were used correctly - user direction survived the workflow - the result is coherent across stages
4. Finished-product judgment - would the creator or user keep, use or export it? - is it meaningful, controllable and differentiated? - is it better than the current production behavior?
Across every layer: - quality - reliability - latency - cost
An Eval Case Is a Frozen Product Situation
Scenario - user intent - project fixture - source media - relevant history
System snapshot - knowledge and asset versions - retrieval configuration - workflow and prompt versions - model-routing policy
Judgment - hard requirements - quality rubric - unacceptable failure modes - reference or comparative outputs when useful
Evidence - final artifact - intermediate artifacts - retrieved knowledge - model and tool trace - latency and cost