Building a voice assistant with Orbz

Orbz is the visual output of a voice experience. It does not capture audio, transcribe speech, call a model, synthesize a response, or manage a session. That separation lets the same component work with any provider or custom pipeline.

Map the session lifecycle

A common voice turn maps cleanly to the five public states:

These are product choices, not hidden behavior. If your assistant waits for a wake word, for example, asleep may mean “wake word armed” instead of “offline.” Describe the meaning in nearby text.

Centralize the mapping

Keep the provider-specific lifecycle in your application and map it once:

import type { OrbzState } from "@neongate-ai/orbz";
 
type VoicePhase =
  | "ready"
  | "requesting-permission"
  | "capturing"
  | "transcribing"
  | "generating"
  | "playing"
  | "offline"
  | "error";
 
export function toOrbzState(phase: VoicePhase): OrbzState {
  switch (phase) {
    case "capturing":
      return "listening";
    case "transcribing":
    case "generating":
      return "thinking";
    case "playing":
      return "speaking";
    case "offline":
      return "asleep";
    default:
      return "idle";
  }
}

Permission prompts and errors require explanatory UI; the orb returns to a neutral visual state while the application presents the actual message and recovery action.

Drive the native element

import "@neongate-ai/orbz/browser";
import type { OrbzElement } from "@neongate-ai/orbz";
import { toOrbzState } from "./voice-state";
 
const orb = document.querySelector<OrbzElement>("#assistant-orb");
const status = document.querySelector<HTMLElement>("#assistant-status");
 
export function renderVoicePhase(phase: VoicePhase) {
  if (orb) orb.state = toOrbzState(phase);
  if (status) status.textContent = describeVoicePhase(phase);
}

The important part is the one-way flow: session state changes first, then the view derives both Orbz state and accessible text from it. Do not use the orb as the source of truth for the session.

React example

"use client";
 
import { Orbz } from "@neongate-ai/orbz/react";
import { toOrbzState } from "./voice-state";
 
export function VoicePresence({ phase }: { phase: VoicePhase }) {
  const state = toOrbzState(phase);
 
  return (
    <div role="status" aria-live="polite">
      <Orbz
        state={state}
        size={300}
        preset="neongate"
        reducedMotion="system"
      />
      <span>{describeVoicePhase(phase)}</span>
    </div>
  );
}

The adapter does not need to know which audio or AI SDK produced phase.

Align speaking with playback

Set speaking when users can actually hear response audio, not merely when text tokens begin arriving. For an HTMLAudioElement, use playback events:

audio.addEventListener("play", () => renderVoicePhase("playing"));
audio.addEventListener("ended", () => renderVoicePhase("ready"));
audio.addEventListener("pause", () => renderVoicePhase("ready"));
audio.addEventListener("error", () => renderVoicePhase("error"));

For streamed synthesis, use the equivalent “first audio frame,” “playback ended,” and failure signals from your audio layer.

Handle interruption and cancellation

Voice turns race. A user may interrupt playback and immediately begin a new recording. Model requests may finish after the active turn has been cancelled. Prevent stale work from changing the visible state:

let activeTurn = 0;
 
async function runTurn(audio: Blob) {
  const turn = ++activeTurn;
  renderVoicePhase("transcribing");
 
  try {
    const response = await requestAssistant(audio);
    if (turn !== activeTurn) return;
 
    await playResponse(response);
    if (turn === activeTurn) renderVoicePhase("ready");
  } catch (error) {
    if (turn === activeTurn) renderVoicePhase("error");
    throw error;
  }
}
 
function interrupt() {
  activeTurn += 1;
  stopCaptureAndPlayback();
  renderVoicePhase("ready");
}

Use AbortController as well when your network and audio APIs support it.

Pause means visual pause

orb.pause() and the paused attribute freeze Orbz animation only. They do not pause microphone capture, model streaming, or audio playback. Name product controls according to what they actually do:

• “Mute microphone” changes the capture layer.
• “Stop response” changes playback and session state.
• “Reduce motion” changes reduced-motion.
• paused is useful for demos, screenshots, or a product-specific visual hold.

Accessible voice UI

The orb cannot replace the rest of the interface. A complete experience also needs:

• a visible and keyboard-operable start/stop control
• microphone permission and error messages
• an announced status or transcript where appropriate
• a way to stop response audio
• reduced-motion behavior that preserves meaning
• a non-voice path for essential tasks

Keep those controls in the host application. Orbz stays visual, portable, and provider-agnostic.