> ## Documentation Index
> Fetch the complete documentation index at: https://docs.avocadostudio.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Chat troubleshooting

> Playbook for investigating prompt failures, wrong operations, and regressions in the chat assistant.

This playbook is for the self-hoster or developer investigating prompt failures,
wrong operations, and regressions in the editor's chat. For background on the editing pipeline, see [How It Works](/how-it-works). For the full telemetry event reference, see [Chat Telemetry Events](/observability/chat-telemetry-events).

## What telemetry is captured

The orchestrator records structured chat telemetry events for each request:

* `received`
* `milestone`
* `forced_plan`
* `deterministic_plan_generated`
* `plan_attempt_failed`
* `plan_generated`
* `plan_apply_failed`
* `repair_attempt`
* `repair_generated`
* `tool_call`
* `result`

Each event includes:

* request id
* session
* requested/effective slug
* model key/model used
* planner source (`openai`, `anthropic`, `gemini`, or `demo`)
* prompt hash (stable fingerprint)
* prompt excerpt (short preview)
* prompt length
* intent/op types/op count (when available)
* outcome + error category (when available)

## Persistence

Telemetry is persisted to NDJSON so it survives restarts.

* Default file: `.data/chat-telemetry.ndjson` in the data directory
* Env override: `CHAT_TELEMETRY_FILE`
* Disable persistence: `CHAT_TELEMETRY_PERSIST=0`. It is already off under `NODE_ENV=test`.
* In-memory buffer size: `CHAT_TELEMETRY_LIMIT` (default `500`)

On startup, the orchestrator loads recent telemetry entries from disk.

## APIs for debugging

These endpoints are served by the standalone server. In library mode, read the
NDJSON file or the `chat_telemetry` lines in the site's server log instead. The
standalone server does not gate them, and they return prompt excerpts.

### 1) Raw telemetry stream (filtered)

`GET /telemetry/chat`

Query params:

* `limit` (default `100`, max `1000`)
* `session`
* `phase`
* `outcome`

Example:

```bash theme={null}
curl -s "http://127.0.0.1:4200/telemetry/chat?session=dev&limit=200" | jq
```

### 2) Review summary for manual test runs

`GET /telemetry/chat/review`

Query params:

* `limit` (default `300`, max `2000`)
* `session`

Returns:

* analyzed count
* applied/failed counts
* failure rate
* failure breakdown by outcome
* failure breakdown by reason category
* top failed prompts (grouped by prompt hash)
* automatic recommendations

Example:

```bash theme={null}
curl -s "http://127.0.0.1:4200/telemetry/chat/review?session=dev&limit=500" | jq
```

## UI debug mode (for screenshots)

Enable debug metadata directly in assistant response cards:

1. Open **Settings** in the editor's top bar. On a public demo editor, add
   `?dev=1` to the URL first.
2. Turn on **Developer mode**, then the **Debug mode** toggle it reveals.
3. Run the prompt and capture a screenshot.

Each assistant card includes a `Debug` panel with:

* `traceId`
* `promptHash`
* `outcome`
* `reason` category, and the detail when it differs
* `intent` and `plannerTier`
* the model and provider used
* `opCount`, `ops`, and any skipped ops with the reason each was skipped
* tokens and estimated cost
* the timeline of the turn
* prompt excerpt

Use this panel when sharing failures so engineering can reproduce and convert them into regression tests quickly.

## Standard workflow after manual UI testing

1. Use a dedicated session for a test batch (for example: `manual-2026-03-01-a`).
2. Run your manual prompts in UI.
3. Pull review summary:

```bash theme={null}
curl -s "http://127.0.0.1:4200/telemetry/chat/review?session=manual-2026-03-01-a&limit=1000" | jq
```

4. Inspect top failures:
   * high `schema_violation`: normalization/repair gaps
   * high `not_found`: slug/block resolution gaps
   * high `ambiguity`: clarification prompts too weak
5. Drill into raw events for one problematic prompt hash:

```bash theme={null}
curl -s "http://127.0.0.1:4200/telemetry/chat?session=manual-2026-03-01-a&limit=1000" \
| jq '.rows | map(select(.promptHash=="<hash>"))'
```

6. Convert top failed prompts into regression tests in:
   * `apps/orchestrator/src/nlp-ops.test.ts`

## Recommended operating rules

* Always run manual tests with an explicit session id.
* Keep a fixed prompt suite for weekly regression checks.
* Add a test for every new recurring failed prompt family.
* Track failure rate trend (`/telemetry/chat/review`) before/after changes.

## Quick checklist when behavior is wrong

* Did the model fail to produce valid schema?
* Did normalization repair aliases correctly?
* Was deterministic fallback expected but not triggered?
* Was the apply step blocked by not-found or no-effective-change?
* Did clarification context leak across unrelated intents?

## "Chat won't edit one prop on my custom block"

The shape of this report is specific and worth recognising, because every check
you would run to diagnose it passes.

Symptoms, all at once:

* `POST /chat` answers `200` with `status: "info"`, an empty `changes` array,
  and a summary that names your block but describes the failure as a *styling*
  limitation.
* `POST /ops` with the identical `update_props` patch applies instantly.
* `/blocks/manifest` is correct, the block is selected, and the schema is right.
* One prop fails and its siblings succeed. `subheading` works; `heading` does not.

That last asymmetry is the tell: a model that could not read the schema would
fail on every prop, not one. Something deterministic is rewriting a name.

**Check the server log for the strip.** The line is emitted at `warn`:

```
Planner generated an unsupported prop; stripped before apply
  { blockId: 'b1', blockType: 'heroImage', propName: 'title',
    allowedProps: ['heading','subheading', ...] }
```

`propName` is what the planner asked for and `allowedProps` is what your block
declares. If `propName` is a name from Avocado's own catalogue (`title`, `q`,
`a`, `quote`) and your block uses a different one, the planner's output was
rewritten on the way through — see below. If `propName` is something visual
(`backgroundColor`, `animation`), the model genuinely promised a capability the
block does not have, which is the case that message was written for.

**Two things to check on your side:**

1. **Is the prop declared?** The strip compares against the block's registered
   schema. A prop that only exists behind `.catchall()` is not declared, and is
   stripped even though `POST /ops` would accept it. Declare every prop you want
   the chat to reach.
2. **Set `CHAT_ADAPTIVE_SCHEMA_CONTEXT=1`.** Without it, a single-field edit is
   sent to the model with no block contracts at all, so it has to guess your
   prop names from its priors — and its priors are the built-in catalogue. See
   [Block schema contracts](/specs/block-schema-contracts).

Before 0.3.2 the normalizer also renamed `heading` → `title` on every block type
not literally called `Hero`, and rewrote `question`/`answer`/`testimonial`/
`review` inside every list prop of every block, regardless of type. Both now ask
the registry first and only rename when the block cannot take the key the
planner used and can take the one it would be rewritten to. If you are pinned
below 0.3.2 and hit this, upgrading is the fix.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.