Agent & workflow testing

1. Setup. The fixture is written into your tenant with ids unique to the run (t1a2b3c4-patient). Setup writes are silent — no webhooks — and, for built-in scenarios, don't count toward your write quota.
2. Stimulus.If the scenario has one (a new lab result, a portal message, an appointment marked no-show), it's written like any other write, so it doesfire your webhook subscriptions — that's what starts a Keragon workflow.
3. Brief.In agent mode you get a brief to hand to your agent: the task, in plain words, with the patient's id.
4. Grading. Every 15 seconds Blue Banana grades what changed since setup. The run passes as soon as every required expectation holds, or times out at its deadline (default 5 minutes). Call finish to grade and end it early.
5. Evals. Run a set of scenarios under one label (“triage-agent v3”) — each one up to 5 times, and optionally in every persona variant — for a pass rate with its 95% confidence interval, reliability across repeats, a per-persona breakdown and the most-failed checks. Compare two evals to see what regressed.

Modes. Agent: something reads the brief and acts (an assistant over MCP, your own agent over the API). Workflow: nothing gets a brief — your automation reacts to the stimulus.

Scores. Each expectation has a weight (default 1). The score is the weighted share that passed; the run passes only when every required expectation passes. Bonus expectations add to the score without blocking a pass. AI-scored rubric checks are always bonus: they can lower a score, never decide pass or fail.

Real patients don't write like test fixtures. A scenario can carry variants: the same situation, the same expected results, with the patient written differently. Run one variant on its own ("variant": "spanish"), or every variant in an eval ("personas": "all") to get a pass rate per persona. Variants are written ahead of time and reviewed — nothing is generated while a run is live — so a variant run is as reproducible as the baseline.

All personas are fictional writing styles, never real people.

Some scenarios leave out what the system needs to know — “I can't make my appointment, can we change it?” — and give the patient a hidden goal and hidden facts instead. The only way to pass reliably is to ask. A scenario opts in with patientSim.

• Message the patient with a FHIR Communication whose recipient is the patient (REST POST /Communication, the MCP tool message_patient, or a Keragon action).
• The patient replies within a few seconds with a Communication whose sender is the patient and inResponseTois your message, in the run's persona. The reply is a normal write, so it fires message.received — a workflow can continue from it. Read the thread with GET /Communication?patient=… or the MCP tool get_patient_messages.
• The patient reveals a fact only when asked about it, never acts as staff, and stops after 6 replies (a scenario can allow up to 12). Replies are tagged http://bluebananaehr.com/source|simulated-patient.
• Replies come from a language model when one is configured, otherwise from a scripted stand-in that answers whatever your message asks about — the run page says which.

Open Testing(top right of the clinician workspace). Pick a mode, repeats and personas, and tick one scenario to start a run, or several for an eval; a scenario's details list its variants, each with its own Run button. A run page shows the brief, the resources set up for it (linked into the chart), the conversation with a simulated patient, each expected result with its evidence, and every API / MCP / workspace call made while it ran. An eval page shows reliability per scenario × persona and can be compared with an earlier eval. Run reference solutionreplays the scenario's own solution so you can see what passing looks like; Delete run dataremoves the run's patients and resources afterwards.

Use an API key from /account.

# 1. Start a run — returns the brief and fixture ids, never the expected results
curl -s https://bluebananaehr.com/api/testing/runs \
  -H "Authorization: Bearer bb_…" -H "Content-Type: application/json" \
  -d '{"scenarioId":"inbox.message-triage-urgent","mode":"agent","label":"triage-agent v3"}'

# 2. Give run.brief to your agent; it works through /api/fhir/r4/* as usual.

# 3. Finish and read the verdict
curl -s -X POST https://bluebananaehr.com/api/testing/runs/run_…/finish \
  -H "Authorization: Bearer bb_…"

# A single persona variant
curl -s https://bluebananaehr.com/api/testing/runs \
  -H "Authorization: Bearer bb_…" -H "Content-Type: application/json" \
  -d '{"scenarioId":"inbox.message-triage-urgent","mode":"agent","variant":"spanish"}'

# An eval of every workflow scenario, every persona, three times each
curl -s https://bluebananaehr.com/api/testing/evals \
  -H "Authorization: Bearer bb_…" -H "Content-Type: application/json" \
  -d '{"mode":"workflow","label":"keragon-workflows 2026-10-01","repeats":3,"personas":"all"}'

# Compare it with an earlier eval
curl -s "https://bluebananaehr.com/api/testing/evals/eval_…/compare?baseline=eval_…" \
  -H "Authorization: Bearer bb_…"

While a run is live, GET /api/testing/runs/{id} reports only progress— the verdicts appear once it's final. Full reference: the Testing section of the API docs.

• Repeats. Agents don't behave the same way twice. With repeats (1–5) every scenario × persona runs several times. pass^k is the share where every repeat passed — the number to watch — and pass@k the share where at least one did. A scenario that passed some repeats and failed others is flaky.
• Confidence.The pass rate comes with a 95% Wilson interval. Cancelled and errored runs are left out (they're infrastructure, not your agent); timed-out runs count as failures.
• Queue. An eval can be up to 100 runs. Past 20 live runs the rest wait as queued and start as slots free (a queued run that never starts is cancelled after 24 hours). In agent mode, poll the eval for runs that have started and hand each brief to your agent.
• Compare. GET /api/testing/evals/{id}/compare?baseline={evalId} gives each scenario × persona a verdict — regressed (its pass rate fell by a third or more, or it stopped passing every repeat), improved, same, new or removed — and says whether the overall difference is bigger than the noise.

Run an eval on every change to your Keragon workflows and fail the build on regressions. scripts/testing-eval-ci.ts in the Blue Banana repository starts an eval, waits for it, compares it with the latest eval whose label starts with a prefix, writes a Markdown summary and exits 1 on any regression or a pass rate under your minimum.

# .github/workflows/blue-banana-eval.yml (in your workflow repo)
name: Blue Banana eval
on: [pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { repository: KeragonGrowth/bluebananaehr, path: bb }
      - uses: actions/setup-node@v4
        with: { node-version: 22 }
      - run: npx --yes tsx bb/scripts/testing-eval-ci.ts --mode workflow --repeats 3 --personas all \
               --label "pr-${{ github.event.number }} ${{ github.sha }}" --baseline label:main --min-pass-rate 0.8
        env:
          BB_API_KEY: ${{ secrets.BB_API_KEY }}

Run the same command with --label "main $(git rev-parse --short HEAD)" on your main branch so pull requests have a baseline. Keep one sandbox per system under test: runs in one tenant share patient-less calls.

With the MCP server connected, ask your assistant: “List the Blue Banana test scenarios, take the critical-lab one, and tell me your score.” It calls list_test_scenarios, start_test_run (which returns the brief, and takes a variant), does the work with the other tools — message_patient and get_patient_messages to talk to a simulated patient — then finish_test_run.

1. Build the workflow on the Blue Banana connector's trigger — e.g. results.critical-lab wants: on a final DiagnosticReport with a critical result, create an urgent result-review Task owned by the ordering provider. Each scenario's Workflow to build note says what it expects.
2. Start the scenario in workflow mode. Its stimulus fires your trigger exactly like a real write; your workflow calls back into Blue Banana.
3. Watch the run page: checks turn green as your workflow's writes land. It passes on its own when everything holds.

Import JSON on the Testing tab or POST /api/testing/scenarios. It's validated for shape, symbols, time tokens and FHIRPath before it's saved.

{
  "id": "custom.book-annual",
  "title": "Book an annual physical",
  "category": "scheduling",
  "modes": ["agent"],
  "summary": "A patient asks for their annual physical with Dr. Chen.",
  "brief": "{{patient.name}} (patient id {{patient.id}}) wants their annual physical with Dr. Sarah Chen next week. Book 30 minutes.",
  "fixture": [
    { "resourceType": "Patient", "id": "@patient",
      "name": [{ "given": ["Alex"], "family": "Rivera" }], "birthDate": "1970-05-01", "gender": "male" }
  ],
  "expectations": [
    { "id": "booked", "type": "resource", "label": "A 30-minute visit with Dr. Chen next week",
      "resourceType": "Appointment", "count": { "min": 1, "max": 1 },
      "where": ["status = 'booked'",
                "participant.actor.reference contains 'Practitioner/demo-canonical-prac-chen'",
                "start.toDateTime() >= %runStart.toDateTime() + 3 days"] },
    { "id": "looked-first", "type": "api_call", "label": "Checked availability first",
      "method": "GET", "path": "/{Slot,Appointment/$find}", "before": "booked", "required": false }
  ],
  "timeoutSeconds": 300,
  "oracle": [
    { "method": "POST", "path": "/Appointment",
      "body": { "resourceType": "Appointment", "status": "booked",
                "start": { "$time": "+5wd@10:00" }, "end": { "$time": "+5wd@10:30" },
                "participant": [{ "actor": { "reference": "@patient" }, "status": "accepted" },
                                { "actor": { "reference": "Practitioner/demo-canonical-prac-chen" }, "status": "accepted" }] } }
  ]
}

Symbols

Fixture and stimulus ids are symbols like @patient. Each run replaces them with fresh ids. Use { "reference": "@patient" } anywhere a reference goes; any other string equal to a symbol becomes the bare id. A stimulus resource may reuse a fixture symbol to updateit (e.g. the fixture's booked appointment → noshow).

Time tokens

{ "$time": "+1wd@10:00" } → an ISO time relative to the run start, in UTC. Units: d days, wd weekdays (+0wd= today if it's a weekday still ahead, else the next one), h, m; @HH:MM sets the clock. { "$date": "-20d" } gives a date.

Brief tokens

{{patient.name}}, {{patient.id}}, {{appt.start}}, {{appt.ref}}, or any top-level string field.

resource

Resources changed during the run (on the fixture patients; "scope": "tenant" for anywhere) matching every where FHIRPath, counted against count (default at least one; { "max": 0 }means “must not happen”).

field

A fixture resource, as it reads now, matches every where.

webhook

A successful delivery of the named event for a fixture resource.

api_call

Calls in the run's trace matching method and a path glob (*, **, {a,b}). MCP tools count as the REST call they stand for. beforeonly counts calls made before another expectation's evidence.

any_of

Passes when any of its child expectations passes.

rubric

An AI judge scores the text your system wrote during the run — messages to the patient (Communication), task notes and plain-text documents — against rubric, from 0 to 1; it passes at threshold (default 0.7). Always bonus, graded once when the run ends (shown as pending until then). of limits where it reads. Without a model configured, a heuristic stands in and says so in the result.

variants

[{ "persona": "spanish", "summary": "…", "overrides": { "@msg": { "description": "…" } }, "brief": "…" }] — each override is a JSON merge patch on the resource with that symbol; brief replaces the brief. Every variant is validated as a full scenario.

patientSim

{ "patient": "@patient", "goal": "…", "facts": [{ "topic": "…", "keywords": ["when", "day"], "answer": "…" }], "maxTurns": 6 } — a patient who replies to messages. Facts are revealed only when a message asks about them (the scripted stand-in matches keywords).

FHIRPath variables

%fixture.patient (the id), %current.appt.start (a fixture resource as it reads now), %runStart, %now.

oracle

The reference solution: FHIR writes (/Type, /Type/@symbol), /api/eligibility/$check, and GET lookups. It must pass, and doing nothing must fail.

• Up to 20 live runs per tenant (eval runs past that are queued); up to 100 runs per eval; deadlines from 30 seconds to 30 minutes.
• Simulated-patient replies and AI-scored checks made by a model share an allowance of 300calls an hour per sandbox. Past it, the scripted patient and the heuristic judge take over, and the reply's engine tag or the check's detail says so — a run never stalls on it.
• Setup writes for built-in scenarios are free. A custom scenario's setup, the reference solution and your system's own writes count toward the write quota as usual.
• Runs in the same tenant share calls that aren't about a patient (a slot lookup, say), so run evals of one system at a time. Anyone holding your API key can read scenarios and replay reference solutions — keep the agent under test on its own key if that matters.
• The request trace is recorded only while a run is live, and kept for 7 days. Test data is tagged http://bluebananaehr.com/source|test-run.
• Emails or texts your system sends outside Blue Banana can't be graded. A simulated patient only reads and writes Communications in the sandbox.