---
name: cqa-authoring-tests
description: Author ContextQA test cases that actually RUN — typed NLP steps with correct wire format, real locators (from an existing case, a browser, or an AI bootstrap), data-driven and environment binding, then execute-and-fix. Hand-authoring is the goal (so you can edit); AI steps are a bootstrap and a fallback, never a replace-the-whole-case shortcut.
---

# ContextQA — Authoring Runnable Test Cases

The goal is a test case made of **typed NLP steps you can edit** (navigate / enter / click /
verify …), not one opaque AI-agent step. A typed case is inspectable, diffable, and
repairable one step at a time. An AI step is a black box you can only regenerate.

**The one rule that matters: never "fix" a case by deleting it and dropping in a single AI
step.** That destroys every typed step and all the progress in them (the old MCP did this and
it was the worst failure mode). **Edit steps** with `manage_test_step action="update"`.

AI has exactly two legitimate uses here:
1. **Bootstrap** — `create_test_case(task_description=..., environment_id=N)` makes one AI step;
   **executing it converts the case in place into typed NLP steps with real captured locators.**
   Use that to learn the wire format and harvest locators for a page, then hand-author.
2. **Fallback** — an `ai_*` step for an action typed steps genuinely can't express.

## Before you start

1. **Refresh the API reference.** Endpoints, and especially `naturalTextActionId` values, are
   instance/version-specific. Pull your instance's OpenAPI to `cqa-api-docs.json` (it is
   gitignored and meant to be refreshed) and grep it for the paths/actions you'll use. Do this
   at the start of every authoring session — stale ids silently produce wrong steps.
2. **Confirm the org and the workspace.** `get_current_user` names the org (tenant), the
   signed-in user and the active workspace in one call — never infer any of them from the API
   host or from returned data. A case's workspace is baked into its record; you cannot move it
   later. Author in the workspace that will run it, in the org you meant.
3. **Confirm the workspace can execute.** Execution needs a **default test plan**; a workspace
   version without one 400s on `execute_test_case` (see "Known platform quirks"). Prove it with
   one throwaway run before authoring 15 cases into a workspace that can't run them.

## The wire-format ground truth (this is what RUNS)

A typed step that executes needs ALL of this. The single most common mistake is a step that
**saves cleanly and then times out at runtime** because the `event`/locators are missing.

Believe this shape (captured from an executed, passing step), not intuition:

**`enter` step** (`naturalTextActionId: 70`):
```json
{
  "action": "Enter <span data-key=\"test-data\" data-event-key=\"value\" class=\"test_data\">*|username|</span> in the <span data-key=\"label\" data-event-key=\"label\" class=\"element\">E-mail address</span>",
  "element": "E-mail address",
  "testDataType": "raw",
  "testData": "*|username|",
  "environmentId": 497,
  "requestList": {},
  "event": {
    "customEvent": "enter", "label": "E-mail address", "value": "*|username|",
    "isIFrame": false, "iFrameLocator": null,
    "pwLocator": ["#v-0-0-0", "xpath=...", "input[type='email'][placeholder='...']", "div.form-input__wrapper > input[placeholder='...']"],
    "selector": "#v-0-0-0"
  },
  "dataMapBean": {"test-data": "*|username|", "test-data-type": "raw", "element": "E-mail address", "condition_if": [], "for_loop": {"startIndex": null, "endIndex": null, "testDataId": null}},
  "dataMapJson": {"testDataType": "raw", "testData": "*|username|", "element": "E-mail address", "conditionIf": []}
}
```

Rules that trip everyone:
- **`testDataType` is `"raw"`, NOT `"environment"`** — even for environment variables. `*|url|`
  resolves against the bound `environmentId` regardless. `"environment"` produces a dead step.
- **`navigateToUrl` needs `event.href` AND `requestList.url`** — e.g.
  `event:{customEvent:"navigateToUrl","href":"*|url|"}` + `requestList:{"url":"https://..."}`.
  Without the URL it fails in ~188ms reporting "network idle" (a misleading message — the step
  simply has no URL).
- **`enter`/`click`/`verify` need the full `event`** — `label`, `value` (enter), and
  **`pwLocator[]` + `selector`**. Missing locators → 30s timeout: `" is not clickable on the
  page…"` (the leading space is the empty element name).
- **Do NOT include `contenteditable` in the action HTML.** The platform sometimes emits it, but
  the authoring API's XSS validator rejects it (`Disallowed attribute 'contenteditable'`). Strip
  it when copying an AI-generated step. Allowed attrs: `class`, `data-*`.
- Action ids seen on one tenant: `navigateToUrl 425, enter 70, click 26, clickIfPresent 154,
  verifyTextContains 105, ai_verify 583`. **These are tenant-specific — confirm from the API
  reference, don't hardcode.**

## Locators — five independent sources (pick per situation)

You cannot invent ids like `#v-0-0-0` blind. Get them from one of:

1. **Edit an existing case** — locators are already on its steps. No tooling. The common case.
2. **Copy from a sibling recorded step** for the same page/app.
3. **AI bootstrap → execute → it converts to typed steps with captured locators.** Works even on
   complex/dynamic flows (menus, modals) and auto-handles onboarding interstitials you'd never
   anticipate. Cost: ~100-200s per run. Read the converted steps with `manage_test_step list`
   and reuse their locators.
4. **Browser recon** — drive the page yourself and read the locators (see next section). The most
   self-sufficient: no test run, works on any page including authenticated ones.
5. **Semantic descriptors are the resilience layer over all of the above.** The `element` field is
   a *meaning* ("Name input field"), and the platform re-resolves it at runtime when the raw
   selector drifts. **Prefer stable semantic/BEM/text selectors over generated ids** — Vue-style
   auto-ids (`#v-1-1`) shift across renders, modal portals shift absolute xpaths, and grids that
   render twice (frozen + scroll pane) make CSS selectors match duplicates.

## Browser recon (use if a browser is configured; else configure Playwright)

Recon = navigate the target page, read the DOM, extract exact locators. **If your MCP client has
a browser tool configured (a Playwright MCP server, an in-client browser, etc.), use it.** If
not, set up Playwright Chrome locally — this repo already depends on Playwright's trace tooling,
so Chromium is one command away:

```bash
pip install playwright            # or: uv pip install playwright
python -m playwright install chromium
```

Minimal recon script (extract the same multi-strategy locator set the platform captures):
```python
# recon.py  —  python recon.py https://app.example.com/login
import sys, json
from playwright.sync_api import sync_playwright

def info(el):
    if not el: return None
    return el.evaluate("""e => {
      function xp(n){ if(n.id) return '//*[@id=\"'+n.id+'\"]';
        let p=[]; for(; n && n.nodeType===1; n=n.parentNode){ let i=1,s=n.previousSibling;
          for(; s; s=s.previousSibling) if(s.nodeType===1 && s.nodeName===n.nodeName) i++;
          p.unshift(n.nodeName.toLowerCase()+'['+i+']'); } return '/'+p.join('/'); }
      return { id:e.id||null, type:e.getAttribute('type'), placeholder:e.getAttribute('placeholder'),
        cssId: e.id?('#'+e.id):null,
        cssAttr:(e.getAttribute('type')&&e.getAttribute('placeholder'))?
          ("input[type='"+e.getAttribute('type')+"'][placeholder='"+e.getAttribute('placeholder')+"']"):null,
        xpath:'xpath='+xp(e), text:(e.innerText||e.value||'').slice(0,40) };
    }""")

with sync_playwright() as p:
    b = p.chromium.launch()             # headless
    pg = b.new_page(); pg.goto(sys.argv[1]); pg.wait_for_load_state("networkidle")
    out = {"email": info(pg.query_selector("input[type='email']")),
           "password": info(pg.query_selector("input[type='password']")),
           "submit": info(pg.query_selector("button:has-text('Login'), button[type='submit']"))}
    print(json.dumps(out, indent=2)); b.close()
```
Feed the resulting `cssId`/`cssAttr`/`xpath` into a step's `event.pwLocator`/`selector`. For
authenticated pages, log in first in the same script (see the credential gate below), or reuse a
saved storage state (`context = b.new_context(storage_state="auth.json")`).

**Vue/SPA fill caveat:** setting an input's `.value` directly often doesn't update the framework's
model (the field resets on re-render). Use real keyboard events — Playwright `page.fill()` or
`.type()`, not raw DOM `value=`.

## Logging into the app under test — the credential consent gate

For QA the app password usually MUST reach the automation to log in. Handle it with an explicit
gate, never silently:

1. **Warn** the user the credential will be used to log in.
2. **Get explicit permission.**
3. **Confirm it is NOT production** and there are **no billing repercussions** (test/sandbox
   account, free tier).
4. Then the automation may use it. **Never write the password to a file, a committed file, or the
   transcript/logs** (filter `password`/`token`/`authorization`/`bearer`/`apikey`/`cookie` from any
   log output).
5. **Always offer the alternative:** the user logs in themselves (open a visible browser, they
   authenticate, you take over the session), so the password never reaches the automation.

## Data-driven cases

- Bind a profile with `manage_test_case action="bind_test_data"` (id, is_data_driven=True,
  test_data_id, start/end index).
- **Profile-column tokens are BARE in the step** — `login_email`, NOT `*|login_email|`. Only
  environment variables use the `*|name|` wrapper. A wrapped column name → `"login_email variable
  is not present in the … Environment"`.
- `testDataType` for a profile column is `"test_data"`; for an env var it's `"raw"`.

## Execute and fix (the loop)

1. `execute_test_case(id, browser="chromium")` waits for the run by default and returns the
   verdict in one call. Pass `wait=False` for the run handle instead, then poll
   `get_execution_status(session_id=<session_id>)`.
2. **Read `result` off the payload** — it is the executor's own verdict (`SUCCESS`/`FAILURE`),
   and it is `null` while `is_completed` is false. Never treat "a new result row appeared" as a
   pass; that inference is what once reported a FAILURE as a pass. For the per-step story call
   `get_execution_step_details(result_id)`, whose `failure_reason` says exactly what broke.
3. **For DATA-DRIVEN cases, `get_execution_step_details` returns `steps: []`** — read the verdict
   from `get_test_case(id).last_run` (`total_steps`/`passed_steps`/`result`) instead.
4. On failure, read `failure_reason`, fix the offending step's fields, re-run. Advance one step per
   fix — a well-formed step turns red→green and the next step becomes the new failure.
5. Cap at ~3 fix attempts per case; if still red, report the real reason rather than thrashing or
   faking a pass. Do not delete a failing case to tidy the numbers.

## Known platform quirks to expect (work around these; they are logged for the platform team)

- **No default test plan → `execute_test_case` 400 with empty body `{}`.** That workspace version
  can't run anything. Use one that has a default plan.
- **`verifyAttribute` can fail to resolve an element that `enter` succeeded on one step earlier**
  (post-input re-render). Prefer `verifyTextContains` for post-input assertions.
- **`get_test_cases` uses `version_id`**, not `workspace_version_id` like every other tool.
- **`manage_environment` reports HTTP 202 as an error** though the write succeeded.
- **Results can carry a wrong `test_case_name`** (correct id, stale name) — cosmetic.
- **`export_to_playwright` writes to the server's filesystem**, not yours; and
  **`export_test_case_as_code` currently targets a dead endpoint.** Don't rely on either for a
  usable local project yet.

## Companion skills

- `cqa-copy-test-case` — reconstruct a case across environments from primitives (the completeness
  test for the authoring surface).
- `cqa-bug-fix`, `n8n-testing`.