--- name: contextqa-platform description: How ContextQA actually works — tenancy and workspace scoping, the test-case object graph, the variable token grammar, how runs really execute, where results and evidence live, and the integrations/PR-impact surface. Read this first if you are driving the ContextQA MCP and do not already know the platform; it is the orientation skill every other ContextQA skill assumes. --- # ContextQA — how the platform works ContextQA is an AI-native test automation platform. Test cases are written as **natural-language steps** that a runner turns into real browser, mobile or API actions. This skill is the mental model you need before you touch anything: what the objects are, how they are scoped, how a run actually happens, and which traps silently produce a test case that saves but never runs. Read this once, then read the task-specific skill (`cqa-authoring-tests`, `cqa-pr-impact`, `cqa-copy-test-case`, `cqa-bug-fix`, `cqa-export-playwright`) for the workflow you are on. --- ## 1. Identity and tenancy Three nested concepts, and confusing them is the most common source of "the API returned someone else's data": | Concept | What it is | Where it comes from | |---|---|---| | **Organization** (tenant) | The customer account. A subdomain: `.contextqa.com`. | Chosen at login when a user belongs to more than one. | | **Workspace** | A product/app under that org (a web app, a mobile app, an API). Carries a `workspaceType`: `WebApplication`, `MobileApplication`, `RestApi`. | Listed by `list_workspaces`. | | **Workspace version** | A *version* of a workspace. **This is the id nearly every API wants.** | `workspace_version_id`, per user. | **`workspace_version_id` is per-user, not per-deployment.** There is no environment variable for it. It is resolved after authentication — the browser login page lets the user pick one, and the headless path derives it from `GET /user_preferences`. Use `get_current_workspace` to see which one you are in and `switch_workspace` to move. Every id you read (a test case, an environment, a test-data profile) belongs to exactly one workspace version. ### The login flow Two-step, because of multi-org: 1. `POST /login/orgs {username, password, recaptchaToken}` → `{preAuthToken, userName, orgs[]}` where each org is `{tenant, companyName, companyImageUrl, lastAccessAt}`. 2. `POST /login/month` with the `preAuthToken` and the chosen tenant → the session token. A single-org account skips the picker. `LEGACY_AUTH=true` selects the older one-call `POST /login/month {username, password}` → `{token, tenant}` for backends not yet upgraded. Over MCP you never do this by hand: the server runs the flow at `/login` and keys a session to your bearer token. **If a tool reports a missing session, the fix is to re-authenticate — never to fall back to service-account credentials**, which would run as a different identity in a different workspace. --- ## 2. Workspace scoping — the biggest trap There is **no single convention.** The platform scopes requests four different ways, and the wrong one does not error — it silently returns another workspace's rows. | Style | Looks like | Used by | |---|---|---| | `query` filter (**default**) | `?query=workspaceVersionId:76` | most list endpoints | | `versionId` | `?versionId=76` | `/test_cases/filter/{id}`, `/test_cases/coverage_summary` | | bare `workspaceVersionId` | `?workspaceVersionId=76` | `/dashboard`, `/api/upload_requirements` | | both | `?versionId=76&workspaceId=76` | `/test_cases/data-grouping/summary` | Two rules that follow from this: - **A bare `workspaceVersionId` on an endpoint that wants the `query` form is ignored, not rejected.** `/test_data?workspaceVersionId=76` returned workspace 1's profiles. The `query=` form is correct-by-default for anything new. - **A required parameter is not evidence of scoping.** `/api/jira/issues` *requires* `workspaceVersionId` and then ignores it — the Jira integration is resolved org-wide. Only a with-data-vs-empty probe proves scoping. The MCP centralises this in `app/services/workspace_scope.py`. When you add a client method, let `scope_params()` do it rather than hand-writing the param, and add an isolation test if the endpoint is an exception. Why it cannot be fixed piecemeal: one argument resolver reads **only** the `query` parameter and hands the result to a specification builder. Any other request parameter is invisible to those endpoints by construction — and an empty builder produces *no WHERE clause at all*, which is why a bare `?workspaceVersionId=` does not merely fail to scope, it returns the whole tenant. Re-verified 2026-09-25: still true for `/test_cases`, `/environments`, `/test_suites`, `/test_data`, `/elements`, `/elements_screen_name` and `/global_variables`. Two specifics worth carrying: - **AI insights deliberately removed its bare `workspaceVersionId`** (a breaking change). A caller still sending one gets **HTTP 200 with zeroed figures**, not a 400 — it fails silently. `?query=` is the only correct form. - **`/test_data` is an open question.** Its specification special-cases the key `versionId`, and the row carries *both* a `versionId` and a `workspaceVersionId` column, of which `versionId` is the one the product writes. The MCP currently filters on `workspaceVersionId`, which may be the wrong column. Probe it before relying on a test-data listing being complete. Separately: **`GET /test_cases` has no `deleted` guard** and lists deleted cases; `/test_cases/filter/2` does. A count that disagrees with the UI is usually deletions, not missing rows. And a listing asymmetry: **`GET /workspaces` applies no membership filter** — it returns every workspace in the organisation, including ones you cannot open — while `GET /workspaces/{id}` *does* check and 403s. So a workspace can appear in a list and then refuse to be switched into. There is no "workspaces I can access" endpoint; membership has to be read off each record. --- ## 3. The object graph ``` Organization (tenant) └── Workspace (WebApplication | MobileApplication | RestApi) └── Workspace version ← workspace_version_id ├── Test case BROWSER | MOBILE | API_TESTCASE │ ├── Steps (ordered, 1-based positions) │ │ ├── typed NLP step ← the goal: editable, deterministic │ │ ├── AI-agent step ← a bootstrap/fallback, not a shortcut │ │ ├── step group ← a reusable sub-case │ │ └── REST step ← API_TESTCASE only │ ├── Prerequisite test case(s) ← run before this one (a tree) │ ├── Element references ← named locators │ ├── Variable section ← values for ${name} │ ├── Test-data profile binding ← data-driven rows │ ├── Environment binding ← REQUIRED, always │ ├── Folder, tags, module name │ └── Requirement links ├── Element name → locator(s) (incl. self-healed pwLocator) ├── Screen groups elements by page ├── Environment named variables, permissionMode READ | READ_WRITE ├── Global variable workspace-scoped, resolved by name ├── Test-data profile → rows (columns are referenced bare in step text) ├── Requirement → generated/linked test cases, coverage ├── Test suite a set of cases (membership) └── Test plan suites + devices/environment → the runnable unit └── Run (test_plan_result) → test_case_result → test_step_result ``` ### Non-obvious rules - **Every test case must carry an environment**, even one with no environment tokens. `create_test_case` binds the `isDefault` environment, else the first in the workspace, else it creates one called `Default`. - **Test-data profiles need a follow-up PUT.** `POST /test_data` accepts a profile but **ignores `rows`**. Create, then `PUT` the rows, or the profile is empty. It is also the one object whose create must set `versionId`, or the profile is invisible in the UI. - **A test plan is the runnable unit** for anything scheduled or multi-case; a single case can be executed directly. - Step **positions are 1-based** and `position` is ignored on create — steps append. Reorder afterwards. - A test case's workspace lives **on its record**, not on your session. Acting on someone else's case means passing that case's workspace. --- ## 4. Variable token grammar Four different things look like variables. Getting one wrong is the classic "saved fine, failed at run time": | Token | Kind | What you must pass alongside | |---|---|---| | `*\|name\|` | **Environment** variable | An `environment_id` that *defines* `name`. Omitted → the workspace default is bound and then checked. | | `${name}` | **Runtime / test-case** variable | A value in `variables`, registered as the case's variable section. An undeclared `${name}` is rejected *before* create. | | `{{name}}` | **Global** variable | **Nothing.** Workspace-scoped, resolved by name. No id, no `variables` entry, no environment. | | `name` (bare) | **Test-data profile column** | The bound profile must have that column. | | `${result.body.id}` | **API response chaining** | Not a variable — a reference to an earlier REST step's response. | --- ## 5. How a run actually happens **This is the single most commonly mis-implemented part of the platform.** Execution does **not** go through the portal API. It goes to the **session-manager** (the "node" service): - One test case: `POST {node_url}/tc/{test_case_id}/execute` - A test plan: `GET {node_url}/tp/{test_plan_id}/execute` - Re-run a plan: `GET {node_url}/tp_rerun/{execution_id}/execute` `dry_test_plans` is **not** the execution path — it is the payload builder used for run *overrides*. Two consequences worth knowing: - An **environment override is only possible via `dry_test_plans`**, and that path excludes `watch_live`. Test data can never be overridden at run time. - `/plan_run/org_status` counts **plans, not runs** — do not read it as capacity. A started run returns a handle: `session_id`, `run_ref`, `live_url`. **Share the live link immediately** — a link to a live run is worthless once the run ends. Sessions are reaped shortly after completion, after which the result row is the only record left. ### Reading results and evidence ``` test_case_result (result_id) ├── test_step_results → per-step verdicts │ └── children (nested) → step groups / AI sub-steps, fetched by depth ├── console logs, network logs ├── video → plays inline └── Playwright trace → a deep link into the trace viewer ``` Tools: `get_execution_status`, `get_test_case_results`, `get_test_step_results`, `get_step_children_details`, `get_console_logs`, `get_network_logs`, `get_trace_url`, `investigate_failure`. --- ## 6. Authoring: the rule that matters **Hand-authored typed NLP steps are the goal, not AI steps.** A typed step (`navigate`, `enter`, `click`, `verify`, …) is deterministic and *editable*; you can fix one selector later. An AI-agent step is a black box that re-derives its own actions each run. So: use an AI step to *bootstrap* (discover locators on a page you have never seen) or as a fallback for something genuinely dynamic — never as a "describe the whole case in a paragraph" shortcut. A case built entirely of AI steps cannot be reviewed, diffed or repaired, and exports to a Playwright file full of manual checkpoints. Granular typed steps need the **action HTML** and an **environment binding**; action ids resolve automatically from `displayName` (they vary per tenant, so never hard-code one — `list_natural_text_actions` is the lookup). Read `cqa-authoring-tests` before writing steps. Known authoring defects there include `task_description` collapsing, `<>` being stripped, `bulk_operation` silently no-opping, and step lists coming back unsorted. --- ## 7. Integrations, PR impact and identity Three services, and confusing them causes real security mistakes: | Service | Owns | How you reach it | |---|---|---| | **ts-portal-server** | test cases, runs, requirements, billing, integration *settings* | your tenant session (JWT or cookie) | | **cqa-integrations** | the OAuth *connection* lifecycle, provider webhooks, PR-impact forwarding, bug reproduction | a shared `Internal-Service-Key` — **server-to-server only** | | **cqa-accounts** | signup, email verification, tenant registration, login | its own app, not an API you call | **Never call cqa-integrations directly from a client.** Its management routes take one static `Internal-Service-Key` that carries *no tenant claim* — a holder can read and write every customer's connections. The portal keeps that key server-side and re-exposes the same data as tenant-scoped facades. The browser UI follows this rule for exactly the same reason, and so does the MCP. ### Connecting a provider `github`, `gitlab`, `slack` and `linear` connect by one-click OAuth. **`jira` does not** — it is a site URL + username + API token form. The OAuth flow *cannot* be completed headlessly: the provider shows a consent screen, and the handshake created at `/oauth//start` is consumed exactly once by the callback (and expires in ~10 minutes). So the only useful thing an agent can do is build the link and hand it over — `manage_integrations(action="connect_url", ...)`. Two traps: - **Always send `workspaceVersionId` on the start URL.** Omit it and the connection binds to the tenant's *oldest* workspace version, which is almost never the one the user is looking at. Reconnecting from the right workspace heals a wrongly-bound connection. - **A connection row can outlive the provider-side install.** `list` will happily show "connected" for an app that was uninstalled on GitHub. Only `action="registry"` sees the truth. That is the first thing to check when someone says "it says connected but nothing happens". Disconnecting is worse than it looks: `DELETE /settings/integrations/{id}` **cascade-deletes every enrolled repository** under it, with no 409 to warn you, and reconnecting does not bring the enrolments back. ### PR impact analysis A PR is identified by **(connected repo, PR number)**. The head sha plays no part — there is one analysis row per PR forever, and a new commit overwrites it in place. ``` enrol a repo ──▶ PR webhook ──▶ analysis ──▶ review items ──▶ run impacted cases │ │ preflight check pre- or post-merge │ comment + check posted back on the PR ``` `testRunTiming` decides whether anything runs at all: | Value | Behaviour | |---|---| | `OFF` (default) | analyse and report, run nothing | | `POST_MERGE` | run after merge, once `deploymentWaitMinutes` elapses or a deployment webhook arrives early | | `PRE_MERGE` | run against the PR's own build, matched on the head commit | **Four things here read as success when they are not:** 1. **`preflight == null` means "not checked", not "clean."** A real report with `casesWithFindings == 0` is the earned all-clear. Render them differently. 2. **A run that `COMPLETED` with 0 passed and 0 failed executed nothing.** Not a pass. 3. **`FAILED_TO_START` is an infrastructure problem, not a red build.** 4. **`decisionCarry == null` means nothing was carried, not that everything survived.** And two naming traps: the run block is still on the wire as `postMergeRun` even for a **pre-merge** run (the class was renamed, the JSON key deliberately was not) — read its `timing` field. And `preventSelfApproval`, `requiredApproversQuorum`, `archiveApproversQuorum` and `autoDemoteOnRejections` are stored and returned but **not enforced by the platform** — never describe them to a user as active controls. The preflight itself is a static, advisory scan asking "would a run against this environment actually exercise the PR's build?". Its findings are ordered by how badly they mislead: `FIXED_URL` (a literal address in a step — the case *passes*, having tested the old deployment) is worse than `MISSING_ENV_KEY` (fails loudly, for a reason unrelated to the PR). Settings are an **org-wide singleton** with no workspace scoping, and `PUT` is a **full replace** whose validator wants nearly every field non-null — so change them by reading, merging, then writing. `manage_pr_impact(action="update_settings")` does this for you; hand-rolling a partial PUT blanks the rest of the org's configuration. ### `analyze_impact` is a different tool `analyze_impact` is an ad-hoc semantic query — give it a title and a diff and it reasons over the case repository, returning `MUST_UPDATE` / `MUST_RERUN` / `SHOULD_RERUN` per case in 1–2 minutes. It runs nothing, stores nothing, and needs no repository enrolled. Use it for "what would this change break?". Use `manage_pr_impact` for the actual PR pipeline. ### Identity and onboarding There is **no new identity service and no new token format** — it is the same session auth, with a multi-org selection step in front of it (§1). What is new is **onboarding v2**: `APP_DETAILS → CONTEXT → CONNECT_CODE → BUG_TRACKER_INTEGRATION → INVITE_TEAM`, each `NOT_STARTED`, `COMPLETED`, `FAILED` or `SKIPPED`. Two report-only preflights are worth knowing, because both are safe to call and both are easy to misread: - **App-URL check** — only a 5xx, timeout or DNS failure makes `reachable` false. A login-walled app answering 401 is *reachable*, and that is correct. - **Repo support check** — `supported` is **tri-state**: `true`, `false`, or `null` meaning undecided. Null **fails open**; do not report it as unsupported. Signup, verification and tenant registration are not in this API at all. Note for anyone validating org names: registration allows **3–20** characters, lowercase, no underscores — stricter than the rule for resolving existing tenants, deliberately, so older long or underscored tenants still work. ### Credits Work costs credits: a PR-impact run, a crawl, an AI generation. Two facts decide whether an answer about them is right: - **Every credit figure on the wire is integer millicredits.** 1 credit = 1000. A balance of `4500000` is 4,500 credits. Plan prices are the currency's **minor** unit and divide by **100** instead — confusing the two puts a 1000x error next to a real price. - **`configured: false` (and a 404) means the organisation has no credit account at all**, typically an on-prem or unmetered install. It is *not* a balance of zero and must never be reported as one. `/credits/**` and `/billing/**` are **owner-only**, so an ordinary member gets a permission error rather than an empty result — a different answer again. Two more shapes that mislead: `summary.available` is **not windowed** (the same figure whatever dates you pass), and the summary deliberately **does not sum to the ledger** — it excludes internal and record-only rows that the ledger lists. In the ledger a refund is its own row and `creditsCharged` is a magnitude, so the row's `type` carries the direction; summing the magnitudes counts refunds as spend. `manage_credits` converts all of this and says which caveat applies. There is no invoices endpoint — the billing portal link is where invoices live. ### Requirements Requirements reads are exposed **read-only on purpose**: the id-addressed portal endpoints have *no server-side workspace check at all*, so the MCP verifies ownership before returning a record. Defect/Jira data is **org-level** — the endpoint requires `workspaceVersionId` and then ignores it (§2). ## 8. How the platform rejects things Reading an error correctly saves more time here than anything else, because the most common shape says nothing on its own. **Five envelopes:** | Trigger | Body | What to read | |---|---|---| | Bean Validation (`@Valid`) | `{"error": "One or more fields are invalid.", "fieldErrors": [...]}` | **`fieldErrors`** — the `error` string is boilerplate. No `code`. | | Hand-thrown validation | `{"error": "", "code": ...}` | The sentence. | | Unparseable body | `{"error": "Malformed request body: ..."}` | It names the field. | | In-use conflict (409) | `{"count": 2, "testSuites": [...]}` — **no `error` key at all** | The list of what still references the record. | | Credits exhausted (429) | `{..., "requestedMillicredits", "availableMillicredits"}` | Convert: these are millicredits. | **Do not branch on `code`.** The platform's numeric error enum is frozen; new errors carry their message as the contract and most arrive with `code: null`. Match on the message, or better, on the field. The MCP funnels all of this through one describer, so a tool error is already a sentence. When you see `field: must not be blank (got '')`, that is the platform's own field-level detail, not a guess. **Permissions produce two different 403s**, and the distinction matters: - **Role permission** — the caller's role lacks something. Many reads now need `TEST_READ`; AI insights need `AI_INSIGHTS`; requirements need `UPLOAD_REQUIREMENT`; credits and billing are **owner-only** (superadmin only — a role holding a permission is still refused). - **Workspace membership** — the record belongs to a workspace the caller is not a member of. Adding a role permission will not fix it. **The one that wastes the most time: a missing `workspaceVersionId` is a 500, not a 400.** It falls through a null id into Spring Data, which raises "The given id must not be null!", and the handler flattens that to `{"error": "Internal Server Error"}` — a 500 that names nothing. It is reachable from test-case create, test-step writes, bulk update, upload-requirements, crawl, AI insights, onboarding and log streaming. So: **never let a workspace id be absent.** The MCP resolves it or refuses the call before sending anything, which is why you get a clear message instead of a 500. ### Validation that was added, and validation that was taken away Recently **added** (a call that used to work may now be refused): - A blank or whitespace-only **environment name** is rejected; names are also trimmed before the uniqueness check, so `" QA "` and `"QA"` now collide. - A **step group that would contain its own test case** is rejected at save instead of recursing forever at execution. - **Permanently deleting a test case still linked to a suite** is a 409, as is deleting a suite still used by a plan. - `MOBILE` test type on a plan without the mobile entitlement. - Organisation identifiers at **registration**: `^[a-z][a-z0-9-]{2,19}$` — 3–20 characters, lowercase, **no underscores**. (Looser for resolving *existing* tenants, deliberately, so older names still work.) - `POST /schedule_test_plans` requires a parseable schedule time. - Permission gates on around forty routes that used to be open to any authenticated user. Recently **removed** — do not pre-reject these, the server accepts them again: - The **variable-reference check** (unresolved `${name}` / `*|key|` / `{{global}}`) is commented out on main as a temporary client hotfix. Its four error messages are unreachable today. The surrounding service still runs its **workspace access check**, so that part still 403s. - Server-side **test-step `event` / `testDataList` validation** is gone entirely — the validator, its token pattern and its error codes were deleted. - `parent_id` on test-suite folders is nullable at the database level (though the request DTO still declares it required, so the API is stricter than the schema). - Unknown JSON fields are **silently ignored**, and `null` still coerces to `0`/`false` for primitive fields. A typo in a field name is not an error — it is a field that never arrives, which is why the MCP rejects unknown keys on the settings writes it merges. --- ## 9. Known defects and sharp edges Things that will waste your time if you do not know them: | Area | What bites | |---|---| | Test data | `POST /test_data` ignores `rows`; create then PUT. Create must set `versionId` or the profile is invisible in the UI. | | Listing cases | `GET /test_cases` returns deleted cases; `filter/2` does not. | | Scoping | A bare `workspaceVersionId` is silently ignored on `query`-style endpoints. | | Jira/defects | Requires `workspaceVersionId`, ignores it. Org-wide. | | Step authoring | `position` ignored on create; `<>` stripped from text; step lists unsorted. | | Filters | Several documented test-case filters 500; `IN` is silently ignored on `status`. | | Auth | A missing OAuth session must fail loudly; falling back to env credentials runs as a different identity. | | Execution | Environment overrides only via `dry_test_plans`; test data never overridable. | --- ## 10. Where to go next | You want to… | Read | |---|---| | Write a test case that actually runs | `cqa-authoring-tests` | | Drive the PR impact pipeline | `cqa-pr-impact` | | Move a case between environments/tenants | `cqa-copy-test-case` | | Go from a ticket to a reproduced, fixed bug | `cqa-bug-fix` | | Turn cases into a Playwright project | `cqa-export-playwright` | | Generate cases from an n8n workflow | `n8n-testing` | Call `list_contextqa_skills` for the live catalogue — this server serves whatever it is running, which may be newer than this table.