--- name: cqa-pr-impact description: Drive ContextQA's PR impact pipeline — enrol a repository, connect GitHub/GitLab, read an analysis without misreading it, review items, and run the impacted tests pre- or post-merge. Covers the four result shapes that look like success and are not, and the settings PUT that silently blanks an org's configuration. --- # ContextQA — PR impact analysis Work out which existing tests a pull request endangers, then run them and report back onto the PR. Read `contextqa-platform` §7 first if you do not know how the three services fit together. **This is not `analyze_impact`.** That tool is an ad-hoc semantic query — a title and a diff in, per-case `MUST_UPDATE` / `MUST_RERUN` / `SHOULD_RERUN` out, in 1–2 minutes, storing nothing. This skill is the pipeline: enrolled repositories, webhooks, reviewable items, real runs, a comment on the PR. --- ## The shape of it ``` connect GitHub/GitLab ──▶ enrol a repo ──▶ PR opened/updated │ analysis runs │ ┌──────────────────┴─────────────────┐ review items preflight check (approve / reject / apply) (advisory, report-only) │ run impacted cases ──▶ comment + check posted on the PR (PRE_MERGE or POST_MERGE) ``` --- ## 1. Check what is actually connected Start here, always. An integration row can outlive the provider-side install, so "connected" in a list is not evidence. ``` manage_integrations(action="list") # what the tenant has configured manage_integrations(action="registry", provider="github") # what is REALLY mapped ``` If `list` shows GitHub but `registry` shows nothing mapped, the app was uninstalled on the provider side. Reconnect — do not debug the analysis. To connect one: ``` manage_integrations(action="connect_url", provider="github", return_to="https://.contextqa.com/settings/plugins") ``` You get a URL back, not a connection. Hand it to the user to open in a browser: the provider shows a consent screen and the handshake is single-use and expires in about ten minutes, so it cannot be completed headlessly. `github`, `gitlab`, `slack` and `linear` work this way; **`jira` does not** — it takes a site URL, username and API token instead. --- ## 2. Enrol a repository ``` manage_integrations(action="repositories", connection_id="") # what it can reach manage_pr_impact(action="register_repo", integration_id=21, name="northwind-pay", repo_full_name="indivatools/northwind-pay", provider_repo_id="...", default_branch="main") manage_pr_impact(action="list_repos") ``` A repository registered without a workspace blocks every analysis start. The tool always binds the current workspace, so do not try to omit it — but *do* check you are in the workspace you mean (`get_current_workspace`). --- ## 3. Find and read an analysis A PR is identified by **(repo, PR number)**. The head sha plays no part: one analysis per PR forever, overwritten in place by each new commit. ``` manage_pr_impact(action="find_by_pr", repo_id=3, pr_number="482") manage_pr_impact(action="get_analysis", analysis_id=91) ``` `get_analysis` returns a `summary` plus the raw `analysis`. **Read the summary.** It exists because four fields in the raw payload read as success when they are not: | What you see | What it actually means | |---|---| | `preflight: null` | **Not checked** — the analysis predates the check. NOT clean. | | `preflight.casesWithFindings: 0` with a real report | The earned all-clear. | | run `COMPLETED`, `passed: 0, failed: 0` | **Nothing executed.** Not a pass. | | run `FAILED_TO_START` | An infrastructure problem, **not a red build**. | | `decisionCarry: null` | Nothing was carried. NOT "everything survived". | Two naming traps: - The run block is on the wire as **`postMergeRun` even for a pre-merge run** — the class was renamed, the JSON key deliberately was not. Read its `timing`. - An analysis still in progress has no run block at all. Absence means nothing ran, not that a run passed. Pass `raw=True` if you specifically need the unsummarised payload. --- ## 4. Review the items ``` manage_pr_impact(action="list_items", analysis_id=91) # everything manage_pr_impact(action="list_items", analysis_id=91, item_class="UPDATE") manage_pr_impact(action="approved_items", analysis_id=91) ``` Classes are `ADD`, `UPDATE`, `ARCHIVE`, `RERUN`. **Branch on `analysisClass` before acting**: a `RERUN` item implies no test-case change at all, so treating the approved feed as a to-do list of edits will have you rewriting cases that only needed running. `approved_items` excludes items already applied, so the feed empties as you work. ``` manage_pr_impact(action="set_item_status", analysis_id=91, item_id=5, status="APPROVED") manage_pr_impact(action="set_item_status", analysis_id=91, item_id=7, status="APPLIED", created_test_case_id=28114) ``` `AUTO_APPROVED` is set only by the platform and is always rejected here. Once an analysis is finalized, every transition is refused. --- ## 5. Run the impacted cases ``` manage_pr_impact(action="run", analysis_id=91, environment_id=627) ``` Give an environment unless the org has `targetEnvId` configured. Then poll `get_analysis` and read the run line from the summary. Whether runs happen automatically is one setting: | `testRunTiming` | Behaviour | |---|---| | `OFF` (default) | Analyse and report. Runs nothing. | | `POST_MERGE` | Run after merge, once `deploymentWaitMinutes` elapses — or earlier if a deployment webhook arrives. | | `PRE_MERGE` | Run against the PR's own build, matched on the head commit. | **Before recommending `PRE_MERGE`, read the preflight.** It is a static advisory scan asking "would a run against this environment actually exercise the PR's build?", and its findings are ordered by how badly they mislead: | Finding | Why it matters | |---|---| | `FIXED_URL` | A literal address in a step. The case **passes**, having tested the old deployment. Worst of the lot. | | `MISSING_ENV_KEY` | Reads an env key the target environment lacks. Fails every case for a reason unrelated to the PR. | | `URL_IN_AI_STEP` | An address inside AI-agent prose. Passes — the agent reads the destination from the prose. | | `URL_IN_STEP_GROUP_ARG` | A literal address passed into a step group. Passes, and invisible in both case and group. | | `GLOBAL_IN_URL` | A `{{global}}` used as an address. Passes — globals have no environment dimension. | | `UNDECLARED_CASE_VARIABLE` | A `${name}` only the case can supply. Fails, unrelated to the PR. | A clean preflight is what makes a pre-merge run trustworthy. `FIXED_URL` findings mean a green run proves nothing. --- ## 6. Settings ``` manage_pr_impact(action="get_settings") manage_pr_impact(action="update_settings", changes={"testRunTiming": "PRE_MERGE", "deploymentWaitMinutes": 15}) ``` Settings are an **org-wide singleton** — not per-workspace. Changing them changes them for every workspace in the org; say so before you do it. The underlying `PUT` is a **full replace** whose validator wants nearly every field non-null, so `update_settings` reads the current object, merges your keys and writes the result back. Never hand-roll a partial PUT: it either 400s or blanks the rest of the configuration. `deploymentWaitMinutes` is bounded 0–720. `deploymentEnvironment` blank means "any environment". **Do not describe these as working controls:** `preventSelfApproval`, `requiredApproversQuorum`, `archiveApproversQuorum` and `autoDemoteOnRejections` are stored and returned but **not enforced** anywhere in the platform. Telling a user their approval quorum is protecting them would be wrong. --- ## 7. Where results get reported You do not post the PR comment or the check yourself — the platform does, driven by `postPrComment` and `postGhCheck`. Run updates re-render the *one* analysis comment with a run column rather than posting a second comment. For Slack, point the workspace at a channel: ``` manage_notifications(action="channels") # find the id manage_notifications(action="set", channel_id="C0BS12345", channel_name="qa-alerts") ``` `channel_id` is Slack's id, never a `#name`. The choice is stored per **workspace**, not per workspace version, so two versions share one channel. A legacy pasted-token Slack integration reports `connected: false` here even though it appears in `manage_integrations(action="list")` — it needs the one-click connection. --- ## 8. When it is not working | Symptom | Look at | |---|---| | "Connected" but no analyses | `registry` — the provider-side install may be gone | | Analysis start fails | The repo has no workspace bound; re-register it | | Every case fails for an unrelated reason | Preflight `MISSING_ENV_KEY` / `UNDECLARED_CASE_VARIABLE` | | Runs pass but prove nothing | Preflight `FIXED_URL` — the old deployment was tested | | Post-merge runs always wait the full timer | Deployment events are not arriving (the GitHub App needs Deployments: Read plus the deployment-status event) | | Nothing runs at all | `testRunTiming` is `OFF`, which is the default | | A run "completed" with nothing to show | `passed: 0, failed: 0` — nothing executed |