KQ-00Engineering reference

PRAuto & ArcQA

Automated PR pipeline controls and the application QA factory. Current status, what's in review, and what's next.

Internal - not indexed

KQ-01Status overview

System

PRAuto

12 live 3 in progress 0 not built

System

ArcQA

1 live 0 in PR 5 not started

KQ-02PRAuto

PR Automation Pipeline

GitHub Actions that automate the pull request lifecycle. Architecture pivot: K.33–K.36 were originally planned as n8n webhook factories but were implemented as native GitHub Actions, removing the n8n import dependency. End state: PRs open, pass review, and merge autonomously.

Runner Infrastructure All workflows run on self-hosted ArcHelm runners (archelm-1 through archelm-4, 10.20.10.29), running as systemd user services on the Arch Linux K8s control plane. Migrated from ubuntu-latest GitHub-hosted runners Aug 2026.

Service management: systemctl --user restart github-runner-{1..4}. Services are defined per-runner at ~/.config/systemd/user/github-runner-N.service.

Pre-job hook: ACTIONS_RUNNER_HOOK_JOB_STARTED=/home/archelm/fix-perms.sh is set in each runner's .env file. The hook runs before every job and uses sudo to fix root-owned files in _work/ left over from Docker-based jobs. Without it, subsequent actions/checkout steps fail with EACCES: permission denied, unlink.

Known issue: Jobs that run Docker containers (e.g. Semgrep) write root-owned directories into _work/. The fix-perms hook mitigates this pre-job, but if a runner gets into a bad state mid-run the workspace may need manual recovery.

Workspace recovery: sudo rm -rf /home/archelm/actions-runners/runner-N/_work/ followed by systemctl --user restart github-runner-N. The runner rebuilds its workspace and tool cache on the next job (first run after a nuke is slower due to Node.js tool cache rebuild).

Live in CI

auto-pr.yml

Auto-Draft PR on Push

Any branch push that isn't main creates a draft PR against main, derives a title from the branch name (feat/surface-pm-flows → feat: surface pm flows), and pre-fills the PR template body. No manual PR creation needed.

Live

K.17 · branch-age.yml

Branch Age Warning

Weekly cron (Monday 9am UTC). Scans open PRs and posts a warning comment on any open more than 7 days. Nudges toward short-lived branches without hard enforcement.

Live

K.29 · GitHub Settings

Copilot Auto-Reviewer

GitHub Copilot posts inline review comments on every PR diff before human review begins. Ruleset "Copilot review for default branch" is active. Complements the Claude-based AI reviewer (K.15), which runs a deeper security pass and triggers auto-merge on approval.

Live

Live in CI - core review pipeline

K.15 · 02-ai-review.yml · covers K.34

AI PR Reviewer + Verdict Status

Runs claude-sonnet-4-6 on every non-draft, non-bot PR diff. Posts inline comments and a VERDICT summary. Publishes ai-review/verdict commit status. Security patterns always flagged: 0.0.0.0 binding, bare except, SQL/shell string concatenation, hardcoded credentials, eval() / exec() / pickle.loads(). Verdict drives the automerge controller (04-automerge.yml) and the auto-fixer (05-auto-fix.yml). Supersedes K.34.

Live

K.12 · lint-guard.yml

Lint Suppression Gate

Diffs the PR and greps added lines for # noqa and # type: ignore. Posts a comment listing each suppression and fails CI if any are introduced. Catches AI-generated code that silences type and lint errors instead of fixing the underlying design problem.

Live

K.26 · security.yml

Dependency + SAST Scanning

pip-audit scans Python dependencies against the OSV/PyPI advisory database and fails on CVSS ≥ 7.0; bandit SAST at HIGH/MEDIUM severity catches what ruff ignores: eval(), exec(), os.system(), subprocess(shell=True), pickle.loads(), path traversal in string concatenation. Excludes node_modules/.next.

Live

K.33 · pr-contract.yml

PR Contract Validator + Tier Labeling

Validates every PR against the contract spec: max 700 lines changed, max 15 files, scope/slug branch format, conventional commit title prefix. Also enforces the K.52 tier system: labels each PR with its tier on open, posts a structured pass/fail comment listing specific violations, and fails CI. Built as GitHub Actions instead of the originally planned n8n webhook factory.

Live

K.35 · wip-check.yml

WIP Limiter

Triggers on every PR open event. Checks open PR count via GitHub API and fails CI with a comment if it exceeds 15. Prevents queue pile-up as agent factories begin opening PRs autonomously. Requires DISCORD_SCOUT_QUEUE_WEBHOOK secret for optional #scout-queue notification.

Live

K.36 · stale-sweep.yml

Stale PR Sweeper

Daily cron (9am CDT) via actions/github-script. Labels PRs stale at 48 hours; closes stale-labeled PRs at 96 hours. Prevents abandoned agent branches from occupying the queue. Requires the stale label to exist in the GitHub repo before the first run.

Live

auto-fix.yml · mechanical ruff

Ruff Auto-Fix

Separate from the AI fixer loop (05-auto-fix.yml). On ruff failure, checks out the PR branch, runs ruff check --fix && ruff format, commits as chore: auto-fix ruff violations, and pushes back to re-trigger CI. Handles the purely mechanical lint case without AI involvement.

Live

Live in CI - advanced pipeline

03-risk-classifier.yml

Risk Classifier

Labels PRs risk/low, risk/medium, or risk/high based on path patterns in .github/risk-paths.yml plus size (>400 lines escalates one tier). Uses API fetch rather than sparse-checkout. Labels drive the automerge controller and SLO tracking.

Live

04-automerge.yml

Auto-Merge Controller

Arms or disarms GitHub native auto-merge by tier: risk/low → armed immediately; risk/medium → armed but gated on ai-review/verdict passing; risk/high → auto-merge disabled. Requires "Allow auto-merge" enabled in GitHub repo settings to take effect (pending GitHub configuration).

Live

K.51 · 05-auto-fix.yml

AI Auto-Fixer Loop

When ai-review returns REQUEST_CHANGES on a risk/low or risk/medium PR, a fixer agent implements the findings and commits back to the branch. Maximum 2 iterations, then escalates to NEEDS_HUMAN. Never touches guarded CODEOWNERS paths. Handles the mechanical fix case without human interruption.

Live

07-stale-rebase.yml

Stale Branch Rebase

Detects branches that have fallen behind main and initiates automated rebase. Companion to stale-sweep.yml: handles the rebase case before the sweeper closes the PR.

Live

08-review-slo-digest.yml

SLO Digest + Escalation

Hourly SLO check escalates overdue risk/high PRs. Daily merge-health digest posted as a GitHub issue. Keeps long-lived human-required PRs from going dark in the queue.

Live

K.52 · 10-evidence-gate.yml

Evidence Gate

Agent-authored PRs must carry an EVIDENCE block in the PR body. Human PRs are exempt. Publishes evidence/bundle commit status so branch protection can require it for bot-opened PRs. Tier 1 = auto-merge eligible; Tier 2 = human required for CODEOWNERS paths or >400 lines.

Live

12-suggestion-applier.yml

Suggestion Applier

Applies GitHub review suggestion blocks as file patches and commits them back to the branch. Never touches guarded CODEOWNERS paths. Allows reviewers to suggest code changes that apply automatically without author intervention.

Live

Pending ArcFlow build

N.14 · factories/shared/qa-review-loop/

QA Review Loop (shared factory)

Callable by any agent factory as a shared quality gate. Two-round architecture: local analysis model scores output against a per-service rubric (0.0–1.0); if below threshold, claude-haiku generates a targeted revision; second eval pass decides pass or Discord escalation. Rubrics already authored for Aria, Scout, Dax, Odo, and Louis. Platform: ArcFlow (replacing n8n). Existing workflow.json is in n8n format and serves as the logic spec for the ArcFlow build. Requires LITELLM_BASE_URL, LITELLM_API_KEY, DISCORD_BOT_TOKEN, per-service channel vars.

Pending ArcFlow

Live - GitHub issue pipeline

K.49

Roadmap-to-GitHub Issue Sync

scripts/sync_roadmap_to_github.py - parses docs/roadmap.md and creates or updates GitHub issues via gh CLI: one issue per task, labeled by phase (phase:K, phase:QA, etc.), status emoji maps to open/closed. Idempotent: updates existing issues rather than duplicating. Run after any roadmap edit alongside sync_roadmap_to_neo4j.py so GitHub and Neo4j stay in sync.

Live

K.50

PRAuto Failure Issue Creation

Extends the GitHub Actions workflows to auto-create tracked GitHub issues on key failure conditions: ai-review.yml request_changes verdict → issue labeled ai-review,needs-fix assigned to PR author; security.yml failure → issue labeled security; stale-sweep.yml close → issue labeled stale,needs-reopen summarizing dropped work. Prevents failures from disappearing into CI logs with no tracking artifact.

Live

Not yet built

K.13

Phantom Code Detection

Weekly CI job using Python AST scan to flag functions defined in 3 or more files with near-identical signatures. AI-generated code batches consistently produce duplicate utility functions across scripts/, litellm/hooks/, and agent files.

Not built

KQ-03ArcQA

Application QA Factory

A four-layer automated testing harness for the Surface application. Owner: Vera. The goal is to remove the founder from the manual QA loop entirely - no code merges without passing scripted tests; no tenant onboards without all four layers green.

L1 · Scripted Regression

Playwright Browser Tests

Full route manifest coverage with ARIA snapshots. Every production bug gets a regression test before the fix. No test imports a tool directly - everything goes through the harness interface.

Blocks merges at QA.A1

L2 · API Invariants

OpenAPI + Schemathesis

OpenAPI 3.x spec drives Schemathesis property-based tests - schema conformance, response contract validation, and edge-case generation at zero marginal cost per new endpoint.

Blocks merges at QA.A1

L3 · Exploratory

Agent-Driven QA Sessions

Playwright MCP + local ArcRig inference. Agent explores the app in bounded weekly sessions, surfacing anomalies the scripted suite won't find. Includes fault simulation.

Weekly from QA.A3

L4 · Multi-Tenant

Choreographed Isolation

11 scenarios: 6 flow family + 5 isolation family (data never crosses tenant boundaries). No override path. Gate for any external custodied tenant.

Blocks at QA.A2 · no override

Gate sequence

QA.A0 · e2e.yml

Scaffold + Top 20 Routes

Playwright configured in the Surface app (playwright.config.ts). Full 32-route manifest defined with auth/dynamic flags. Two smoke suites: unauthenticated tests run fully automated (login page renders, protected routes redirect to /login); authenticated tests skip gracefully when E2E_SUPABASE_TOKEN is absent. CI workflow (.github/workflows/e2e.yml) triggers on surface/** changes, posts JUnit summary as PR comment. Authenticated coverage activates at QA.A1 once secrets are configured.

Live

QA.A1

Full Coverage + L2 Blocking

100% route manifest covered by L1 tests. OpenAPI spec authored. L2 business-invariant suite live. Both layers block PR merges. Requires E2E_SUPABASE_TOKEN, NEXT_PUBLIC_SUPABASE_URL, and NEXT_PUBLIC_SUPABASE_ANON_KEY GitHub secrets. Gates on QA.A0.

Not started

QA.A2

Two-Tenant Choreographed Harness

L4 flow family (6 scenarios) and isolation family (5 scenarios) live and blocking. Isolation harness configured with no override path. Minimum gate for QA.A4 (first external custodied tenant). Gates on QA.A1.

Not started

QA.A3

Exploratory + Fault Suite

L3 agent-driven exploratory QA running weekly on local ArcRig inference. Fault simulation suite covering connection drops, slow responses, and malformed payloads. Requires ArcRig inference routable from the test environment. Gates on QA.A2.

Not started

QA.A4

First External Custodied Tenant

QA harness gates first external tenant onboarding. All four layers must be green. Gates on QA.A2 minimum; QA.A3 strongly recommended first.

Not started

QA.A5

ArcQA-to-GitHub Issue Pipeline

When any Playwright layer (L1–L4) fails in CI, auto-create a GitHub issue: title = QA: <test name> failed on <branch>, body = error output + Playwright trace link + CI run URL, labels = arcqa + layer label (smoke / invariant / exploratory / isolation), assignee = PR author. Tier classification follows K.52: L1 smoke failures are Tier 2 (human review required before merge); L2+ failures are Tier 3 (Charles reviews). Routes through N.14 ArcFlow QA Review Loop for triage and severity scoring once ArcFlow is operational. Script: scripts/qa_issue_reporter.py called from e2e.yml. Gate: QA.A0 merged + K.50 live.

Not started

KQ-04Action items

What's next

The pipeline is fully built. The remaining blockers are two GitHub repo settings and two platform builds.

Charles / GH Settings Enable branch protection ruleset on main. GitHub Settings → Rulesets → add required checks: lint, test, gitleaks, ai-review/verdict, evidence/bundle. Zero required human approvals for Tier 1. CODEOWNERS routes Tier 2+ paths to Charles automatically. Without this, the full merge gate is not enforced.

Charles / GH Settings Enable auto-merge in repo settings. GitHub Settings → General → "Allow auto-merge". The 04-automerge.yml workflow is live and arms native auto-merge by risk tier, but the feature must be enabled at the repo level before it can take effect on any PR.

Build - QA.A5 QA.A5: Playwright failure → issue pipeline. When any Playwright layer (L1–L4) fails in CI, auto-create a GitHub issue via scripts/qa_issue_reporter.py called from e2e.yml. Title = QA: <test name> failed on <branch>; body = error output + trace link + CI run URL; labels = arcqa + layer label; assignee = PR author. Tier classification follows K.52. Gate: QA.A0 live (done) + K.50 live (done).

ArcFlow build Author N.14 in ArcFlow (qa-review-loop). Existing workflow.json is in n8n format and serves as the logic spec. Two-round architecture: local analysis model scores output against a per-service rubric; if below threshold, claude-haiku generates a targeted revision; second eval pass decides pass or Discord escalation. Once live, QA.A5 routes Playwright failures through it for severity scoring and triage. Requires LITELLM_BASE_URL, LITELLM_API_KEY, DISCORD_BOT_TOKEN, and per-service channel vars.