PROJECT STATUSPrototype / Feasibility Bench
CORE ECONOMIC STRATEGYDeterministic Fast-Path Replay
RECORDING RUNTIMEObserver + Playwright
EVALUATION MODELDOM Analysis + Judge Engine
01. THE PROBLEM
The Fragility and Cost Dilemma of Modern Web Testing
End-to-end browser automation suites have historically suffered from brittle locator strategies. Scripts written against traditional CSS selectors, transient test IDs, or dynamic XPath strings like div > button.btn-39x9 fail frequently when frontends mutate. Framework upgrades, responsive redesigns, and CSS-in-JS hashing routinely break suites without any underlying business-logic defect, imposing substantial maintenance overhead on QA and engineering teams.
The emerging alternative—delegating entire testing sessions to autonomous vision agents (such as raw BrowserUse or multi-modal LLM reasoning on every step)—introduces a severe economic and operational penalty:
- Prohibitive API Latency & Token Costs: Invoking vision-capable LLMs on every browser interaction incurs significant latency (1.5–4 seconds per click) and token expenditure, making regression suites with hundreds of test cases economically impractical.
- Non-Deterministic Flakiness: Pure probabilistic models can interpret similar screens inconsistently across runs, introducing stochastic variance into automated CI/CD pipelines.
- The Risk of Masking Real Application Defects: If an autonomous agent alters test flows whenever a target button fails to respond—without evaluating whether a real backend error or validation failure caused the issue—it risks silently bypassing bugs rather than catching them.
02. THE ORIGINAL VISION
Cost-Efficient, Self-Healing Hybrid Automation
IntentIQ was conceived to bridge the gap between brittle deterministic scripts and expensive autonomous agents. The central architectural hypothesis was straightforward: deterministic execution handles known workflows for free, while AI reasoning is reserved strictly for planning, recovery, and outcome evaluation.
THE INTENDED 7-STEP LIFECYCLE- Demonstration Recording: An engineer performs a target workflow once inside an instrumented Chromium browser session.
- Blueprint Compilation: An in-browser observer captures DOM interactions, mutation patterns, and ranked locator candidates into a reusable YAML blueprint.
- Deterministic Fast-Path Replay: Playwright executes the blueprint at native browser speed (<50ms per step) with zero token costs for routine regression runs.
- Selective AI Recovery Loop: When a recorded element changes or fails resolution, BrowserUse and ChatGPT inspect the rendered DOM and visual context, reconnect the target element, validate recovery, and update the blueprint.
- Outcome Evaluation: The system verifies completion conditions, DOM mutations, and network activity rather than relying solely on HTTP status codes.
- Defect vs. Automation Failure Discrimination: Distinguish locator drift from true application regressions, preventing the healing engine from mutating blueprints around genuine bugs.
- Evidence-Based Verdicts: Output structured diagnostic reports with root causes and actionable remediation advice.
03. ARCHITECTURE & IMPLEMENTATION STATUS
Subsystems: What Is Implemented vs. What Remains Architectural Vision
A static architectural investigation across the IntentIQ codebase established clear boundaries between verified working subsystems, partially wired modules, and aspirational designs.
IMPLEMENTED & VERIFIED IN CODEBASEVerified Active
- Shared Event Collector: Injected
intent_observer.js (417 lines) capturing clicks, inputs, network requests, and generating 10-tier candidate locators (placeholder, text match, name, id, test-id, aria, tag+class, xpath) with confidence scoring. - Live Playwright Recorder:
recorder.py opens Chromium on the host, tracks real user interactions, and streams structured DOM events. - Blueprint Synthesizer:
synthesizer.py isolates authentication paths into login.yaml, prunes noise, and synthesizes structured YAML blueprints. - Candidate Resolver & Executor:
executor.py iterates through ranked locator candidates in descending confidence with dynamic wait retries—executing without LLM intervention. - DOM Completion & Output Extraction:
completion.py and output_extractor.py monitor network-idle stability and extract structured HTML tables/cards. - Judge Engine:
judge/engine.py scores normalized outputs and execution errors against rubrics using Moonshot, Gemini Flash, or heuristic fallbacks to output structured verdicts. - macOS Native Dialog Watcher:
browser_runner.py runs background AppleScript daemons to automatically dismiss native OS-level dialogs (password save, location alerts).
PARTIALLY IMPLEMENTED / DISCONNECTED IN RUNTIMEBench Constrained
- Gated Fallback Routing: In
browser_runner.py, ENABLE_AI_FALLBACK = False was set during prototype test-bench runs, meaning locator failures return an error rather than falling back to autonomous BrowserUse browsing. - Hardcoded Test Target: Runtime execution logic in
browser_runner.py hardcodes app_name = "Browser_Agent", constraining quick execution outside the verified test harness. - Standalone AI Healer Module:
backend/recovery/healer.py defines an LLM-based single-element recovery prompt, but it is currently not invoked by the BlueprintExecutor runtime loop. - Execution Evolution Counter:
WorkflowEvolution.record_execution() is defined for tracking usage counts and triggering A/B competition, but is never invoked on live mission runs.
ARCHITECTURAL VISION & PLANNEDRoadmap
- Closed-Loop Blueprint Self-Updating: Seamless, automated patching of YAML blueprints on disk following successful AI-driven recovery without manual review.
- Automated Defect Isolation: Full automated attribution distinguishing frontend visual modifications from application server errors.
04. HOW IT WORKS · 3 TECHNICAL CHAPTERS
Step-by-Step Implementation with Authentic Video Evidence
CHAPTER 01 · RECORDING & BLUEPRINT SYNTHESIS
Observing Human Interaction to Compile Multi-Candidate Blueprints
When a recording session starts, Playwright launches a Chromium instance and injects intent_observer.js. The observer listens to click, input, and mutation events, calculating multi-tier candidate locators with confidence scores for every interacted node. The synthesizer separates authentication flows into login.yaml and functional tasks into target blueprints.
Chapter 1 Evidence: Recording user workflows, capturing 10-tier candidate locators, and synthesizing structured YAML blueprints.
Actual Blueprint Schema Example (Create_Mission.yaml generated by synthesizer)
workflow: manual_training_session_4414
start_url: https://browseragentai.aisuites.io/new-mission
quality_score: 80
execution_ready: true
fields:
mission_title:
required: false
candidates:
- type: placeholder
value: "[placeholder='e.g., Lead Gen — Dentists Miami']"
confidence: 0.9
source: observed
- type: text
value: "\"e.g., Lead Gen — Dentists Miami\""
confidence: 0.9
source: generated
- type: xpath
value: /*[@id="root"]/div[1]/main[1]/div[1]/div[1]/div[2]/input[1]
confidence: 0.8
source: observed
- type: css_class
value: input._inputBase_197v1_1
confidence: 0.25
source: observedCHAPTER 02 · DETERMINISTIC EXECUTION & AI RECOVERY
Deterministic Fast-Path Replay and Selective AI Recovery
Execution is designed around an economic hierarchy:
- Deterministic Fast-Path: The
CandidateResolver iterates through candidate locators in descending order of confidence (placeholder > text > xpath > css_class) with dynamic retries up to 5 seconds. This runs at native Playwright speed (<50ms per step) with zero token costs. - Selective AI Recovery Vision: If all deterministic candidates fail, the intended architecture invokes BrowserUse and ChatGPT/LLM. The agent inspects the accessibility tree and visual state, re-identifies the target element, executes the action, and queues the candidate for blueprint updates.
- Test-Bench Prototype Status: In the current prototype codebase,
CandidateResolver handles the multi-tier deterministic execution, while the automated fallback router was set to ENABLE_AI_FALLBACK = False during initial bench evaluation to isolate deterministic locator robustness.
CHAPTER 03 · OUTCOME EVALUATION & DEFECT DISCRIMINATION
The Judge Engine: Distinguishing Automation Failures from Real Bugs
A critical failure mode in naive self-healing systems is the tendency to bypass real product defects. If an application form fails because an API returned an unhandled 500 error or a red error notification appeared, self-healing must not modify the test to click alternative elements.
IntentIQ addresses this through post-execution evaluation. The DomCompletionDetector validates state stability and network idle, while the OutputExtractor parses rendered structured components. Finally, the JudgeEngine scores the run against rubrics to generate an evidence-based verdict.
Chapter 3 Evidence: Post-execution evaluation, log and DOM inspection, and structured verdict reporting.
Judge Engine Schema (backend/judge/engine.py)
{
"verdict": "FAIL_APPLICATION_BUG", // PASS | FAIL_TASK | FAIL_APPLICATION_BUG | FAIL_UI | FAIL_AUTH | FAIL_TIMEOUT
"summary": "Form submission failed due to unhandled server response in borrower payload",
"root_cause": "Endpoint /api/v1/missions returned HTTP 500; submit button disabled after click",
"evidence": [
"DOM error notification present: 'Internal server error'",
"Network log: POST /api/v1/missions returned status 500"
],
"severity": "CRITICAL",
"actionable_fixes": [
"Check backend mission controller for null pointer handling in borrower payload"
]
}05. REAL ENGINEERING TRADE-OFFS & LIMITATIONS
Engineering Decisions, Technical Debt, and Architectural Boundaries
Cost-Efficiency vs. Autonomous Adaptability
Executing via Playwright and multi-tier locators costs $0 in tokens and runs in milliseconds, but cannot dynamically adapt if an entire workflow changes structure. Conversely, full vision agents adapt fluidly to structural redesigns but are too slow and expensive for high-frequency CI pipelines. IntentIQ opts for a hybrid hierarchy, accepting that recovery should only fire when deterministic confidence is exhausted.
Healing vs. Defect Masking Paradox
In QA automation, adapting a test to make it pass is catastrophic if the failure was caused by a real defect. IntentIQ's evaluation layer must enforce strict boundaries: if error banners, network 5xx responses, or missing assertions are detected, the system records a FAIL_APPLICATION_BUG and halts rather than rewriting the blueprint.
Single-Session Concurrency Constraints
The current prototype uses a process-level singleton BrowserManager that maintains an active browser session. While suitable for single-user recording and local replay testing, concurrent execution requires isolated browser contexts or containerized runners to prevent cross-test session collisions.
Canvas & Accessibility Boundaries
IntentIQ relies on inspectable DOM nodes, semantic roles, and accessibility trees. Applications rendered heavily within HTML5 Canvas or WebGL contexts lack DOM representations for intent_observer.js to rank, which would require fallback to visual coordinate clicking.
06. FUTURE ROADMAP
Next Architectural Phases & Deepened Evaluation
Addressing the gaps uncovered during prototype benchmarking defines the engineering priorities for IntentIQ:
PLANNED CAPABILITIES:- 1. Deep Runtime Telemetry & Interaction Auditing:Capture browser console errors (
console.error), uncaught JavaScript runtime exceptions, and element interaction failures directly into the execution trace to provide unambiguous evidence for failure classification. - 2. API Exposure & Security Checks:Inspect client-side network request payloads during execution to detect leaked API tokens, unauthorized endpoints, insecure cookies, and improper CORS exposure during test flows.
- 3. Robust Root-Cause Diagnosis & Evaluation:Connect the disconnected
healer.py module to the execution exception pipeline, wire the dormant WorkflowEvolution execution counters, and generate reproducible bug reports with developer remediation hints.