# Browser comparison for UI Palooza

**Recommendation: keep Playwright + Chrome as the reference renderer and evaluator. Pilot SleepyHollow as an optional managed preview tool and WebKit check. Do not use this Obscura release for benchmark screenshots or authoritative interaction evaluation.**

This is a completed local experiment, not an assessment based only on project READMEs. [Open the visual report](index.html) to inspect all three renderings, compare overlays, and inspect the saved screenshots. The generated sites remain local.

## Scope and controls

Twelve existing artifacts cover City journal, Forma frontend, Team inbox, and Weekend discovery. Each brief contributes one Whacky design, one GPT-5.6 Sol native baseline, and one GLM-5.3 Taste-skill design. The Whacky examples span GLM-5.3, DeepSeek, GLM-5.3 Flash, and Muse. This is a deliberately varied convenience sample, not a random or model-balanced study.

The same originals were captured at 1440 × 1000 and 390 × 844, at 1× screenshot scale. There are 72 primary screenshots and 24 extra repeatability captures: 96 total. Sources were hashed before capture and verified afterward. No model was invoked, no saved design was edited, and no production evaluator or run status was changed.

All engines used a shared local proxy, the same readiness probe (items present, fonts ready, minimum one second after load, with a 12-second fallback), CSS animations/transitions disabled, light theme, ephemeral browser state, and a same-origin CSP. The CSP is supplied equally, but its enforcement is not independently audited across engines. These are narrow desktop-engine viewports, not touch-device emulation. The same axe-core 4.10.2 script and rule tags were used for the accessibility smoke comparison.

## Measured results

| Measurement                                                  |        Chrome |       Obscura |  SleepyHollow |
| ------------------------------------------------------------ | ------------: | ------------: | ------------: |
| Successful PNG capture commands                              |         32/32 |         32/32 |         32/32 |
| Median primary end-to-end capture                            |       1.626 s |       1.292 s |       1.294 s |
| Primary capture range                                        | 1.592–2.053 s | 1.231–1.474 s | 1.230–1.454 s |
| Material visual differences against Chrome, assistant review |     Reference |   20/24 views |    0/24 views |
| Median pixels differing by >24 in any RGB channel            |     Reference |        15.16% |         5.18% |
| Search/open/close outcome assertions                         |         35/36 |         28/36 |         35/36 |
| Horizontal overflow reported                                 |    1/24 views |    6/24 views |    1/24 views |
| axe execution completed                                      |         12/12 |         12/12 |         12/12 |

PNG success only means the command returned successfully and wrote an image of the expected dimensions. Obscura's most visibly incorrect screenshots still returned success. Pixel differences include rasterization, native controls, typography, and alignment; they are neither design scores nor defect percentages. The qualitative material-difference count comes from one assistant inspecting every desktop/mobile triptych, with full-size inspection of the first Whacky case. It is not a blinded human panel.

All repeated Whacky desktop images had identical decoded RGB pixels within each engine: four artifacts × three engines, each captured three times. That demonstrates short-run repeatability in this environment, not stability across OS versions or machines.

Timing includes fresh browser/CLI startup, navigation, settling, and screenshot production. Models were still generating in the background, and part of capture overlapped interaction probes. Engine order rotated between cases. This is an exploratory end-to-end measurement, not a cold-cache or load-saturation benchmark. No statistically robust performance ranking or aggregate memory claim is made. WebKit's system-managed helper processes make simple parent-process RSS misleading.

## Visual findings

**SleepyHollow preserved the overall composition and visible content in the inspected viewports.** Typography and image rasterization differ, and native selects are often shorter. City Hours has different mobile form sizing; Pigeonhole's shorter select shifts the cards upward. These differences matter for pixel baselines, even though none met this review's material-difference threshold. A zero count is limited to this corpus and these viewport captures.

**Obscura materially changes designs we are trying to judge.** Examples:

- **Sidewalk / City journal Whacky (case-01):** text disappears from the doors, their proportions change, and background/imagery treatment differs on desktop and mobile.
- **Side Street / City journal Taste (case-03):** grayscale imagery becomes color; mobile headings and descriptions fail to wrap within the viewport.
- **Forma Taste (case-06):** hero proportions and image crop change substantially; the mobile headline overflows.
- **Pigeonhole / Team inbox Whacky (case-07):** search/select controls become dark and lose contrast.
- **Elsewhere / Weekend native baseline (case-11):** map labels disappear and paths/shapes differ.

These would bias the benchmark: a model could be penalized or rewarded for the replacement renderer's behavior. Obscura was closer on some conventional layouts, particularly City Hours and Docket, but that is insufficient for the creative designs this project intentionally explores.

## Interaction and accessibility findings

The 36 outcome checks per engine are a smoke subset: searching for unmatched text must yield zero visible items and an empty state; opening must show the detail; closing must hide it. They do not cover all production checks, detail identity/body correctness, focus restoration, saved-state persistence, or mobile touch interactions. An open check alone is weak if an engine already misreports hidden content as visible; therefore it must not be interpreted as complete interaction parity.

All three engines expose the same existing search problem in **Forma Taste (case-06)**: the empty state appears but six item boxes remain visible according to the shared geometry predicate. This is not counted as a browser regression. Chrome and SleepyHollow otherwise agree on these assertions. Obscura adds seven failed close/hidden-state checks (cases 02, 03, 04, 06, 07, 08, 11); these establish evaluator incompatibility, without attributing every failure to a specific renderer or DOM implementation detail.

Chrome and SleepyHollow produced identical summarized axe results on all twelve cases: violation IDs, severity, affected-node counts, and incomplete-rule IDs. Obscura's summaries differ, including additional scrollability violations and missing contrast findings. Execution success is not audit equivalence. axe is a partial automated audit, and incomplete results are retained.

A deliberately constructed canary gives stronger evidence about tool semantics:

| Canary                  | Chrome / Playwright         | Obscura / Playwright                                 | SleepyHollow CLI                                                    |
| ----------------------- | --------------------------- | ---------------------------------------------------- | ------------------------------------------------------------------- |
| Normal selector click   | Activates button            | Activates button                                     | Activates button through untrusted synthetic event                  |
| Covered selector click  | Refuses; identifies overlay | Returns success, but covered button is not activated | Activates the covered button despite overlay                        |
| Disabled selector click | Refuses                     | Refuses                                              | Refuses                                                             |
| Enter on focused button | Activates button            | Command returns, button not activated                | No native CLI key action used; synthetic JS would not be equivalent |

SleepyHollow's **coordinate-click** path correctly hits the overlay and leaves the covered button untouched. A future adapter could explicitly verify `elementFromPoint` and use coordinate clicks. That mitigates this particular selector-click failure; it does not establish native keyboard or accessibility interaction fidelity.

The first SleepyHollow interaction pass also exposed an integration trap: one-shot `--fill` splits its argument at the first `=`, including an equals sign inside an attribute selector. The corrected adapter uses `input[data-testid]` and asserts the selected element is the search field and received the exact intended value. Final counts use the corrected pass; initial diagnostics remain locally under `experiments/browser-comparison/raw/`. Vendor code was not patched.

## Sandbox and operational fit

SleepyHollow compiled locally without installing a global package or browser. A basic screenshot succeeded under a temporary Seatbelt profile mirroring our runner's read/write/signal restrictions, with a copied binary and isolated HOME/TMPDIR. The PNG, raw output, and sandbox profile are retained locally; only the outcome is published. This narrows the earlier concern from its documentation: it works in this specific basic sandbox test. It does not prove every operation, named session, or actual model integration works.

SleepyHollow's ephemeral CLI is a promising fit for managed agent previews. Its system WebKit dependency ties results to macOS updates; pinning only the CLI commit is insufficient. It also needs a different adapter than the current Playwright pipeline. Its computed accessibility output should not be assumed to be a native assistive-technology audit.

Obscura connected through Playwright/CDP and completed all captures with little adapter work, but API compatibility did not imply matching rendering, hidden-state behavior, or keyboard interaction.

## Recommended next step

1. **Keep the current Chrome evaluator and screenshot baseline.** Pin a browser build when formalizing the next benchmark revision. Fixing export and lifecycle issues remains separate from choosing a renderer.
2. **If we add model-visible previews, pilot a preinstalled SleepyHollow wrapper in a new, explicitly labeled experiment condition.** Give every compared model the same access, bounded operations, and documented semantics. Prefer coordinate/hit-tested actions and check outcomes. Do not silently add it to existing vanilla runs.
3. **Use SleepyHollow as additional WebKit coverage if useful.** Keep its screenshots and audit results labeled by engine rather than mixing them into Chrome baselines. More testing is needed for full-page capture, scrolling/sticky content, touch behavior, complex keyboard flows, and saturation at our six-worker load.
4. **Do not migrate to Obscura v0.2.2 for this use case.** Revisit a pinned newer version only after it reproduces the problematic cases here. Its smaller runtime may suit other automation, but visual fidelity is the decisive requirement for a taste benchmark.

The measured time saving is roughly 0.33 seconds per primary capture. It does not explain or resolve a model spending its 30-minute generation budget installing/debugging browsers. The architectural improvement is controlled, consistently available tooling, not a wholesale renderer swap.

## Reproduction and evidence

Instructions and scripts: `experiments/browser-comparison/README.md` in the repository. Public design cases: [corpus.json](corpus.json). Exact run IDs and source hashes are retained locally. Browser versions and general platform: [environment.json](environment.json). Measurements: [results.json](results.json), [analysis.json](analysis.json), [interactions.json](interactions.json), [extra-checks.json](extra-checks.json). Qualitative assessment: [review.json](review.json). Validation: [validation.json](validation.json).

Tested builds: [Obscura v0.2.2](https://github.com/h4ckf0r0day/obscura/releases/tag/v0.2.2) and [SleepyHollow commit 0e721864c930](https://github.com/bensyverson/sleepyhollow/tree/0e721864c930d6cf20475474e33c16ddd440de4b). Conclusions apply to these builds, this Mac, and this corpus; project main branches may have newer behavior.