FrankenBench

The initial milestone H harness runs the same versioned cases through selected provider and router arms:

python scripts/frankenbench.py --arms http,router --rounds 1 --output results/frankenbench/run

The manifest is scripts/frankenbench-au.json. Cases require content assertions; a transport success with missing or mismatched content is an assertion failure. Reports retain receipts, verification outcomes, latency and reported software costs. Missing token measurements remain null. Release gates G1–G6 remain unmeasured until the complete specification comparisons exist.

Manifest 0.2 contains two Australian structured catalogue discovery cases and two exact detail/gallery cases. Category/page identity, exact listing IDs and decoding of every declared gallery URL have explicit assertions. Broader site classes, independent gallery completeness and current-state checks, all installed providers, a browser snapshot baseline, route-memory learning and three-client equivalence remain pending. Repeated router arms currently start fresh runtimes; they do not prove learning. FRANKENBENCH_INITIAL_SNAPSHOT.json records the initial live comparison.

Before issuing requests, the harness validates the full manifest, unique safe case IDs, supported assertion operators, required comparison values and each arm’s effective WebPolicy. Invalid definitions fail before acquisition. Boolean assertions distinguish true from numeric 1; missing fields and invalid list indices fail checks. Negative, non-finite and boolean cost or latency values remain unmeasured rather than entering comparison metrics.

Reports separate workload classes so catalogue success cannot hide detail/gallery failures. gallery_decoded requires every declared URL to have a decoded image result; empty galleries fail. This verifies declared coverage, not independently complete photos or current stock. FRANKENBENCH_AU_DETAIL_GALLERY_SNAPSHOT.json records the first manifest-0.2 comparison: all four cases passed through HTTP and router arms. Release gates remain unmeasured.

Manifest 0.3 declares discovery cases for all ten public sites in the Australian scope. A coverage matrix lists pending discovery, pagination, detail, gallery and state workloads per site. Facebook Marketplace is explicitly identity required and has no anonymous execution case. Declared cases are coverage, not evidence of reliable access. Current-state verification remains pending on every site. Full-spec classes and release-gate comparisons still require work.

FRANKENBENCH_ALL_PUBLIC_HTTP_SNAPSHOT.json records the first HTTP-only expanded run: seven of twelve assertion-defined cases passed. The failures remain in the denominator and receipts. This report does not compare every provider or prove router superiority.

The first expanded manifest-0.3 router arm passed all twelve cases; the HTTP arm passed seven. FRANKENBENCH_HTTP_ROUTER_COMPARISON.json retains both arms and receipts, checked against the same case IDs, classes and assertion shapes. Gumtree, eBay AU, Trading Post, Depop and Carsales discovery passed in the router arm after failing the HTTP arm. This is one sequential round, not a confidence estimate, a best-provider comparison or evidence that release gate G1 passes. All gates remain unmeasured. All-site pagination and state truth remain pending.

Manifest 0.4 adds exact Grays and Lloyds detail/gallery cases. Both passed HTTP and router arms, including exact listing IDs and every declared photo decoded. The dated evidence is FRANKENBENCH_AUCTION_GALLERY_SNAPSHOT.json. This focused comparison excludes the other twelve cases; it is not a full manifest-0.4 run or proof of independently complete galleries or current state.

Use --cases grays-exact-gallery,lloyds-exact-gallery for a focused canary. Unknown or duplicate IDs fail before acquisition. Reports expose selected and excluded case IDs alongside declared site coverage. Omit --cases to run all manifest cases.

Manifest 0.5 adds two-page Cash Converters and ALLBIDS pagination cases. The harness follows returned same-origin continuation URLs and retains both receipts. Continuations have their own category/page assertions; listing IDs are compared for new candidates, overlaps and no progress. Failed, stale or repeated pages cannot pass a distinct-page assertion. Both page receipts contribute to reported workload latency and cost; incomplete measurements remain null. The focused HTTP/router run passed all four cases, recorded in FRANKENBENCH_PAGINATION_SNAPSHOT.json. Other sites’ pagination, current-state truth and longitudinal reliability remain pending.

Manifest 0.6 adds Grays, Lloyds, Gumtree, eBay AU, Trading Post and Depop pagination. Cases may declare a separate continuation adapter (Depop’s public catalogue cursor), with its own assertions. Acquisition failures on either step retain typed failure codes rather than becoming generic assertion errors.

The focused router run passed Grays (40 new IDs), Lloyds (20) and Gumtree (24), each without overlap. eBay returned 57 new IDs and three overlaps; Depop returned 22 new IDs and two overlaps. Trading Post was blocked before case completion. These are three passes out of six under the distinct-page assertion, recorded in FRANKENBENCH_EXPANDED_PAGINATION_SNAPSHOT.json. Overlap counts and failures remain explicit. This run excludes other manifest cases and provider arms; current-state truth, independently complete galleries and reliability remain unverified. Vehicle pagination remains outside these continuation-URL cases.

Use --policy scripts/frankenbench-escalation-policy.json to benchmark an explicit public provider sequence with block escalation enabled. Policy-file values override case policy; freshness remains now, explicit baseline arms retain their single provider, and sequences apply to the router arm. Every observation records its effective policy. Invalid overrides fail preflight before acquisition. This configuration does not authorize identity fallback.

The 2026-10-02 eight-case pagination run with the escalation policy passed five: Cash Converters, ALLBIDS, Grays, Lloyds and Gumtree. Gumtree recovered through Scrapling after both HTTP and Camoufox were blocked on each page. eBay and Depop returned two repeated IDs each and failed strict distinct-page assertions. Trading Post’s initial page recovered through Camoufox, but its continuation was blocked through all three candidates. This contradicts treating two earlier Trading Post passes as reliability proof. The committed-code run is preserved in FRANKENBENCH_PAGINATION_ESCALATION_SNAPSHOT.json; challenge-token query parameters are omitted. Carsales, Bikesales and identity-bound Facebook pagination remain outside these eight declared cases.

An audit of the saved eBay/Depop overlap evidence is recorded in AU_PAGINATION_OVERLAP_AUDIT.json. eBay repeated cards contain obfuscated label text, but accessible Sponsored labels are not consistently available. Depop repeated records have no verified promotion marker. Neither site’s repeats are excluded as advertisements. The audit is a reanalysis of existing evidence, not a new live sample. Cross-page identity deduplication and independent promotion verification remain needed; strict failures stand.

Runtime.paginate(url, adapter, policy, continuation_adapter=...) traverses same-origin HTTPS continuation URLs up to WebPolicy.max_pages (default 2, caller-configurable). It returns unique listings by adapter listing ID, all raw page projections and overlap count. Original identity policy is retained on every page. Cycles, invalid continuations, failures and page-budget exhaustion remain explicit. Exhausting a site’s continuation does not prove catalogue completeness or current listing state. Strict benchmark assertions are unchanged.

A committed-code live traversal on 2026-10-02 returned 112 unique eBay listings from two 60-card pages, preserving eight repeated IDs; Depop returned 46 unique listings from two 24-card pages, preserving two repeats. Both stopped at the configured two-page budget with a continuation URL and unknown catalogue completeness. Receipts and policy are saved in AU_DEDUPLICATED_PAGINATION_LIVE.json. This verifies deduplication on one query per site, not full traversal or longitudinal reliability.

Manifest 0.7 adds two exact eBay detail/gallery cases (24 cases total). A fresh Camoufox canary ran both items twice: each round decoded all four and eight photos respectively, with canonical-item/carousel identity checks passing. AU_EBAY_CAROUSEL_LIVE.json preserves all four receipts and image results. These fixed-item samples verify the repair, while broader sampling, independent completeness truth, current state and longitudinal reliability remain pending.

--state-mode warm reuses each arm/case’s local state across rounds while keeping acquisitions fresh. Default cold uses independent round directories. Use a new output directory for a new experiment; warm mode can retain existing observations there. Reports record the mode and effective policy.

A warm Grays/Lloyds pagination canary on 2026-10-02 passed all eight observations over four rounds. Later receipts explicitly selected HTTP through route memory. FRANKENBENCH_WARM_ROUTE_CANARY.json preserves the results. HTTP was already the starting provider, so this verifies persistence/selection, not an improvement in latency or success, broad learning performance or G6 completion.

Manifest 0.8 expands to 29 cases with exact Gumtree, Trading Post and Depop detail/gallery projections plus Carsales/Bikesales detail cases. In the first fresh default-router run, the three public gallery cases and Bikesales detail passed; Carsales detail failed. Receipts are preserved in FRANKENBENCH_EXPANDED_DETAILS_SNAPSHOT.json. Vehicle detail cases require a gallery endpoint but do not count as gallery decoding. Gumtree gallery decoding checks declared URLs; it does not independently prove all photos were declared. Every site’s current-state truth and longitudinal coverage remain pending.

Manifest 0.9 adds separate Carsales/Bikesales gallery endpoints (31 cases total), requiring exact network IDs, declared endpoint completeness and decoding every photo URL. Both passed a fresh default-router sample on 2026-10-02; full receipts and photo counts are in FRANKENBENCH_VEHICLE_GALLERY_SNAPSHOT.json. Endpoint success does not erase the Carsales detail-navigation failure. Gallery truth remains endpoint-declared, and varied longitudinal coverage is incomplete.

Manifest 0.10 retains the failing default Carsales detail case and adds a separate explicit page-two case (32 cases total). A fresh page-two Camoufox canary reached the exact Mazda item by an observed category link and validated matching detail data. Evidence is in AU_CARSALES_PAGE2_DETAIL_CANARY.json. This identifies a navigation-scope gap: default first-page search does not reach this item. Automatic bounded category traversal remains unimplemented; the explicit pass does not replace the default failure.

The full manifest 0.10 cold default-router run on 2026-10-02 passed 27 of 32 cases. Gumtree discovery and pagination failed; eBay, Trading Post and Depop strict pagination failed. Every declared exact detail/gallery case passed, including repaired automatic Carsales navigation and both vehicle galleries. FRANKENBENCH_AU_FULL_CURRENT_SNAPSHOT.json preserves all observations. This 27/32 result describes one declared-case run, not a maturity percentage or all-site reliability. Facebook identity workloads, vehicle pagination, independent gallery/state truth and varied longitudinal samples remain missing. All release gates remain unmeasured.

Manifest 0.11 adds Carsales/Bikesales numbered-control pagination (34 cases total). navigation_continuation declares the next policy page; the harness compares initial/continuation IDs, checks page provenance and preserves the second receipt. Continuations retain identity/provider authority. Router arms now honor explicit providers declared by a case, while explicit baseline arms remain pinned to their provider; effective policies make this visible.

The first live numbered run passed Carsales and failed Bikesales, preserved in FRANKENBENCH_NUMBERED_VEHICLE_PAGINATION.json. Declared vehicle pagination is now covered by cases, but reliable live Bikesales continuation remains open. This does not replace earlier full-manifest results or complete release gates.

AU manifest v0.12 adds three exact ALLBIDS anonymous update cases. All 18 fresh acquisitions passed across HTTP and router arms, three lots and three rounds on 2026-10-02 (FRANKENBENCH_ALLBIDS_UPDATE_SNAPSHOT.json). These are state-evidence projections counted under detail coverage; independently verified listing-state coverage stays pending. A historical detail canary exposed the gallery adapter’s rejection of the site’s archive image host (AU_ALLBIDS_ARCHIVE_GALLERY_GAP.json). Rounds occurred in one short run and do not establish longitudinal reliability.

AU manifest v0.13 adds an exact ALLBIDS ended reference, including its archived photo. Four fresh acquisitions passed in Camoufox/router arms over two rounds (FRANKENBENCH_ALLBIDS_ENDED_SNAPSHOT.json). This declares one ended-state case; live, pending and sold state variation and longitudinal validation remain unfinished. It does not establish overall ALLBIDS listing-state reliability.

AU manifest v0.14 adds a live ALLBIDS reference alongside the ended reference. The live lot’s declared close is 2026-10-06; it must be refreshed when the auction ends rather than weakening its live-state assertion. Neither reference proves pending/sold coverage or reliability over multiple days.

All eight fresh live/ended acquisitions passed in Camoufox and router arms over two rounds on 2026-10-02, including declared gallery decoding. Results are in FRANKENBENCH_ALLBIDS_LIVE_ENDED_SNAPSHOT.json; the rounds were minutes apart.

AU manifest v0.15 adds the Cash Converters live camera reference. All eight fresh Cash-live/ALLBIDS-ended cases passed over two rounds in Camoufox and router arms, including gallery decoding (FRANKENBENCH_CASH_ALLBIDS_STATE_SNAPSHOT.json). AU_CASH_RENDERED_STATE_REFERENCE.json records the preceding screenshot audit. Cash sold/ended coverage, state variation across listings and days, and the remaining sites’ listing-state workloads are unfinished.

AU manifest v0.16 adds an exact eBay live-state reference with eight-photo gallery checks. Two of four fresh acquisitions passed (Camoufox/router, two rounds); both second-round acquisitions were BLOCKED before extraction. The passing observations recognized live state and decoded the complete declared gallery. FRANKENBENCH_EBAY_LIVE_STATE_SNAPSHOT.json retains all attempts, and AU_EBAY_RENDERED_STATE_REFERENCE.json records the preceding screenshot audit. This demonstrates state recognition on accessible pages, not reliable repeated eBay access. Sold/ended identity workloads and longitudinal validation remain pending.

AU manifest v0.17 removes obsolete unknown-only availability assertions from older ALLBIDS and Cash detail/gallery cases. Their workload assertions continue to check exact IDs, galleries and price semantics; dedicated listing state references still require live or ended explicitly. v0.16 full-run unknown-only failures must not be described as adapter failures when the projection supplied corroborated live state.

New benchmark runs save the exact loaded manifest to manifest.json and include its canonical SHA-256 plus startup hashes of the package Python files and harness in report provenance. source_changed_since_start flags a difference in these files at each checkpoint. This does not claim a clean Git checkout or capture external provider binaries. A fresh HTTP ALLBIDS update canary passed with stable source hashes (FRANKENBENCH_PROVENANCE_CANARY.json). Older running reports do not retroactively acquire this provenance.

The completed first round of the ongoing v0.16 full run passed 29/41 cases (FRANKENBENCH_AU_V016_ROUND1_CHECKPOINT.json). It retains two obsolete unknown-only detail assertion failures rather than retroactively counting them as passes. ALLBIDS live/ended and Cash live references passed; eBay live was blocked, Bikesales numbered pagination failed, and Depop/Trading Post strict pagination found overlaps. The second round remains in progress; this checkpoint is not a final two-round report or a reliability claim.

The full v0.16 cold router run completed 82 acquisitions across two rounds: 59 passed (29/41 and 30/41). Four observations retain obsolete unknown-only assertion failures already corrected in v0.17; no result is retroactively upgraded. FRANKENBENCH_AU_V016_REPEATED_SNAPSHOT.json preserves the complete report and explicit retrospective manifest reconstruction. eBay discovery, strict pagination, exact galleries and live state were blocked in both rounds. Gumtree discovery varied; its pagination failed both rounds. Depop pagination overlapped both rounds; Trading Post pagination passed only the second. Carsales numbered pagination passed twice, but its forced page-two detail timed out once. Bikesales numbered pagination was blocked on category navigation twice. ALLBIDS live/ended and Cash live state passed twice. This same-day test does not satisfy longitudinal reliability or remaining authenticated workloads.

AU manifest v0.18 adds opt-in shared-context Bikesales pagination using the versioned provider/adapter pair. The fresh integrated canary validated 22 new page-two listings, zero overlap, exact AU scope, DOM/API ID agreement and a 200 continuation response (AU_BIKESALES_SHARED_CONTEXT_CANARY.json). In the next two fresh benchmark rounds, one passed and one was blocked on initial category navigation (FRANKENBENCH_BIKESALES_SHARED_CONTEXT_SNAPSHOT.json). The original separate-acquisition pagination case remains; no failures are removed. Shared context repairs the extra initial-page reload within an operation, but initial access and repeated reliability remain unresolved.