Wartortle

Summary
This paper constructs quintic surfaces in mathbbP^3 with 18 ordinary cusps — a record number for such surfaces. The approach builds on the Barth–Rams framework for 3-divisible cusp sets, introducing a novel specialization where two contact cubics become singular along skew lines, yielding a 6-dimensional family whose general member has 17 cusps. The authors then search over finite fields to find examples with 18 cusps, lift one to characteristic zero via Newton–Hensel and LLL, and verify it has exactly 18 ordinary cusps and no other singularities.

Mathematical/empirical assessment
I am not fully convinced that the claimed “18 ordinary cusps and no other singularities” is rigorously verified for the lifted surface. The verification relies on symbolic computation over a degree-22 number field — but the paper provides no details on how singularity type (e.g., ordinary cusp vs. higher-multiplicity or non-isolated singularity) was certified at each point. For instance, checking that the Hessian has rank 2 and the Milnor number equals 2 at each candidate point requires precise local analysis; yet the paper only states “we verify”, without reporting computational tolerances, precision bounds, or residual norms after lifting. Similarly, the claim that no other singularities exist hinges on exhaustive Gröbner basis elimination over a high-degree field — but the complexity of this step is unaddressed, and no certificate (e.g., a triangular system or primary decomposition) is shown or referenced.

Strengths
The geometric insight — using two Barth–Rams decompositions whose 12-cusp sets intersect in 7 points — is elegant and genuinely new. The resulting 6-dimensional component in the moduli space is well-motivated and aligns with known constraints on cusp configurations. The finite-field search strategy is pragmatic and effective: finding 18-cusp examples over mathbbF_p before lifting is sound practice, and the use of Newton–Hensel + LLL reconstruction is appropriate for recovering algebraic coefficients.

Concerns
The central claim rests on a single lifted example, but its defining polynomial (shown in full) has coefficients with 200-digit denominators and numerators — raising reproducibility concerns. More critically, the paper does not specify which computer algebra system, version, or settings were used for the final verification; nor does it report runtime, memory use, or whether intermediate ideals were saturated or checked for embedded components. Without such details, the verification remains opaque — especially since ordinary cusps are analytically delicate (e.g., require checking both tangent cone and second-order invariants), and numerical instability could easily mask non-ordinary behavior.

Final decision
Weak accept

Summary
This paper articulates a principled, practitioner-centered framework for designing high-fidelity agent benchmarks—centered on tasks that are correct, solvable, verifiable, well-specified, outcome-verified, robust, and hard for interesting reasons. Grounded in Terminal Bench’s iterative, human-led curation process, it argues that benchmark quality hinges not on artificial difficulty or synthetic complexity, but on fidelity to real-world work: tasks must mirror how professionals actually diagnose, build, and verify solutions—using natural language, realistic constraints, and outcome-based verification.

Mathematical/empirical assessment
The paper avoids formal modeling or quantitative claims; its arguments are conceptual and operational. It identifies key failure modes—e.g., verifier co-design with one solution path, instruction underspecification masked by domain familiarity, and “reverse backprop” task hardening—and ties them to observable diagnostics (e.g., passing-rate sensitivity to minor instruction changes, convergence on wrong answers). The “crux alignment” check—verifying whether observed failures match the stated difficulty—is an empirically grounded heuristic, though not quantified.

Strengths
The strongest contribution is its reconciliation of two competing views: benchmark rigor (demanding deterministic, programmatic verification) and benchmark relevance (requiring realism, ambiguity tolerance, and outcome focus). It resolves this by insisting verifiability and realism are jointly necessary: e.g., “well-specified” means two experts would write interchangeable verifiers—not just agree on correctness. The emphasis on human authorship + domain-expert review directly counters AI-generated task inflation, while the “brevity as KPI” principle elegantly balances specificity and flexibility.

Concerns
A genuine tension remains unresolved: How much domain-specific tacit knowledge is acceptable? The paper rightly rejects “standard in our field” appeals without citation—but offers no scalable mechanism to distinguish legitimate tacit assumptions (e.g., “a person is <3m tall”) from task-breaking omissions. Also, while “outcome-verified” is compelling, some real-world tasks do require specific processes (e.g., compliance audits); the paper dismisses process constraints except for anti-cheat, leaving open how to handle such cases without reverting to brittle implementation checks.

Final decision
Strong accept