05 / detailed analysis
The recorded model implemented SVM, GMM and LightGrad in two scenarios, with three fresh prompt-layout trials per cell: 18 trials per language and at most three repairs each. Quidra returned an initial answer in every trial but never produced a fully correct program. The rows distinguish that observed outcome from token cost, worker judgments and physical measurements; they describe this model and frozen run rather than a general language limitation.
Recorded run · 27 September 2026. These scores evaluate Quidra v0.3.0 at 38b7137. The site’s current compiler release is v0.5.0.
Run 2026-09-27-38b7137-gh31 · Methodology reference · Published score breakdown
Run’s declared inference model: claude-sonnet-5. Any separate review stages in the archived assessments are identified in the evidence notes.
Score records belong to this run; methodology links point to a later source snapshot. Its full template hash differs from the run’s recorded template hash. The explanations distinguish verified rubric inputs and score calculations from missing original observations.
18 requirements / 0–100 scores
Higher is better in every row. “Best other” is the highest-scoring non-Quidra language for that requirement, with every tie included. Δ is Quidra’s score minus the best other score. Published scores.
Scroll horizontally to see every column.
| Requirement | Quidra | Best other | Other score | Δ |
|---|---|---|---|---|
| Generation success rate | 100.00 | Python, C++, Rust, Go, Java, TypeScript, Kotlin, Swift, Zig | 100.00 | 0.00 |
| Compile / parse success rate | 0.00 | Java | 94.44 | −94.44 |
| Correct@1 | 0.00 | Python | 88.89 | −88.89 |
| Correct@N | 0.00 | Python, C++, Rust, Go, Kotlin, Swift | 100.00 | −100.00 |
| Test pass rate | 0.00 | Python | 86.96 | −86.96 |
| Repair success rate | 0.00 | Python, C++, Rust, Go, Kotlin, Swift | 100.00 | −100.00 |
| Repair efficiency | 0.00 | Python | 95.83 | −95.83 |
| Diagnosis efficiency | 0.00 | Rust, Go | 100.00 | −100.00 |
| Silent-bug resistance | 0.00 | Python, C++, Rust, Go, Kotlin | 100.00 | −100.00 |
| Syntax hallucination resistance | 0.00 | Python, TypeScript | 100.00 | −100.00 |
| Specification compliance | 0.00 | Python, C++, Kotlin | 100.00 | −100.00 |
| Prompt robustness | 0.00 | Java | 83.33 | −83.33 |
| Unseen-case generalization | 0.00 | Python | 90.48 | −90.48 |
| Source token efficiency | 24.85 | Python | 90.38 | −65.53 |
| Total token efficiency | 5.37 | Python | 90.16 | −84.79 |
| Generated code performance | 0.00 | Rust | 95.16 | −95.16 |
| Generated code memory efficiency | 0.00 | Rust | 99.77 | −99.77 |
| Generated code compile performance | 0.00 | Python | 100.00 | −100.00 |
Whether the model returns a non-empty, complete initial answer.
100 × qualifying initial completions / 18 trials. Compilation and correctness are evaluated separately. Score breakdown · Methodology reference.
All ten languages tie at 100. Producing an answer earns this row's credit even when that answer cannot compile.
Quidra
All 18 initial requests returned a qualifying non-empty completion, giving 100 × 18/18 = 100. All initial programs failed compilation, and all 18 trials consumed three repairs without reaching the public repair gate.
| Qualifying initial completions | 18 / 18 |
|---|---|
| Scored workloads | SVM; GMM; LightGrad |
| Scenarios per workload | 2 |
Best other · 9 tied languages
All 18 initial requests returned a qualifying non-empty completion, giving 100 × 18/18 = 100. Sixteen initial programs passed; the SVM specification trial needed one repair and the GMM specification trial needed two.
| Qualifying initial completions | 18 / 18 |
|---|---|
| Scored workloads | SVM; GMM; LightGrad |
| Scenarios per workload | 2 |
Whether the initial generated program passes the frozen build or parse command.
100 × first completions that compile or parse / 18 trials. A later successful repair does not change this first-attempt metric. Score breakdown · Methodology reference.
Java leads this requirement even though Python leads the overall proficiency evaluation.
Quidra
None of the 18 initial sources passed the real Quidra build/parse gate. The worker reports invented C/Rust-style forms, inappropriate imports and entrypoint or declaration syntax errors. The first-attempt fraction is therefore 0/18.
| Initial compile / parse successes | 0 / 18 |
|---|---|
| Repair allowance | 3 turns per trial |
Evidence limits
Some later attempts compiled but produced no required output. The worker gives an inconsistent count for those attempts, so that count is not used here.
Best other
Seventeen of the 18 initial Java programs compiled. The sole reported compile failure was a GMM source containing literal Markdown fences, which javac rejected. The two SVM output mismatches did compile and therefore do not lower this row.
| Initial compilations | 17 / 18 |
|---|---|
| Initial compile failures | 1 |
| Calculation | 100 × 17/18 = 94.44 |
Whether the first program passes every public and hidden oracle case.
100 × fully correct first completions / 18 trials. A public example alone is insufficient. Score breakdown · Methodology reference.
The gap comes from first-attempt working programs: Python passed 16 of 18, while no initial Quidra program compiled.
Quidra
All 18 first completions failed compilation, so none could pass the complete oracle. The recorded syntax and declaration failures prevented any first-attempt correctness credit.
| Fully correct initial programs | 0 / 18 |
|---|---|
| Initial compile successes | 0 / 18 |
Best other
Sixteen of the 18 initial Python programs passed the complete oracle. The two specification-to-implementation trials that needed repair were SVM trial 2 and GMM trial 2, giving 100 × 16/18 = 88.89.
| Fully correct initial programs | 16 / 18 |
|---|---|
| Trials initially needing repair | 2 |
| Calculation | 100 × 16/18 = 88.89 |
Whether a trial produces a fully correct program within the fixed attempt budget.
100 × trials with a fully correct source at any allowed attempt / 18. N is at most four: one initial answer and three repairs. Score breakdown · Methodology reference.
All six tied leaders eventually produced a fully correct source in every trial. Quidra exhausted the same repair allowance without one.
Quidra
The retained raw record marks 0 of 18 trials as having a fully correct source within the allowed attempts. All initial programs failed compilation, and all 18 trials consumed three repairs without reaching the public repair gate.
| Trials with a fully correct source | 0 / 18 |
|---|---|
| Maximum attempts per trial | 4 |
Best other · 6 tied languages
The retained raw record marks 18 of 18 trials as having a fully correct source within the allowed attempts. Sixteen initial programs passed; the SVM specification trial needed one repair and the GMM specification trial needed two.
| Trials with a fully correct source | 18 / 18 |
|---|---|
| Maximum attempts per trial | 4 |
The proportion of oracle cases passed across all scored initial and repair calls.
100 × passed public and hidden cases / all scored public and hidden cases. Failed attempts remain in the denominator. Score breakdown · Methodology reference.
This rate is below Python's final success rate because earlier failed attempts still count. The case totals below are reconstructed from the frozen case counts and recorded repair history.
Quidra
No attempt produced an oracle-passing output. All four allowed calls were consumed in every trial, so the frozen case counts imply 240 scored cases and zero passes.
| Reconstructed oracle cases passed | 0 / 240 |
|---|---|
| Recorded model calls | 72 |
| Final successful trials | 0 / 18 |
Evidence limits
The complete per-call oracle traces are not committed; the denominator is reconstructed from the frozen case set and the documented full repair budget.
Best other
Python's 18 initial calls cover 60 cases. One additional SVM call and two additional GMM calls add 9; the final published ratio corresponds to 60 of 69 passing case executions. Earlier compile failures remain in that denominator even though every trial was eventually solved.
| Reconstructed case executions passed | 60 / 69 |
|---|---|
| Recorded model calls | 21 |
| Calculation | 100 × 60/69 = 86.96 |
Evidence limits
The worker's prose describes final public success as 100, but that is not the published all-attempt case metric. Full case traces are not committed; the displayed counts are reconstructed from the task and repair records.
How often an initially failing program reaches the public repair gate after feedback.
100 × later public-gate successes / trials that initially failed compilation or public tests. Hidden cases never create a repair opportunity; no needed repairs would score 100. Score breakdown · Methodology reference.
The six tied leaders repaired every eligible failure, but they needed different numbers of turns. Repair efficiency and diagnosis efficiency show that difference.
Quidra
Of 18 trials that failed the initial public repair gate, 0 reached it on a later allowed completion. All initial programs failed compilation, and all 18 trials consumed three repairs without reaching the public repair gate.
| Eligible trials repaired | 0 / 18 |
|---|---|
| Maximum repair turns | 3 |
Best other · 6 tied languages
Of 2 trials that failed the initial public repair gate, 2 reached it on a later allowed completion. Sixteen initial programs passed; the SVM specification trial needed one repair and the GMM specification trial needed two.
| Eligible trials repaired | 2 / 2 |
|---|---|
| Maximum repair turns | 3 |
Reward successful completion earlier within the repair budget.
Per trial: first-attempt public-gate success scores 100; success after repair k scores 100 × (1 − k/4); an unrepaired failure scores 0. Average all 18 trials. Score breakdown · Methodology reference.
Python's two repaired trials cost one and two turns. Quidra never crossed the public repair gate, so every trial contributed zero.
Quidra
Every trial remained below the public repair gate after all three repairs. Each unrepaired failure contributes zero, so the mean of the 18 trial scores is zero.
| Unrepaired trials | 18 / 18 |
|---|---|
| Repair calls consumed | 54 |
| Mean trial credit | 0 |
Best other
Sixteen first-attempt successes contribute 100 each. The SVM repair succeeds at k=1 for 75 points, and the GMM repair at k=2 for 50 points. Their mean is (16 × 100 + 75 + 50) / 18 = 95.83.
| First-attempt trial credits | 16 × 100 |
|---|---|
| Repaired trial credits | 75; 50 |
| Calculation | 1,725 / 18 = 95.83 |
Evidence limits
The worker's explanatory sentence incorrectly prints 98.06 for this arithmetic. Its listed inputs and the published score agree on 95.83.
Whether the first trusted diagnostic is enough for a successful repair.
100 × trials passing the public repair gate on the first repair / trials initially needing repair. No needed repairs would score 100. Score breakdown · Methodology reference.
Rust and Go fixed every eligible failure on their first repair. Their observed failures were mostly source-formatting defects.
Quidra
The first repair succeeded in 0 of 18 eligible trials. All initial programs failed compilation, and all 18 trials consumed three repairs without reaching the public repair gate.
| Successful first repairs | 0 / 18 |
|---|---|
| Diagnostic source | Compiler errors and public test failures |
Best other · 2 tied languages
The first repair succeeded in 4 of 4 eligible trials. Four initial sources included Markdown fences that rustc rejected; all four passed after the first repair removed the formatting defect.
| Successful first repairs | 4 / 4 |
|---|---|
| Diagnostic source | Compiler errors and public test failures |
Avoid programs that run successfully but return an incorrect oracle result.
Among first completions that compile and execute oracle cases: 100 × (eligible trials − silent-bug trials) / eligible trials. With no eligible trials, score 0. Score breakdown · Methodology reference.
Quidra's zero is the specified no-eligible-program result. It does not mean that every Quidra program ran and silently returned a wrong answer.
Quidra
No initial source compiled, leaving zero first completions eligible for a silent-bug check. The frozen formula explicitly returns zero in that situation rather than awarding unearned full resistance.
| Eligible initial programs | 0 |
|---|---|
| Observed initial runtime population | No compiled programs |
| No-eligible-trials rule | Score 0 |
Best other · 5 tied languages
The worker reports no successful-exit wrong-output run among 16 compiling first completions. The published runner score gives full resistance for that eligible population; initial formatting or compile failures are excluded from it.
| Eligible first completions reported | 16 |
|---|---|
| Silent-bug trials reported | 0 |
| Calculation | 100 × (16 − 0)/16 = 100 |
Evidence limits
Worker summaries expose public outcomes, while the complete hidden-oracle traces are not committed for independent case-by-case inspection.
Avoid inventing language constructs or APIs that do not exist in the evaluated language.
Worker judgment from preserved source and diagnostics. This is one of the two proficiency metrics that are not computed entirely by the runner. Score breakdown · Methodology reference.
TypeScript ties Python here despite failing every initial build: its worker attributed those errors to Node typing gaps rather than fabricated syntax.
Quidra
The worker judged resistance as zero after observing invented foreign-language syntax, imports of automatically available standard namespaces, entrypoint misuse and incorrect declaration forms. Those source-level mistakes explain the judgment beyond the numeric compile-failure rate.
| Judged score | 0 / 100 |
|---|---|
| Initial build failures | 18 / 18 |
| Observed defect types | Invented syntax; namespace and declaration misuse |
Evidence limits
This is a worker judgment; no separate numeric hallucination rubric is committed.
Best other · 2 tied languages
The worker assigned Python 100 and described the solutions as compact, idiomatic single-file programs. Sixteen initially passed and all eighteen eventually succeeded; the record does not attribute the two initial failures to invented Python constructs.
| Judged score | 100 / 100 |
|---|---|
| Initially correct trials | 16 / 18 |
| Eventually correct trials | 18 / 18 |
Evidence limits
The worker does not provide a separate per-trial hallucination checklist, so this 100 cannot be reconstructed as an independent mechanical count.
Honor the task's protocol and requirements, including constraints beyond a single output example.
Worker judgment from preserved programs and task evidence; the published score is not a separately complete mechanical compliance checklist. Score breakdown · Methodology reference.
Python, C++ and Kotlin receive full credit in this judged row. Their final programs succeeded even when initial programs required repair.
Quidra
The worker awarded zero because no trial produced a program with the required output or a successful public repair result. Invented declarations and incomplete working bodies prevented the attempted solutions from satisfying the task contract.
| Judged compliance | 0 / 100 |
|---|---|
| Trials with a fully correct source | 0 / 18 |
Evidence limits
The worker uses observed program failure as its compliance basis; no separate non-output constraint checklist is committed.
Best other · 3 tied languages
The worker awarded full compliance after the final solutions met the workload protocol. Sixteen initial programs passed; the SVM specification trial needed one repair and the GMM specification trial needed two.
| Judged compliance | 100 / 100 |
|---|---|
| Trials with a fully correct source | 18 / 18 |
| Initially correct trials | 16 / 18 |
Evidence limits
No separate checklist of every non-output constraint is committed. The final-success evidence supports the judgment but does not independently reproduce the complete compliance rubric.
Maintain first-attempt correctness when the same task is presented in different orders.
For each of three equivalent prompt layouts, compute full-oracle first-attempt successes across six workload/scenario cells; publish the lowest of the three rates. Score breakdown · Methodology reference.
Java's weakest layout still passed five of six cells. All three Quidra layouts had zero successful first attempts.
Quidra
Canonical, scenario-first and contract-first prompts each had zero fully correct initial programs across their six cells. Taking the minimum of those three zero rates produces zero.
| Canonical layout | 0 / 6 |
|---|---|
| Scenario-first layout | 0 / 6 |
| Contract-first layout | 0 / 6 |
Best other
Each of Java's three prompt layouts passed five of six cells. The two SVM numeric-output failures and the GMM fenced-source failure were spread across the three layouts, so the weakest rate is also 5/6 = 83.33.
| Canonical layout | 5 / 6 |
|---|---|
| Scenario-first layout | 5 / 6 |
| Contract-first layout | 5 / 6 |
Evidence limits
Some other totals in the worker's narrative are inconsistent. These layout results agree with the published first-attempt total of 15/18 and robustness score.
Pass hidden inputs that were not supplied as prompt examples.
100 × hidden cases passed by first completions / 42 frozen hidden cases. The denominator is six trials each for two SVM, two GMM and three LightGrad hidden inputs. Score breakdown · Methodology reference.
This first-attempt metric excludes repair gains. Python's published full-precision score corresponds to 38 of 42 hidden cases; Quidra passed none.
Quidra
No first completion compiled, so none passed a hidden case. The frozen denominator still includes all 42 hidden inputs across the 18 trials; build failures do not remove cases or earn generalization credit.
| Hidden cases passed | 0 / 42 |
|---|---|
| Initial compilations | 0 / 18 |
Best other
The published full-precision ratio and the frozen 42-case denominator give 38 hidden passes. The two initially failing specification trials account for the four missing SVM/GMM hidden-case passes; successful repairs do not change this first-attempt metric.
| Hidden cases passed | 38 / 42 |
|---|---|
| Calculation | 100 × 38/42 = 90.48 |
| Measurement point | First completion only |
Evidence limits
The numerator is reconstructed from the published full-precision score and fixed denominator. Complete hidden-case execution logs are not committed.
How compact the terminal source completion is in provider-counted output tokens.
For each exact trial cell, score 100 × lowest source-token count / this language's count; average the 18 cell scores. Count failed terminal attempts as well as successful ones. Score breakdown · Methodology reference.
Normalize each matched trial before averaging. A ratio of overall mean token counts would produce a different result, and a short failed program can still receive token-efficiency points.
These raw measurements were retained from the 25 September run for the same language versions and reused in this 27 September publication. Their certified workload and trial identities match, and recomputing the per-cell ratios reproduces the published score.
Quidra
The terminal source completions used 1,231–16,384 tokens. The requirement averages the source-token ratios within all 18 matched trials; each final attempt is counted even when it is incorrect.
The model's compact failed attempts still receive some token credit; none of these 18 trials produced a correct program.
| Raw range across trials | 1,231–16,384 tokens |
|---|---|
| Mean raw measurement | 7,638.78 tokens |
| Lowest raw value in matched cell | 0 / 18 |
| Fully correct generated programs | 0 / 18 |
Best other
The terminal source completions used 566–3,611 tokens. The requirement averages the source-token ratios within all 18 matched trials; each final attempt is counted even when it is incorrect.
The successful terminal programs are comparatively short. This row measures their completion size; repair overhead is included separately in total token efficiency.
| Raw range across trials | 566–3,611 tokens |
|---|---|
| Mean raw measurement | 1,112.78 tokens |
| Lowest raw value in matched cell | 7 / 18 |
| Fully correct generated programs | 18 / 18 |
How many model-visible tokens the complete task consumes, including feedback and repairs.
For each trial, sum provider input, cache and output tokens through stopping; score 100 × lowest task-token count / this language's count; average all 18 trials. Score breakdown · Methodology reference.
Quidra used all three repairs in every trial. Python needed repairs in only two trials, producing a much lower recorded task-token cost.
These raw measurements were retained from the 25 September run for the same language versions and reused in this 27 September publication. Their certified workload and trial identities match, and recomputing the per-cell ratios reproduces the published score.
Quidra
The complete trials consumed 30,209–70,621 tokens each, including prompts, repeated context, feedback and outputs. The score averages the 18 comparisons with the cheapest language in each matching trial.
All 18 trials consumed the initial call plus three repairs: 72 calls in total. That repeated cost drives the score close to zero.
| Raw range across trials | 30,209–70,621 tokens |
|---|---|
| Mean raw measurement | 49,133.89 tokens |
| Lowest raw value in matched cell | 0 / 18 |
| Fully correct generated programs | 0 / 18 |
Evidence limits
The worker estimated roughly 10,000–20,000 tokens per trial. The retained provider-counted raw values are substantially higher and are the measurements used here.
Best other
The complete trials consumed 1,528–10,270 tokens each, including prompts, repeated context, feedback and outputs. The score averages the 18 comparisons with the cheapest language in each matching trial.
Sixteen trials needed only one completion; just two needed repairs. That limited the repeated prompts, diagnostics and outputs included in total task cost.
| Raw range across trials | 1,528–10,270 tokens |
|---|---|
| Mean raw measurement | 3,164.89 tokens |
| Lowest raw value in matched cell | 12 / 18 |
| Fully correct generated programs | 18 / 18 |
Runtime of the first fully correct program produced in each trial.
Use median oracle-case wall time for each fully correct source, normalize as 100 × fastest valid time / measured time in that trial, then average 18 trials. A trial with no fully correct source scores 0. Score breakdown · Methodology reference.
Quidra's zero is a failure to produce a measurable correct program. It is not a runtime-speed measurement of the language itself.
These raw measurements were retained from the 25 September run for the same language versions and reused in this 27 September publication. Their certified workload and trial identities match, and recomputing the per-cell ratios reproduces the published score.
Quidra
None of the 18 trials produced a fully correct source. The runner therefore assigns zero to every trial in this physical-efficiency metric; the raw measurement fields are empty because there is no valid program to time or profile.
| Fully correct generated programs | 0 / 18 |
|---|---|
| Valid physical measurements | 0 |
| Score rule applied | 18 failed trials × 0 |
Evidence limits
This zero does not measure the speed, memory use or build latency of a correct Quidra implementation.
Best other
All 18 trials produced correct programs. Their median oracle-case wall times ranged from 1.226–6.271 ms; the language was fastest in 15 of 18 matched cells, which explains its high mean normalized runtime score.
| Raw range across trials | 1.226–6.271 ms |
|---|---|
| Mean raw measurement | 1.848 ms |
| Lowest raw value in matched cell | 15 / 18 |
| Fully correct generated programs | 18 / 18 |
Peak memory of the fully correct generated programs under the frozen harness.
Use median child peak RSS from separate oracle-case replays; normalize 100 × lowest valid memory / measured memory per trial, then average all 18 trials. No fully correct source gives 0. Score breakdown · Methodology reference.
Rust leads by a small measured margin. Quidra has no valid generated-program memory sample because none of its trials produced a fully correct source.
These raw measurements were retained from the 25 September run for the same language versions and reused in this 27 September publication. Their certified workload and trial identities match, and recomputing the per-cell ratios reproduces the published score.
Quidra
None of the 18 trials produced a fully correct source. The runner therefore assigns zero to every trial in this physical-efficiency metric; the raw measurement fields are empty because there is no valid program to time or profile.
| Fully correct generated programs | 0 / 18 |
|---|---|
| Valid physical measurements | 0 |
| Score rule applied | 18 failed trials × 0 |
Evidence limits
This zero does not measure the speed, memory use or build latency of a correct Quidra implementation.
Best other
All 18 correct programs had recorded memory measurements between 11,728,896–11,837,440 bytes. The language had the lowest peak-RSS value in 6 cells and stayed close to the minimum in the rest, producing the highest mean score.
| Raw range across trials | 11,728,896–11,837,440 bytes |
|---|---|
| Mean raw measurement | 11,776,796.44 bytes |
| Lowest raw value in matched cell | 6 / 18 |
| Fully correct generated programs | 18 / 18 |
Evidence limits
The runner reports 99.77 for Rust and 99.75 for Python. This small margin describes the frozen harness measurements, not an established general memory advantage.
Time spent in the frozen build or parse step for fully correct generated programs.
Normalize each successful build/parse wall time as 100 × fastest valid time / measured time, then average all 18 trials. No fully correct source gives 0. Score breakdown · Methodology reference.
Python's parse/build command was fastest in all 18 matched cells. This compares the prescribed commands, whose work differs across compiled and interpreted languages.
These raw measurements were retained from the 25 September run for the same language versions and reused in this 27 September publication. Their certified workload and trial identities match, and recomputing the per-cell ratios reproduces the published score.
Quidra
None of the 18 trials produced a fully correct source. The runner therefore assigns zero to every trial in this physical-efficiency metric; the raw measurement fields are empty because there is no valid program to time or profile.
| Fully correct generated programs | 0 / 18 |
|---|---|
| Valid physical measurements | 0 |
| Score rule applied | 18 failed trials × 0 |
Evidence limits
This zero does not measure the speed, memory use or build latency of a correct Quidra implementation.
Best other
The frozen build/parse command took 33.385–39.229 ms for the 18 correct programs. It was fastest in every matched cell, so each cell scored 100 and the average is 100.
| Raw range across trials | 33.385–39.229 ms |
|---|---|
| Mean raw measurement | 35.185 ms |
| Lowest raw value in matched cell | 18 / 18 |
| Fully correct generated programs | 18 / 18 |
