02 / detailed analysis

LLM Learnability

These six conditions test how the recorded model applies supplied language rules under renamed vocabulary, unfamiliar surfaces and new rules. The published comparisons use the frozen Quidra 0.3.0 run. The worker records also expose differing exercises, scoring shortcuts and missing arithmetic; those limits are stated alongside the concrete successes and failures.

Quidra’s evaluation rank: 10th of 10
Evaluation score: 71.97 / 100

Recorded run · 27 September 2026. These scores evaluate Quidra v0.3.0 at 38b7137. The site’s current compiler release is v0.5.0.

Run 2026-09-27-38b7137-gh31 · Methodology reference · Published score breakdown

Run’s declared inference model: claude-sonnet-5. Any separate review stages in the archived assessments are identified in the evidence notes.

Score records belong to this run; methodology links point to a later source snapshot. Its full template hash differs from the run’s recorded template hash. The explanations distinguish verified rubric inputs and score calculations from missing original observations.

6 requirements / 0–100 scores

Per-requirement comparison

Higher is better in every row. “Best other” is the highest-scoring non-Quidra language for that requirement, with every tie included. Δ is Quidra’s score minus the best other score. Published scores.

Scroll horizontally to see every column.

All 6 LLM Learnability requirements. Select a requirement to read its evidence.
Requirement	Quidra	Best other	Other score	Δ
Keyword anonymization	96.00	Python, C++, Rust, Go, Java, TypeScript, Kotlin, Swift	100.00	−4.00
Vocabulary anonymization	57.33	Python, Rust, Java, TypeScript, Kotlin, Swift	100.00	−42.67
Structural surface perturbation	100.00	Python, C++, TypeScript, Swift	100.00	0.00
Novel-rule generalization	100.00	Python, C++, TypeScript, Swift	100.00	0.00
Held-out rule composition	34.00	Rust, Go, Java, Swift	100.00	−66.00
Prior-conflict resistance	12.00	Rust, Go, Java, Swift	100.00	−88.00

01 / condition.i1_keyword_anonymization

Evaluation weight 20%

Keyword anonymization

Apply grammar rules after familiar keywords have been replaced by supplied pseudo-words.

How it is measured

The specified method uses three independent seeds. Each trial combines twelve task metrics with fixed weights; the condition is their mean. The records below state where the published worker arithmetic or trial count does not fully follow that specification. Score breakdown · Methodology reference.

The observed first-attempt successes alone do not explain Quidra's four-point gap: its worker applied an additional discount. The individual records below preserve that distinction.

Quidra

Quidra 0.3.0

96.00 / 100

All three recorded sum-above-threshold programs compiled and printed 26 on the first attempt. The worker nevertheless reported 96 after applying a four-point discount for token efficiency that it had not separately measured.

Recorded evidence
First-attempt successes	3 / 3
Recorded seed scores	100, 100, 100
Repairs used	0

Evidence limits

The seed scores average 100, while the published condition is 96. The fixed token-efficiency weights total only three points; the worker's four-point discount is not an independently reproducible application of the specified formula.

Best other · 8 tied languages

Python 3.12.3

100.00 / 100

Each of three keyword mappings produced a working top_two function on the first attempt. Every function passed all five held-out tests, so the worker awarded 100 to every seed and reported their mean as 100.

Recorded evidence
First-attempt successes	3 / 3
Held-out tests	15 / 15
Seed scores	100, 100, 100

Evidence limits

The worker treated short, correct single-function solutions as ceiling token efficiency; it did not publish comparative token measurements for this condition.

C++ 18.1.3 100.00 Evidence

The keyword-anonymized sumEven, maxOfThree and countPositive tasks all compiled on the first attempt and produced their expected outputs, 30, 7 and 4. No repair was needed, and all three recorded seed scores were 100.

Recorded evidence
First-attempt successes	3 / 3
Expected outputs matched	30; 7; 4
Repairs used	0

Evidence limits

The worker described token efficiency as nominal from completion lengths; a separate twelve-component numeric record is not committed.

Rust 1.95.0 100.00 Evidence

All three translations from pseudo-keywords compiled and ran without repair. The tasks exercised a sum and parity check, a slice maximum, and enum matching for areas; the worker assigned 100 to each seed.

Recorded evidence
First-attempt successes	3 / 3
Seed scores	100, 100, 100
Example result	Maximum of slice: 9

Evidence limits

For the geometry seed, the expected result was re-derived from the generated source rather than listed as an independently frozen oracle. The full component arithmetic is not recorded.

Go 1.26.3 100.00 Evidence

The three reverse-mapped programs built and matched the expected SumEven, MaxOf and ReverseString outputs: 12, 9 and olleh. The worker converted each compile-and-output pass to a seed score of 100.

Recorded evidence
Compile and output passes	3 / 3
Seed scores	100, 100, 100
Expected outputs matched	12; 9; olleh

Evidence limits

The worker explicitly used one binary pass/fail result as a proxy for all twelve metrics; repair, token and other component scores were not independently measured.

Java 26.0.1 100.00 Evidence

All three programs correctly translated the keyword glossary, compiled with javac and printed 30 for the even-number sum. They used the real standard output call and required no repair, which the worker scored as 100 for each seed.

Recorded evidence
First-attempt successes	3 / 3
Output per seed	30
Repairs used	0

Evidence limits

The worker states that all twelve components received full credit, but does not supply comparative token counts or separate unseen-case results.

TypeScript 7.0.2 100.00 Evidence

Each of three translated solve functions returned distinct even numbers in ascending order. All compiled with tsc and passed five unseen cases covering empty input, negatives, duplicates, odd-only input and descending order.

Recorded evidence
First-attempt successes	3 / 3
Unseen tests	15 / 15
Seed scores	100, 100, 100

Evidence limits

Token efficiency was assigned a 100 baseline for short single-shot completions rather than calculated from comparative token counts.

Kotlin 2.3.21 100.00 Evidence

All three clampValue implementations translated the pseudo-keywords correctly, compiled, and printed 5 0 10. No pseudo-keyword leaked into the Kotlin source and no repair was needed; each seed received 100.

Recorded evidence
First-attempt successes	3 / 3
Output per seed	5 0 10
Repairs used	0

Evidence limits

The worker declares every applicable component maximal but does not preserve a separate numeric record for all twelve components.

Swift 6.2.3 100.00 Evidence

The primality, factorial and digit-reversal programs all compiled and matched their expected outputs on the first attempt. The record shows three seed scores of 100 and no repair calls.

Recorded evidence
First-attempt successes	3 / 3
Tasks	Primality; factorial; digit reversal
Repairs used	0

Evidence limits

The record supports the build and output successes, but does not itemize all twelve weighted component scores.

02 / condition.i2_vocabulary_anonymization

Evaluation weight 20%

Vocabulary anonymization

Learn renamed standard-library and builtin operations as well as renamed grammar keywords.

How it is measured

Quidra eventually solved all three tasks, but its initial attempts and additional repairs lowered the published composite. The six tied leaders recorded first-attempt success for all three of their seeds.

Quidra

Quidra 0.3.0

57.33 / 100

All three initial sortedEvens attempts failed to compile. The model invented free-function append/sorted calls instead of methods; two seeds also introduced Python-style colons or C-style braces and loops. All three became correct after 1, 1 and 2 repairs. The recorded seed totals 62, 62 and 48 average to 57.33.

Recorded evidence
Initial successes	0 / 3
Final successes	3 / 3
Repairs by seed	1, 1, 2
Seed scores	62, 62, 48

Evidence limits

The record supplies seed totals and qualitative penalties, but not the twelve numeric component scores needed to verify how 62 and 48 were derived.

Best other · 6 tied languages

Python 3.12.3

100.00 / 100

The three vocabulary-anonymized top_two solutions all parsed and passed five held-out tests on the first attempt. No repair or invented API was reported, and every seed received 100.

Recorded evidence
First-attempt successes	3 / 3
Held-out tests	15 / 15
Repairs used	0

Evidence limits

The worker awarded ceiling token efficiency for concise correct solutions without publishing comparative token measurements.

Rust 1.95.0 100.00 Evidence

The model correctly translated the supplied vocabulary into vector sorting, word counting with a hash map, and descending collection operations. All three programs compiled and ran on the first attempt, with no repairs and three recorded scores of 100.

Recorded evidence
First-attempt successes	3 / 3
Seed scores	100, 100, 100
Example result	[6, 5, 4, 3, 2, 1]

Evidence limits

The worker describes successful programs, but does not publish the separate twelve-metric arithmetic behind the composite.

Java 26.0.1 100.00 Evidence

All three programs mapped the renamed output and modulo operations back to System.out.println and %. They compiled and printed 30 without using invented zant.blenk or wibmod APIs, earning three recorded scores of 100.

Recorded evidence
First-attempt successes	3 / 3
Output per seed	30
Repairs used	0

Evidence limits

All components were declared maximal; comparative token counts and separate unseen-case results are not recorded.

TypeScript 7.0.2 100.00 Evidence

The model correctly reconstructed length, deduplication, sorting and filtering from behavioral descriptions. All three solve functions compiled and passed all five unseen edge-case tests on their first attempt, so each seed received 100.

Recorded evidence
First-attempt successes	3 / 3
Unseen tests	15 / 15
Repairs used	0

Evidence limits

Token efficiency was assigned a 100 baseline for short single-shot code, without comparative token measurements.

Kotlin 2.3.21 100.00 Evidence

All three sumOfSortedRange implementations used real Kotlin operations for the renamed printing, length, range, sort and map vocabulary. They compiled and printed [0, 1, 2, 3, 4] followed by 10 without repair.

Recorded evidence
First-attempt successes	3 / 3
Seed scores	100, 100, 100
Repairs used	0

Evidence limits

The worker declares every applicable component maximal but does not provide a separate twelve-component numeric record.

Swift 6.2.3 100.00 Evidence

The model reconstructed filtering, sorting, counting, ranges and mapping from renamed vocabulary. Each of the three programs compiled and matched its expected result on the first attempt, including the filtered and sorted even values [2, 8].

Recorded evidence
First-attempt successes	3 / 3
Seed scores	100, 100, 100
Repairs used	0

Evidence limits

The build and output results are recorded; all twelve weighted component values are not itemized.

03 / condition.i3_structural_surface_perturbation

Evaluation weight 15%

Structural surface perturbation

Follow unfamiliar delimiters, separators or operator spellings while preserving the original program meaning.

How it is measured

The specified method uses two independently generated transformation sets. Each trial combines twelve task metrics with fixed weights; the condition is their mean. The records below state where the published worker arithmetic or trial count does not fully follow that specification. Score breakdown · Methodology reference.

Quidra ties the four highest comparison languages. All five records report two first-attempt successes, although the transformations and validation detail differ between workers.

Quidra

Quidra 0.3.0

100.00 / 100

Both transformation sets used three reversible substitutions involving call separators, grouping and an operator. After reverse mapping, the generated programs ran successfully and printed 25 and then 1 followed by 0. Both recorded scores were 100.

Recorded evidence
Successful sets	2 / 2
Substitutions per set	3
Repairs used	0

Evidence limits

The worker records output checks but does not supply every component needed to reproduce the twelve-metric composite.

Best other · 4 tied languages

Python 3.12.3

100.00 / 100

The two perturbed programs were correctly translated into clamp and sum_evens functions. The clamp tests covered an interior value and both bounds; sum_evens passed an ordinary input and an empty list. Both sets passed without repair.

Recorded evidence
Successful sets	2 / 2
Recorded function assertions	5 / 5
Repairs used	0

Evidence limits

No complete twelve-component score breakdown is committed for these two sets.

C++ 18.1.3 100.00 Evidence

Both ordinary C++ translations compiled and preserved the specified computation after delimiter and declaration-marker perturbations. The sum-of-squares task printed 91 and the product task printed 120, with no repairs.

Recorded evidence
Successful sets	2 / 2
Expected outputs matched	91; 120
Repairs used	0

Evidence limits

The worker says the runner will compute the complete composite, but its committed record contains no twelve-component runner input.

TypeScript 7.0.2 100.00 Evidence

Both sets changed three surface elements. The translations passed strict tsc checks on the first attempt, and the worker marked both as preserving the intended semantics, leading to the reported 100.

Recorded evidence
Strict compile checks	2 / 2
Substitutions per set	3
Repairs used	0

Evidence limits

The record gives compile commands and semantic-preservation flags, but no executed-case results or complete twelve-component calculation.

Swift 6.2.3 100.00 Evidence

Both translations compiled and produced the required sums and products, 7 and 30, without repair. The worker recorded 100 for each set.

Recorded evidence
Successful sets	2 / 2
Expected outputs matched	7; 30
Repairs used	0

Evidence limits

The worker describes three transformation axes in one set and four in the other, and notes partial retention of ordinary surface forms in the second. It does not publish the complete component calculation.

04 / condition.i4_novel_rule_generalization

Evaluation weight 20%

Novel-rule generalization

Apply a newly invented marker or naming rule that is supplied in the reference pack and removed before ordinary execution.

How it is measured

The specified method uses one primary trial. Each trial combines twelve task metrics with fixed weights; the condition is their mean. The records below state where the published worker arithmetic or trial count does not fully follow that specification. Score breakdown · Methodology reference.

Quidra and all four tied leaders satisfied their recorded synthetic rule on the first attempt. This condition preserves recognizable language syntax, so structural familiarity can still help.

Quidra

Quidra 0.3.0

100.00 / 100

The generated program correctly marked the reassigned total_mut binding and left the unchanged threshold unmarked. The worker validated the overlay before stripping it; the resulting Quidra program printed 5 and 100 on the first attempt.

Recorded evidence
Primary trials	1
Marker validation	Passed
Output after stripping	5; 100

Evidence limits

The successful rule and output checks are explicit; a complete twelve-component numeric derivation of 100 is absent.

Best other · 4 tied languages

Python 3.12.3

100.00 / 100

The accumulate function obeyed the new mutation-marker rule and produced 7 for inputs 3 and 4. An AST validator checked the marker, and the worker also records positive and deliberately unmarked negative controls.

Recorded evidence
Primary trials	1
Marker validation	Passed
Positive and negative controls	Passed

Evidence limits

The committed record documents functional and marker success, but does not itemize the twelve weighted component values.

C++ 18.1.3 100.00 Evidence

The square function received the required PURE-RETURN comment and the reference-mutating increment function received MUTATING. The program compiled and printed 36; removing the comments preserved its meaning.

Recorded evidence
Primary trials	1
Marker checks	2 / 2
Expected output matched	36

Evidence limits

The record supports rule compliance and execution but does not include the full twelve-component arithmetic.

TypeScript 7.0.2 100.00 Evidence

The model added qz_ to the numeric function containing mutable let state and correctly withheld it from the pure function. The rule checker and strict tsc compilation both passed on the first attempt.

Recorded evidence
Primary trials	1
Functions classified correctly	2 / 2
Strict compile check	Passed

Evidence limits

The worker records the rule checker and compilation, but no complete twelve-component calculation or runtime-case details.

Swift 6.2.3 100.00 Evidence

The model placed return-type marker comments above square and isEven, and omitted one above the Void-returning greet function. The mechanical checker passed, and the compiled program produced 16, Hello, World! and false.

Recorded evidence
Primary trials	1
Function marker checks	3 / 3
Repairs used	0

Evidence limits

The complete twelve-component score calculation is not itemized in the committed summary.

05 / condition.i5_held_out_rule_composition

Evaluation weight 15%

Held-out rule composition

Combine individually taught rules in combinations that were withheld from the worked examples.

How it is measured

Quidra's worker describes a failed composition and a successful one, while the four tied leaders report successful combinations. The reported 34 and 100 totals have specific replication or arithmetic limitations identified below.

Quidra

Quidra 0.3.0

34.00 / 100

One recorded task combined mutable references, elif branches and compound assignment successfully, printing 45, 70, 95 and 25. The other attempted those rules but invented a to-based loop and an implicit main entrypoint. The worker penalized that syntax failure and reported a composite of 34.

Recorded evidence
Recorded successful trials	1 / 2
Successful combination	Mutable reference + elif + compound assignment
Failed attempt	Invented loop and entrypoint syntax

Evidence limits

The record lists two trials although the frozen primary configuration specifies one. The first failure was judged from source inspection without a compiler invocation, and the twelve numeric component scores behind 34 are missing.

Best other · 4 tied languages

Rust 1.95.0

100.00 / 100

The single program correctly combined doubling, filter-and-reverse, and windowed sums into the withheld A+C, B+C and A+B+C cases. It compiled and produced the hand-checked outputs without repair; the worker published 100.

Recorded evidence
Recorded trials	1
Withheld combinations passed	3 / 3
Repairs used	0

Evidence limits

The listed components give source and total token efficiency 90, with every other component 100. Applying the specified weights yields 99.70, not the published 100; the discrepancy is retained as an evidence limitation.

Go 1.26.3 100.00 Evidence

Both recorded programs correctly composed the separately taught INC, DBL and NEG operations. Each printed 6, -1 and -3 for the three held-out programs, leading the worker to assign 100.

Recorded evidence
Recorded successful trials	2 / 2
Compositions per trial	3
Expected outputs matched	6; -1; -3

Evidence limits

Two trials are recorded where the primary configuration specifies one. The worker also redistributed repair weights instead of giving no-repair trials the specified full repair credit; all remaining values were 100, so this shortcut did not change its reported total.

Java 26.0.1 100.00 Evidence

The single trial combined negation-hoisting, sum-flattening and product-order rules into the previously unseen A+B+C case. The Java harness compiled and verified the exact canonical expression mul(j, add(neg(g), h, i)), supporting the reported 100.

Recorded evidence
Primary trials	1
Held-out A+B+C combination	Passed
Compile and harness checks	Passed

Evidence limits

Repair components are marked not needed rather than given numeric values. The record reports 100 for all other components, without separate comparative token evidence.

Swift 6.2.3 100.00 Evidence

The single trial correctly combined repeatCollect, filterKeep and joinDash into A+C, B+C and A+B+C functions. The generated Swift passed assertions for all three combinations and printed I5_OK without repair.

Recorded evidence
Primary trials	1
Withheld combinations passed	3 / 3
Repairs used	0

Evidence limits

The worker explicitly calculated this total as Correct@1 × 100. That is a reported shortcut, not an independently recorded twelve-metric composite.

06 / condition.i6_prior_conflict_resistance

Evaluation weight 10%

Prior-conflict resistance

Follow a supplied counterfactual mapping when familiar tokens or names suggest a conflicting meaning.

How it is measured

Quidra's recorded attempts reverted to foreign syntax under keyword swaps. The leaders followed their respective counterfactual rules, but the records use different kinds of mapping and do not support treating every 100 as an identical test.

Quidra

Quidra 0.3.0

12.00 / 100

Neither recorded attempt produced valid Quidra. A for/while swap led to C-style braces, semicolons and loops; an int/string swap produced func declarations, arrow return types and toString calls. The worker awarded 12 for partial task engagement and algorithmic intent despite those failures.

Recorded evidence
Recorded compile successes	0 / 2
Mappings	for/while; int/string
Observed failure	Reversion to foreign syntax

Evidence limits

Two trials are recorded where the primary configuration specifies one, and the component values behind 12 are missing. A worker's manual rewrite only checked algorithmic intent; it was not a successful model completion.

Best other · 4 tied languages

Rust 1.95.0

100.00 / 100

The model followed deliberately misleading function names: sum_all returned the maximum, filter_max negated values, and identity_map sorted descending. The single program compiled and produced all four expected results without repair, and the worker reported 100.

Recorded evidence
Primary trials	1
Counterfactual functions checked	4
Repairs used	0

Evidence limits

The listed token components are 92 with all others 100, which weights to 99.76 rather than 100. This task remapped function meanings rather than permuting Rust grammar keywords.

Go 1.26.3 100.00 Evidence

Both programs obeyed the swapped plus/minus meanings and strict left-to-right evaluation. Each produced the four expected results, 7, 13, 17 and 11, with no observed prior-conflict failure; the worker assigned 100.

Recorded evidence
Recorded successful trials	2 / 2
Checks per trial	4
Expected outputs matched	7; 13; 17; 11

Evidence limits

The primary configuration specifies one trial, while this record includes two. The task swaps arithmetic operations in a small interpreter, so its surface challenge differs from Quidra's grammar-keyword swaps.

Java 26.0.1 100.00 Evidence

The model correctly treated the toy language's if token as a loop and its for token as a conditional. The translated Java compiled and printed 0, 1 and 2, matching the supplied counterfactual specification on the single trial.

Recorded evidence
Primary trials	1
Expected output matched	0; 1; 2
Compile and run checks	Passed

Evidence limits

Repair components are marked not needed and the remaining components 100; separate comparative token evidence is not supplied.

Swift 6.2.3 100.00 Evidence

The model followed conflicting meanings assigned to loop, branch, sum and product in the task wording. The generated compute function matched an independent reference on two arrays, producing 30 for each, without repair.

Recorded evidence
Primary trials	1
Reference comparisons	2 / 2
Repairs used	0

Evidence limits

The mapping was applied only to English task wording, not to Swift source tokens. The worker calculated Correct@1 × 100 rather than preserving the complete twelve-metric composite.

Read the result in context

These are descriptive results for the frozen benchmark sample. They do not establish statistical superiority beyond that sample. Each evaluation stands on its own; the project publishes no cross-evaluation total.

The analysis above reads the recorded evidence alongside the scores. The machine-readable rankings and score breakdown are the published result. The methodology reference describes the linked source snapshot; its full template is not identical to the run’s recorded template.

Return to all ten languages in LLM Learnability