# S2 Pro vs S2.1 Pro: multilingual blind listening study

Completed 2026-10-26T17:59:24.607294+00:00: 40 characters, 8 languages, 640 generated WAVs, 320 matched pairs, 672 complete Gemini 3.1 Pro reviews including controls.

Equal-language macro: No clear S2.1 Pro naturalness advantage: the descriptive primary interval crosses zero. 141/320 pairs changed preference across presentation order. Identical-audio controls tied 32/32 times; that does not establish order invariance on differing recordings.

Language-specific paired estimates are primary interpretation. The equal-language macro averages within each character and resamples whole character vectors, preserving dependence across languages. Absolute scores across languages are not directly comparable; the macro is descriptive only.

## Language-specific naturalness

| Language | Characters | S2 Pro | S2.1 Pro | Difference | Descriptive 95% interval | S2 wins | S2.1 wins | Ties | Inconsistent |
|---|---:|---:|---:|---:|---|---:|---:|---:|---:|
| Mandarin Chinese (Simplified Chinese) | 40 | 2.975 | 3.000 | 0.025 | [-0.212, +0.263] | 10 | 10 | 5 | 15 |
| Japanese | 40 | 3.175 | 2.888 | -0.287 | [-0.475, -0.100] | 22 | 3 | 2 | 13 |
| Korean | 40 | 3.100 | 3.263 | 0.163 | [-0.075, +0.400] | 11 | 12 | 3 | 14 |
| Spanish (Spain) | 40 | 2.425 | 2.513 | 0.087 | [-0.075, +0.237] | 8 | 5 | 8 | 19 |
| French (France) | 40 | 2.350 | 2.538 | 0.188 | [+0.037, +0.338] | 2 | 9 | 9 | 20 |
| German (Germany) | 40 | 2.725 | 2.737 | 0.013 | [-0.125, +0.150] | 4 | 5 | 7 | 24 |
| Brazilian Portuguese | 40 | 2.812 | 2.575 | -0.237 | [-0.400, -0.087] | 11 | 3 | 7 | 19 |
| Modern Standard Arabic | 40 | 2.712 | 2.650 | -0.062 | [-0.163, +0.037] | 7 | 7 | 9 | 17 |

Descriptive equal-language macro: S2 Pro 2.784, S2.1 Pro 2.770; candidate-minus-control -0.014, interval [-0.100, +0.073].

Differences compare the models within the same language. The 40 characters, not the hundreds of requests, are the sampling units. Within each language each model has two order-reversed scores per character. The macro averages eight language means within character, then resamples the whole character vector, stratified by gender, 5,000 times. Missing scores require complete-case exclusion from that metric; nothing is imputed. Language slices and secondary dimensions are multiple descriptive comparisons without multiplicity correction or significance claims.

Main pair preferences: S2 Pro 75, S2.1 Pro 54, both-order ties 50, inconsistent 141. Stable winners require the same underlying recording to win both orders; tie-versus-winner also counts as inconsistent.

## Calibration and order reliability

Identical audio: 32/32 tie responses; 0 non-tie responses, 0 unassessable. All control judgments are retained. Passing identical-byte checks does not prove order invariance for differing recordings.

| Language | Identical-audio ties | Opposite winners | Tie vs winner | Presented A / B / tie |
|---|---:|---:|---:|---|
| Mandarin Chinese (Simplified Chinese) | 4/4 | 6 | 9 | 36 / 25 / 19 |
| Japanese | 4/4 | 3 | 10 | 38 / 28 / 14 |
| Korean | 4/4 | 8 | 6 | 34 / 34 / 12 |
| Spanish (Spain) | 4/4 | 5 | 14 | 31 / 19 / 30 |
| French (France) | 4/4 | 5 | 15 | 18 / 29 / 33 |
| German (Germany) | 4/4 | 4 | 20 | 30 / 16 / 34 |
| Brazilian Portuguese | 4/4 | 3 | 16 | 27 / 23 / 30 |
| Modern Standard Arabic | 4/4 | 2 | 15 | 26 / 21 / 33 |

Confidence values average two uncalibrated model self-reports; they are not probabilities of a real-world win.

All selected characters remain visible. Unassessable sample reviews (0): [].

## Direct cold WebSocket timing

Direct cold WebSocket requests from this test host; no warm connection pool, not end-to-end production call latency.

| Requested model | Samples | Median TTFA incl. handshake | p95 TTFA | Median after start event | Failed attempts |
|---|---:|---:|---:|---:|---:|
| S2 Pro | 320 | 534.9ms | 698.8ms | 289.2ms | 0 |
| S2.1 Pro | 320 | 514.8ms | 1286.1ms | 279.0ms | 0 |

Timings describe successful attempts. Failed attempts are counted separately; requests were not concurrent within a matched pair.

## Scope and evidence

Top 20 per gender by distinct real registered callers in positive-duration private call transcripts over last 14 days; public non-retired characters with active Fish voice. Tie-break by call count then character id.
Selection window: 2026-10-12T16:42:47.757Z to 2026-10-26T16:42:47.757Z. The same 40-character cohort is used for every language. 42 distinct references may include existing language-specific voices; within each pair the reference is identical.

One synthetic localized passage per language exercises conversational fluency, emotional delivery and numbers/times. Exact source scripts and the predeclared design are in protocol.json; no private customer utterances were used.

Existing locale-specific voice where present, else existing English bank slot, else active Fish slot; manually selected/cloned voices preserved. Each model pair always uses the exact same reference. This tests current character voices speaking the target language, not freshly selected native reference voices.

Only one character has stored non-English language-specific reference slots; the other 39 use current cross-lingual character voices, not proven native-reference accents. Scores describe this existing-voice setup, not native-reference model capability.

Direct Fish WebSocket. One text message and flush for the full passage in every language, then stop; Fish may chunk internally. At most two synthesis sockets. No explicit language parameter: identical automatic language detection for both models. Fresh connection per sample.

The earlier English study used sentence-level flushes; this multilingual extension uses one full-passage flush. Timing across the two studies is therefore not a matched comparison.

Judge receives opaque pair ID, target language/locale, exact text/directions and audio only; no model names, identities, portraits, popularity, timings or file paths. Each pair gets two separate reversed-order judgments. Orchestrator holds labels; not independent human double-blind.

Selection and English baseline are reused from the completed English study. Multilingual scripts, masking, control selection, transport settings and aggregation rules are frozen before new synthesis or judging.

Masked-result SHA-256: `3d4cff7d7e7c59905a07d5d96c43995208e15e91e1b21f7e85d9f7dfa8e77487`. All source model/audio request hashes, final WAV hashes, full-decode receipts, returned Gemini model versions and STOP completion are verified before publishing. The orchestrator knew assignments; this is blinded model evaluation, not independent human double-blind.

Judge versions: {"gemini-3.1-pro-preview": 672}. Successful response tokens: {"promptTokenCount": 1363094, "candidatesTokenCount": 313616, "thoughtsTokenCount": 375079, "totalTokenCount": 2051789}. 10 evaluations required retry: 13 failed API attempts plus 672 successful calls, at least 685 model calls in total. Failed attempts, including HTTP500 responses, do not establish billing charges. Failed provider API attempts were retried with unchanged inputs, including one request recovered after three HTTP500 failures. Failed attempts do not establish a billed charge; no quality-based rejudging occurred.

## Limits

- One AI judge is not a native-speaker panel and does not establish human preference; reported language-specific errors require independent listening before product decisions.
- One passage and one synthesis per model/reference/language do not characterize all scenarios, accents or stochastic variation.
- Same-audio reversed order is a reliability check, not independent replication. Unstable decisions are not forced into wins.
- Absolute scores across languages are not directly comparable; the equal-language macro is descriptive and cannot justify a universal rollout.
- No reference recording was supplied, so identity/clone fidelity is not assessed. Existing language-specific references remain paired fairly.
- Cold direct-provider timing is not warm-pool or end-to-end production call latency.
- No production voice configuration, model rollout or active A/B experiment was changed.

Artifacts: gallery/, results.csv, summary.json. Private source requests, raw model responses, model mappings and frozen masked results remain under private/. Public outputs exclude voice references, credentials and customer/account identifiers.