Skip to main content

​
Intro

When comparing three or more speech synthesis models, a ranking evaluation is an effective method for determining the relative quality of each model. Rather than comparing pairs individually, evaluators listen to a set of audio samples generated from the same script and rank them from best to worst. The Ranking evaluation is flexible in its evaluation criteria. You can rank models based on naturalness, overall preference, clarity, expressiveness, or any other quality dimension that matters to your use case.
  • Objective: Determine the relative ordering of multiple models by having evaluators rank them.
  • Use Case: Ideal for comparing TTS providers, model versions, or synthesis configurations side by side.
  • Type: RANKING in the SDK, or RANKING_REF when every set should be judged against a reference.
ranking

​
Example

In this example, we compare three different TTS providers by generating speech from the same scripts and submitting them for ranking evaluation. Here is a code example that you can immediately execute:
python
Ok, let’s go line by line.
1

Create a Client

Let’s first create a new instance of Client.
python
2

Create an Evaluator

Then, you create a new instance of Evaluator with type='RANKING':
python
3

Generate speech and add ranking sets

For each script, generate speech from all providers and add a ranking set. Each ranking set contains one audio file per provider.
python
4

Close

Finally, close the Evaluator object.
python
With this, you can rank multiple TTS providers via podonos from real human evaluators. Once these steps finish, you can check the results in your Workspace.

​
Ranking Against a Reference

Sometimes “best” only means something relative to a target. If you are matching a specific voice, restoring a recording, or reproducing a reference performance, the question is not which sample sounds nicest but which one is closest to the reference. RANKING_REF adds exactly that: each ranking set carries one reference audio in addition to the candidates. Evaluators hear the reference in every match of that set, and it is never itself a candidate.
  • Objective: Rank models by how closely they match a per-script reference.
  • Use Case: Voice cloning, speech restoration, dubbing, any task with a target.
  • Type: RANKING_REF in the SDK.
The code is the ranking example with one extra File:
python

​
Rules

1

Exactly one reference per set

Every set needs one File with is_ref=True and at least two candidates. A set without a reference, or with two, is rejected when you call add_ranking_set().
2

The reference goes anywhere in the list

The SDK sorts it into position for you, so you can build the list however is convenient. These are equivalent:
python
3

One reference model, not one per script

The reference audio changes from script to script, but its model_tag must stay the same across every set. It names the role, not the file. Using Reference_1, Reference_2, … makes your workspace treat each one as a separate model and splits the reference into one row per tag.
python
4

The reference tag must differ from every candidate tag

Reference and Provider_A are fine together; a reference tagged Provider_A is not.
Candidates are still ranked only against each other. The reference is a listening aid, so it never appears in the Bradley-Terry scores and never shows up as a choice. A set of N candidates produces N-1 matches whether or not a reference is present.

​
Use Case

Consider a scenario where you are evaluating multiple TTS providers to decide which one to integrate into your product. Each provider may have different strengths. One might excel at naturalness while another handles proper nouns better. Using the Ranking evaluation, you can have human evaluators directly compare all providers on the same scripts and produce a clear ordering, giving you confidence in your selection.

​
How It Works

Rankings are computed using the Bradley-Terry (BT) model, a well-established statistical method for deriving global rankings from pairwise comparisons. Each pair of models is compared by human evaluators, and the aggregated results are used to estimate a score for every model via maximum likelihood estimation. A key challenge in pairwise ranking is deciding which pairs to compare. With many models, the number of possible pairs grows quickly, but evaluation budgets are limited. Comparing pairs with obvious quality differences wastes valuable evaluations, while neglecting certain pairs leaves gaps in the data. To address this, Podonos uses adaptive pairing based on Fisher information. Fisher information quantifies how much a given comparison will improve the overall ranking accuracy. Comparisons between similarly-ranked models yield the most information, while lopsided matchups yield little. The system dynamically balances exploration (ensuring all pairs are observed) and exploitation (focusing on the most informative pairs), adapting automatically as data accumulates throughout the evaluation.