SDK Template Usage

Create Evaluations from Templates

For developers, the SDK provides two powerful ways to create evaluations programmatically using templates:

Option 1: Using Template ID

Copy the Template ID from the Web

Navigate to the “Templates” Tab on the Workspace, select the template you want to use, and copy the Template ID.

Initialize the SDK

Initialize with your API key. For details, refer to Get API Key

client = podonos.init("<API_KEY>")

Create an evaluation using the template

Use the Template ID to create a new evaluation.

evaluator = client.create_evaluator_from_template(
    name="Evaluate the english voice",
    desc="model_1_and_model_2",
    num_eval=10,
    template_id="HTGRqU"  # Evaluation template ID from the Workspace
)

Add Files to the Evaluation

Add files with relevant metadata to the evaluator.

for i in range(60):
    file = File(path=f"path/speech_{i}.mp3", tags=["man", "noisy"],
                model_tag="Model 1")
    evaluator.add_file(file)

for i in range(60):
    file0 = File(path=f"path2/speech_{i}.mp3", tags=["man", "gentle"],
                 model_tag="Model 1")
    file1 = File(path=f"path2/speech_{i}.mp3", tags=["woman", "mild"],
                 model_tag="Model 2")
    evaluator.add_files(file0, file1)

Finalize the evaluation

Close the evaluator to finalize the setup.

evaluator.close()

Option 2: Using Template JSON

You can create evaluations by defining the template structure either directly in your code using a JSON dictionary or by loading it from a JSON file.

Intro

A template consists of three main sections:

Questions Section: Contains the core evaluation questions
- SCORED: Rating scale questions (e.g., 1-5 Likert scale)
- NON_SCORED: Multiple/Single choice questions
- COMPARISON: Comparison questions (for double evaluations)
Instructions Section (Optional): Contains guidance messages to the evaluators.
- These messages help ensure that evaluators understand how to conduct the evaluation properly. Providing clear and appropriate guidance can significantly enhance the quality of the evaluation results.
- Use one of the following types: DO, WARNING, DONT, EXAMPLE
Annotations Section (Optional): Collects free-form text feedback from evaluators.
- Allows evaluators to provide detailed written feedback beyond standard ratings.
- Use related_model to specify which audio the annotation applies to.

Types

Scored Question
Non-Scored Question
Comparison Question
Instruction
Annotation

{
    "type": "SCORED",
    "question": "Please evaluate the overall quality of the audio",
    "description": "You can ignore the wrong pronunciation of the audio",
    "options": [
        {"label_text": "Excellent"},
        {"label_text": "Good"},
        {
            "label_text": "Fair",
            "reference_file": "path/to/reference/file" # Helps the evaluator to select the quality of the audio
        },
        {"label_text": "Poor"},
        {"label_text": "Bad"},
    ]
}

For options in SCORED questions: - score is automatically generated only for SCORED questions. If there are 5 options, the first option in the list receives a score of 5, the second option receives a score of 4, and so on, down to a score of 1 for the last option. - order is the index of the option in the list, starting from 0. - For more detailed explanations, please refer to the reference documentation.

{
    "type": "NON_SCORED",
    "question": "Please select all the bad audio characteristics",
    "description": "Select only the audio characteristics that are distracting or undesirable.",
    "options": [
        {"label_text": "Background Noise"},
        {
            "label_text": "Echo",
            "reference_file": "path/to/reference/file" # Can tell why this audio has echo
        },
        {"label_text": "Distortion"}
    ],
    "allow_multiple": true, # If the evaluator can select multiple options
    "has_other": true # If the evaluator can select other options
}

For options in NON_SCORED questions:

allow_multiple is a boolean value that determines if the evaluator can select multiple options.
has_other is a boolean value that determines if the evaluator can select other options.
For more detailed explanations, please refer to the reference documentation.

{
    "type": "COMPARISON",
    "question": "Listen to the speech/audio samples and rate which one you prefer.",
    "description": "Ignore the background noise and audio quality",
    "scale": 5 # Required. 5-point comparison scale
    "anchor_label": {
        "title": "Preferences", # Optional
        "label_text": {
            "left": "is better", # The prefix (A) is automatically added to the left label
            "right": "is better" # The prefix (B) is automatically added to the right label
        }
    }
}

For scale in COMPARISON questions:

scale is an required integer value between 2 and 9.
anchor_label should be provided for giving context to the evaluator.
For more detailed explanations, please refer to the reference documentation.

{
    "type": "EXAMPLE",
    "instruction": "Use the provided reference file to assess the quality of the audio.",
    "description": "Audio quality is not considered for the evaluation",
    "reference_files": [
        {
            "path": "./aws/speech_1_Joanna.mp3",
            "type": "audio"
        },
        {
            "path": "./aws/reference_1.mp3",
            "type": "reference"
        },
        {
            "path": "./aws/target_1.mp3",
            "type": "target"
        }
    ]
}

For reference_files in Instruction:

reference_files is an optional array that helps the evaluator to select the quality of the audio.
Each file must have a path and a type. The type must be one of: audio, reference, or target.
Use the EXAMPLE type when you want to provide users with more examples to better understand the evaluation context.
For more detailed explanations, please refer to the reference documentation.

Annotation questions collect free-form text feedback from evaluators beyond standard rating scales.

{
    "type": "ANNOTATION",
    "question": "Describe any issues you noticed in the audio",
    "description": "e.g., noise, pronunciation errors, unnatural intonation",
    "related_model": "ALL"
}

For double stimulus evaluations, specify which audio the annotation applies to:

{
    "annotations": [
        {
            "type": "ANNOTATION",
            "question": "Describe issues in Audio A",
            "related_model": "MODEL_A"
        },
        {
            "type": "ANNOTATION",
            "question": "Describe issues in Audio B",
            "related_model": "MODEL_B"
        }
    ]
}

For annotations:

Annotations are defined in a separate annotations array, not in questions.
related_model must be "ALL" for single stimulus, or "MODEL_A"/"MODEL_B" for double stimulus.
You can also enable annotations automatically with use_annotation=True parameter instead of defining the array.
Cannot use both use_annotation=True and annotations array together.
For more detailed explanations, please refer to the reference documentation.

Example

Here’s a step-by-step guide to create an evaluation using the template JSON:

Prepare Template

Define your template as a Python dictionary or save it as a JSON file:

# Single evaluation template example
TEMPLATE = {
    "instructions": [
        {
            "type": "DO",
            "instruction": "Use the provided reference file to assess the quality of the audio.",
            "reference_files": [
                {
                    "path": "./aws/speech_1_Joanna.mp3",
                    "type": "audio"
                },
                {
                    "path": "./aws/speech_1_Joanna.mp3",
                    "type": "reference"
                }
            ]
        },
        {
            "type": "WARNING",
            "instruction": "Please evaluate only the audio's natural pronunciation",
            "description": "Just focus on the audio's naturalness, the background noise or audio quality is not considered for the evaluation",
        },
    ],
    "questions": [
        {
            "type": "SCORED",
            "question": "Please evaluate the overall quality of the audio",
            "description": "You can ignore the wrong pronunciation of the audio",
            "options": [
                {"label_text": "Excellent"},
                {"label_text": "Good"},
                {"label_text": "Fair", "reference_file": "path/to/reference/file"},
                {"label_text": "Poor"},
                {"label_text": "Bad"},
            ]
        },
        {
            "type": "NON_SCORED",
            "question": "Select all the bad audio characteristics",
            "description": "Select only the audio characteristics that are distracting or undesirable.",
            "options": [
                {"label_text": "Background Noise"},
                {"label_text": "Echo", "reference_file": "path/to/reference/file"},
                {"label_text": "Distortion"}
            ],
            "allow_multiple": true,
            "has_other": true
        }
    ],
    "annotations": [
        {
            "type": "ANNOTATION",
            "question": "If you rated poorly, please describe the issues",
            "description": "e.g., noise, pronunciation errors, unnatural intonation",
            "related_model": "ALL"
        }
    ]
}

# Double evaluation template example
TEMPLATE = {
    "instructions": [
        {
            "type": "EXAMPLE",
            "instruction": "Evaluation Guidelines",
            "description": "Important points to consider when evaluating audio",
            "reference_files": [
                {
                    "path": "./aws/speech_1_Joanna.mp3",
                    "type": "audio"
                },
                {
                    "path": "./aws/speech_1_reference.mp3",
                    "type": "reference"
                },
                {
                    "path": "./aws/speech_1_target.mp3",
                    "type": "target"
                }
            ]
        },
        {
            "type": "DONT",
            "instruction": "Please do not consider background noise when evaluating the audio's clarity.",
            "description": "Just focus on the audio's naturalness, the background noise or audio quality is not considered for the evaluation"
        }
    ],
    "questions": [
        {
            "type": "COMPARISON",
            "question": "Which audio sample is clearer?",
            "description": "Focus on the clarity of the speech",
            "scale": 5,
            "anchor_label": {
                "title": "Clarity",
                "label_text": {
                    "left": "is clearer",
                    "right": "is clearer"
                }
            }
        },
        {
            "type": "NON_SCORED",
            "question": "Why did you choose the sample?",
            "options": [
                {"label_text": "Good pronunciation"},
                {"label_text": "Naturalness"},
                {"label_text": "Emotionality"}
            ],
            "allow_multiple": false,
            "has_other": true,
            "related_model": "MODEL_A"
        }
    ],
    "annotations": [
        {
            "type": "ANNOTATION",
            "question": "Describe any issues in Audio A",
            "related_model": "MODEL_A"
        },
        {
            "type": "ANNOTATION",
            "question": "Describe any issues in Audio B",
            "related_model": "MODEL_B"
        }
    ]
}

{
    "instructions": [
        {
            "type": "DO",
            "instruction": "Use the provided reference file to assess the quality of the audio.",
            "reference_files": [
                {
                    "path": "./aws/speech_1_Joanna.mp3",
                    "type": "audio"
                },
                {
                    "path": "./aws/speech_1_Joanna.mp3",
                    "type": "reference"
                }
            ]
        },
        {
            "type": "WARNING",
            "instruction": "Please evaluate only the audio's natural pronunciation",
            "description": "Just focus on the audio's naturalness, the background noise or audio quality is not considered for the evaluation"
        }
    ],
    "questions": [
        {
            "type": "SCORED",
            "question": "Please evaluate the overall quality of the audio",
            "description": "You can ignore the wrong pronunciation of the audio",
            "options": [
                {"label_text": "Excellent"},
                {"label_text": "Good"},
                {"label_text": "Fair", "reference_file": "path/to/reference/file"},
                {"label_text": "Poor"},
                {"label_text": "Bad"}
            ]
        },
        {
            "type": "NON_SCORED",
            "question": "Select all the bad audio characteristics",
            "description": "Select only the audio characteristics that are distracting or undesirable.",
            "options": [
                {"label_text": "Background Noise"},
                {"label_text": "Echo", "reference_file": "path/to/reference/file"},
                {"label_text": "Distortion"}
            ],
            "allow_multiple": true,
            "has_other": true
        }
    ],
    "annotations": [
        {
            "type": "ANNOTATION",
            "question": "If you rated poorly, please describe the issues",
            "description": "e.g., noise, pronunciation errors, unnatural intonation",
            "related_model": "ALL"
        }
    ]
}

{
  "instructions": [
    {
      "type": "EXAMPLE",
      "instruction": "Evaluation Guidelines",
      "description": "Important points to consider when evaluating audio",
      "reference_files": [
        {
          "path": "./aws/speech_1_Joanna.mp3",
          "type": "audio"
        },
        {
          "path": "./aws/speech_1_reference.mp3",
          "type": "reference"
        },
        {
          "path": "./aws/speech_1_target.mp3",
          "type": "target"
        }
      ]
    },
    {
      "type": "DONT",
      "instruction": "Please do not consider background noise when evaluating the audio's clarity.",
      "description": "Just focus on the audio's naturalness, the background noise or audio quality is not considered for the evaluation"
    }
  ],
  "questions": [
    {
      "type": "COMPARISON",
      "question": "Which audio sample is clearer?",
      "description": "Focus on the clarity of the speech",
      "scale": 5,
      "anchor_label": {
        "title": "Clarity",
        "label_text": {
          "left": "is clearer",
          "right": "is clearer"
        }
      }
    },
    {
      "type": "NON_SCORED",
      "question": "Why did you choose the sample?",
      "options": [
        { "label_text": "Good pronunciation" },
        { "label_text": "Naturalness" },
        { "label_text": "Emotionality" }
      ],
      "allow_multiple": false,
      "has_other": true,
      "related_model": "MODEL_A"
    }
  ],
  "annotations": [
    {
      "type": "ANNOTATION",
      "question": "Describe any issues in Audio A",
      "related_model": "MODEL_A"
    },
    {
      "type": "ANNOTATION",
      "question": "Describe any issues in Audio B",
      "related_model": "MODEL_B"
    }
  ]
}

{
  "instructions": [
    {
      "type": "DO",
      "instruction": "Listen to every sample all the way through before ranking.",
      "description": "Rank the samples from best to worst on overall naturalness."
    }
  ],
  "questions": [
    {
      "type": "COMPARISON",
      "question": "Which sample sounds more natural?",
      "description": "Focus on naturalness, not recording quality",
      "anchor_label": {
        "title": "Naturalness",
        "label_text": {
          "left": "is more natural",
          "right": "is more natural"
        }
      }
    }
  ]
}

{
  "instructions": [
    {
      "type": "DO",
      "instruction": "Listen to the reference first, then to both samples.",
      "description": "Judge which sample is closer to the reference, not which one you prefer."
    }
  ],
  "questions": [
    {
      "type": "COMPARISON",
      "question": "Which sample is closer to the reference?",
      "anchor_label": {
        "title": "Similarity to reference",
        "label_text": {
          "left": "is closer",
          "right": "is closer"
        }
      }
    }
  ]
}

Ranking templates (RANKING and RANKING_REF) accept only COMPARISON questions and instructions. A SCORED or NON_SCORED question raises when the template is validated. This is because ranking is derived from pairwise matches rather than per-file scores.

When using a JSON file:

Save the file with .json extension
Ensure proper JSON formatting
Use UTF-8 encoding

Create Evaluation

For single and double evaluations (evaluating one or two files at a time):

# Single evaluation template example
evaluator = client.create_evaluator_from_template_json(
    json=SINGLE_TEMPLATE,
    name="Single Audio Quality Test",
    desc="model_a_and_model_b",
    num_eval=10,
    custom_type="SINGLE" # Specify SINGLE type
)

# Add single files

for i in range(3):
file = File(path=f"./audio/speech\_{i}.mp3",
model_tag="Model A",
tags=["clean", "male"])
evaluator.add_file(file)

# Double evaluation template example
evaluator = client.create_evaluator_from_template_json(
    json=DOUBLE_TEMPLATE,
    name="Double Audio Quality Test",
    desc="model_a_and_model_b",
    num_eval=10,
    custom_type="DOUBLE" # Specify DOUBLE type
)

# Add file pairs
for i in range(3):
    file_a = File(path=f"./audio/model_a_{i}.mp3",
                  model_tag="Model A",
                  tags=["enhanced"])
    file_b = File(path=f"./audio/model_b_{i}.mp3",
                  model_tag="Model B",
                  tags=["baseline"])
    evaluator.add_files(file_a, file_b)

# Single evaluation template example
evaluator = client.create_evaluator_from_template_json(
    json_file="path/to/single_evaluation.json",
    name="Single Audio Quality Test",
    custom_type="SINGLE"  # Specify SINGLE type
)

# Add single files
for i in range(3):
    file = File(path=f"./audio/speech_{i}.mp3",
                model_tag="Model A",
                tags=["clean", "male"])
    evaluator.add_file(file)

# Double evaluation template example
evaluator = client.create_evaluator_from_template_json(
    json_file="path/to/double_evaluation.json",
    name="Audio Comparison Test",
    custom_type="DOUBLE"  # Specify DOUBLE type
)

# Add file pairs
for i in range(3):
    file_a = File(path=f"./audio/model_a_{i}.mp3",
                  model_tag="Model_A",
                  tags=["enhanced"])
    file_b = File(path=f"./audio/model_b_{i}.mp3",
                  model_tag="Model_B",
                  tags=["baseline"])
    evaluator.add_files(file_a, file_b)

# Ranking template example. Ranking templates may only carry COMPARISON
# questions and instructions -- SCORED and NON_SCORED are rejected.
evaluator = client.create_evaluator_from_template_json(
    json=RANKING_TEMPLATE,
    name="TTS Provider Ranking",
    num_eval=10,
    custom_type="RANKING"  # Specify RANKING type
)

# Add one ranking set per script
for i in range(3):
    evaluator.add_ranking_set([
        File(path=f"./audio/model_a_{i}.mp3", model_tag="Model_A"),
        File(path=f"./audio/model_b_{i}.mp3", model_tag="Model_B"),
        File(path=f"./audio/model_c_{i}.mp3", model_tag="Model_C"),
    ])

# Same as RANKING, plus one reference per set. The reference is heard in every
# match of that set and is never a candidate.
evaluator = client.create_evaluator_from_template_json(
    json=RANKING_TEMPLATE,
    name="Ranking Against a Reference",
    num_eval=10,
    custom_type="RANKING_REF"  # Specify RANKING_REF type
)

for i in range(3):
    evaluator.add_ranking_set([
        File(path=f"./audio/model_a_{i}.mp3", model_tag="Model_A"),
        File(path=f"./audio/model_b_{i}.mp3", model_tag="Model_B"),
        File(path=f"./audio/model_c_{i}.mp3", model_tag="Model_C"),
        # Same model_tag in every set: it names the role, not the file.
        File(path=f"./audio/reference_{i}.mp3", model_tag="Reference", is_ref=True),
    ])

custom_type accepts SINGLE, DOUBLE, SINGLE_REF, RANKING, and RANKING_REF. Use add_file() for SINGLE, add_files() for DOUBLE and SINGLE_REF, and add_ranking_set() for RANKING and RANKING_REF.model_tag accepts letters, digits, hyphens and underscores only — "Model A" raises, "Model_A" works.

For single evaluations:

Use add_file() method for adding individual files
Each file is evaluated independently
Avoid using COMPARISON type questions

For double evaluations:

Use add_files() method to add pairs of files
Files are always evaluated in pairs
COMPARISON type questions are supported
Consider using clear model_tag for distinguishing each file

Finalize the Evaluation

Close the evaluator to finalize the setup:

evaluator.close()

Tip: Using the SDK is ideal for integrating evaluations into automated workflows.

Get Started

Basics

Details

Use Cases

Roadmap

SDK References

SDK Template Usage

Create Evaluations from Templates

Option 1: Using Template ID

Option 2: Using Template JSON

Intro

Types

Example

​Create Evaluations from Templates

​Option 1: Using Template ID

​Option 2: Using Template JSON

​Intro

​Types

​Example

Create Evaluations from Templates

Option 1: Using Template ID

Option 2: Using Template JSON

Intro

Types

Example