Reference

Podonos

This is a base module. You can import:

python

import podonos
from podonos import *

init()

Initialize the module and return an instance of Client.

string

API key you obtained from the workspace. For details, see the Get API key. If this is not set, the package tries to read PODONOS_API_KEY from the environment variable. Throws an error if both of them are not available.

string

default:"https://prod.podonosapi.com"

API base URL. You usually do not need to set this unless you are using a private, staging, or development API endpoint.

Returns an instance of Client.

python

client = podonos.init(api_key="<API_KEY>")

# Optional: pass the production API endpoint explicitly
client = podonos.init(api_key="<API_KEY>", api_url="https://prod.podonosapi.com")

Client

Client manages one or more Evaluator instances and evaluation history.

create_evaluator()

Create a new instance of Evaluator. One evaluator supports a single type of evaluation throughout its life cycle. If you want multiple types of evaluation, create multiple evaluators by calling create_evaluator() multiple times.

string

Name of this evaluation session. If empty, a random name is automatically generated and used.

string

Description of this evaluation session. This field is for your records, so later you can see how you generated the output files or trained your model.

string

default:"NMOS"

Evaluation type. One of the following:

Type	Description
`NMOS`	Naturalness Mean Opinion Score
`QMOS`	Quality Mean Opinion Score
`CMOS`	Reference-based comparison (compare target audio against reference)
`CSMOS`	Comparative similarity test between two target audios and one reference
`SMOS`	Similarity Mean Opinion Score
`P808`	Speech quality by ITU-T P.808
`PREF`	Preference test between two audios/speeches
`CUSTOM_SINGLE`	Custom single-stimulus evaluation
`CUSTOM_DOUBLE`	Custom double-stimulus evaluation
`RANKING`	Ranking evaluation across multiple stimuli
`RANKING_REF`	Ranking across multiple stimuli against one reference per set

string

default:"en-us"

Specific language and locale of the speech. Currently we support:

Code	Description
`en-us`	English (United States)
`en-gb`	English (United Kingdom)
`en-au`	English (Australia)
`en-ca`	English (Canada)
`en-in`	English (India)
`ko-kr`	Korean (Korea)
`zh-cn`	Mandarin (China)
`es-es`	Spanish (Spain)
`es-mx`	Spanish (Mexico)
`fr-fr`	French (France)
`fr-ca`	French (Canada)
`de-de`	German (Germany)
`ja-jp`	Japanese (Japan)
`it-it`	Italian (Italy)
`pl-pl`	Polish (Poland)
`pt-pt`	Portuguese (Portugal)
`pt-br`	Portuguese (Brazil)
`id-id`	Indonesian (Indonesia)
`nl-nl`	Dutch (Netherlands)
`sv-se`	Swedish (Sweden)
`hi-in`	Hindi (India)
`ta-in`	Tamil (India)
`kn-in`	Kannada (India)
`ml-in`	Malayalam (India)
`si-lk`	Sinhala (Sri Lanka)
`bn-in`	Bengali (India)
`gu-in`	Gujarati (India)
`mr-in`	Marathi (India)
`te-in`	Telugu (India)
`ar-eg`	Arabic (Egypt)
`ar-ae`	Arabic (UAE)
`ar-sa`	Arabic (Saudi Arabia)
`audio`	General audio file

We will add more soon. Please check later again.

float

default:"1.0"

Granularity of evaluation scales. Supported values are 1.0 and 0.5.

int

default:"10"

Number of evaluations per sample. For example, if this is 10 for NMOS type evaluation, each audio file will be assigned to 10 humans, and the statistics of the evaluation output will be computed and presented in the final report.

int

default:"12"

Expected due time of the final report in hours. Must be at least 12. Depending on the hours, the pricing may change.

bool

default:"False"

Enable annotation to collect free-form text feedback from evaluators.

bool

default:"True"

Enable loudness normalization to ensure consistent audio volume levels during evaluation.

bool

default:"False"

If True, close() waits for the uploaded files to finish processing, then starts the evaluation and charges your workspace balance. It raises if it cannot start. If False, close() returns as soon as the uploads are done and nothing is charged until you start the evaluation yourself in Workspace. Requires podonos 0.46.0 or newer; earlier versions accept the parameter and silently ignore it.Pair it with resume_upload=True and an upload_state_path. If the start fails or times out, resume_evaluator() is the only way back to that evaluation; without a ledger you have to start it from Workspace instead.

int

default:"20"

Maximum number of upload worker threads. If you experience a slow upload, please increase the number of workers.

int

default:"100"

Batch size for file verification API calls. Must be between 1 and 1000.

tuple[float, float]

default:"(5, 30)"

Timeout tuple for API requests, in seconds.

tuple[float, float]

default:"(5, 120)"

Timeout tuple for file verification requests, in seconds.

tuple[float, float]

default:"(10, 300)"

Timeout tuple for direct file upload requests, in seconds.

bool

default:"False"

Enable SDK-local upload ledger/resume support. The ledger records upload progress and the original evaluation contract so interrupted uploads can be resumed safely.

string

default:"None"

Optional SQLite upload ledger path when resume_upload=True. Use the same path with resume_evaluator() if the upload is interrupted.

float

default:"1800"

Seconds close() waits when auto_start=True. Must be positive. It limits when the SDK stops issuing new requests, so a request already in flight can finish somewhat after it. Lower it in CI, where a blocking close() costs runner time.

Returns an instance of Evaluator.

etor = client.create_evaluator()

etor = client.create_evaluator(
    name="sim_iter100",
    desc="new_model_vs_sota",
    type="SMOS",
)

etor = client.create_evaluator(
    name="sim_iter100",
    desc="new_model_vs_sota",
    type="SMOS",
    lan="en-us",
    granularity=1.0,
    num_eval=100,
    due_hours=24,
    use_loudness_normalization=True,
    auto_start=True,
    verify_batch_size=100,
    resume_upload=True,
    upload_state_path="./podonos-upload-ledger.sqlite3",
    start_timeout=1800,
)

resume_evaluator()

Resume uploads for an existing evaluation using an SDK upload ledger. Add files again in the same order as the interrupted run so the ledger can recover each file’s remote object identity.

Resume is safety-checked against the local ledger. The ledger must contain the original evaluation contract and session configuration for evaluation_id; otherwise the SDK refuses to resume. The SDK restores the original session details from the ledger where available, so do not use resume_evaluator() to change the evaluation type, language, template, batch size, or file identities.During resume, completed ledger rows are reused only when the current local file still matches the recorded path, content hash, size, and upload manifest. If a completed metadata or verification row no longer matches the local file, start a fresh evaluation or remove the stale ledger row before resuming.

string

required

Evaluation ID to resume.

string

required

Path to the SDK upload ledger created by a previous upload run. It must contain the original evaluation contract for evaluation_id.

string

Optional session name.

string

Optional session description.

string

default:"NMOS"

Evaluation type. Uses the same supported values as create_evaluator().

string

default:"en-us"

Language code. Uses the same supported values as create_evaluator().

float

default:"1.0"

Granularity of evaluation scales.

int

default:"10"

Number of evaluations per sample.

int

default:"12"

Expected due time of the final report in hours.

bool

default:"False"

Enable annotation to collect free-form text feedback from evaluators.

bool

default:"True"

Enable loudness normalization for evaluation.

bool

default:"False"

Restored from the resumed session’s ledger, so the value you pass here is ignored. If the original run set it, close() blocks and starts the evaluation — charging your workspace balance — even when you pass False. The SDK logs a warning whenever the restored value disagrees with the one you passed. Requires podonos 0.46.0 or newer; earlier versions accept the parameter and silently ignore it.

int

default:"20"

Maximum number of upload worker threads.

int

default:"100"

Batch size for file verification API calls.

tuple[float, float]

default:"(5, 30)"

Timeout tuple for API requests, in seconds.

tuple[float, float]

default:"(5, 120)"

Timeout tuple for file verification requests, in seconds.

tuple[float, float]

default:"(10, 300)"

Timeout tuple for direct file upload requests, in seconds.

float

default:"1800"

Seconds close() waits when the restored auto_start is True. Unlike auto_start this is not restored from the ledger, so the value you pass here wins.

python

etor = client.resume_evaluator(
    evaluation_id="<EVALUATION_ID>",
    upload_state_path="./podonos-upload-ledger.sqlite3",
)

create_evaluator_from_template()

When you create an evaluation using a template, all the questions and options defined in the template are automatically assigned to the new evaluation. This ensures consistency and saves time by reusing pre-defined content.

string

required

Name of this evaluation session. Required and must be non-empty.

string

required

The unique identifier of the template to base the new evaluation on.

string

Description of this evaluation session. This field is for your records.

int

default:"10"

Number of evaluations per sample.

bool

default:"False"

Enable annotation to collect free-form text feedback from evaluators. Cannot be used together with an annotations array in template JSON.

bool

default:"True"

Enable loudness normalization to ensure consistent audio volume levels during evaluation.

bool

default:"False"

int

default:"20"

Maximum number of upload worker threads.

int

default:"100"

Batch size for file verification API calls.

tuple[float, float]

default:"(5, 30)"

Timeout tuple for API requests, in seconds.

tuple[float, float]

default:"(5, 120)"

Timeout tuple for file verification requests, in seconds.

tuple[float, float]

default:"(10, 300)"

Timeout tuple for direct file upload requests, in seconds.

bool

default:"False"

Enable SDK-local upload ledger/resume support. The ledger records upload progress and the original evaluation contract so interrupted uploads can be resumed safely.

string

default:"None"

Optional SQLite upload ledger path when resume_upload=True. Use the same path with resume_evaluator() if the upload is interrupted.

float

default:"1800"

etor = client.create_evaluator_from_template(
    name="Voice naturalness evaluation",
    desc="new_model_vs_competitor_model",
    num_eval=10,
    template_id="abcdef",
    use_loudness_normalization=True,
)

create_evaluator_from_template_json()

Create a new evaluation using a JSON template. This allows you to define custom evaluation structures programmatically.

Dict

Template JSON as a dictionary. Optional if json_file is provided.

string

Path to the JSON template file. Optional if json is provided.

string

Name of this evaluation session. Optional; if omitted, the SDK generates a name.

string | CustomType

default:"SINGLE"

Type of evaluation. Accepts either a string or CustomType enum value.

Value	Description	batch_size	File Configuration
`SINGLE`	Single stimulus evaluation	1	1 stimulus
`DOUBLE`	Double stimulus evaluation	2	2 stimuli (no reference)
`SINGLE_REF`	Reference-based comparison (CMOS style)	2	1 reference + 1 stimulus
`RANKING`	Ranking evaluation	2+	2+ stimuli
`RANKING_REF`	Ranking against a reference	3+	2+ stimuli + 1 reference

You can use string values ("SINGLE", "DOUBLE", "SINGLE_REF", "RANKING", "RANKING_REF") or the CustomType enum from podonos.common.enum:

from podonos.common.enum import CustomType

custom_type=CustomType.SINGLE_REF
custom_type="SINGLE_REF"

string

Description of this evaluation session. Optional.

string

default:"en-us"

Language for evaluation. See supported languages in create_evaluator().

int

default:"10"

Number of evaluations per sample.

bool

default:"False"

Enable annotation to collect free-form text feedback from evaluators. Cannot be used together with an annotations array in template JSON. When using custom annotation questions, define them in the annotations array instead.

bool

default:"True"

Enable loudness normalization to ensure consistent audio volume levels during evaluation.

bool

default:"False"

int

default:"20"

Maximum number of upload workers. Must be a positive integer.

int

default:"100"

Batch size for file verification API calls.

tuple[float, float]

default:"(5, 30)"

Timeout tuple for API requests, in seconds.

tuple[float, float]

default:"(5, 120)"

Timeout tuple for file verification requests, in seconds.

tuple[float, float]

default:"(10, 300)"

Timeout tuple for direct file upload requests, in seconds.

bool

default:"False"

Enable SDK-local upload ledger/resume support. The ledger records upload progress and the original evaluation contract so interrupted uploads can be resumed safely.

string

default:"None"

Optional SQLite upload ledger path when resume_upload=True. Use the same path with resume_evaluator() if the upload is interrupted.

float

default:"1800"

# Using JSON dictionary
template = {
    "questions": [
        {
            "type": "SCORED",
            "question": "How natural is the voice?",
            "description": "Rate the quality of the voice",
            "options": [
                {"label_text": "Excellent"},
                {"label_text": "Good"},
                {"label_text": "Fair"},
                {"label_text": "Poor"},
                {"label_text": "Bad"},
            ],
        }
    ]
}

evaluator = client.create_evaluator_from_template_json(
    json=template,
    name="Quality Test",
    custom_type="SINGLE",
)

# Using JSON file
evaluator = client.create_evaluator_from_template_json(
    json_file="./template.json",
    name="Quality Test",
    custom_type="DOUBLE",
)

from podonos import File
from podonos.common.enum import CustomType

# CMOS-style reference comparison
template = {
    "questions": [
        {
            "type": "SCORED",
            "question": "How similar is the synthesized audio to the reference?",
            "options": [
                {"label_text": "Identical"},
                {"label_text": "Very similar"},
                {"label_text": "Somewhat similar"},
                {"label_text": "Different"},
                {"label_text": "Completely different"},
            ],
        }
    ]
}

evaluator = client.create_evaluator_from_template_json(
    json=template,
    name="CMOS Style Evaluation",
    custom_type=CustomType.SINGLE_REF,  # or "SINGLE_REF"
)

# Add files: 1 stimulus + 1 reference
evaluator.add_files(
    file0=File(path="synthesized.wav", model_tag="tts"),
    file1=File(path="reference.wav", model_tag="human", is_ref=True),
)

from podonos import File

# Ranking templates take COMPARISON questions only.
template = {
    "questions": [
        {
            "type": "COMPARISON",
            "question": "Which sample is closer to the reference?",
            "anchor_label": {"label_text": {"left": "is closer", "right": "is closer"}},
        }
    ]
}

evaluator = client.create_evaluator_from_template_json(
    json=template,
    name="Ranking Against a Reference",
    custom_type="RANKING_REF",
)

# One set per script: 2+ stimuli plus exactly one reference.
# The reference keeps the same model_tag in every set.
for i in range(3):
    evaluator.add_ranking_set([
        File(path=f"model_a_{i}.wav", model_tag="model_a"),
        File(path=f"model_b_{i}.wav", model_tag="model_b"),
        File(path=f"model_c_{i}.wav", model_tag="model_c"),
        File(path=f"reference_{i}.wav", model_tag="reference", is_ref=True),
    ])

Returns an instance of Evaluator. Here’s the JSON template for reference:

Question
Option
Anchor Label
Instruction
Annotation

Question: Represents the main question posed to evaluators about the audio being assessed. It guides evaluators on the specific aspect of the audio they should focus on during the evaluation.

Parameter	Description	Required	Notes
`type`	Type of question. Options: `SCORED`, `NON_SCORED`, `COMPARISON`	Yes	Determines the structure and requirements of the question
`question`	The main question text	Yes	Must be provided for all question types
`description`	Additional details or context for the question	No	Optional for all question types
`options`	List of possible options. Only for `SCORED` and `NON_SCORED` types	Conditional	Must have between 1 and 9 options for `SCORED` and `NON_SCORED` types
`scale`	Scale for comparison. Only for `COMPARISON` type	Conditional	Must be an integer between 2 and 9 for `COMPARISON` type
`allow_multiple`	Allows multiple selections. Only for `NON_SCORED` type	Yes for `NON_SCORED`	Enables multiple choice selection
`has_other`	Includes an “Other” option. Only for `NON_SCORED` type	No	Adds an option for evaluators to specify an unlisted choice
`has_none`	Includes a “None” option. Only for `NON_SCORED` type	No	Adds an option for evaluators to select none of the listed choices
`related_model`	Related model for the question. Only for `Double` Evaluation type.	Conditional	Select which model the question is related to.
`anchor_label`	Labels for the ends of the comparison scale. Only for `COMPARISON` type.	Conditional	Provides context for what each end of the scale represents.

Important Notes:

SCORED and NON_SCORED questions can have a maximum of 9 options.
NON_SCORED questions must specify allow_multiple.
COMPARISON type questions must have a scale between 2 and 9.
related_model consists of ALL, MODEL_A and MODEL_B. Default is ALL. The related_model is only used for the question (not for instructions).

Option: Represents a possible answer or choice in a question. It provides evaluators with a range of options to choose from.

Parameter	Description	Required	Notes
`label_text`	The text displayed for the option	Yes	This is the text shown to the evaluator
`reference_file`	A reference audio file for the option. It helps the evaluator understand the quality of the audio	No	This file serves as a benchmark and is saved in the evaluation results

For options in SCORED and NON_SCORED questions:

score is automatically generated only for SCORED questions. If there are 5 options, the first option in the list receives a score of 5, the second option receives a score of 4, and so on, down to a score of 1 for the last option.
order is the index of the option in the list, starting from 0.

Anchor Label: Only for COMPARISON type. Provides context for what each end of the scale represents, enhancing evaluator understanding.

Parameter	Description	Required	Notes
`title`	The title of the anchor label	No	Provides a clear and concise title for the anchor label
`label_text`	The text displayed for the anchor label. `left` and `right` should be provided for `COMPARISON` type.	Yes	This is the text shown to the evaluator

For COMPARISON type:

left and right should be provided for label_text.
title is optional.

{
  "anchor_label": {
    "title": "Clarity",
    "label_text": {
      "left": "Much worse",
      "right": "Much better"
    }
  }
}

Instruction: Provides key guidance to evaluators on how to approach the evaluation.

Parameter	Description	Required	Notes
`type`	Type of instruction. Options: `DO`, `WARNING`, `DONT`, `EXAMPLE`	Yes	Guides evaluators on how to approach the evaluation
`instruction`	The main instruction text	Yes	Provides key guidance to evaluators
`description`	Additional details or context for the instruction	No	Optional for all instruction types
`reference_files`	A list of reference files to set a benchmark for evaluators	No	Up to 3 items. Each item must be a dictionary with `path` and `type`, where `type` is one of `audio`, `reference`, or `target`.

Additional Notes:

Instruction is optional.
description and reference_files are optional but can provide valuable context.
reference_files uses this shape: [{"path": "./sample.wav", "type": "audio"}].

Annotation: Collects free-form text feedback from evaluators. Defined in the annotations array.

Parameter	Description	Required	Notes
`type`	Must be `ANNOTATION`	Yes	Identifies this as an annotation question
`question`	The question text shown to evaluators	Yes	Cannot be empty
`related_model`	Which audio the annotation applies to	Conditional	`ALL` for single stimulus, `MODEL_A` or `MODEL_B` for double stimulus
`description`	Additional guidance for evaluators	No	Optional hint for what feedback to provide

Important Notes:

The annotations array is separate from the questions array.
For single stimulus evaluations, use related_model: "ALL" or omit it.
For double stimulus evaluations, related_model must be "MODEL_A" or "MODEL_B".
Cannot use annotations array together with use_annotation=True parameter.

Example:

{
  "questions": [...],
  "annotations": [
    {
      "type": "ANNOTATION",
      "question": "Describe any issues you noticed",
      "related_model": "ALL",
      "description": "e.g., noise, pronunciation errors"
    }
  ]
}

flash_eval()

Run automatic evaluation on an audio file and return a FlashEvalResult.

string

required

Path to the audio file to evaluate.

string

Language code for model routing. Currently available: en-us, es-es. When omitted, the default en-us model is used. Ignored when category="noise_quality".

string

Evaluation category. Defaults to naturalness when omitted. Currently available: naturalness, noise_quality.

Returns a FlashEvalResult with these fields:

Field	Description
`naturalness`	Naturalness score when `category="naturalness"`
`noise_quality`	Noise quality score when `category="noise_quality"`
`file`	The evaluated `File` object
`id`	Optional evaluation ID
`message`	Optional service message

python

result = client.flash_eval(file_path="path/to/audio.wav")
print(result.naturalness)

spanish_result = client.flash_eval(file_path="path/to/audio_es.wav", language="es-es")
noise_result = client.flash_eval(file_path="path/to/audio.wav", category="noise_quality")
print(noise_result.noise_quality)

get_evaluation_list()

Returns a JSON containing all your evaluations, each with its current status and progress.

evaluations = client.get_evaluation_list()
print(evaluations)

The output JSON looks like:

[
  {
    "id": "<UUID>",
    "title": "How natural my synthetic voices are",
    "internal_name": null,
    "description": "Used latest internal model. Epoch 10, alpha 0.1",
    "batch_size": 1,
    "status": "ACTIVE",
    "progress": 52.5,
    "started_time": "2024-06-25T02:03:11.005Z",
    "ended_time": null,
    "created_time": "2024-06-25T01:40:43.429Z",
    "updated_time": "2024-06-26T13:21:34.801Z"
  }
]

progress, started_time and ended_time are reported by podonos 0.47.0 or newer. started_time and ended_time are null until the evaluation starts and ends, and all three are null when the service does not report them, so check for null before comparing.

An evaluation is finished when status is "COMPLETED", never when progress reaches 100. progress is capped at 90 while the evaluation is running and stays at 90 during report review, so it cannot tell “almost done” from “done”. Use status to decide, and progress for display only.To check a single evaluation instead of listing the whole workspace, see get_evaluation_progress().

get_evaluation_progress()

Returns the current status and progress of one evaluation, without listing the rest of the workspace. Requires podonos 0.47.0 or newer.

string

required

Evaluation id. See get_evaluation_list(), or Evaluator.get_evaluation_id() on an evaluator you created.

Returns an EvaluationProgress with these fields. Call .to_dict() for the same values as a JSON-style dictionary.

Field	Type	Description
`id`	`str`	The evaluation id
`status`	`str`	One of `DRAFT`, `SUBMITTED`, `SCHEDULED`, `ACTIVE`, `COMPLETED`, `CANCELED`, `DELETED`
`progress`	`float`	0 to 100. Capped at 90 while running; 100 only once `status` is `COMPLETED`
`started_time`	`datetime` or `None`	UTC. `None` until the evaluation starts
`ended_time`	`datetime` or `None`	UTC. `None` until the evaluation ends

import time
from podonos.errors import EvaluationNotFoundError

pending = {etor.get_evaluation_id() for etor in my_evaluators}
results = {}

while pending:
    for evaluation_id in list(pending):
        try:
            progress = client.get_evaluation_progress(evaluation_id)
        except EvaluationNotFoundError:
            pending.discard(evaluation_id)   # not in this workspace any more
            continue
        if progress.status == "COMPLETED":
            results[evaluation_id] = client.get_stats_json_by_id(evaluation_id)
            pending.discard(evaluation_id)
        elif progress.status in {"CANCELED", "DELETED"}:
            pending.discard(evaluation_id)
    time.sleep(max(len(pending), 10))

progress = client.get_evaluation_progress("<EVALUATION_ID>")
print(progress.status, progress.progress)   # e.g. ACTIVE 52.5
print(progress.to_dict())

Stop polling on COMPLETED, CANCELED or DELETED. A deleted or hidden evaluation still reports its real status here, even though get_evaluation_list() no longer lists it.Decide on status, never on progress >= 100, for the reason given under get_evaluation_list().

The service allows 60 requests per minute per API key, counted across all evaluation ids rather than per evaluation. Polling N evaluations therefore needs an interval of at least N seconds; the example above sleeps max(N, 10). The SDK retries a rate-limited request with backoff, and those retries spend the same budget, so a sustained overrun surfaces as a podonos.common.exception.HTTPError with status 429.Two other exceptions are possible:

podonos.errors.EvaluationNotFoundError when the id is not in your API key’s workspace. The service does not distinguish an id that does not exist from one that belongs to another workspace.
podonos.common.exception.HTTPError for any other failure, including a key that is missing or revoked, and a service that does not serve evaluation progress. In the latter case, get_evaluation_list() still reports progress for every evaluation in the workspace.

get_eval_template_info()

Gets detailed information about an evaluation template by its ID.

string

The unique identifier of the evaluation template to retrieve information for.

Returns a Python dictionary containing detailed template information.

template_info = client.get_eval_template_info("abcdef")
print(template_info)

The returned dictionary has this shape:

{
  "id": "<UUID>",
  "code": "abcdef",
  "title": "Voice Quality Assessment",
  "description": "Template for evaluating voice naturalness and quality",
  "language": Language.ENGLISH_AMERICAN,
  "eval_type": "Single",
  "created_time": datetime.datetime(2024, 6, 25, 1, 40, 43, 429000, tzinfo=datetime.timezone.utc),
  "updated_time": datetime.datetime(2024, 6, 26, 13, 21, 34, 801000, tzinfo=datetime.timezone.utc)
}

Field	Description
`id`	Unique identifier (UUID) of the template
`code`	Template code used for identification
`title`	Display name of the template
`description`	Detailed description of the template’s purpose
`language`	`Language` enum value from the template, such as `Language.ENGLISH_AMERICAN` for `en-us`
`eval_type`	Type of evaluation: `Single`, `Double`, or `Triple`
`created_time`	`datetime.datetime` value for when the template was created
`updated_time`	`datetime.datetime` value for when the template was last modified

get_stats_json_by_id()

Returns a list of JSONs containing the statistics of each stimulus for the evaluation referenced by the id.

string

Evaluation id. See get_evaluation_list().

string

default:"question"

Group by criteria. Options are “question”, “script”, or “model”. Default is “question”. Note that “script” and “model” are only available for single-question evaluations.

evaluations = client.get_evaluation_list()
for eval in evaluations:
    stats = client.get_stats_json_by_id(eval['id'], group_by='question')
    print(stats)

Field	Description	SCORED	NON_SCORED
`frequency`	List of score counts: `{"score": number, "count": number}[]`	✓	-
`mean`	Average score	✓	-
`median`	Median score	✓	-
`std`	Standard deviation	✓	-
`sem`	Standard error of the mean	✓	-
`ci_95`	95% confidence interval	✓	-
`options`	Each option name as key with integer value	✓	✓
`OTHER`	The number of evaluators who selected “Other”	✓	✓

For NON_SCORED questions:

The integer value is the number of evaluators who selected the option.
All options are included in the response regardless of their value

Single
Double (including CSMOS)

You can get the statistics of each question by calling get_stats_json_by_id() with group_by set to question, script, or model.

{
  "question": string,
  "description": string,
  "order": int,
  "responses": [
    {
      "name": string,
      "model_tag": string,
      "tags": string[],
      "type": "A" | "B" | "REF",
      "script": string | null,
      "frequency": [
        {
          "score": number,
          "count": number
        }
      ],
      "mean": float | null, // null if the question is not SCORED
      "median": float | null, // null if the question is not SCORED
      "std": float | null, // null if the question is not SCORED
      "sem": float | null, // null if the question is not SCORED
      "ci_95": float | null, // null if the question is not SCORED
    }
  ]
}

{
  "script": string,
  "responses": [
    {
      "question": string,
      "description": string,
      "order": int,
      "responses": [
        {
          "name": string,
          "model_tag": string,
          "tags": string[],
          "type": "A" | "B" | "REF",
          "script": string | null,
          "frequency": [
            {
              "score": number,
              "count": number
            }
          ],
          "mean": float | null, // null if the question is not SCORED
          "median": float | null, // null if the question is not SCORED
          "std": float | null, // null if the question is not SCORED
          "sem": float | null, // null if the question is not SCORED
          "ci_95": float | null, // null if the question is not SCORED
        },
        ...
      ]
    },
    ...
  ]
}

{
  "model": string,
  "responses": [
    {
      "question": string,
      "description": string,
      "order": int,
      "responses": [
        {
          "name": string,
          "model_tag": string,
          "tags": string[],
          "type": "A" | "B" | "REF",
          "script": string | null,
          "frequency": [
            {
              "score": number,
              "count": number
            }
          ],
          "mean": float | null, // null if the question is not SCORED
          "median": float | null, // null if the question is not SCORED
          "std": float | null, // null if the question is not SCORED
          "sem": float | null, // null if the question is not SCORED
          "ci_95": float | null, // null if the question is not SCORED
        },
        ...
      ]
    },
    ...
  ]
}

The group_by is set to question by default. Other options (script, model) cannot be used.

{
  "question": string,
  "description": string,
  "order": int,
  "responses": [
    {
      "targets": [
        {
          "name": string,
          "model_tag": string,
          "tags": string[],
          "type": "A" | "B" | "REF",
          "script": string | null,
        }
      ],
      "frequency": [
        {
          "score": number,
          "count": number
        }
      ],
      "mean": float | null, // null if the question is not SCORED
      "median": float | null, // null if the question is not SCORED
      "std": float | null, // null if the question is not SCORED
      "sem": float | null, // null if the question is not SCORED
      "ci_95": float | null, // null if the question is not SCORED
    }
  ]
}

download_evaluation_files_by_evaluation_id()

Download all files associated with a specific evaluation, identified by its evaluation_id, from the Podonos evaluation service. It saves these files to a specified directory on the local file system and generates a metadata file describing the downloaded files. Return a string indicating the status of the download operation. This could be a success message or an error message if the download fails.

string

Evaluation id. See get_evaluation_list().

string

The directory path where the downloaded files will be saved. This should be a valid path on the local file system where the user has write permissions.

client.download_evaluation_files_by_evaluation_id(
  evaluation_id="12345",
  output_dir="./output",
)

{
  "files": [
    {
      "file_path": "./output/model1/4c318c0fd5a492d3e20cb8142ac5344b.wav",
      "original_name": "file1.wav",
      "model_tag": "model1",
      "tags": ["tag1", "tag2"]
    }
  ]
}

Field	Description
`file_path`	The path produced by joining `output_dir` with `{model_tag}/{hashed_file_name}`. It can be relative or absolute depending on the `output_dir` you pass.
`original_name`	The original name of the file before downloading.
`model_tag`	The model tag associated with the file, used for categorization.
`tags`	A list of tags associated with the file, providing additional context or categorization.

File Naming Convention:Each downloaded file is saved in the format {output_dir}/{model_tag}/{file_name}. This means that files are organized into subdirectories named after their model_tag, and the original file name is hashed formatted.

File

A class representing one file, used for adding files in Evaluator.

string

required

Path to the file to evaluate. For audio files, we support wav, mp3, and flac formats.

string

required

Name of your model (e.g., WhisperTTS) or any unique name (e.g., human).

list[string]

A list of string tags for the file designated by path. You can use this field as properties of the file such as original, synthesized, tom, maria, and so on. Later you can look up or group files with particular tags in the output report.

string

Text script of the input audio file.

bool

default:"False"

True if this file works as a reference in a comparative evaluation.

Dict[str, str | int | float | bool | None]

Optional metadata dictionary for the file. Keys must be strings, and values must be JSON-primitive values.

list[string]

Optional tags for the script. These are useful for ranking and script-level grouping.

python

file = File(
    path="/path/to/speech.wav",
    model_tag="my_model",
    tags=["synthesized", "male"],
    script="hello there",
    meta_data={"speaker_id": "spk-001"},
    script_tags=["greeting"],
)

Evaluator

Evaluator manages a single type of evaluation.

add_file()

Add one file to evaluate in a single evaluation question. For a single file evaluation like NMOS, one file to evaluate is added.

File

required

Input File. This field is required if type is NMOS, QMOS, P808, or CUSTOM_SINGLE.

etor.add_file(
    file=File(
        path="/path/to/speech_0_0.wav",
        model_tag="my_model",
        tags=["synthesized", "male", "ver1234"],
    )
)

etor.add_file(
    file=File(
        path="/path/to/speech_0_0.wav",
        model_tag="my_model",
        tags=["synthesized", "female", "ver1234"],
    )
)

add_files()

Add multiple files for evaluations that require comparison.

File

required

First input file.

File

required

Second input file.

File

Optional third input file. Used for CSMOS evaluations with one reference and two stimuli.

The file requirements depend on evaluation type:

Type	File requirements
`PREF`, `CUSTOM_DOUBLE`	Two stimulus files, in any order
`SMOS`	Two stimulus files, in any order
`CMOS`	One stimulus file and one reference file, in any order
`CSMOS`	Two stimulus files in any order, followed by one reference file (`file2`)

You do not need to keep a consistent argument order, and you should not shuffle it yourself. The SDK normalizes what it stores: stimuli are sorted by model_tag (numeric-aware and case-insensitive) and the reference is stored last. The same model therefore occupies the same position in every group, which is what lets the platform attribute scores back to it.Presentation order is randomized per evaluator regardless — see Bias Minimization.The one positional rule left: for CSMOS the reference must be passed as file2.

file0 = File(path="/path/to/speech0.wav", model_tag="human", tags=["original", "male"])
file1 = File(path="/path/to/speech1.wav", model_tag="my_model", tags=["synthesized", "male", "ver1234"])
etor.add_files(file0=file0, file1=file1)

reference = File(path="/path/to/reference.wav", model_tag="human", is_ref=True)
model_a = File(path="/path/to/model_a.wav", model_tag="model_a")
model_b = File(path="/path/to/model_b.wav", model_tag="model_b")
etor.add_files(file0=model_a, file1=model_b, file2=reference)

add_ranking_set()

Add one ranking set for a RANKING or RANKING_REF evaluation.

list[File]

required

Candidate files for one ranking group. For RANKING_REF, include exactly one file with is_ref=True; its position in the list does not matter.

Constraints enforced across calls:

All groups must have the same number of files.
The order of stimulus model_tag must be identical across groups.
Within a group, every model_tag must be unique, case-insensitively. The reference is included in this check, so it cannot reuse a candidate’s tag.
RANKING: every file must be a stimulus. Setting is_ref=True raises.
RANKING_REF: exactly one file with is_ref=True and at least two stimuli. The reference’s model_tag must be the same in every group.

etor = client.create_evaluator(type="RANKING", name="Ranking test")

etor.add_ranking_set([
    File(path="/path/to/model_a_001.wav", model_tag="model_a", script_tags=["script_001"]),
    File(path="/path/to/model_b_001.wav", model_tag="model_b", script_tags=["script_001"]),
    File(path="/path/to/model_c_001.wav", model_tag="model_c", script_tags=["script_001"]),
])

etor = client.create_evaluator(type="RANKING_REF", name="Ranking against a reference")

etor.add_ranking_set([
    File(path="/path/to/model_a_001.wav", model_tag="model_a", script_tags=["script_001"]),
    File(path="/path/to/model_b_001.wav", model_tag="model_b", script_tags=["script_001"]),
    File(path="/path/to/model_c_001.wav", model_tag="model_c", script_tags=["script_001"]),
    File(path="/path/to/reference_001.wav", model_tag="reference", script_tags=["script_001"], is_ref=True),
])

model_tag accepts letters, digits, hyphens and underscores only. "Provider A" raises; use "Provider_A".

close()

Close the evaluation session. Once this function is called, all the evaluation files will be sent to the Podonos evaluation service, the files will go through a series of processing, and delivered to evaluators. Returns a JSON object containing the uploading status.

python

status = etor.close()

With auto_start=True on podonos 0.46.0 or newer, close() does not return once the uploads finish. It waits for the files to become processable, starts the evaluation — charging your workspace balance — and returns only after the evaluation is running.

With auto_start=True, close() can block for up to start_timeout (30 minutes by default) and can raise where it previously always returned. Two different exception types are possible, and they do not share a base class:

podonos.common.exception.HTTPError when the service refuses the start, for example when the workspace balance is insufficient. The message carries the server’s error code.
the builtin TimeoutError when start_timeout elapses first. except HTTPError will not catch this one.

A timeout does not mean the evaluation did not start. The SDK reports it as unconfirmed because a start whose response was lost may well have succeeded and been charged. Check the evaluation in Workspace before running it again, rather than paying twice.

auto_start

etor = client.create_evaluator(
    name="nightly_regression",
    type="NMOS",
    auto_start=True,
    start_timeout=600,          # 10 minutes is usually plenty in CI
    resume_upload=True,
    upload_state_path="./podonos-upload-ledger.sqlite3",
)
etor.add_file(File(path="audio1.wav", model_tag="model_a"))
etor.add_file(File(path="audio2.wav", model_tag="model_b"))

etor.close()   # blocks through processing, then starts the evaluation

Calling close() with no files added raises a ValueError.

Get Started

Basics

Details

Use Cases

Roadmap

SDK References

Podonos

init()

Client

create_evaluator()

resume_evaluator()

create_evaluator_from_template()

create_evaluator_from_template_json()

flash_eval()

get_evaluation_list()

get_evaluation_progress()

get_eval_template_info()

get_stats_json_by_id()

download_evaluation_files_by_evaluation_id()

File

Evaluator

add_file()

add_files()

add_ranking_set()

close()

​Podonos

​init()

​Client

​create_evaluator()

​resume_evaluator()

​create_evaluator_from_template()

​create_evaluator_from_template_json()

​flash_eval()

​get_evaluation_list()

​get_evaluation_progress()

​get_eval_template_info()

​get_stats_json_by_id()

​download_evaluation_files_by_evaluation_id()

​File

​Evaluator

​add_file()

​add_files()

​add_ranking_set()

​close()

Podonos

init()

Client

create_evaluator()

resume_evaluator()

create_evaluator_from_template()

create_evaluator_from_template_json()

flash_eval()

get_evaluation_list()

get_evaluation_progress()

get_eval_template_info()

get_stats_json_by_id()

download_evaluation_files_by_evaluation_id()

File

Evaluator

add_file()

add_files()

add_ranking_set()

close()