Key Concepts

​

Intro

The Podonos SDK has three major components: Client, Evaluator, and File.

​

Three Key Components

​

Client

Client manages multiple evaluations and can create multiple Evaluators. Each evaluator corresponds to one types of evaluation (e.g., naturalness, preferences). So if you want to evaluate your inputs along multiple dimensions, you will create multiple evaluators from one client like:

There are key parameters:

• name: Name of the evaluation.
• desc: Detailed description of the evaluation.
• type: One of the evaluation types. For details, see type in create_evaluator().
• lan: Language to be evaluated. Real human who fluently speaks the language and lives in the country will evaluate.
• num_eval: The number of people who evaluates one file. The higher this number is, the more reliable the evaluation will be.
• use_annotation: This is for uncovering the evaluation details.

You can also get the whole list of evaluations you created so far by calling get_evaluation_list().

eval_list = client.get_evaluation_list()
print(eval_list)

In addition, you can get the detailed statistics of one evaluation:

eval_detail_list = client.get_stats_dict_by_id(evaluation_id='<EVALUATION_ID>')
print(eval_detail_list)

For more details, see get_stats_dict_by_id().

​

Evaluator

Evaluator object manages the details of one evaluation type. With this object, you can add files to evaluate.

​

File

File represents one file and its details.

File(path='/path/to/1.mp3', model_tag='my_new_model',
     tags=['male', 'american', 'narrative'])

Here are the key parameters:

• path: Path to the file to evaluate.
• model_tag: Name of your model (or any unique name). In the evluation report, you will see the statistics grouped by this model name.
• tags: List of multiple tags strings. In the evaluation report, you can see the statistics grouped by one or more tags.

For more details, see File.