Automated
Audio AI Evaluation

Uncover your AI model performance. Iterate ultrafast with trust.

Start your project

Contact Us

Trusted by top AI companies

Audio AI model evaluation takes long. We do the hard work, you move fast.

Audio AI model evaluation takes long. We do the hard work, you move fast.

Multi Language Evaluation.

Evaluations over 8 languages, 12 locale at once.

my_speech_model_latest.pt

Hello

Pass

Hello

Pass

Hola

Pass

안녕하세요

Fail

Bonjour

Pass

你好

Pass

Hola

Pass

Hallo

Pass

Hello

Pass

こんにちは

Fail

Multi Language Evaluation.

Evaluations over 8 languages, 12 locale at once.

my_speech_model_latest.pt

Hello

Pass

Hello

Pass

Hola

Pass

안녕하세요

Fail

Bonjour

Pass

你好

Pass

Hola

Pass

Hallo

Pass

Hello

Pass

こんにちは

Fail

Multi Language Evaluation.

Evaluations over 8 languages, 12 locale at once.

my_speech_model_latest.pt

Hello

Pass

Hello

Pass

Hola

Pass

안녕하세요

Fail

Bonjour

Pass

你好

Pass

Hola

Pass

Hallo

Pass

Hello

Pass

こんにちは

Fail

Detailed and reliable analytics.

We support multiple standards for trustworthy evaluation.
You improve your model fast with trust.

Total

150,000 Evaluators

Around the world from 12 locale.

Automated Evaluator Training & Screening

5 screenings and tests

Pre-screening, device test, intelligibility test, attention test. All included.

Evaluation Types

World standard evaluation methods

Naturalness, quality, similarity, preferences, and many others.

Detailed and reliable analytics.

We support multiple standards for trustworthy evaluation.
You improve your model fast with trust.

Total

150,000 Evaluators

Around the world from 12 locale.

Automated Evaluator Training & Screening

5 screenings and tests

Pre-screening, device test, intelligibility test, attention test. All included.

Evaluation Types

World standard evaluation methods

Naturalness, quality, similarity, preferences, and many others.

Detailed and reliable analytics.

We support multiple standards for trustworthy evaluation.
You improve your model fast with trust.

Total

150,000 Evaluators

Around the world from 12 locale.

Automated Evaluator Training & Screening

5 screenings and tests

Pre-screening, device test, intelligibility test, attention test. All included.

Evaluation Types

World standard evaluation methods

Naturalness, quality, similarity, preferences, and many others.

Get results faster than ever.

Maximum 12 hours to evaluate.

Which is 5x faster than typical process.

Get results faster than ever.

Maximum 12 hours to evaluate.

Which is 5x faster than typical process.

Get results faster than ever.

Maximum 12 hours to evaluate.

Which is 5x faster than typical process.

Intuitive analytics.

Performance compared to the other model

ChatGPT

Gemini

previous_model

latest_model

Performance by timeline

speech_synthesis_latest.pt

Reference 1

Reference 2

Ours

4/24

5/1

5/15

Today

Performance scores by language.

Intuitive analytics.

Performance compared to the other model

ChatGPT

Gemini

previous_model

latest_model

Performance by timeline

speech_synthesis_latest.pt

Reference 1

Reference 2

Ours

4/24

5/1

5/15

Today

Performance scores by language.

Intuitive analytics.

Performance compared to the other model

ChatGPT

Gemini

previous_model

latest_model

Performance by timeline

speech_synthesis_latest.pt

Reference 1

Reference 2

Ours

4/24

5/1

5/15

Today

Performance scores by language.

Three simple steps for evaluations and insights.

Three simple steps for evaluations and insights.

REQUEST

Connect with 5 lines of code.

View Docs

Python

1 import podonos
2 client = podonos.init(api_key='my_api_key')
3 etor = client.create_evaluator(name='syn_p2_head8')
4 etor.add_file(path='/eval/output/1.wav')
5 etor.close()

REQUEST

Connect with 5 lines of code.

View Docs

Python

1 import podonos
2 client = podonos.init(api_key='my_api_key')
3 etor = client.create_evaluator(name='syn_p2_head8')
4 etor.add_file(path='/eval/output/1.wav')
5 etor.close()

EVALUATE

Ultrafast and reliable evaluation.

NMOS, QMOS, ITU-T P.808, SMOS, and many others.

Your Data
7 stages for world's highest quality evaluation.

The entire precess takes less than 12 hours.

Pre-Screening
Device Test
Intelligibility Test
Attention Test
Evaluation
Data Collection
Analysis

Up to 70% higher accuracy than other competitors.

Report

EVALUATE

Ultrafast and reliable evaluation.

NMOS, QMOS, ITU-T P.808, SMOS, and many others.

Your Data
7 stages for world's highest quality evaluation.

The entire precess takes less than 12 hours.

Pre-Screening
Device Test
Intelligibility Test
Attention Test
Evaluation
Data Collection
Analysis

Up to 70% higher accuracy than other competitors.

Report

INSIGHT

Get insight from multiple charts.

You will get the detailed insight along evaluator's region, language, performances, and comparisons.

Performance compared to the other model.

Evaluation date : 2024/06/28

ChatGPT

Gemini

previous_model

latest_model

Performance analysis by timeline.

speech_synthesis_latest.pt

Reference 1

Reference 2

Ours

4/24

5/1

5/15

Today

INSIGHT

Get insight from multiple charts.

You will get the detailed insight along evaluator's region, language, performances, and comparisons.

Performance compared to the other model.

Evaluation date : 2024/06/28

ChatGPT

Gemini

previous_model

latest_model

Performance analysis by timeline.

speech_synthesis_latest.pt

Reference 1

Reference 2

Ours

4/24

5/1

5/15

Today

Save time and cost, Improve model quality fast.

Save time and cost, Improve model quality fast.

Up to

5x

Faster product development.

Up to

50%

Cost reduction.

Up to

80%

Work time reduction.

Genuine voices from our users.

Genuine voices from our users.

Setting up an evaluation session for a speech synthesis model is quite cumbersome and time consuming for our machine learning engineers: we otherwise would have had to find the right evaluation contractors, educate them, set up the pipeline properly, and finally analyze the evaluation results ourselves. Without such hassles, we now get clear insight on how to further improve our AI models from Podonos' evaluation reports.

Tim Jung

CEO @XL8

Evaluation of AI-enhanced speech is painful. It is especially difficult if you follow industry standards, such as ITU-R P.835, for rigorous evaluations. Podonos service carefully satisfies all the requirements in such standards, quickly evaluated our enhanced audio files, and sent us the final evaluation results within 9 hours. It is an incredible speed! Without the Podonos service, we would need to implement the evaluation tools, connect with existing services such as MTurk, deal with dishonest participants, and analyze the results, which easily takes more than a week.

Stanisław Andrzej Raczyński

CEO @Revoize

Pricing Plans.

Pricing Plans.

Free

Everything you need to kick off with Podonos.

$0

$0

$0

/rating
/rating
/rating

Get started for free

Get started for free

Get started for free

Access to SDK interface

Access to Workspace

Unlimited private reports

Up to 100 ratings

1 GB file storage

Pay-as-you-go

Ideal for AI researchers and engineers looking to evaluate single AI model.

$0.32

$0.32

$0.32

/rating

Start your project

Start your project

Start your project

Access to SDK interface

Access to Workspace

Unlimited private reports

Unlimited team members

3 GB file storage

10 public reports every month

Dedicated support channel

Frequently Asked Questions.

We can use other crowdsourcing platforms. Why use Podonos?

You can access a large number of crowds via existing services. However, you need to set up the details of multi-language evaluations and secret questions, prescreen the evaluators, occasionally reject poor evaluators, and analyze the results, which takes days to weeks, but the quality may not be as good as you expect. Podonos solves all of these automatically following the industry standards.

Which evaluation types do you support?

We support NMOS (Naturalness), QMOS (Quality), CMOS (Comparison), ITU-T P.808 (Quality), speaker similarity, and preferences. If you want more, please let us know at hello@podonos.com

Which languages do you support?

We support 8 languages and 13 locales including English (US, UK, Australia, India, Canada), Spanish (Spain, Mexico), Chinese (Mandarin), Korean, French, German, Japanese and Italian. We are adding 25 more languages. Let us know which languages you want us to support.

How do you confirm the evaluators are qualified?

We have more than 150k worldwide qualified evaluators ready. In every evaluation session, we check their equipment, ambient noise, language skills, and attention level. Additionally, we automatically filter low-quality evaluations hurting overall reliability.

Is Podonos secure?

Yes. We securely keep your data and files using AES-256 encryption. Your data is used only for running evaluation. Optionally your data may be in public only when you share the evaluation result with everyone in the world.

Can I use the evaluation results for academic publications?

Yes. Regardless of their location, ethnicity, gender, and their social status, the global evaluators are treated equally and rewarded relevantly. We control the session length to avoid potential abuse.

We can use other crowdsourcing platforms. Why use Podonos?

You can access a large number of crowds via existing services. However, you need to set up the details of multi-language evaluations and secret questions, prescreen the evaluators, occasionally reject poor evaluators, and analyze the results, which takes days to weeks, but the quality may not be as good as you expect. Podonos solves all of these automatically following the industry standards.

Which evaluation types do you support?

We support NMOS (Naturalness), QMOS (Quality), CMOS (Comparison), ITU-T P.808 (Quality), speaker similarity, and preferences. If you want more, please let us know at hello@podonos.com

Which languages do you support?

We support 8 languages and 13 locales including English (US, UK, Australia, India, Canada), Spanish (Spain, Mexico), Chinese (Mandarin), Korean, French, German, Japanese and Italian. We are adding 25 more languages. Let us know which languages you want us to support.

How do you confirm the evaluators are qualified?

We have more than 150k worldwide qualified evaluators ready. In every evaluation session, we check their equipment, ambient noise, language skills, and attention level. Additionally, we automatically filter low-quality evaluations hurting overall reliability.

Is Podonos secure?

Yes. We securely keep your data and files using AES-256 encryption. Your data is used only for running evaluation. Optionally your data may be in public only when you share the evaluation result with everyone in the world.

Can I use the evaluation results for academic publications?

Yes. Regardless of their location, ethnicity, gender, and their social status, the global evaluators are treated equally and rewarded relevantly. We control the session length to avoid potential abuse.

Ready to dive in?

Sign up and simply start your project.

Start your project

Docs

Still have questions?

If you need more information, Book a demo session.

or

Book a demo