MihaiPopa-1/common_voice_22_0_toki_pona_parquet

Common Voice 22.0 - Toki Pona Subset!

My own Parquet conversion of Toki Pona's subset of Fsicoli's reupload of Common Voice 22 so we don't have to downgrade to Datasets 3.6 anymore!

Why?

Because the original dataset required Hugging Face Datasets 3.6 or older because it has Python code and it's in TAR shards.

This is in Parquet and works with any recent version of Hugging Face Datasets!

Details

Dataset Structure

DatasetDict({
    train: Dataset({
        features: ['client_id', 'path', 'audio', 'sentence', 'up_votes', 'down_votes', 'age', 'gender', 'accent', 'locale', 'segment', 'variant'],
        num_rows: 2630
    })
    validation: Dataset({
        features: ['client_id', 'path', 'audio', 'sentence', 'up_votes', 'down_votes', 'age', 'gender', 'accent', 'locale', 'segment', 'variant'],
        num_rows: 2217
    })
    test: Dataset({
        features: ['client_id', 'path', 'audio', 'sentence', 'up_votes', 'down_votes', 'age', 'gender', 'accent', 'locale', 'segment', 'variant'],
        num_rows: 2166
    })
    other: Dataset({
        features: ['client_id', 'path', 'audio', 'sentence', 'up_votes', 'down_votes', 'age', 'gender', 'accent', 'locale', 'segment', 'variant'],
        num_rows: 4997
    })
    invalidated: Dataset({
        features: ['client_id', 'path', 'audio', 'sentence', 'up_votes', 'down_votes', 'age', 'gender', 'accent', 'locale', 'segment', 'variant'],
        num_rows: 358
    })
})

Duration in Hours

This excludes the invalidated set from this calculation!

train: 2.55 hours
validation: 2.61 hours
test: 2.50 hours
other: 7.14 hours
TOTAL: 14.80 hours

Code to Reproduce

By GPT-5.6 Luna and Gemini 2.5 Flash:

pip install "datasets>=3,<4" huggingface_hub

from datasets import load_dataset

ds = load_dataset(
    "fsicoli/common_voice_22_0",
    "tok",
    trust_remote_code=True,
)

print(ds)

Then:

import os

def get_duration_batched(examples):
    durations = []
    for audio_item in examples["audio"]:
        # Calculating duration still requires accessing the audio array to get its length
        durations.append(len(audio_item["array"]) / audio_item["sampling_rate"])
    examples["duration"] = durations
    return examples

hours = 0

for split in ["train", "validation", "test", "other"]:
    # Use batched=True to process multiple examples at once
    # Use remove_columns=["audio"] to free up memory after duration is calculated
    # Use num_proc for parallel processing (adjust based on your system's cores and memory)
    tmp = ds[split].map(
        get_duration_batched,
        batched=True,
        remove_columns=["audio"],
        num_proc=os.cpu_count() # Consider reducing if memory remains an issue
    )
    split_hours = sum(tmp["duration"]) / 3600

    print(f"{split}: {split_hours:.2f} hours")
    hours += split_hours

print(f"\nTOTAL: {hours:.2f} hours")

Extra Info

License (this repo): Apache 2.0

License (original dataset and Fsicoli's reupload): CC0 (Public Domain)