plaid.storage.hf_datasets.writer

HF Datasets writer module.

This module provides functionality for writing and managing datasets in Hugging Face Datasets format for the PLAID library. It includes utilities for generating datasets from sample generators, saving to disk, uploading to Hugging Face Hub, and configuring dataset cards with metadata.

Key features: - Dataset generation from generators with parallel processing - Disk saving with automatic sharding - Hub uploading with optimized sharding - Dataset card configuration and updating

Functions

save_datasetdict_to_disk(→ None)	Save a Hugging Face DatasetDict to disk.
generate_datasetdict_to_disk(→ None)	Generates and saves a DatasetDict to disk from sample generators.
push_datasetdict_to_hub(→ None)	Push a Hugging Face DatasetDict to the Hugging Face Hub.
push_local_datasetdict_to_hub(→ None)	Pushes a local DatasetDict to Hugging Face Hub.
configure_dataset_card(→ None)	Configures and updates a dataset card on Hugging Face Hub for HF datasets backend.

Module Contents

save_datasetdict_to_disk(path: str | pathlib.Path, hf_datasetdict: datasets.DatasetDict, **kwargs) → None

Save a Hugging Face DatasetDict to disk.

This function serializes the provided DatasetDict and writes it to the specified directory, preserving its features, splits, and data for later loading.

Parameters:

• path (Union[str, Path]) – Directory path where the DatasetDict will be saved.

• hf_datasetdict (datasets.DatasetDict) – The Hugging Face DatasetDict to save.

• **kwargs – Keyword arguments forwarded to [DatasetDict.save_to_disk](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.DatasetDict.save_to_disk).

Returns:

None

generate_datasetdict_to_disk(output_folder: str | pathlib.Path, generators: dict[str, Callable[Ellipsis, Generator[plaid.Sample, None, None]]], variable_schema: dict[str, dict], gen_kwargs: dict[str, dict[str, list[plaid.types.IndexType]]] | None = None, num_proc: int = 1, verbose: bool = False) → None

Generates and saves a DatasetDict to disk from sample generators.

Parameters:

• output_folder (Union[str, Path]) – Base directory to save the dataset.

• generators (dict[str, Callable[..., Generator[Sample, None, None]]]) – Dictionary of split names to generator functions.

• variable_schema (dict[str, dict]) – Schema describing variables.

• gen_kwargs (Optional[dict[str, dict[str, list[IndexType]]]]) – Optional generator arguments for parallel processing.

• num_proc (int) – Number of processes for generation.

• verbose (bool) – Whether to enable verbose output.

push_datasetdict_to_hub(repo_id: str, hf_datasetdict: datasets.DatasetDict, **kwargs) → None

Push a Hugging Face DatasetDict to the Hugging Face Hub.

This is a thin wrapper around datasets.DatasetDict.push_to_hub, allowing you to upload a dataset dictionary (with one or more splits such as “train”, “validation”, “test”) to the Hugging Face Hub.

Note

The function automatically handles sharding of the dataset by setting num_shards for each split. For each split, the number of shards is set to the minimum between the number of samples in that split and such that shards are targetted to approx. 500 MB. This ensures efficient chunking while preventing excessive fragmentation. Empty splits will raise an assertion error.

Parameters:

• repo_id (str) – The repository ID on the Hugging Face Hub (e.g. “username/dataset_name”).

• hf_datasetdict (datasets.DatasetDict) – The Hugging Face dataset dictionary to push.

• **kwargs – Keyword arguments forwarded to [DatasetDict.push_to_hub](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.DatasetDict.push_to_hub).

Returns:

None

push_local_datasetdict_to_hub(repo_id: str, local_dir: str | pathlib.Path, num_workers: int = 1) → None

Pushes a local DatasetDict to Hugging Face Hub.

Parameters:

• repo_id (str) – The repository ID on Hugging Face Hub.

• local_dir (Union[str, Path]) – Local directory containing the dataset.

• num_workers (int) – Number of workers for uploading.

configure_dataset_card(repo_id: str, infos: dict[str, dict[str, str]], local_dir: str | pathlib.Path | None = None, viewer: bool = False, pretty_name: str | None = None, dataset_long_description: str | None = None, illustration_urls: list[str] | None = None, arxiv_paper_urls: list[str] | None = None) → None

Configures and updates a dataset card on Hugging Face Hub for HF datasets backend.

This function downloads the existing README.md (dataset card) from the specified Hugging Face repository, modifies it by adding metadata such as license, viewer settings, task categories, tags, and optional descriptions/illustrations. It then pushes the updated card back to the repository.

Parameters:

• repo_id (str) – The Hugging Face repository ID where the dataset card is located and will be updated.

• infos (dict[str, dict[str, str]]) – Dictionary containing dataset metadata, including legal information like license.

• local_dir (Optional[Union[str, Path]]) – Unused parameter for local directory path.

• variable_schema (Optional[dict]) – Unused parameter for variable schema.

• viewer (bool) – Whether to enable the dataset viewer. Defaults to False, which sets ‘viewer: false’ in the card.

• pretty_name (Optional[str]) – A human-readable name for the dataset to display in the card.

• dataset_long_description (Optional[str]) – A detailed description of the dataset to include in the card.

• illustration_urls (Optional[list[str]]) – List of URLs to images that illustrate the dataset, displayed in the card.

• arxiv_paper_urls (Optional[list[str]]) – List of arXiv URLs for papers related to the dataset, included as sources.

Returns:

This function does not return a value; it updates the dataset card

directly on Hugging Face Hub.

Return type:

None