plaid.storage.hf_datasets.writer ================================ .. py:module:: plaid.storage.hf_datasets.writer .. autoapi-nested-parse:: HF Datasets writer module. This module provides functionality for writing and managing datasets in Hugging Face Datasets format for the PLAID library. It includes utilities for generating datasets from sample generators, saving to disk, uploading to Hugging Face Hub, and configuring dataset cards with metadata. Key features: - Dataset generation from generators with parallel processing - Disk saving with automatic sharding - Hub uploading with optimized sharding - Dataset card configuration and updating Functions --------- .. autoapisummary:: plaid.storage.hf_datasets.writer.save_datasetdict_to_disk plaid.storage.hf_datasets.writer.generate_datasetdict_to_disk plaid.storage.hf_datasets.writer.push_datasetdict_to_hub plaid.storage.hf_datasets.writer.push_local_datasetdict_to_hub plaid.storage.hf_datasets.writer.configure_dataset_card Module Contents --------------- .. py:function:: save_datasetdict_to_disk(path: Union[str, pathlib.Path], hf_datasetdict: datasets.DatasetDict, **kwargs) -> None Save a Hugging Face DatasetDict to disk. This function serializes the provided DatasetDict and writes it to the specified directory, preserving its features, splits, and data for later loading. :param path: Directory path where the DatasetDict will be saved. :type path: Union[str, Path] :param hf_datasetdict: The Hugging Face DatasetDict to save. :type hf_datasetdict: datasets.DatasetDict :param \*\*kwargs: Keyword arguments forwarded to [`DatasetDict.save_to_disk`](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.DatasetDict.save_to_disk). :returns: None .. py:function:: generate_datasetdict_to_disk(output_folder: Union[str, pathlib.Path], generators: dict[str, Callable[Ellipsis, Generator[plaid.Sample, None, None]]], variable_schema: dict[str, dict], gen_kwargs: Optional[dict[str, dict[str, list[plaid.types.IndexType]]]] = None, num_proc: int = 1, verbose: bool = False) -> None Generates and saves a DatasetDict to disk from sample generators. :param output_folder: Base directory to save the dataset. :type output_folder: Union[str, Path] :param generators: Dictionary of split names to generator functions. :type generators: dict[str, Callable[..., Generator[Sample, None, None]]] :param variable_schema: Schema describing variables. :type variable_schema: dict[str, dict] :param gen_kwargs: Optional generator arguments for parallel processing. :type gen_kwargs: Optional[dict[str, dict[str, list[IndexType]]]] :param num_proc: Number of processes for generation. :type num_proc: int :param verbose: Whether to enable verbose output. :type verbose: bool .. py:function:: push_datasetdict_to_hub(repo_id: str, hf_datasetdict: datasets.DatasetDict, **kwargs) -> None Push a Hugging Face `DatasetDict` to the Hugging Face Hub. This is a thin wrapper around `datasets.DatasetDict.push_to_hub`, allowing you to upload a dataset dictionary (with one or more splits such as `"train"`, `"validation"`, `"test"`) to the Hugging Face Hub. .. note:: The function automatically handles sharding of the dataset by setting `num_shards` for each split. For each split, the number of shards is set to the minimum between the number of samples in that split and such that shards are targetted to approx. 500 MB. This ensures efficient chunking while preventing excessive fragmentation. Empty splits will raise an assertion error. :param repo_id: The repository ID on the Hugging Face Hub (e.g. `"username/dataset_name"`). :type repo_id: str :param hf_datasetdict: The Hugging Face dataset dictionary to push. :type hf_datasetdict: datasets.DatasetDict :param \*\*kwargs: Keyword arguments forwarded to [`DatasetDict.push_to_hub`](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.DatasetDict.push_to_hub). :returns: None .. py:function:: push_local_datasetdict_to_hub(repo_id: str, local_dir: Union[str, pathlib.Path], num_workers: int = 1) -> None Pushes a local DatasetDict to Hugging Face Hub. :param repo_id: The repository ID on Hugging Face Hub. :type repo_id: str :param local_dir: Local directory containing the dataset. :type local_dir: Union[str, Path] :param num_workers: Number of workers for uploading. :type num_workers: int .. py:function:: configure_dataset_card(repo_id: str, infos: dict[str, dict[str, str]], local_dir: Optional[Union[str, pathlib.Path]] = None, variable_schema: Optional[dict] = None, viewer: bool = False, pretty_name: Optional[str] = None, dataset_long_description: Optional[str] = None, illustration_urls: Optional[list[str]] = None, arxiv_paper_urls: Optional[list[str]] = None) -> None Configures and updates a dataset card on Hugging Face Hub for HF datasets backend. This function downloads the existing README.md (dataset card) from the specified Hugging Face repository, modifies it by adding metadata such as license, viewer settings, task categories, tags, and optional descriptions/illustrations. It then pushes the updated card back to the repository. :param repo_id: The Hugging Face repository ID where the dataset card is located and will be updated. :type repo_id: str :param infos: Dictionary containing dataset metadata, including legal information like license. :type infos: dict[str, dict[str, str]] :param local_dir: Unused parameter for local directory path. :type local_dir: Optional[Union[str, Path]] :param variable_schema: Unused parameter for variable schema. :type variable_schema: Optional[dict] :param viewer: Whether to enable the dataset viewer. Defaults to False, which sets 'viewer: false' in the card. :type viewer: bool :param pretty_name: A human-readable name for the dataset to display in the card. :type pretty_name: Optional[str] :param dataset_long_description: A detailed description of the dataset to include in the card. :type dataset_long_description: Optional[str] :param illustration_urls: List of URLs to images that illustrate the dataset, displayed in the card. :type illustration_urls: Optional[list[str]] :param arxiv_paper_urls: List of arXiv URLs for papers related to the dataset, included as sources. :type arxiv_paper_urls: Optional[list[str]] :returns: This function does not return a value; it updates the dataset card directly on Hugging Face Hub. :rtype: None