plaid.storage.writer
====================

.. py:module:: plaid.storage.writer

.. autoapi-nested-parse::

   PLAID storage writer module.

   This module provides high-level functions for saving PLAID datasets to local disk and pushing
   them to Hugging Face Hub. It supports multiple storage backends including CGNS, HF Datasets,
   and Zarr, abstracting the backend-specific implementations.

   Key features:
   - Unified interface for saving datasets across different backends
   - Automatic preprocessing and schema extraction
   - Metadata and problem definition handling
   - Hub integration with dataset cards and metadata



Attributes
----------

.. autoapisummary::

   plaid.storage.writer.AVAILABLE_BACKENDS
   plaid.storage.writer.generate_datasetdict_to_disk
   plaid.storage.writer.push_local_datasetdict_to_hub
   plaid.storage.writer.configure_dataset_card


Functions
---------

.. autoapisummary::

   plaid.storage.writer.save_to_disk
   plaid.storage.writer.push_to_hub


Module Contents
---------------

.. py:data:: AVAILABLE_BACKENDS
   :value: ['cgns', 'hf_datasets', 'zarr']


.. py:data:: generate_datasetdict_to_disk

.. py:data:: push_local_datasetdict_to_hub

.. py:data:: configure_dataset_card

.. py:function:: save_to_disk(output_folder: Union[str, pathlib.Path], generators: dict[str, Callable[Ellipsis, Generator[plaid.Sample, None, None]]], backend: str, infos: Optional[dict[str, Any]] = None, pb_defs: Optional[Union[plaid.ProblemDefinition, collections.abc.Iterable[plaid.ProblemDefinition]]] = None, gen_kwargs: Optional[dict[str, dict[str, Any]]] = None, num_proc: int = 1, verbose: bool = False, overwrite: bool = False) -> None

   Save a PLAID dataset to local disk using the specified backend.

   This function preprocesses the dataset generators, extracts schemas, and saves the dataset
   to disk using the chosen backend. It also saves metadata, infos, and problem definitions.

   :param output_folder: Path to the output directory where the dataset will be saved.
   :param generators: Dictionary mapping split names to sample generators.
   :param backend: Storage backend to use ('cgns', 'hf_datasets', or 'zarr').
   :param infos: Optional additional information to save with the dataset.
   :param pb_defs: Optional problem definitions to save.
   :param gen_kwargs: Optional keyword arguments for the generators.
   :param num_proc: Number of processes to use for preprocessing.
   :param verbose: If True, enables verbose output during processing.
   :param overwrite: If True, overwrites existing output directory.


.. py:function:: push_to_hub(repo_id: str, local_dir: Union[str, pathlib.Path], num_workers: int = 1, viewer: bool = False, pretty_name: Optional[str] = None, dataset_long_description: Optional[str] = None, illustration_urls: Optional[list[str]] = None, arxiv_paper_urls: Optional[list[str]] = None) -> None

   Push a local PLAID dataset to Hugging Face Hub.

   This function uploads a previously saved dataset from local disk to Hugging Face Hub,
   including data, metadata, infos, and problem definitions. It automatically detects the
   backend used for saving and configures the dataset card.

   :param repo_id: Hugging Face repository ID (e.g., 'username/dataset-name').
   :param local_dir: Local directory containing the saved dataset.
   :param num_workers: Number of workers for parallel upload.
   :param viewer: If True, enables dataset viewer on Hub.
   :param pretty_name: Optional pretty name for the dataset.
   :param dataset_long_description: Optional detailed description.
   :param illustration_urls: Optional list of illustration URLs.
   :param arxiv_paper_urls: Optional list of arXiv paper URLs.