Skip to content

PLAID Benchmarks

PLAID Benchmarks

We provide interactive benchmarks hosted on Hugging Face, in which anyone can test their own SciML method. These benchmarks involve regression problems posed on datasets provided in PLAID format. Some of these datasets have been introduced in the MMGP (Mesh Morphing Gaussian Process) paper 1 and the PLAID paper 2. A ranking is automatically updated based on a score computed on the testing set of each dataset. For the benchmarks to be meaningful, the outputs on the testing sets are not made public.

The relative RMSE is the considered metric for comparing methods. Let \(\{ \mathbf{U}^i_{\rm ref} \}_{i=1}^{n_\star}\) and \(\{ \mathbf{U}^i_{\rm pred} \}_{i=1}^{n_\star}\) be the test observations and predictions, respectively, of a given field of interest. The relative RMSE is defined as

\[ \mathrm{RRMSE}_f(\mathbf{U}_{\rm ref}, \mathbf{U}_{\rm pred}) = \left( \frac{1}{n_\star}\sum_{i=1}^{n_\star} \frac{\frac{1}{N^i}\|\mathbf{U}^i_{\rm ref} - \mathbf{U}^i_{\rm pred}\|_2^2}{\|\mathbf{U}^i_{\rm ref}\|_\infty^2} \right)^{1/2}, \]

where \(N^i\) is the number of nodes in the mesh \(i\), and \(\max(\mathbf{U}^i_{\rm ref})\) is the maximum entry in the vector \(\mathbf{U}^i_{\rm ref}\). Similarly for scalar outputs:

\[ \mathrm{RRMSE}_s(\mathbf{w}_{\rm ref}, \mathbf{w}_{\rm pred}) = \left( \frac{1}{n_\star} \sum_{i=1}^{n_\star} \frac{|w^i_{\rm ref} - w_{\rm pred}^i|^2}{|w^i_{\rm ref}|^2} \right)^{1/2}. \]

Resources

Dataset Dataset links Benchmark
Tensile2d Zenodo Hugging Face Benchmark
2D_MultiScHypEl Zenodo Hugging Face Benchmark
2D_ElPlDynamics Zenodo Hugging Face Benchmark
Rotor37 Zenodo Hugging Face Benchmark
2D_profile Zenodo Hugging Face Benchmark
VKI-LS59 Zenodo Hugging Face Benchmark

AirfRANS 3 is an additional dataset provided in PLAID format and various variants. Since the outputs on the testing sets are public, no benchmark application is provided for this dataset.

Dataset Dataset links
AirfRANS original Zenodo Hugging Face
AirfRANS clipped Zenodo Hugging Face
AirfRANS remeshed Zenodo Hugging Face

Benchmark results

As of August 5, 2025

Dataset MGN MMGP Vi-Transf. Augur FNO MARIO
Tensile2d 0.0673 0.0026 0.0116 0.0154 0.0123 0.0038
2D_MultiScHypEl 0.0437 ❌ 0.0325 0.0232 0.0302 0.0573
2D_ElPlDynamics 0.1202 ❌ 0.0227 0.0346 0.0215 0.0319
Rotor37 0.0074 0.0014 0.0029 0.0033 0.0313 0.0017
2D_profile 0.0593 0.0365 0.0312 0.0425 0.0972 0.0307
VKI-LS59 0.0684 0.0312 0.0193 0.0267 0.0215 0.0124

❌: Not compatible with topology variation

The code to reproduce these results is available in this repository.

Benchmark interpretation notes

  • MMGP does not support variable mesh topologies, which limits its applicability to certain datasets and often necessitates custom preprocessing for new cases. However, when morphing is either unnecessary or inexpensive, it offers a highly efficient solution, combining fast training with good accuracy (e.g., Tensile2d and Rotor37).
  • MARIO is computationally expensive to train but achieves consistently a very strong performance across most datasets. Its result on 2D_MultiScHypEl is slightly worse than other tested methods, which may reflect the challenge of capturing complex shape variability in these cases.
  • Vi-Transformer and Augur perform well across all datasets, showing strong versatility and generalization capabilities.
  • FNO suffers on datasets featuring unstructured meshes with pronounced anisotropies, due to the loss of accuracy introduced by projections to and from regular grids (e.g., Rotor37 and 2D_profile). Additionally, the use of a 3D regular grid on Rotor37 results in substantial computational overhead.

  1. F. Casenave, B. Staber, and X. Roynard, “MMGP: a Mesh Morphing Gaussian Process-based machine learning method for regression of physical problems under non-parameterized geometrical variability,” Thirty-seventh Conference on Neural Information Processing Systems, 2023. arXiv:2305.12871. ↩

  2. F. Casenave, X. Roynard, B. Staber, A. Devaux-Rivière, W. Piat, M. A. Bucci, N. Akkari, A. Kabalan, X. M. V. Nguyen, L. Saverio, R. Carpintero Perez, A. Kalaydjian, S. Fouché, T. Gonon, G. Najjar, T. Daniel, E. Menier, M. Nastorg, G. Catalani, and C. Rey, “PLAID: A Unified Data Model for Machine Learning on Heterogeneous Physics Simulations,” ICML AI4Physics Workshop, 2026. arXiv:2505.02974. ↩

  3. F. Bonnet, J. Mazari, P. Cinnella, and P. Gallinari, “AirfRANS: High Fidelity Computational Fluid Dynamics Dataset for Approximating Reynolds-Averaged Navier–Stokes Solutions,” Advances in Neural Information Processing Systems, vol. 35, pp. 23463–23478, 2022. arXiv:2212.07564. ↩