Test error by split
What the training set already contains
Predicted against DFT
Every compound of one element held out
Element
The same elements in another study
Does a relative in the training set help?
Error by kind of material
Experimental structures
How far the test materials are from the training set
Error against distance
Checks of the numbers
How the test sets are built
Model. scikit-learn HistGradientBoostingRegressor (learning rate 0.05, at most 5000 iterations, 63 leaves, early stopping on a validation set) on 136 composition and 14 structure features, the best model of the baselines experiment.
Group splits. 10-fold cross-validation; every group lies in one fold, and the groups are dealt out from the largest to the smallest into the fold with the fewest materials. Fold k is the test set, fold k + 1 the validation set, the other eight folds the training set (80/10/10). Every material is predicted once.
Leave one element out. For each element in at least 200 materials, all its compounds form the test set; the rest is split by chemical system into training (90 %) and validation (10 %) sets.
Added after 2021. Trained on the 2021.8.18 snapshot (split by chemical system 90/10), tested on every material of the 2025.9.24 snapshot that is not in the 2021 one. Both snapshots agree on all shared materials.
Prototype label. spglib space group (tolerance 0.1 Å) and, per species, its count in the reduced formula and its Wyckoff letters, made independent of the choice of origin and axes with the Wyckoff-set table of PyXtal.
Differences from standard practice
No hyperparameter search per split; the random-split model reaches the iteration limit; one cross-validation pass and single runs (the spread is between folds); a strict prototype label (no fuzzy structure matching, so the prototype split is milder than one with it); one distance measure on hand-made features; the comparison with Li et al. uses another model, other descriptors and the 2022.12.12 snapshot. The materials added after 2021 were chosen by the JARVIS project and many are hypothetical: a test of the next batch of a database, not of experimental discovery. Details in the experiment README.
Sources
Data: JARVIS-DFT (K. Choudhary et al., npj Comput. Mater. 6, 173, 2020), snapshots 2021.8.18 and 2025.9.24 via jarvis-tools.
Comparison: K. Li et al., arXiv:2406.06489 (2024), per-element R² of XGBoost on JARVIS 2022.12.12, read from their Supplementary Figure 2.
Symmetry: spglib; Wyckoff sets of the normalizers from PyXtal (MIT licence; Bilbao Crystallographic Server, WYCKSETS).