Seungjae Shin, Heesun Bae, DongHyeok Shin, Weonyoung Joo, Il-Chul Moon, Loss Curvature Matching for Dataset Selection and Condensation, International Conference on Artificial Intelligence and Statistics (AISTATS 2023), Valencia, Spain, Apr 25-27, 2023

Abstract

Training neural networks on a large dataset requires substantial computational costs. Dataset reduction selects or synthesizes data instances based on the large dataset, while minimizing the degradation in generalization performance from the full dataset. Existing methods utilize the neural network during the dataset reduction procedure, so the model parameter becomes important factor in preserving the performance after reduction. By depending upon the importance of parameters, this paper introduces a new reduction objective, coined LCMat, which Matches the Loss Curvatures of the original dataset and reduced dataset over the model parameter space, more than the parameter point. This new objective induces a better adaptation of the reduced dataset on the perturbed parameter region than the exact point matching. Particularly, we identify the worst case of the loss curvature gap from the local parameter region, and we derive the implementable upper bound of such worst-case with theoretical analyses. Our experiments on both coreset selection and condensation benchmarks illustrate that LCMat shows better generalization performances than existing baselines. 

@inproceedings{shin2023loss, 

title={Loss-Curvature Matching for Dataset Selection and Condensation}, 

author={Shin, Seungjae and Bae, Heesun and Shin, Donghyeok and Joo, Weonyoung and Moon, Il-Chul}, 

booktitle={International Conference on Artificial Intelligence and Statistics}, 

pages={8606--8628}, 

year={2023}, 

organization={PMLR} 

} 

Source Website:

https://proceedings.mlr.press/v206/shin23a/shin23a.pdf