It has been recently demonstrated that optimal neural networks operate near the asymptotic edge of chaos for state-of-the-art feed-forward neural networks, where its generalization power is maximal due to the highest number of asymptotic metastable states. However, how to leverage this principle to improve the model training process remains open. Here, by mapping the model evolution during training to the phase diagram in the classic analytic result of Sherrington–Kirkpatrick model in spin glasses, we illustrate on a simple neural network model that one can provide principled training of the network without manually tuning the training hyper-parameters. In particular, we provide a semi-analytical method to set the optimal weight decay strength, such that the model will converge toward the edge of chaos during training. Consequently, such hyper-parameter setting leads the model to achieve the highest test accuracy. Another benefit for restricting the model at the edge of chaos is its robustness against the common practical problem of label noise, as we find that it automatically avoids fitting the shuffled labels in the training samples while maintaining good fitting to the correct labels, providing simple means of achieving good performance on noisy labels without any additional treatment.