// MODEL OPTIMIZATION AND PROMPT SYNTAX TERM
Early Stopping
A technique used to prevent a machine learning model from overfitting by stopping the training process when the model's performance on a separate validation set starts to worsen.
TECHNICAL DEFINITION
A regularization technique applied during iterative model training, particularly in neural networks, where training is halted when the model's performance on a designated validation set ceases to improve or begins to degrade, thereby preventing overfitting and optimizing generalization.
BACKGROUND
Prompt engineering is the process of structuring natural language inputs to produce specified outputs from a generative artificial intelligence (GenAI) model. Context engineering is the related area of software engineering that focuses on the management of non-prompt and prompt contexts supplied to the GenAI model, such as system instructions, metadata, API tools and tokens.
READ MORE ON WIKIPEDIASYNONYMS & ALIASES
- Stop training early
- regularization by stopping
- validation-based stopping
USAGE NOTE
Early stopping is a common practice to find the optimal point in a model's training without excessive computation.
DEVELOPERS
Organizations developing technology related to Early Stopping.
As the developer of the Keras and TensorFlow deep learning frameworks, Google provides a direct and widely-used implementation via the `EarlyStopping` callback, which monitors a specified metric and stops training when it ceases to improve, thus preventing overfitting.
The Hugging Face `Trainer` API, a standard for fine-tuning Transformer models, includes built-in arguments for enabling and configuring early stopping. This allows developers to easily halt training when the validation loss stops decreasing, optimizing the fine-tuning process.
Within its SageMaker platform, AWS provides automatic model tuning capabilities that support early stopping. It can be configured to stop hyperparameter tuning jobs that are not showing improvement, saving significant compute time and cost.
The Azure Machine Learning platform offers several early termination policies for its hyperparameter tuning services. Policies like the Bandit policy, Median stopping policy, and Truncation selection policy stop poorly performing runs, increasing the efficiency of model training.
While Meta's PyTorch provides the building blocks, PyTorch Lightning (now Lightning AI) offers a high-level wrapper that formalizes training loops. It has a built-in, highly configurable `EarlyStopping` callback that is a staple for training robust models in the PyTorch ecosystem.
The Weights & Biases MLOps platform facilitates early stopping by providing real-time tracking and visualization of model metrics. Its hyperparameter optimization feature, Sweeps, incorporates early-stopping algorithms like HyperBand to automatically halt unpromising training runs.
As a foundational machine learning library in Python, scikit-learn implements early stopping in many of its iterative models, such as gradient boosting machines (e.g., `HistGradientBoostingClassifier`). This feature is built-in to prevent overfitting and find the optimal number of boosting iterations.