Cross-Validation
Automated parameter selection via cross-validation.
Overview
Cross-validation helps select optimal parameters (especially fraction) by evaluating performance on held-out data.
K-Fold Cross-Validation
Split data into K folds, train on K-1, validate on 1.
using FastLOWESS
using Random, Statistics
rng = MersenneTwister(42)
x = collect(range(0, 2π, length=100))
y = sin.(x) .+ randn(rng, 100) .* 0.3
model = Lowess(; cv=(method="kfold", k=5, fractions=[0.2, 0.3, 0.5, 0.7])
)
result = fit(model, x, y)
println("Selected fraction: ", result.fraction_used)
println("CV scores: ", result.cv_scores)Selected fraction: 0.3
CV scores: [0.5949774044659284, 0.5632616684189621, 0.5985308031883003, 0.6288938314833663]Leave-One-Out (LOOCV)
Each point is held out once. Most thorough but slowest.
using FastLOWESS
using Random, Statistics
rng = MersenneTwister(42)
x = collect(range(0, 2π, length=100))
y = sin.(x) .+ randn(rng, 100) .* 0.3
model = Lowess(; cv=(method="loocv", fractions=[0.2, 0.3, 0.5, 0.7])
)
result = fit(model, x, y)
println("Selected fraction (CV): ", result.fraction_used)Selected fraction (CV): 0.2Seeded Randomization
Set a seed for reproducible fold assignments:
using FastLOWESS
using Random, Statistics
rng = MersenneTwister(42)
x = collect(range(0, 2π, length=100))
y = sin.(x) .+ randn(rng, 100) .* 0.3
model = Lowess(; cv=(method="kfold", k=5, fractions=[0.3, 0.5, 0.7]), seed=42)
result = fit(model, x, y)
println("Selected fraction (CV): ", result.fraction_used)Selected fraction (CV): 0.3Comparison
| Method | Folds | Speed | Variance | Bias |
|---|---|---|---|---|
| KFold(5) | 5 | Fast | Moderate | Low |
| KFold(10) | 10 | Medium | Lower | Lower |
| LOOCV | N | Slow | Lowest | Lowest |
Use 5-fold or 10-fold CV for most applications. LOOCV is only worth it for small datasets (N < 100).
CV Metrics
Cross-validation uses MSE (Mean Squared Error) by default:
MSE = mean((y_true - y_pred)^2)Lower MSE indicates better fit on held-out data.
Interpreting Results
using FastLOWESS
using Random, Statistics
rng = MersenneTwister(42)
x = collect(range(0, 2π, length=100))
y = sin.(x) .+ randn(rng, 100) .* 0.3
# Example output
model = Lowess(; cv=(method="kfold", k=5, fractions=[0.1, 0.3, 0.5, 0.7]))
result = fit(model, x, y)
# Fraction | CV Score (MSE)
# 0.1 | 0.0542 ← Undersmoothed
# 0.3 | 0.0231 ← Best
# 0.5 | 0.0298
# 0.7 | 0.0412 ← Oversmoothed
println("Selected fraction (CV): ", result.fraction_used)Selected fraction (CV): 0.3The fraction with lowest CV score is automatically selected.
Availability
| Feature | Batch | Streaming | Online |
|---|---|---|---|
| K-Fold CV | ✓ | ✗ | ✗ |
| LOOCV | ✓ | ✗ | ✗ |
Best Practices
- Test a range: Include fractions from 0.1 to 0.9
- Use enough folds: 5-10 folds balance speed and accuracy
- Set a seed: For reproducible results
- Check the curve: CV optimizes MSE, but visual inspection matters