Cross-Validation

Automated parameter selection via cross-validation.

Overview

Cross-validation helps select optimal parameters (especially fraction) by evaluating performance on held-out data.

Cross-Validation

K-Fold Cross-Validation

Split data into K folds, train on K-1, validate on 1.

import fastlowess.Lowess;
import fastlowess.Options;
import fastlowess.Result;
import fastlowess.CVOptions;

public class Example {
    public static void main(String[] args) {
        int n = 100;
        double[] x = new double[n];
        double[] y = new double[n];
        for (int i = 0; i < n; i++) {
            x[i] = i * 2 * Math.PI / (n - 1);
            y[i] = Math.sin(x[i]) + 0.1;
        }

        Options options = Options.builder()
                .cv(CVOptions.builder().method("kfold").k(5)
                    .fractions(0.2, 0.3, 0.5, 0.7).build())
                .build();

        try (Lowess model = new Lowess(options)) {
            Result result = model.fit(x, y);

            // The best fraction was automatically selected
            System.out.println("Selected fraction: " + result.fractionUsed());
            System.out.println("CV scores: " + java.util.Arrays.toString(result.cvScores().orElseThrow()));
        }
    }
}
Selected fraction: 0.3
CV scores: [0.3442028395147172, 0.34364341192477826, 0.39986523402252333, 0.47601021763016]

Leave-One-Out (LOOCV)

Each point is held out once. Most thorough but slowest.

import fastlowess.Lowess;
import fastlowess.Options;
import fastlowess.Result;
import fastlowess.CVOptions;

public class Example {
    public static void main(String[] args) {
        int n = 100;
        double[] x = new double[n];
        double[] y = new double[n];
        for (int i = 0; i < n; i++) {
            x[i] = i * 2 * Math.PI / (n - 1);
            y[i] = Math.sin(x[i]) + 0.1;
        }

        Options options = Options.builder()
                .cv(CVOptions.builder().method("loocv")
                    .fractions(0.2, 0.3, 0.5, 0.7).build())
                .build();

        try (Lowess model = new Lowess(options)) {
            Result result = model.fit(x, y);
            System.out.println("Selected fraction (CV): " + result.fractionUsed());
        }
    }
}
Selected fraction (CV): 0.2

Seeded Randomization

Set a seed on the outer options for reproducible fold assignments. The same seed also controls residual bootstrap when intervals enable it:

import fastlowess.Lowess;
import fastlowess.Options;
import fastlowess.Result;
import fastlowess.CVOptions;

public class Example {
    public static void main(String[] args) {
        int n = 100;
        double[] x = new double[n];
        double[] y = new double[n];
        for (int i = 0; i < n; i++) {
            x[i] = i * 2 * Math.PI / (n - 1);
            y[i] = Math.sin(x[i]) + 0.1;
        }

        Options options = Options.builder()
                .cv(CVOptions.builder().method("kfold").k(5)
                    .fractions(0.3, 0.5, 0.7).build())
                .seed(42L)
                .build();

        try (Lowess model = new Lowess(options)) {
            Result result = model.fit(x, y);
            System.out.println("Selected fraction (CV): " + result.fractionUsed());
        }
    }
}
Selected fraction (CV): 0.3

Comparison

Method Folds Speed Variance Bias

KFold(5)

5

Fast

Moderate

Low

KFold(10)

10

Medium

Lower

Lower

LOOCV

N

Slow

Lowest

Lowest

Recommendation: Use 5-fold or 10-fold CV for most applications. LOOCV is only worth it for small datasets (N < 100).


CV Metrics

Cross-validation uses MSE (Mean Squared Error) by default:

MSE = mean((y_true - y_pred)^2)

Lower MSE indicates better fit on held-out data.


Interpreting Results

import fastlowess.CVOptions;
import fastlowess.Lowess;
import fastlowess.Options;
import fastlowess.Result;

public class Example {
    public static void main(String[] args) {
        int n = 100;
        double[] x = new double[n];
        double[] y = new double[n];
        for (int i = 0; i < n; i++) {
            x[i] = i * 2 * Math.PI / (n - 1);
            y[i] = Math.sin(x[i]) + 0.1;
        }

        Options options = Options.builder()
                .cv(CVOptions.builder().method("kfold").k(5)
                    .fractions(0.1, 0.3, 0.5, 0.7).build())
                .build();

        // Fraction  | CV Score (MSE)
        // 0.1       | 0.0542  <- Undersmoothed
        // 0.3       | 0.0231  <- Best
        // 0.5       | 0.0298
        // 0.7       | 0.0412  <- Oversmoothed

        try (Lowess model = new Lowess(options)) {
            Result result = model.fit(x, y);
            System.out.println("Selected fraction (CV): " + result.fractionUsed());
        }
    }
}
Selected fraction (CV): 0.3

The fraction with lowest CV score is automatically selected.


Availability

Batch Mode Only: Cross-validation is only available in Batch mode.

Feature Batch Streaming Online

K-Fold CV

✓

✗

✗

LOOCV

✓

✗

✗


Best Practices

  1. Test a range: Include fractions from 0.1 to 0.9

  2. Use enough folds: 5-10 folds balance speed and accuracy

  3. Set a seed: For reproducible results

  4. Check the curve: CV optimizes MSE, but visual inspection matters