StreamingLowess API

For datasets that don’t fit in memory or arrive in chunks. Processes data incrementally, merging overlapping regions between chunks.

See also: API

When to Use

  • Dataset >100,000 points

  • Memory-constrained environments

  • Batch processing pipelines

Class

StreamingLowess

The StreamingLowess class processes data in chunks, suitable for very large datasets or streaming applications.

Constructor:

StreamingOptions options = StreamingOptions.builder()
        .chunkSize(2000)
        .overlap(200)
        .build();

try (StreamingLowess model = new StreamingLowess(options)) {
    // ...
}
  • new StreamingLowess(StreamingOptions options) creates a new streaming model with the given options.

  • options: A StreamingOptions instance (exposes all Options settings plus chunkSize, overlap, and mergeStrategy).

Methods:

processChunk(double[] x, double[] y) Result

Feeds one chunk of data into the model. Each chunk is fit together with the trailing overlap points buffered from the previous call, then only the points that are fully resolved are returned — the tail of the chunk (the next overlap points) is held back internally, since it will be refit once the following chunk arrives and its estimate reconciled via mergeStrategy. This is what lets the adapter process a dataset far larger than memory allows, one bounded-size chunk at a time, without ever materializing the whole dataset at once.

Result partial = model.processChunk(x, y);

finish() Result

Flushes the overlap points still buffered from the last processChunk call. Because each call withholds its tail until the next chunk arrives to resolve it, the final chunk’s tail would never be emitted otherwise — always call finish() once after the last chunk to retrieve it.

Result result = model.finish();
  • StreamingLowess.close() releases native resources. Safe to call multiple times. Implements AutoCloseable.

Options Structure

StreamingOptions

Setting Type Default Description

fraction

double

0.67

Smoothing fraction (bandwidth)

iterations

int

3

Number of robustifying iterations

weightFunction

String

"tricube"

Weight function name

robustnessMethod

String

"bisquare"

Robustness method name

delta

double

NaN

Interpolation distance (NaN auto-sets it to 0.0 in Streaming, i.e. interpolation disabled)

zeroWeightFallback

String

"use_local_mean"

Zero-weight handling

boundaryPolicy

String

"extend"

Boundary handling policy

scalingMethod

String

"mad"

Residual scaling method

autoConverge

double

NaN

Auto-convergence tolerance

missing

String

"error"

Policy for non-finite (NaN/Infinity) values in each chunk

chunkSize

int

5000

Data chunk size

overlap

int

chunk_size / 10

Overlap between chunks

mergeStrategy

String

"weighted_average"

Strategy for blending overlap regions

parallel

boolean

true

Enable parallel execution

outputs(String…​)

String…​

empty

Select se, diagnostics, residuals, weights, and/or derivative

intervals(IntervalsOptions)

IntervalsOptions

disabled

Confidence/prediction levels and per-chunk bootstrap refits

seed(long)

long

unset

Reproducible bootstrap draws for each combined chunk

Cross-validation, GPU backend, customWeights, and the sorted output are Batch-only and not available here; see API for those. Standard errors and intervals are computed per combined chunk (including the previous overlap) and blended across overlaps via mergeStrategy; they are local chunk intervals, not whole-stream intervals.

Options

fraction

fraction is the most important parameter: it controls the size of the local neighbourhood used at each point.

Range Effect Use case

0.1-0.3

Fine detail

Rapidly changing signals

0.3-0.5

Balanced

General purpose

0.5-0.7

Heavy smoothing

Noisy data

0.7-1.0

Very smooth

Trend extraction

iterations

iterations controls robustness to outliers, at the cost of speed.

Value Effect Performance

0

No robustness

Fastest

1-3

Moderate

Recommended

4-6

Strong

Contaminated data

7+

Very strong

Heavy outliers

weightFunction

  • "tricube" (default)

  • "epanechnikov"

  • "gaussian"

  • "uniform" (alias: "boxcar")

  • "biweight" (alias: "bisquare")

  • "triangle" (alias: "triangular")

  • "cosine"

robustnessMethod

See: Robustness

  • "bisquare" (default; alias: "biweight")

  • "huber"

  • "talwar"

delta

Points within delta of each other on the x-axis share the same local fit instead of each computing its own regression — an interpolation shortcut that trades a small amount of accuracy for a large speedup on dense, evenly-spaced data. NaN (default) auto-sets it to 0 in Streaming mode, i.e. interpolation is disabled and every point is fit exactly.

zeroWeightFallback

Behavior when all neighborhood weights are zero:

Option Behavior

"use_local_mean" (default; aliases: "local_mean", "mean")

Use the mean of the neighborhood

"return_original" (alias: "original")

Return the original y value

"return_none" (alias: "none")

Return NaN

boundaryPolicy

  • "extend" (default; alias: "pad")

  • "reflect" (alias: "mirror")

  • "zero"

  • "noboundary" (alias: "none")

scalingMethod

  • "mad" (default; alias: "median_absolute_deviation")

  • "mar" (alias: "median_absolute_residual")

  • "mean" (alias: "mean_absolute_residual")

autoConverge

See: Robustness

Convergence tolerance for early stopping of robustness iterations. NaN (default) disables early stopping.

missing

Policy for handling non-finite (NaN/Infinity) values within each chunk:

Option Behavior

"error" (default)

Throw if any value in the chunk is non-finite

"drop"

Silently remove rows where x or y is non-finite before merging the chunk with the overlap buffer

A length mismatch between x and y always throws, even under "drop".

chunkSize

Number of points processed per chunk. Larger chunks reduce per-chunk overhead and give each local fit more surrounding context, at the cost of higher peak memory; smaller chunks bound memory tightly but increase the fraction of points that fall in overlap regions. A good starting point is balancing available memory against how much processing overhead per chunk is acceptable — match it to your file-read buffer or message-batch size to avoid unnecessary copying.

overlap

Number of points retained from the previous chunk as context, so the neighbourhood at chunk boundaries isn’t artificially truncated. Points inside the overlap zone are fitted twice (once by each chunk) and reconciled via mergeStrategy. A good starting point is 10-20% of chunkSize: too little overlap causes visible boundary artefacts, while too much wastes computation refitting the same points twice.

  • -1 (default) — computes chunk_size / 10, clamped to at least 1 and less than chunkSize

  • Any integer >= 1 and < chunkSize

mergeStrategy

Strategy Behavior

"weighted_average" (default)

Distance-weighted blend

"average"

Average overlapping values

"take_first"

Keep left chunk values

"take_last"

Keep right chunk values

parallel

Enable multi-threaded execution via Rayon.

  • true (default) — parallelizes the local regression fits across CPU cores

  • false — forces single-threaded execution

outputs: diagnostics

Select "diagnostics" to include a Diagnostics object (RMSE, MAE, R², residual_sd) in the result. effectiveDf/aic/aicc require standard errors, which are Batch-only, so they’re always empty here.

outputs: residuals

Select "residuals" to include per-point residuals (y - fitted) in the result.

outputs: weights

Select "weights" to include the final per-point robustness weights (from the last robustness iteration) in the result.

outputs: derivative

Each point’s local WLS fit already computes a slope internally; this exposes that per-point slope (rate of change of the smoothed curve) via Result.derivative() at effectively no extra computation cost. Derivative values in the overlap region are merged across chunk boundaries the same way y() is, via mergeStrategy.

intervals

See: Intervals

Build with IntervalsOptions.builder(). confidence(level) and prediction(level) add bounds; bootstrap(n) (at least 2) replaces analytic intervals with residual-bootstrap refits of each combined chunk. With parallel(true), refits run concurrently.

seed

Seeds bootstrap draws. Each combined chunk restarts from the same seed. It does not enable bootstrap by itself; 0 is valid.

Result Structure

Result

Returned by processChunk and finish.

Accessor Type Populated when

x(), y()

double[]

Always.

standardErrors()

Optional<double[]>

"se" output or any interval

confidenceLower(), confidenceUpper()

Optional<double[]>

intervals confidence set

predictionLower(), predictionUpper()

Optional<double[]>

intervals prediction set

residuals()

Optional<double[]>

"residuals" output

robustnessWeights()

Optional<double[]>

"weights" output

cvScores()

Optional<double[]>

Always empty (Batch only)

fractionUsed()

double

Always.

iterationsUsed()

OptionalInt

Always (empty if not available).

diagnostics()

Optional<Diagnostics>

"diagnostics" output

derivative()

Optional<double[]>

"derivative" output

Diagnostics

Accessor Type Description

rmse()

double

Root Mean Squared Error

mae()

double

Mean Absolute Error

rSquared()

double

R-squared

residualSd()

double

Cumulative sample SD of emitted residuals

effectiveDf()

Optional<Double>

Always empty (requires standard errors, Batch only)

aic()

Optional<Double>

Always empty (requires effectiveDf, Batch only)

aicc()

Optional<Double>

Always empty (requires effectiveDf, Batch only)

Example

import fastlowess.Result;
import fastlowess.StreamingLowess;
import fastlowess.StreamingOptions;

public class Example {
    public static void main(String[] args) {
        final int n = 20;
        double[] x = new double[n];
        double[] y = new double[n];
        for (int i = 0; i < n; i++) {
            x[i] = i;
            y[i] = i + 0.1;
        }

        StreamingOptions options = StreamingOptions.builder()
                .chunkSize(10)
                .overlap(2)
                .build();

        try (StreamingLowess model = new StreamingLowess(options)) {
            double[] x1 = java.util.Arrays.copyOfRange(x, 0, 10);
            double[] y1 = java.util.Arrays.copyOfRange(y, 0, 10);
            double[] x2 = java.util.Arrays.copyOfRange(x, 10, 20);
            double[] y2 = java.util.Arrays.copyOfRange(y, 10, 20);
            model.processChunk(x1, y1);
            model.processChunk(x2, y2);

            Result result = model.finish();
            System.out.printf("y[0]: %.4f%n", result.y()[0]);
        }
    }
}
y[0]: 17.9200