Streaming Adapter

Process large datasets in chunks with configurable overlap.

See also: Batch Adapter

When to Use

  • Dataset >100,000 points
  • Memory-constrained environments
  • Batch processing pipelines

Class

StreamingLowess

The StreamingLowess type processes data in chunks, suitable for very large datasets or streaming applications.

Constructor:

using FastLOWESS

model = StreamingLowess(; fraction=0.3, chunk_size=5000, overlap=500)
println(typeof(model))
StreamingLowess

Methods:

process_chunk(model, x, y)

Feeds one chunk of data into the model. Each chunk is fit together with the trailing overlap points buffered from the previous call, then only the points that are fully resolved are returned — the tail of the chunk (the next overlap points) is held back internally, since it will be refit once the following chunk arrives and its estimate reconciled via merge_strategy. This is what lets the adapter process a dataset far larger than memory allows, one bounded-size chunk at a time, without ever materializing the whole dataset at once.

using Random, Statistics

rng = MersenneTwister(42)
x = collect(range(0, 2π, length=100))
y = sin.(x) .+ randn(rng, 100) .* 0.3

process_chunk(model, x, y)
println("Called process_chunk")
Called process_chunk

finalize(model)

Flushes the overlap points still buffered from the last process_chunk call. Because each call withholds its tail until the next chunk arrives to resolve it, the final chunk's tail would never be emitted otherwise — always call finalize once after the last chunk to retrieve it.

result = finalize(model)
println("First smoothed value: ", result.y[1])
First smoothed value: 0.3538812783156377

Options Structure

StreamingLowess keyword arguments (mirrors Lowess)

FieldTypeDefaultDescription
fractionFloat640.67Smoothing fraction (bandwidth)
iterationsInt3Number of robustifying iterations
weight_functionString"tricube"Weight function name
robustness_methodString"bisquare"Robustness method name
deltaFloat64NaNInterpolation distance (NaN auto-sets it to 0.0 in Streaming, i.e. interpolation disabled)
zero_weight_fallbackString"use_local_mean"Zero-weight handling
boundary_policyString"extend"Boundary handling policy
scaling_methodString"mad"Residual scaling method
auto_convergeFloat64NaNAuto-convergence tolerance
missingString"error"Policy for non-finite (NaN/Inf) values in each chunk
chunk_sizeInt5000Points per chunk
overlapIntchunk_size / 10Overlap between chunks
merge_strategyString"weighted_average"Strategy for blending overlap regions
parallelBooltrueEnable parallel execution
outputsVector{String}String[]Select se, diagnostics, residuals, weights, and/or derivative
intervalsNamedTuplenothingGrouped confidence, prediction, and per-chunk bootstrap options
seedUnion{Integer, Nothing}nothingReproducible bootstrap draws for each combined chunk

Cross-validation, GPU backend, custom_weights, and the "sorted" output are Batch-only and not available here; see Batch Adapter for those. Standard errors and confidence/prediction intervals are computed per combined chunk (including the previous overlap), then blended across overlap regions via merge_strategy like y/derivative are; they are local chunk intervals, not whole-stream intervals.

Options

fraction

fraction is the most important parameter: it controls the size of the local neighbourhood used at each point.

RangeEffectUse case
0.1-0.3Fine detailRapidly changing signals
0.3-0.5BalancedGeneral purpose
0.5-0.7Heavy smoothingNoisy data
0.7-1.0Very smoothTrend extraction

iterations

iterations controls robustness to outliers, at the cost of speed.

ValueEffectPerformance
0No robustnessFastest
1-3ModerateRecommended
4-6StrongContaminated data
7+Very strongHeavy outliers

weight_function

See: Weight Functions

  • "tricube" (default)
  • "epanechnikov"
  • "gaussian"
  • "uniform" (alias: "boxcar")
  • "biweight" (alias: "bisquare")
  • "triangle" (alias: "triangular")
  • "cosine"

robustness_method

See: Robustness

  • "bisquare" (default; alias: "biweight")
  • "huber"
  • "talwar"

delta

Points within delta of each other on the x-axis share the same local fit instead of each computing its own regression — an interpolation shortcut that trades a small amount of accuracy for a large speedup on dense, evenly-spaced data. NaN (default) auto-sets it to 0 in Streaming mode, i.e. interpolation is disabled and every point is fit exactly.

zeroweightfallback

Behavior when all neighborhood weights are zero:

OptionBehavior
"use_local_mean" (default; aliases: "local_mean", "mean")Use the mean of the neighborhood
"return_original" (alias: "original")Return the original y value
"return_none" (alias: "none")Return NaN

boundary_policy

See: Boundary Handling

  • "extend" (default; alias: "pad")
  • "reflect" (alias: "mirror")
  • "zero"
  • "noboundary" (alias: "none")

scaling_method

See: Scaling Methods

  • "mad" (default; alias: "median_absolute_deviation")
  • "mar" (alias: "median_absolute_residual")
  • "mean" (alias: "mean_absolute_residual")

auto_converge

See: Robustness

Convergence tolerance for early stopping of robustness iterations. NaN (default) disables early stopping.

missing

Policy for handling non-finite (NaN/Inf) values within each chunk:

OptionBehavior
"error" (default)Raise an error if any value in the chunk is non-finite
"drop"Silently remove rows where x or y is non-finite before merging the chunk with the overlap buffer

Note: A length mismatch between x and y always errors, even under "drop".

chunk_size

Number of points processed per chunk. Larger chunks reduce per-chunk overhead and give each local fit more surrounding context, at the cost of higher peak memory; smaller chunks bound memory tightly but increase the fraction of points that fall in overlap regions. A good starting point is balancing available memory against how much processing overhead per chunk is acceptable — match it to your file-read buffer or message-batch size to avoid unnecessary copying.

overlap

Number of points retained from the previous chunk as context, so the neighbourhood at chunk boundaries isn't artificially truncated. Points inside the overlap zone are fitted twice (once by each chunk) and reconciled via merge_strategy. A good starting point is 10–20% of chunk_size: too little overlap causes visible boundary artefacts, while too much wastes computation refitting the same points twice.

  • -1 (default) — computes chunk_size / 10, clamped to at least 1 and less than chunk_size
  • Any integer >= 1 and < chunk_size

merge_strategy

See: Merge Strategies

StrategyAliasBehavior
"weighted_average" (default)"weighted"Distance-weighted blend
"average""mean"Average overlapping values
"take_first""first"Keep left chunk values
"take_last""last"Keep right chunk values

parallel

Enable multi-threaded execution via Rayon.

  • true (default) — parallelizes the local regression fits across CPU cores
  • false — forces single-threaded execution

outputs: se

See: Intervals

Computes standard errors per chunk the same way Batch does, then merges the overlap region across chunk boundaries the same way y/derivative are, via merge_strategy.

outputs: diagnostics

See: Diagnostics

Select "diagnostics" to include a Diagnostics object (RMSE, MAE, R2, residualsd) in the result. `effectivedf/aic/aiccrequire per-chunk hat-matrix leverage to be threaded into the cumulative diagnostics computation across chunk boundaries, which isn't currently done (even with"se"orintervalsselected), so they're alwaysNaN` here.

outputs: residuals

Include per-point residuals (y - fitted) in the result.

outputs: weights

Include the final per-point robustness weights (from the last robustness iteration) in the result.

outputs: derivative

Each point's local WLS fit already computes a slope internally; this exposes that per-point slope (rate of change of the smoothed curve) in result.derivative at effectively no extra computation cost. Derivative values in the overlap region are merged across chunk boundaries the same way y is, via merge_strategy.

intervals

See: Intervals

A NamedTuple such as (confidence=0.90, prediction=0.99, bootstrap=200), populating result.confidence_lower/result.confidence_upper and result.prediction_lower/result.prediction_upper. The coverage levels are independent. Computed per combined chunk and merged across overlap boundaries via merge_strategy. bootstrap (at least 2) refits each combined chunk from resampled residuals; with parallel=true, refits run concurrently.

seed

Seeds bootstrap draws. Each combined chunk restarts from the same seed. It does not enable bootstrap by itself; 0 is a valid seed.

Result Structure

LowessResult

Returned by process_chunk and finalize.

FieldTypeDescription
xVector{Float64}x values (same order as input)
yVector{Float64}Smoothed y values
fraction_usedFloat64Fraction used
iterations_usedUnion{Int, Nothing}Robustness iterations actually performed
standard_errorsUnion{Vector{Float64}, Nothing}Per-point standard errors (if "se" or any interval was set)
confidence_lowerUnion{Vector{Float64}, Nothing}Lower confidence bounds (if intervals.confidence was set)
confidence_upperUnion{Vector{Float64}, Nothing}Upper confidence bounds (if intervals.confidence was set)
prediction_lowerUnion{Vector{Float64}, Nothing}Lower prediction bounds (if intervals.prediction was set)
prediction_upperUnion{Vector{Float64}, Nothing}Upper prediction bounds (if intervals.prediction was set)
residualsUnion{Vector{Float64}, Nothing}Residuals (if "residuals" was requested)
robustness_weightsUnion{Vector{Float64}, Nothing}Robustness weights (if "weights" was requested)
cv_scoresUnion{Vector{Float64}, Nothing}Always nothing (Batch only)
diagnosticsUnion{Diagnostics, Nothing}Fit metrics (if "diagnostics" was requested)
derivativeUnion{Vector{Float64}, Nothing}Per-point local fit derivative/slope (if "derivative" was requested)

Diagnostics

FieldTypeDescription
rmseFloat64Root Mean Squared Error
maeFloat64Mean Absolute Error
r_squaredFloat64R-squared
residual_sdFloat64Cumulative sample SD of emitted residuals
effective_dfUnion{Float64, Nothing}nothing (cumulative diagnostics don't integrate per-chunk leverage; Batch only)
aicUnion{Float64, Nothing}nothing (requires effective_df; Batch only)
aiccUnion{Float64, Nothing}nothing (requires effective_df; Batch only)

Example

using FastLOWESS
using Random, Statistics

rng = MersenneTwister(42)
x = collect(range(0, 2π, length=100))
y = sin.(x) .+ randn(rng, 100) .* 0.3

model = StreamingLowess(;
    fraction=0.3,
    iterations=2,
    chunk_size=5000,
    overlap=500,
    merge_strategy="average"
)
process_chunk(model, x, y)
result = finalize(model)
println("First smoothed value: ", result.y[1])
First smoothed value: 0.35383694154729933