For datasets that don’t fit in memory or arrive in chunks. Processes data incrementally, merging overlapping regions between chunks.

See also: API

When to Use

  • Dataset >100,000 points
  • Memory-constrained environments
  • Batch processing pipelines

Class

StreamingLowess

The StreamingLowess type processes data in chunks, suitable for very large datasets or streaming applications.

Constructor:

opts := fastlowess.DefaultStreamingOptions()
opts.ChunkSize = 2000
opts.Overlap = 200

model, err := fastlowess.NewStreamingLowess(opts)
if err != nil {
 panic(err)
}
defer model.Close()
  • fastlowess.NewStreamingLowess(opts StreamingOptions) (*StreamingLowess, error) creates a new streaming model with the given options.
  • opts: A StreamingOptions struct.

Methods:

ProcessChunk(x, y []float64) (Result, error)

Feeds one chunk of data into the model. Each chunk is fit together with the trailing Overlap points buffered from the previous call, then only the points that are fully resolved are returned — the tail of the chunk (the next Overlap points) is held back internally, since it will be refit once the following chunk arrives and its estimate reconciled via MergeStrategy. This is what lets the adapter process a dataset far larger than memory allows, one bounded-size chunk at a time, without ever materializing the whole dataset at once.

result, err := model.ProcessChunk(x, y)

Finalize() (Result, error)

Flushes the overlap points still buffered from the last ProcessChunk call. Because each call withholds its tail until the next chunk arrives to resolve it, the final chunk’s tail would never be emitted otherwise — always call Finalize once after the last chunk to retrieve it.

result, err := model.Finalize()
  • (*StreamingLowess) Close() error releases native resources. Safe to call multiple times.

Options Structure

StreamingOptions

FieldTypeDefaultDescription
Fractionfloat640.67Smoothing fraction (bandwidth)
Iterationsint3Number of robustifying iterations
WeightFunctionstring"tricube"Weight function name
RobustnessMethodstring"bisquare"Robustness method name
Delta*float64nilInterpolation distance (nil auto-sets it to 0.0 in Streaming, i.e. interpolation disabled)
ZeroWeightFallbackstring"use_local_mean"Zero-weight handling
BoundaryPolicystring"extend"Boundary handling policy
ScalingMethodstring"mad"Residual scaling method
AutoConverge*float64nilAuto-convergence tolerance
Missingstring"error"Policy for non-finite (NaN/Inf) values in each chunk
ChunkSizeint5000Data chunk size
Overlapintchunk_size / 10Overlap between chunks
MergeStrategystring"weighted_average"Strategy for blending overlap regions
ParallelbooltrueEnable parallel execution
Outputs[]stringnilOptional components: se, diagnostics, residuals, weights, derivative.
Intervals*IntervalsOptionsnilConfidence/prediction levels and per-chunk bootstrap refits
Seed*uint64nilReproducible bootstrap draws for each combined chunk

Cross-validation, GPU Backend, CustomWeights, and the "sorted" output are Batch-only and not available here; see API for those. Standard errors and confidence/prediction intervals are computed per chunk the same way Batch computes them, then blended across overlap regions via MergeStrategy like Y/Derivative are.

Options

Fraction

Fraction is the most important parameter: it controls the size of the local neighbourhood used at each point.

RangeEffectUse case
0.1-0.3Fine detailRapidly changing signals
0.3-0.5BalancedGeneral purpose
0.5-0.7Heavy smoothingNoisy data
0.7-1.0Very smoothTrend extraction

Iterations

Iterations controls robustness to outliers, at the cost of speed.

ValueEffectPerformance
0No robustnessFastest
1-3ModerateRecommended
4-6StrongContaminated data
7+Very strongHeavy outliers

WeightFunction

See: Weight Functions

  • "tricube" (default)
  • "epanechnikov"
  • "gaussian"
  • "uniform" (alias: "boxcar")
  • "biweight" (alias: "bisquare")
  • "triangle" (alias: "triangular")
  • "cosine"

RobustnessMethod

See: Robustness

  • "bisquare" (default; alias: "biweight")
  • "huber"
  • "talwar"

Delta

Points within Delta of each other on the x-axis share the same local fit instead of each computing its own regression — an interpolation shortcut that trades a small amount of accuracy for a large speedup on dense, evenly-spaced data. nil (default) auto-sets it to 0 in Streaming mode, i.e. interpolation is disabled and every point is fit exactly.

ZeroWeightFallback

Behavior when all neighborhood weights are zero:

OptionBehavior
"use_local_mean" (default; aliases: "local_mean", "mean")Use the mean of the neighborhood
"return_original" (alias: "original")Return the original y value
"return_none" (alias: "none")Return NaN

BoundaryPolicy

See: Boundary Handling

  • "extend" (default; alias: "pad")
  • "reflect" (alias: "mirror")
  • "zero"
  • "noboundary" (alias: "none")

ScalingMethod

See: Scaling Methods

  • "mad" (default; alias: "median_absolute_deviation")
  • "mar" (alias: "median_absolute_residual")
  • "mean" (alias: "mean_absolute_residual")

AutoConverge

See: Robustness

Convergence tolerance for early stopping of robustness iterations. nil (default) disables early stopping.

Missing

Policy for handling non-finite (NaN/Inf) values within each chunk:

OptionBehavior
"error" (default)Return an error if any value in the chunk is non-finite
"drop"Silently remove rows where X or Y is non-finite before merging the chunk with the overlap buffer

Note: A length mismatch between X and Y always errors, even under "drop".

ChunkSize

Number of points processed per chunk. Larger chunks reduce per-chunk overhead and give each local fit more surrounding context, at the cost of higher peak memory; smaller chunks bound memory tightly but increase the fraction of points that fall in overlap regions. A good starting point is balancing available memory against how much processing overhead per chunk is acceptable — match it to your file-read buffer or message-batch size to avoid unnecessary copying.

Overlap

Number of points retained from the previous chunk as context, so the neighbourhood at chunk boundaries isn’t artificially truncated. Points inside the overlap zone are fitted twice (once by each chunk) and reconciled via MergeStrategy. A good starting point is 10–20% of ChunkSize: too little overlap causes visible boundary artefacts, while too much wastes computation refitting the same points twice.

  • -1 (default) — computes chunk_size / 10, clamped to at least 1 and less than ChunkSize
  • Any integer >= 1 and < ChunkSize

MergeStrategy

See: Merge Strategies

StrategyAliasBehavior
"weighted_average" (default)"weighted"Distance-weighted blend
"average""mean"Average overlapping values
"take_first""first"Keep left chunk values
"take_last""last"Keep right chunk values

Parallel

Enable multi-threaded execution via Rayon.

  • true (default) — parallelizes the local regression fits across CPU cores
  • false — forces single-threaded execution

Outputs: se

See: Intervals

Select "se" to compute standard errors per chunk the same way Batch does, then merge the overlap region across chunk boundaries via MergeStrategy.

Outputs: diagnostics

See: Diagnostics

Select "diagnostics" to include a Diagnostics object (RMSE, MAE, R², residual_sd) in the result. EffectiveDF/AIC/AICc require per-chunk hat-matrix leverage to be threaded into the cumulative diagnostics computation across chunk boundaries, which isn’t currently done (even with "se" or Intervals set), so they’re always nil here.

Outputs: residuals

Select "residuals" to include per-point residuals (Y - fitted) in the result.

Outputs: weights

Select "weights" to include the final per-point robustness weights (from the last robustness iteration) in the result.

Outputs: derivative

Select "derivative" to expose each point’s local WLS slope in Result.Derivative at effectively no extra computation cost. Derivative values in the overlap region are merged via MergeStrategy.

Intervals

See: Intervals

Set Intervals.Confidence and/or Intervals.Prediction to a level such as 0.95, or leave either nil to disable it. Intervals.Bootstrap (at least 2) refits each combined chunk for percentile bounds and standard errors; overlap values are merged via MergeStrategy. Set the outer Seed for reproducible draws.

Result Structure

Result

Returned by ProcessChunk and Finalize.

FieldTypePopulated when
X, Y[]float64Always.
StandardErrors[]float64"se" output or Intervals
ConfidenceLower, ConfidenceUpper[]float64Intervals.Confidence
PredictionLower, PredictionUpper[]float64Intervals.Prediction
Residuals[]float64"residuals" output
RobustnessWeights[]float64"weights" output
CVScores[]float64Always empty (Batch only)
FractionUsedfloat64Always.
IterationsUsedintAlways (-1 if not available).
Diagnostics*Diagnostics"diagnostics" output
Derivative[]float64"derivative" output

Diagnostics

FieldTypeDescription
RMSEfloat64Root Mean Squared Error
MAEfloat64Mean Absolute Error
RSquaredfloat64R-squared
ResidualSDfloat64Cumulative sample SD of emitted residuals
EffectiveDF*float64nil (cumulative diagnostics don’t integrate per-chunk leverage; Batch only)
AIC*float64nil (requires EffectiveDF; Batch only)
AICc*float64nil (requires EffectiveDF; Batch only)

Example

package main

import (
 "fmt"
 "log"

 "github.com/thisisamirv/lowess-project/bindings/go/fastlowess/v5"
)

func main() {
 const n = 20
 x := make([]float64, n)
 y := make([]float64, n)
 for i := 0; i < n; i++ {
  x[i] = float64(i)
  y[i] = float64(i) + 0.1
 }

 opts := fastlowess.DefaultStreamingOptions()
 opts.ChunkSize = 10
 opts.Overlap = 2

 model, err := fastlowess.NewStreamingLowess(opts)
 if err != nil {
  log.Fatal(err)
 }
 defer model.Close()

 if _, err := model.ProcessChunk(x[:10], y[:10]); err != nil {
  log.Fatal(err)
 }
 if _, err := model.ProcessChunk(x[10:], y[10:]); err != nil {
  log.Fatal(err)
 }

 result, err := model.Finalize()
 if err != nil {
  log.Fatal(err)
 }
 fmt.Printf("y[0]: %.4f\n", result.Y[0])
}
y[0]: 17.9200