StreamingLowess API

See also: fastLowess

When to Use

• Dataset >100,000 points
• Memory-constrained environments
• Batch processing pipelines

Class

StreamingLowess

The StreamingLowess class processes data in chunks, suitable for very large datasets or streaming applications.

Constructor:

const { StreamingLowess } = require('fastlowess');

const stream = new StreamingLowess({ fraction: 0.5 }, { chunk_size: 50, overlap: 10 });
const x = Float64Array.from({ length: 10 }, (_, i) => i);
const y = Float64Array.from({ length: 10 }, (_, i) => i * 0.5);
stream.process_chunk(x, y);
const result = stream.finalize();
console.log("Smoothed", result.y.length, "points via streaming");

Smoothed 10 points via streaming

• options: An object containing StreamingSmoothOptions fields (a subset of the Batch LowessOptions fields — see below).
• streamingOptions: An object containing StreamingOptions fields.

process_chunk(x, y)

Feeds one chunk of data into the model. Each chunk is fit together with the trailing overlap points buffered from the previous call, then only the points that are fully resolved are returned — the tail of the chunk (the next overlap points) is held back internally, since it will be refit once the following chunk arrives and its estimate reconciled via merge_strategy. This is what lets the adapter process a dataset far larger than memory allows, one bounded-size chunk at a time, without ever materializing the whole dataset at once.

const { StreamingLowess } = require('fastlowess');

const n = 100;
const x = Float64Array.from({ length: n }, (_, i) => i * 2 * Math.PI / (n - 1));
const y = Float64Array.from(x, xi => Math.sin(xi) + 0.1);

const stream = new StreamingLowess({ fraction: 0.5 }, { chunk_size: 50, overlap: 10 });
const partialResult = stream.process_chunk(x.slice(0, 50), y.slice(0, 50));
console.log("Fraction used:", partialResult.fraction_used);

Fraction used: 0.5

finalize()

Flushes the overlap points still buffered from the last process_chunk() call. Because each call withholds its tail until the next chunk arrives to resolve it, the final chunk’s tail would never be emitted otherwise — always call finalize() once after the last chunk to retrieve it.

const { StreamingLowess } = require('fastlowess');

const n = 100;
const x = Float64Array.from({ length: n }, (_, i) => i * 2 * Math.PI / (n - 1));
const y = Float64Array.from(x, xi => Math.sin(xi) + 0.1);

const stream = new StreamingLowess({ fraction: 0.5 }, { chunk_size: 50, overlap: 10 });
stream.process_chunk(x.slice(0, 50), y.slice(0, 50));
stream.process_chunk(x.slice(50), y.slice(50));
const finalResult = stream.finalize();
console.log("Fraction used:", finalResult.fraction_used);

Fraction used: 0.5

Options Structure

StreamingSmoothOptions

Field	Type	Default	Description
fraction	number	0.67	Smoothing fraction (bandwidth)
iterations	number	3	Number of robustifying iterations
weight_function	string	"tricube"	Weight function name
robustness_method	string	"bisquare"	Robustness method name
delta	number	NaN	Interpolation distance (NaN auto-sets it to 0.0 in Streaming, i.e. interpolation disabled)
zero_weight_fallback	string	"use_local_mean"	Zero-weight handling
boundary_policy	string	"extend"	Boundary handling policy
scaling_method	string	"mad"	Residual scaling method
auto_converge	number	null	Auto-convergence tolerance
missing	string	"error"	Policy for non-finite (NaN/Inf) values in each chunk
chunk_size	number	5000	Data chunk size
overlap	number	chunk_size / 10	Overlap between chunks
merge_strategy	string	"weighted_average"	Strategy for blending overlap regions
parallel	boolean	true	Enable parallel execution
outputs	string[]	[]	Select se, diagnostics, residuals, weights, and/or derivative
intervals	object	null	Grouped confidence, prediction, and per-chunk bootstrap options
seed	number	null	Reproducible bootstrap draws for each combined chunk; must be an integer from 0 through Number.MAX_SAFE_INTEGER

Cross-validation, GPU backend, custom_weights, and the "sorted" output are Batch-only and not available here; see fastLowess for those. Standard errors and confidence/prediction intervals are computed per combined chunk (including the previous overlap), then blended across overlap regions via merge_strategy like y/derivative are; they are local chunk intervals, not whole-stream intervals.

Options

fraction

fraction is the most important parameter: it controls the size of the local neighbourhood used at each point.

Range	Effect	Use case
0.1-0.3	Fine detail	Rapidly changing signals
0.3-0.5	Balanced	General purpose
0.5-0.7	Heavy smoothing	Noisy data
0.7-1.0	Very smooth	Trend extraction

iterations

iterations controls robustness to outliers, at the cost of speed.

Value	Effect	Performance
0	No robustness	Fastest
1-3	Moderate	Recommended
4-6	Strong	Contaminated data
7+	Very strong	Heavy outliers

weight_function

See: Weight Functions

• "tricube" (default)
• "epanechnikov"
• "gaussian"
• "uniform" (alias: "boxcar")
• "biweight" (alias: "bisquare")
• "triangle" (alias: "triangular")
• "cosine"

robustness_method

See: Robustness

• "bisquare" (default; alias: "biweight")
• "huber"
• "talwar"

delta

Points within delta of each other on the x-axis share the same local fit instead of each computing its own regression — an interpolation shortcut that trades a small amount of accuracy for a large speedup on dense, evenly-spaced data. NaN (default) auto-sets it to 0 in Streaming mode, i.e. interpolation is disabled and every point is fit exactly.

zero_weight_fallback

Behavior when all neighborhood weights are zero:

Option	Behavior
"use_local_mean" (default; aliases: "local_mean", "mean")	Use the mean of the neighborhood
"return_original" (alias: "original")	Return the original y value
"return_none" (alias: "none")	Return NaN

boundary_policy

See: Boundary Handling

• "extend" (default; alias: "pad")
• "reflect" (alias: "mirror")
• "zero"
• "noboundary" (alias: "none")

scaling_method

See: Scaling Methods

• "mad" (default; alias: "median_absolute_deviation")
• "mar" (alias: "median_absolute_residual")
• "mean" (alias: "mean_absolute_residual")

auto_converge

See: Robustness

Convergence tolerance for early stopping of robustness iterations. null (default) disables early stopping.

missing

Policy for handling non-finite (NaN/Inf) values within each chunk:

Option	Behavior
"error" (default)	Throw an error if any value in the chunk is non-finite
"drop"	Silently remove rows where x or y is non-finite before merging the chunk with the overlap buffer

Note: A length mismatch between x and y always errors, even under "drop".

chunk_size

Number of points processed per chunk. Larger chunks reduce per-chunk overhead and give each local fit more surrounding context, at the cost of higher peak memory; smaller chunks bound memory tightly but increase the fraction of points that fall in overlap regions. A good starting point is balancing available memory against how much processing overhead per chunk is acceptable — match it to your file-read buffer or message-batch size to avoid unnecessary copying.

overlap

Number of points retained from the previous chunk as context, so the neighbourhood at chunk boundaries isn’t artificially truncated. Points inside the overlap zone are fitted twice (once by each chunk) and reconciled via merge_strategy. A good starting point is 10–20% of chunk_size: too little overlap causes visible boundary artefacts, while too much wastes computation refitting the same points twice.

• null (default) — computes chunk_size / 10, clamped to at least 1 and less than chunk_size
• Any integer >= 1 and < chunk_size

merge_strategy

See: Merge Strategies

Strategy	Alias	Behavior
"weighted_average" (default)	"weighted"	Distance-weighted blend
"average"	"mean"	Average overlapping values
"take_first"	"first"	Keep left chunk values
"take_last"	"last"	Keep right chunk values

parallel

Enable multi-threaded execution via Rayon.

• true (default) — parallelizes the local regression fits across CPU cores
• false — forces single-threaded execution

outputs: se

See: Intervals

Computes standard errors per chunk the same way Batch does, then merges the overlap region across chunk boundaries the same way y/derivative are, via merge_strategy.

outputs: diagnostics

See: Diagnostics

Select "diagnostics" to include a Diagnostics object (RMSE, MAE, R², residual_sd) in the result. effective_df/aic/aicc require per-chunk hat-matrix leverage to be threaded into the cumulative diagnostics computation across chunk boundaries, which isn’t currently done (even with "se" or intervals selected), so they’re always null here.

outputs: residuals

Include per-point residuals (y - fitted) in the result.

outputs: weights

Include the final per-point robustness weights (from the last robustness iteration) in the result.

outputs: derivative

Each point’s local WLS fit already computes a slope internally; this exposes that per-point slope (rate of change of the smoothed curve) in LowessResult.derivative at effectively no extra computation cost. Derivative values in the overlap region are merged across chunk boundaries the same way y is, via merge_strategy.

intervals

See: Intervals

An object such as { confidence: 0.90, prediction: 0.99, bootstrap: 200 }, populating result.confidence_lower/result.confidence_upper and result.prediction_lower/result.prediction_upper. The coverage levels are independent. Computed per combined chunk and merged across overlap boundaries via merge_strategy. bootstrap (at least 2) refits each combined chunk from resampled residuals; with parallel: true, refits run concurrently.

seed

Seeds bootstrap draws. Each combined chunk restarts from the same seed. It does not enable bootstrap by itself. Seeds must be finite, non-negative safe integers no greater than Number.MAX_SAFE_INTEGER; fractional, negative, and unsafe values throw. 0 is valid.

Result Structure

LowessResult

Returned by process_chunk() and finalize().

Field	Type	Description
x	Float64Array	x values (same order as input)
y	Float64Array	Smoothed y values
fraction_used	number	Fraction used
iterations_used	number | null	Robustness iterations actually performed
standard_errors	Float64Array | null	Per-point standard errors (if "se" or any interval was set)
confidence_lower	Float64Array | null	Lower confidence bounds (if intervals.confidence was set)
confidence_upper	Float64Array | null	Upper confidence bounds (if intervals.confidence was set)
prediction_lower	Float64Array | null	Lower prediction bounds (if intervals.prediction was set)
prediction_upper	Float64Array | null	Upper prediction bounds (if intervals.prediction was set)
residuals	Float64Array | null	Residuals (if "residuals" was requested)
robustness_weights	Float64Array | null	Robustness weights (if "weights" was requested)
cv_scores	Float64Array | null	Always null (Batch only)
diagnostics	Diagnostics | null	Fit metrics (if "diagnostics" was requested)
derivative	Float64Array | null	Per-point local fit derivative/slope (if "derivative" was requested)

Diagnostics

Field	Type	Description
rmse	number	Root Mean Squared Error
mae	number	Mean Absolute Error
r_squared	number	R-squared
residual_sd	number	Cumulative sample SD of emitted residuals
effective_df	number | null	Always null (cumulative diagnostics don’t integrate per-chunk leverage; Batch only)
aic	number | null	Always null (requires effective_df; Batch only)
aicc	number | null	Always null (requires effective_df; Batch only)