StreamingLowess API
For datasets that don’t fit in memory or arrive in chunks. Processes data incrementally, merging overlapping regions between chunks.
See also: API
Class
StreamingLowess
The StreamingLowess class processes data in chunks, suitable for very large datasets or streaming applications.
Constructor:
StreamingOptions options = StreamingOptions.builder()
.chunkSize(2000)
.overlap(200)
.build();
try (StreamingLowess model = new StreamingLowess(options)) {
// ...
}
-
new StreamingLowess(StreamingOptions options)creates a new streaming model with the given options. -
options: AStreamingOptionsinstance (exposes allOptionssettings pluschunkSize,overlap, andmergeStrategy).
Methods:
processChunk(double[] x, double[] y) Result
Feeds one chunk of data into the model. Each chunk is fit together with the trailing overlap points buffered from the previous call, then only the points that are fully resolved are returned — the tail of the chunk (the next overlap points) is held back internally, since it will be refit once the following chunk arrives and its estimate reconciled via mergeStrategy. This is what lets the adapter process a dataset far larger than memory allows, one bounded-size chunk at a time, without ever materializing the whole dataset at once.
Result partial = model.processChunk(x, y);
finish() Result
Flushes the overlap points still buffered from the last processChunk call. Because each call withholds its tail until the next chunk arrives to resolve it, the final chunk’s tail would never be emitted otherwise — always call finish() once after the last chunk to retrieve it.
Result result = model.finish();
-
StreamingLowess.close()releases native resources. Safe to call multiple times. ImplementsAutoCloseable.
Options Structure
StreamingOptions
| Setting | Type | Default | Description |
|---|---|---|---|
|
|
|
Smoothing fraction (bandwidth) |
|
|
|
Number of robustifying iterations |
|
|
|
Weight function name |
|
|
|
Robustness method name |
|
|
|
Interpolation distance ( |
|
|
|
Zero-weight handling |
|
|
|
Boundary handling policy |
|
|
|
Residual scaling method |
|
|
|
Auto-convergence tolerance |
|
|
|
Policy for non-finite (NaN/Infinity) values in each chunk |
|
|
|
Data chunk size |
|
|
|
Overlap between chunks |
|
|
|
Strategy for blending overlap regions |
|
|
|
Enable parallel execution |
|
|
empty |
Select |
|
|
disabled |
Confidence/prediction levels and per-chunk bootstrap refits |
|
|
unset |
Reproducible bootstrap draws for each combined chunk |
Cross-validation, GPU backend, customWeights, and the sorted output are Batch-only and not available here; see API for those. Standard errors and intervals are computed per combined chunk (including the previous overlap) and blended across overlaps via mergeStrategy; they are local chunk intervals, not whole-stream intervals.
Options
fraction
fraction is the most important parameter: it controls the size of the local neighbourhood used at each point.
| Range | Effect | Use case |
|---|---|---|
0.1-0.3 |
Fine detail |
Rapidly changing signals |
0.3-0.5 |
Balanced |
General purpose |
0.5-0.7 |
Heavy smoothing |
Noisy data |
0.7-1.0 |
Very smooth |
Trend extraction |
iterations
iterations controls robustness to outliers, at the cost of speed.
| Value | Effect | Performance |
|---|---|---|
0 |
No robustness |
Fastest |
1-3 |
Moderate |
Recommended |
4-6 |
Strong |
Contaminated data |
7+ |
Very strong |
Heavy outliers |
weightFunction
See: Weight Functions
-
"tricube"(default) -
"epanechnikov" -
"gaussian" -
"uniform"(alias:"boxcar") -
"biweight"(alias:"bisquare") -
"triangle"(alias:"triangular") -
"cosine"
delta
Points within delta of each other on the x-axis share the same local fit instead of each computing its own regression — an interpolation shortcut that trades a small amount of accuracy for a large speedup on dense, evenly-spaced data. NaN (default) auto-sets it to 0 in Streaming mode, i.e. interpolation is disabled and every point is fit exactly.
zeroWeightFallback
Behavior when all neighborhood weights are zero:
| Option | Behavior |
|---|---|
|
Use the mean of the neighborhood |
|
Return the original y value |
|
Return |
boundaryPolicy
See: Boundary Handling
-
"extend"(default; alias:"pad") -
"reflect"(alias:"mirror") -
"zero" -
"noboundary"(alias:"none")
scalingMethod
See: Scaling Methods
-
"mad"(default; alias:"median_absolute_deviation") -
"mar"(alias:"median_absolute_residual") -
"mean"(alias:"mean_absolute_residual")
autoConverge
See: Robustness
Convergence tolerance for early stopping of robustness iterations. NaN (default) disables early stopping.
missing
Policy for handling non-finite (NaN/Infinity) values within each chunk:
| Option | Behavior |
|---|---|
|
Throw if any value in the chunk is non-finite |
|
Silently remove rows where |
A length mismatch between x and y always throws, even under "drop".
|
chunkSize
Number of points processed per chunk. Larger chunks reduce per-chunk overhead and give each local fit more surrounding context, at the cost of higher peak memory; smaller chunks bound memory tightly but increase the fraction of points that fall in overlap regions. A good starting point is balancing available memory against how much processing overhead per chunk is acceptable — match it to your file-read buffer or message-batch size to avoid unnecessary copying.
overlap
Number of points retained from the previous chunk as context, so the neighbourhood at chunk boundaries isn’t artificially truncated. Points inside the overlap zone are fitted twice (once by each chunk) and reconciled via mergeStrategy. A good starting point is 10-20% of chunkSize: too little overlap causes visible boundary artefacts, while too much wastes computation refitting the same points twice.
-
-1(default) — computeschunk_size / 10, clamped to at least 1 and less thanchunkSize -
Any integer
>= 1and< chunkSize
mergeStrategy
See: Merge Strategies
| Strategy | Behavior |
|---|---|
|
Distance-weighted blend |
|
Average overlapping values |
|
Keep left chunk values |
|
Keep right chunk values |
parallel
Enable multi-threaded execution via Rayon.
-
true(default) — parallelizes the local regression fits across CPU cores -
false— forces single-threaded execution
outputs: diagnostics
Select "diagnostics" to include a Diagnostics object (RMSE, MAE, R², residual_sd) in the result. effectiveDf/aic/aicc require standard errors, which are Batch-only, so they’re always empty here.
outputs: weights
Select "weights" to include the final per-point robustness weights (from the last robustness iteration) in the result.
outputs: derivative
Each point’s local WLS fit already computes a slope internally; this exposes that per-point slope (rate of change of the smoothed curve) via Result.derivative() at effectively no extra computation cost. Derivative values in the overlap region are merged across chunk boundaries the same way y() is, via mergeStrategy.
intervals
See: Intervals
Build with IntervalsOptions.builder(). confidence(level) and prediction(level) add bounds; bootstrap(n) (at least 2) replaces analytic intervals with residual-bootstrap refits of each combined chunk. With parallel(true), refits run concurrently.
Result Structure
Result
Returned by processChunk and finish.
| Accessor | Type | Populated when |
|---|---|---|
|
|
Always. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Always empty (Batch only) |
|
|
Always. |
|
|
Always (empty if not available). |
|
|
|
|
|
|
Diagnostics
| Accessor | Type | Description |
|---|---|---|
|
|
Root Mean Squared Error |
|
|
Mean Absolute Error |
|
|
R-squared |
|
|
Cumulative sample SD of emitted residuals |
|
|
Always empty (requires standard errors, Batch only) |
|
|
Always empty (requires |
|
|
Always empty (requires |
Example
import fastlowess.Result;
import fastlowess.StreamingLowess;
import fastlowess.StreamingOptions;
public class Example {
public static void main(String[] args) {
final int n = 20;
double[] x = new double[n];
double[] y = new double[n];
for (int i = 0; i < n; i++) {
x[i] = i;
y[i] = i + 0.1;
}
StreamingOptions options = StreamingOptions.builder()
.chunkSize(10)
.overlap(2)
.build();
try (StreamingLowess model = new StreamingLowess(options)) {
double[] x1 = java.util.Arrays.copyOfRange(x, 0, 10);
double[] y1 = java.util.Arrays.copyOfRange(y, 0, 10);
double[] x2 = java.util.Arrays.copyOfRange(x, 10, 20);
double[] y2 = java.util.Arrays.copyOfRange(y, 10, 20);
model.processChunk(x1, y1);
model.processChunk(x2, y2);
Result result = model.finish();
System.out.printf("y[0]: %.4f%n", result.y()[0]);
}
}
}
y[0]: 17.9200