For datasets that don’t fit in memory or arrive in chunks. Processes data incrementally, merging overlapping regions between chunks.
See also: API
When to Use
- Dataset >100,000 points
- Memory-constrained environments
- Batch processing pipelines
Class
StreamingLowess
The StreamingLowess type processes data in chunks, suitable for very large datasets or streaming applications.
Constructor:
opts := fastlowess.DefaultStreamingOptions()
opts.ChunkSize = 2000
opts.Overlap = 200
model, err := fastlowess.NewStreamingLowess(opts)
if err != nil {
panic(err)
}
defer model.Close()fastlowess.NewStreamingLowess(opts StreamingOptions) (*StreamingLowess, error)creates a new streaming model with the given options.opts: AStreamingOptionsstruct.
Methods:
ProcessChunk(x, y []float64) (Result, error)
Feeds one chunk of data into the model. Each chunk is fit together with the trailing Overlap points buffered from the previous call, then only the points that are fully resolved are returned — the tail of the chunk (the next Overlap points) is held back internally, since it will be refit once the following chunk arrives and its estimate reconciled via MergeStrategy. This is what lets the adapter process a dataset far larger than memory allows, one bounded-size chunk at a time, without ever materializing the whole dataset at once.
result, err := model.ProcessChunk(x, y)Finalize() (Result, error)
Flushes the overlap points still buffered from the last ProcessChunk call. Because each call withholds its tail until the next chunk arrives to resolve it, the final chunk’s tail would never be emitted otherwise — always call Finalize once after the last chunk to retrieve it.
result, err := model.Finalize()(*StreamingLowess) Close() errorreleases native resources. Safe to call multiple times.
Options Structure
StreamingOptions
| Field | Type | Default | Description |
|---|---|---|---|
Fraction | float64 | 0.67 | Smoothing fraction (bandwidth) |
Iterations | int | 3 | Number of robustifying iterations |
WeightFunction | string | "tricube" | Weight function name |
RobustnessMethod | string | "bisquare" | Robustness method name |
Delta | *float64 | nil | Interpolation distance (nil auto-sets it to 0.0 in Streaming, i.e. interpolation disabled) |
ZeroWeightFallback | string | "use_local_mean" | Zero-weight handling |
BoundaryPolicy | string | "extend" | Boundary handling policy |
ScalingMethod | string | "mad" | Residual scaling method |
AutoConverge | *float64 | nil | Auto-convergence tolerance |
Missing | string | "error" | Policy for non-finite (NaN/Inf) values in each chunk |
ChunkSize | int | 5000 | Data chunk size |
Overlap | int | chunk_size / 10 | Overlap between chunks |
MergeStrategy | string | "weighted_average" | Strategy for blending overlap regions |
Parallel | bool | true | Enable parallel execution |
Outputs | []string | nil | Optional components: se, diagnostics, residuals, weights, derivative. |
Intervals | *IntervalsOptions | nil | Confidence/prediction levels and per-chunk bootstrap refits |
Seed | *uint64 | nil | Reproducible bootstrap draws for each combined chunk |
Cross-validation, GPU Backend, CustomWeights, and the "sorted" output are Batch-only and not available here; see API for those. Standard errors and confidence/prediction intervals are computed per chunk the same way Batch computes them, then blended across overlap regions via MergeStrategy like Y/Derivative are.
Options
Fraction
Fraction is the most important parameter: it controls the size of the local neighbourhood used at each point.
| Range | Effect | Use case |
|---|---|---|
| 0.1-0.3 | Fine detail | Rapidly changing signals |
| 0.3-0.5 | Balanced | General purpose |
| 0.5-0.7 | Heavy smoothing | Noisy data |
| 0.7-1.0 | Very smooth | Trend extraction |
Iterations
Iterations controls robustness to outliers, at the cost of speed.
| Value | Effect | Performance |
|---|---|---|
| 0 | No robustness | Fastest |
| 1-3 | Moderate | Recommended |
| 4-6 | Strong | Contaminated data |
| 7+ | Very strong | Heavy outliers |
WeightFunction
See: Weight Functions
"tricube"(default)"epanechnikov""gaussian""uniform"(alias:"boxcar")"biweight"(alias:"bisquare")"triangle"(alias:"triangular")"cosine"
RobustnessMethod
See: Robustness
"bisquare"(default; alias:"biweight")"huber""talwar"
Delta
Points within Delta of each other on the x-axis share the same local fit instead of each computing its own regression — an interpolation shortcut that trades a small amount of accuracy for a large speedup on dense, evenly-spaced data. nil (default) auto-sets it to 0 in Streaming mode, i.e. interpolation is disabled and every point is fit exactly.
ZeroWeightFallback
Behavior when all neighborhood weights are zero:
| Option | Behavior |
|---|---|
"use_local_mean" (default; aliases: "local_mean", "mean") | Use the mean of the neighborhood |
"return_original" (alias: "original") | Return the original y value |
"return_none" (alias: "none") | Return NaN |
BoundaryPolicy
See: Boundary Handling
"extend"(default; alias:"pad")"reflect"(alias:"mirror")"zero""noboundary"(alias:"none")
ScalingMethod
See: Scaling Methods
"mad"(default; alias:"median_absolute_deviation")"mar"(alias:"median_absolute_residual")"mean"(alias:"mean_absolute_residual")
AutoConverge
See: Robustness
Convergence tolerance for early stopping of robustness iterations. nil (default) disables early stopping.
Missing
Policy for handling non-finite (NaN/Inf) values within each chunk:
| Option | Behavior |
|---|---|
"error" (default) | Return an error if any value in the chunk is non-finite |
"drop" | Silently remove rows where X or Y is non-finite before merging the chunk with the overlap buffer |
Note: A length mismatch between X and Y always errors, even under "drop".
ChunkSize
Number of points processed per chunk. Larger chunks reduce per-chunk overhead and give each local fit more surrounding context, at the cost of higher peak memory; smaller chunks bound memory tightly but increase the fraction of points that fall in overlap regions. A good starting point is balancing available memory against how much processing overhead per chunk is acceptable — match it to your file-read buffer or message-batch size to avoid unnecessary copying.
Overlap
Number of points retained from the previous chunk as context, so the neighbourhood at chunk boundaries isn’t artificially truncated. Points inside the overlap zone are fitted twice (once by each chunk) and reconciled via MergeStrategy. A good starting point is 10–20% of ChunkSize: too little overlap causes visible boundary artefacts, while too much wastes computation refitting the same points twice.
-1(default) — computeschunk_size / 10, clamped to at least 1 and less thanChunkSize- Any integer
>= 1and< ChunkSize
MergeStrategy
See: Merge Strategies
| Strategy | Alias | Behavior |
|---|---|---|
"weighted_average" (default) | "weighted" | Distance-weighted blend |
"average" | "mean" | Average overlapping values |
"take_first" | "first" | Keep left chunk values |
"take_last" | "last" | Keep right chunk values |
Parallel
Enable multi-threaded execution via Rayon.
true(default) — parallelizes the local regression fits across CPU coresfalse— forces single-threaded execution
Outputs: se
See: Intervals
Select "se" to compute standard errors per chunk the same way Batch does, then merge the overlap region across chunk boundaries via MergeStrategy.
Outputs: diagnostics
See: Diagnostics
Select "diagnostics" to include a Diagnostics object (RMSE, MAE, R², residual_sd) in the result. EffectiveDF/AIC/AICc require per-chunk hat-matrix leverage to be threaded into the cumulative diagnostics computation across chunk boundaries, which isn’t currently done (even with "se" or Intervals set), so they’re always nil here.
Outputs: residuals
Select "residuals" to include per-point residuals (Y - fitted) in the result.
Outputs: weights
Select "weights" to include the final per-point robustness weights (from the last robustness iteration) in the result.
Outputs: derivative
Select "derivative" to expose each point’s local WLS slope in Result.Derivative at effectively no extra computation cost. Derivative values in the overlap region are merged via MergeStrategy.
Intervals
See: Intervals
Set Intervals.Confidence and/or Intervals.Prediction to a level such as 0.95, or leave either nil to disable it. Intervals.Bootstrap (at least 2) refits each combined chunk for percentile bounds and standard errors; overlap values are merged via MergeStrategy. Set the outer Seed for reproducible draws.
Result Structure
Result
Returned by ProcessChunk and Finalize.
| Field | Type | Populated when |
|---|---|---|
X, Y | []float64 | Always. |
StandardErrors | []float64 | "se" output or Intervals |
ConfidenceLower, ConfidenceUpper | []float64 | Intervals.Confidence |
PredictionLower, PredictionUpper | []float64 | Intervals.Prediction |
Residuals | []float64 | "residuals" output |
RobustnessWeights | []float64 | "weights" output |
CVScores | []float64 | Always empty (Batch only) |
FractionUsed | float64 | Always. |
IterationsUsed | int | Always (-1 if not available). |
Diagnostics | *Diagnostics | "diagnostics" output |
Derivative | []float64 | "derivative" output |
Diagnostics
| Field | Type | Description |
|---|---|---|
RMSE | float64 | Root Mean Squared Error |
MAE | float64 | Mean Absolute Error |
RSquared | float64 | R-squared |
ResidualSD | float64 | Cumulative sample SD of emitted residuals |
EffectiveDF | *float64 | nil (cumulative diagnostics don’t integrate per-chunk leverage; Batch only) |
AIC | *float64 | nil (requires EffectiveDF; Batch only) |
AICc | *float64 | nil (requires EffectiveDF; Batch only) |
Example
package main
import (
"fmt"
"log"
"github.com/thisisamirv/lowess-project/bindings/go/fastlowess/v5"
)
func main() {
const n = 20
x := make([]float64, n)
y := make([]float64, n)
for i := 0; i < n; i++ {
x[i] = float64(i)
y[i] = float64(i) + 0.1
}
opts := fastlowess.DefaultStreamingOptions()
opts.ChunkSize = 10
opts.Overlap = 2
model, err := fastlowess.NewStreamingLowess(opts)
if err != nil {
log.Fatal(err)
}
defer model.Close()
if _, err := model.ProcessChunk(x[:10], y[:10]); err != nil {
log.Fatal(err)
}
if _, err := model.ProcessChunk(x[10:], y[10:]); err != nil {
log.Fatal(err)
}
result, err := model.Finalize()
if err != nil {
log.Fatal(err)
}
fmt.Printf("y[0]: %.4f\n", result.Y[0])
}y[0]: 17.9200