Real-Time Processing

Streaming and online LOWESS for live data.

Overview

When data arrives continuously—from sensors, logs, or streaming pipelines—you need incremental smoothing that doesn’t require reprocessing the entire dataset.


Online Mode: Point-by-Point

For true real-time applications where each point must be processed immediately.

windowCapacity = 25 limits the internal buffer to the 25 most recent observations; each addPoint call costs O(window) rather than growing with total history. minPoints = 5 suppresses output until the window holds enough points for a stable fit — calls made before that threshold return status FILLING. updateMode = "incremental" re-fits only the most recent point rather than the full window, halving typical latency at a modest accuracy cost.

Sensor Data Example

import fastlowess.AddPointResult;
import fastlowess.OnlineLowess;
import fastlowess.OnlineOptions;

public class Example {
    public static void main(String[] args) {
        OnlineOptions options = OnlineOptions.builder()
                .fraction(0.3)
                .windowCapacity(25)
                .minPoints(5)
                .updateMode("incremental")
                .build();

        try (OnlineLowess model = new OnlineLowess(options)) {
            int count = 0;
            for (int i = 0; i < 100; i++) {
                double xi = i;
                double yi = 20.0 + 5.0 * Math.sin(xi / 10.0) + Math.sin(xi * 1.7) * 0.5;

                AddPointResult res = model.addPoint(xi, yi);
                if (res.isReady()) {
                    if (count < 5) {
                        System.out.printf("Time %s: smoothed = %.4f%n", xi, res.point().orElseThrow().y());
                    }
                    count++;
                }
            }
            System.out.println("... (" + (count - 5) + " more)");
        }
    }
}
Time 4.0: smoothed = 22.1941
Time 5.0: smoothed = 22.7964
Time 6.0: smoothed = 22.4733
Time 7.0: smoothed = 22.9120
Time 8.0: smoothed = 24.0164
... (91 more)

Streaming Mode: Chunk Processing

For large datasets that arrive in batches or files.

chunkSize controls how many data points are processed in one pass; matching it to your file-read buffer or message-batch size avoids unnecessary copying. overlap retains that many points from the previous chunk as context so the neighbourhood at chunk boundaries is not artificially truncated. mergeStrategy = "weighted_average" blends the overlapping region smoothly; use "last" if chunk boundaries are guaranteed to be well separated and no blending is needed.

Always call finish(): The streaming adapter buffers overlap data. Call finish() after the last chunk to retrieve the buffered tail.

Log File Processing

import fastlowess.Result;
import fastlowess.StreamingLowess;
import fastlowess.StreamingOptions;

public class Example {
    public static void main(String[] args) {
        double[] chunk1X = new double[50];
        double[] chunk1Y = new double[50];
        double[] chunk2X = new double[50];
        double[] chunk2Y = new double[50];
        for (int i = 0; i < 50; i++) {
            chunk1X[i] = i;
            chunk1Y[i] = Math.sin(chunk1X[i]) + 0.1;
            chunk2X[i] = i + 50;
            chunk2Y[i] = Math.sin(chunk2X[i]) + 0.1;
        }

        StreamingOptions options = StreamingOptions.builder()
                .fraction(0.1)
                .iterations(2)
                .chunkSize(50)
                .overlap(10)
                .mergeStrategy("weighted_average")
                .build();

        try (StreamingLowess model = new StreamingLowess(options)) {
            model.processChunk(chunk1X, chunk1Y);
            model.processChunk(chunk2X, chunk2Y);

            Result result = model.finish();
            System.out.printf("y[0]: %.6f%n", result.y()[0]);
        }
    }
}
y[0]: 0.518853

Real-Time Dashboard Example

The dashboard pattern uses a plain LOWESS fit on a manually managed sliding window rather than OnlineLowess. This is the simplest approach when your UI framework already owns the data buffer and you only need the most recent smoothed value per frame. The trade-off is a full O(window^2) refit on every tick; for high-frequency streams prefer OnlineLowess with updateMode = "incremental" to bound per-frame cost.

import fastlowess.Lowess;
import fastlowess.Options;
import fastlowess.Result;

import java.util.ArrayList;
import java.util.List;

public class Example {
    public static void main(String[] args) {
        int n = 100;
        double[] x = new double[n];
        double[] y = new double[n];
        for (int i = 0; i < n; i++) {
            x[i] = i * 2 * Math.PI / (n - 1);
            y[i] = Math.sin(x[i]) + 0.1;
        }

        List<Double> windowX = new ArrayList<>();
        List<Double> windowY = new ArrayList<>();
        double latest = 0.0;

        for (int i = 0; i < n; i++) {
            windowX.add(x[i]);
            windowY.add(y[i]);
            if (windowX.size() > 50) {
                windowX.remove(0);
                windowY.remove(0);
            }
            if (windowX.size() < 2) continue;

            double[] wx = windowX.stream().mapToDouble(Double::doubleValue).toArray();
            double[] wy = windowY.stream().mapToDouble(Double::doubleValue).toArray();

            Options options = Options.builder().fraction(0.4).build();
            try (Lowess model = new Lowess(options)) {
                Result result = model.fit(wx, wy);
                latest = result.y()[result.y().length - 1];
            }
        }

        System.out.println("Smoothed (dashboard, latest tick): " + latest);
    }
}
Smoothed (dashboard, latest tick): -0.06780546949749054

Choosing Parameters

Online Mode

Parameter Guidance

windowCapacity

Enough history for fraction to work

minPoints

2–5 typically; higher for stability

updateMode

"incremental" for speed, "full" for accuracy

Streaming Mode

Parameter Guidance

chunkSize

Balance memory vs. processing overhead

overlap

10–20% of chunkSize for smooth transitions

mergeStrategy

"weighted_average" for best quality, "average" for simplicity


Performance Considerations

Mode Memory Latency Use Case

Online

Fixed (window)

~1ms/point

Sensors, dashboards

Streaming

~chunkSize

~100ms/chunk

Large files, ETL

Batch

Full dataset

N/A

Analysis, reports


See Also