|
fastlowess-cpp
High-performance LOWESS smoothing
|
Understanding how LOWESS works and when to use it.
LOWESS (Locally Weighted Scatterplot Smoothing) is a nonparametric regression method that fits smooth curves through scatter plots without assuming a global functional form.
Unlike parametric methods (linear regression, polynomial fitting), LOWESS adapts locally to the data structure, making it ideal for:
For each point in your data, LOWESS:
fraction)The fraction (also called bandwidth or span) is the most important parameter. It controls what proportion of data is used for each local fit.
| Fraction | Effect | When to Use |
|---|---|---|
| 0.1–0.3 | Fine detail, follows data closely | Rapidly changing signals |
| 0.3–0.5 | Balanced smoothing | Most applications |
| 0.5–0.7 | Heavy smoothing | Noisy data, trend extraction |
| 0.7–1.0 | Very smooth | Strong noise, global trends |
Rule of Thumb: Start with
fraction=0.67(the default) and adjust based on visual inspection. Use cross-validation for automated selection.
Standard LOWESS is sensitive to outliers. Robustness iterations downweight points with large residuals:
| Iterations | Effect | When to Use |
|---|---|---|
| 0 | No robustness (fastest) | Clean data, speed-critical |
| 1–3 | Moderate robustness | Most applications |
| 4–6 | Strong robustness | Data with outliers |
| 7+ | Very strong | Heavy contamination |
| Interval Type | What It Represents | Width |
|---|---|---|
| Confidence | Uncertainty in the mean curve | Narrow |
| Prediction | Uncertainty for new observations | Wide |
Choose the right mode based on your use case:
| Mode | Use Case | Memory | Features |
|---|---|---|---|
| Batch | Complete datasets | Entire dataset | All features |
| Streaming | Large files (>100K points) | One chunk | Residuals, robustness |
| Online | Real-time data | Fixed window | Incremental updates |
| Situation | Mode |
|---|---|
| Data fits in memory; needs intervals or CV | Batch |
| Data too large for memory or arrives in chunks | Streaming |
| Data arrives point-by-point in real time | Online |
| Feature | LOWESS | Polynomial Regression | Moving Average |
|---|---|---|---|
| No parametric assumptions | ✓ | ✗ | ✓ |
| Adapts to local structure | ✓ | ✗ | Partial |
| Robust to outliers | ✓ | ✗ | ✗ |
| Uncertainty estimates | ✓ | ✓ | ✗ |
| Handles irregular sampling | ✓ | ✓ | ✗ |