Check if `smooth` should hide confidence band for low number of points

[jkrumbiegel]

Seems like ggplot2 doesn't draw the CI band for 6 points or less:

library(ggplot2)

df <- data.frame(
  time = c(0.0, 1.0, 2.0, 3.0, 4.0, 5.0),
  cp = c(1.0, 2.0, 1.5, 1.3, 1.1, 0.9)
)

ggplot(df, aes(time, cp)) +
  geom_smooth(se = TRUE) +
  geom_point()

but

library(ggplot2)

df <- data.frame(
  time = c(0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0),
  cp = c(1.0, 2.0, 1.5, 1.3, 1.1, 0.9, 0.4)
)

ggplot(df, aes(time, cp)) +
  geom_smooth(se = TRUE) +
  geom_point()

I've also noticed the wavy pattern to be a bit unstable between local tests and some CI runner, so it should be investigated if it would be better to hide it.

In AoG the equivalent plots look like this

[andreasnoack]

I looked into the Loess.jl side of this. The output is essentially bit-deterministic on a given machine; cross-LAPACK roundoff propagates at ~1e-14, which won't shift pixels.

The reason Loess.jl and ggplot2 (i.e. R's loess) disagree on the band is R's defaults, not numerical instability. Running the 7-point example:

quantity	Loess.jl	R default	R with surface="direct", statistics="exact"
pred at x = 0…5	match	match	shifted (no kd-tree blending)
pred at x = 6 (boundary)	0.4103288	0.410228	0.4103288
one.delta (δ₁)	0.7239113	0.2845012	0.7239113
s	0.2885308	0.4602410	0.2885308

Two things drive the differences:

1. statistics = "approximate" (R's default) uses Cleveland's lookup approximation for δ₁/δ₂ summed over kd-tree vertices instead of tr((I−L)′(I−L)). R's reported enp = 5.587 doesn't even equal the actual tr(L′L) ≈ 6.026 of its own hat matrix. Loess.jl computes it exactly. That alone makes R's s ~1.6× larger here, which is most of the band-width difference.

2. Outer kd-tree vertices: R extends them slightly past the data (here vert = [-0.03, 6.03]), Loess.jl puts them exactly at the data extrema. That's a deliberate deviation noted in src/kd.jl and explains the boundary prediction gap (0.410228 vs 0.410329).

When R is asked for the same algorithm (surface="direct", statistics="exact") it agrees with Loess.jl to ~6 digits.

About the wavy pattern itself: at vertex x=0 the local quadratic has slope 1.13 (asymmetric neighborhood, captures the rise from 1→2), while at x=1 the slope is 0.34 and at x=2 it's −0.35. The cubic-Hermite blend between those slopes produces the hump. It's real loess behavior at small n with span=0.75, degree=2, not a numerical artifact, and (by the way) the two screenshots above are 6 points vs 7 points, not local-vs-CI — I had originally read that as a cross-machine difference but it isn't.

So my vote: I don't think we should hide it. The math is correct, it's reproducible, and the visual difference from ggplot2 is explained by R's approximate-by-default statistics, not by anything we're getting wrong.

[jkrumbiegel]

Yes, to reiterate, I had a CI refimage failure on the wavy pattern of the 6 point curve. Might have been my mistake, of course, but I had no other reason to suspect than numerical drift at the time. I'll close this though as I agree with the analysis above. In a way, not too much interpretative weight should be given anyway to bounds on such small datasets.