RFF/linear prediction strategy fix (take 2)

[gpleiss]

Attempt 2 at #2671 , but this time adding a naive fantasy strategy for BoTorch models.

The fantasy strategy is O(d^2 (n+m)), so it is no more efficient than re-conditioning the model. Without writing custom cuda kernels, it is unlikely to be more efficient.

[Balandat]

Test failure due to flakiness maybe?

cc @sdaulton as this affects BoTorch's FullyBayesianLinearSingleTaskGP (will test this separately)

[Balandat]

@gpleiss what's the deal with this? Why exactly is this necessary? This seems to be causing some other test failure here when running on a GPU, presumably due to changed numerical behavior: https://github.com/meta-pytorch/botorch/blob/main/test/models/test_fully_bayesian.py#L834

[gpleiss]

We need to resolve the LazyEvaluatedKernelTensor, because otherwise the $\boldsymbol K + \sigma^2 I$ matrix will be an AddedDiagLinearOperator(LazyEvaluatedKernelTensor, DiagLinearOperator) rather than a LowRankRootAddedDiagLinearOperator(LowRankRootLinearOperator, DiagLinearOperator). The latter is necessary to get the structure we want.

I can remove this evaluate_kernel if I add the following line before calling the super constructor:

train_prior_dist = train_prior_dist.__class__(
    train_prior_dist.mean, train_prior_dist.lazy_covariance_matrix.evaluate_kernel()
)

(which I originally added after calling the super constructor; moving it before is probably a better idea). I'm not sure if that will fix the numerical issues on the GPU test though?

[Balandat]

Thanks!