A typical criticism of Gaussian processes is their unfavourable scaling in both compute and memory requirements. Sparse variational Gaussian processes based on inducing variables are commonly used to scale Gaussian processes to large dataset sizes; their inherent compute and memory requirements are dominated by the number of inducing variables used. However, in practise sparse GPs are still limited by the number of datapoints and the number of inducing points one can use to perform matrix operations, making it again challenging to model large complex datasets. In this work we propose a new class of inter-domain variational GP, constructed by projecting the GP onto a set of compactly supported B-Spline basis functions. The key benefit of our approach is that the compact support of the B-Spline basis admits the use of sparse linear algebra to significantly speed up matrix operations and drastically reduce the memory footprint.