Unfertilized buffer strips (BS) generally improve surface water quality. High buffer strip effectiveness (BSE) has been reported for sloping shallow aquifers, but experimental data for plain landscapes with deeply permeable soils is lacking. We tested a novel method to determine BSE on a 20-m-deep, permeable sandy soil. Discharge from soil to ditch was temporarily collected in an in-stream reservoir to measure its quantity and quality, both for a BS and a reference (REF) treatment. Treatments were replicated once for the first, and three times for the next three leaching seasons. No significant BSE was obtained for nitrogen and phosphorus species in the reservoirs. Additionally, water samples were taken from the upper groundwater below the treatments. The effect of BS for nitrate was much bigger in upper groundwater than in the reservoirs that also collected groundwater from greater depths that were not influenced by the treatments. We conclude that measuring changes in upper groundwater to assess BSE is only valid under specific hydrogeological conditions. We propose an alternative experimental set-up for future research, including extra measurements before installing the BS and REF treatments to deal with spatial and temporal variability. The use of such data as covariates will increase the power of statistical tests by decreasing between-reservoir variability.