Somatic cell count (SCC) is one of the most important and widely used mastitis diagnostics. For detecting (sub)clinical mastitis, online SCC related measurements are more and more used in automatic milking systems (AMS). Sensors such as an automated online California Mastitis Test (O-CMT) allow for high frequency screening of high SCC cows within a herd, which makes it potentially powerful to identify episodes of mastitis. However, the performance of O-CMT measurements, as compared to SCC determined in the laboratory (L-SCC), has only scarcely been described. The aims of this study were (1) to assess the agreement between the O-CMT measurement averaged over different time windows and the corresponding L-SCC measurements; (2) to determine the optimal time window for averaging O-CMT as compared to L-SCC; (3) to explore the added value of time-series of frequent O-CMT measurements in individual cow udder health monitoring compared to L-SCC measurements. Data were collected from 50 farms in 6 different countries that were equipped with AMS using O-CMT measurements and also performed regular L-SCC testing. We found that the overall concordance correlation coefficient (CCC) between O-CMT and L-SCC was 0.53 but differed substantially between farms. The CCC between O-CMT and L-SCC improved when averaging O-CMT over multiple milkings, with an optimal time-window of 24 h. Exploration of time series of daily O-CMT recordings show that this is an effective screening tool to find episodes of high SCC. Altogether, we conclude that although O-CMT agrees moderately with L-SCC, because of its high measurement frequency, it is a promising on-farm tool for udder health monitoring.