A number of policies proposed to increase soil organic matter (SOM) content in agricultural land as a carbon sink and to enhance soil fertility. Relations between SOM content and crop yields however remain uncertain. In a recent farm survey across six European countries, farmers reported both their crop yields and their SOM content. For four widely grown crops (wheat, grain maize, sugar beet and potato), correlations were explored between reported crop yields and SOM content (N = 1264). To explain observed variability, climate, soil texture, slope, tillage intensity, fertilisation and irrigation were added as co-variables in a linear regression model. No consistent correlations were observed for any of the crop types. For wheat, a significant positive correlation (p < 0.05) was observed between SOM and crop yields in the Continental climate, with yields being on average 263 ± 4 (95% CI) kg ha−1 higher on soils with one percentage point more SOM. In the Atlantic climate, a significant negative correlation was observed for wheat, with yields being on average 75 ± 2 (95%CI) kg ha−1 lower on soils with one percentage point more SOM (p < 0.05). For sugar beet, a significant positive correlation (p < 0.05) between SOM and crop yields was suggested for all climate zones, but this depended on a number of relatively low yield observations. For potatoes and maize, no significant correlations were observed between SOM content and crop yields. These findings indicate the need for a diversified strategy across soil types, crops and climates when seeking farmers’ support to increase SOM.