Family farming in the tropics suffers from low crop productivity mainly due to a combination of poor soil fertility, low investment capacity, and a variable climate. The Lake Alaotra region of Madagascar is no exception and rainfed production is particularly hard hit. To evaluate the agronomic benefits of conservation agriculture (CA) in a region of erratic rainfall, we analysed four years of yield, management and climatic data from 3803 upland rice fields cultivated by farmers and monitored by researchers. Fields located on rainfed lowlands and hillsides were cultivated with sole rice using conventional tillage (Cv) or rice sown with no-tillage on dead organic mulch and rotated with other cereal/legume combinations (CA) from 2006 to 2011. A first global comparison across seasons, locations and years of adoption showed significantly higher average yields under CA, with no change in variance (on lowland: 2.6 ± 0.9 t ha–1 Cv, 2.8 ± 0.9 t ha–1 CA; on hillside: 2.1 ± 0.8 t ha–1 Cv, 2.4 ± 0.8 t ha–1 CA). Grouping fields according to the number of years under CA (first to fourth) revealed that CA gradually increased average yields and reduced the coefficient of variation in the short and mid-term (on lowland: +0.2 t ha–1 and –6% coefficient of variation; on hillside: +0.7 t ha–1 and –13% coefficient of variation, over four to six years of successive CA cropping). The average yield increase under CA was not associated with an increase in mineral fertiliser use, as farmers used the same amounts of fertilisers (or none) under Cv and CA. The comparison Cv versus CA also highlighted a major benefit of CA regarding climate: it widened the window of possible sowing dates. A classification and regression tree analysis of the entire dataset revealed that rice yield was more affected by agro-environmental factors than management factors (fertilisation, Cv or CA), and extreme climate variability such as the severe drought of 2007–2008 could not be offset by CA. The hypothesis of yield penalties during the first years of implementation of CA cannot be verified with the evidence presented in this study.