In the Central Rift Valley (CRV) of Ethiopia, low productive cereal systems and a declining resource base call for options to increase crop productivity and improve resource use efficiency to meet the growing demand of food. We compiled and analysed a large amount of data from farmers’ fields (>10,000) and experimental data across the CRV from 2004–2009 to quantify yield gaps (Y g) between actual (average and best performing farmers) and experimental (water-limited potential (Y w)) yields of maize and wheat in homogenous farming zones (HFZs). Resource use efficiencies (nutrients and water) of maize and wheat were also analysed to assess spatial variation and scope for improvements. The average (2004–2009) yield gap of maize and wheat in the CRV ranged between 4.2 t ha−1 and 9.2 t ha−1, and 2.5 t ha−1 and 4.7 t ha−1, respectively, across farming zones. The yield gap was lowest in the Central lowlands, where Y w was also lowest, i.e. 6.5 t ha−1 for maize and 4.4 t ha−1 for wheat, compared with Y w in the Eastern highlands (11 t ha−1 for maize and 6.7 t ha−1 for wheat) and Western highlands (10.8 t ha−1 for maize and 5.7 t ha−1 for wheat). The actual nitrogen (N) and phosphorus (P) application in farmers’ fields was low, as about 46% of maize and 27% of wheat fields did not receive fertilizers, while the average applied mineral fertilizer rates across all farmers (2.6–16.5 kg N ha−1 and 2.2–17.3 kg P ha−1 across HFZs and crops) were far below the recommended rate. On average, the best performing farmers applied 8–20 kg N ha−1 and 5–21 kg P ha−1 ranging across HFZs and crops. Increasing N application to recommended rates had only a small effect on narrowing the yield gap under current farmers’ management. Therefore, the yield gap closure strongly depends on improving other aspects of crop management while paying attention to the interaction with nutrient management. Since rain water use efficiency (seasonal rainfall) of water-limited yields was 12–17.3 kg mm−1 for maize and 7.4–10.6 kg mm−1 for wheat and much higher than that of actual yields (2.7–4.3 kg mm−1 for maize and 2.3–3.5 kg mm−1 for wheat), improving the input use and crop management can increase water use efficiency. A large set of experimental and survey data enabled us to gain insight in the spatial and temporal variation in yield gaps and input rates and in differences between average and the best performing farmers.