Improving fertilizer recommendations for farmers is essential to increase food security in smallholder landscapes. Currently, blanket recommendations are provided across agro-ecological zones, although fertilizer response and nutrient use efficiency by maize crop are spatially variable. We aimed to identify factors that could help to refine fertilizer recommendation by analyzing the variability in fertilizer response (FR) and the agronomic nitrogen use efficiency (N-AE). A literature search for on-farm studies across Kenya and Sub-Sahara Africa (SSA), excluding Kenya, yielded 71 publications. The variability in FR was studied using a meta-analysis whereas key factors that influence FR and N-AE were studied with linear regression models. On average, the FR was 2, but it varied considerably from 1 to 28.5 (excluding outliers). In SSA, 18% of the plots were non-responsive plots with an FR < 1. The main factors affecting N-AE for Kenya were P-Olsen, silt content, soil pH, clay and rainfall, whereas only soil pH, exchangeable K and texture were important for SSA. However, our study indicates that available data on soil, climate and management factors could explain only a small part (< 33%) of the variation in FR and N-AE. Soil pH, P-Olsen, silt content, and rainfall had significant but low levels of power in explaining variation in FR and N-AE. Our findings indicate that strategies to refine fertilizer recommendation should include information on soil types and soil properties.