Manual weeding is the predominant weed control practice and the most labor-consuming activity in smallholder, rainfed rice systems in sub-Saharan Africa. This study investigates the technical inefficiency of weeding labor, other labor, and overall inputs, and identifies sources of technical inefficiency of weeding labor in the context of parasitic weed infestation. The analysis applies a two-stage approach. First, a directional input distance function DEA approach was used to compute input-specific technical inefficiencies. Second, sources of technical inefficiency of weeding labor were identified using a truncated bootstrap regression. Data from 406 randomly selected smallholder farmers from Benin (n = 215) and Côte d'Ivoire (n = 191) were used. The technical inefficiency of weeding labor was high in both countries (58% in Côte d'Ivoire and 69% in Benin). This implies that a substantial fraction of weeding labor could be saved without reducing rice productivity or increasing the use of other inputs. A decrease in the technical inefficiency of weeding labor with an increase in production scale was observed. In addition, weeding regime and education level were each associated to significant changes in the technical inefficiency of weeding labor.