Advances in sensor miniaturization are increasing the global popularity of unmanned aerial vehicle (UAV)-based remote sensing applications in many domains of agriculture. Fruit orchards (the source of the fruit industry chain) require site-specific or even individual-tree-specific management throughout the growing season—from flowering, fruitlet development, ripening, and harvest—to tree dormancy. The recent increase in research on deploying UAV in orchard management has yielded new insights but challenges relating to determining the optimal approach (e.g., image-processing methods) are hampering widespread adoption, largely because there is no standard workflow for the application of UAVs in orchard management. This paper provides a comprehensive literature review focused on UAV-based orchard management: the survey includes achievements to date and shortcomings to be addressed. Sensing system architecture focusing on UAVs and sensors is summarized. Then up-to-date applications supported by UAVs in orchard management are described, focusing on the diversity of data-processing techniques, including monitoring efficiency and accuracy. With the goal of identifying the gaps and examining the opportunities for UAV-based orchard management, this study also discusses the performance of emerging technologies and compare similar research providing technical and comprehensive support for the further exploitation of UAVs and a revolution in orchard management.