Insight in the phosphorus (P) flows and P balances in the food chain is largely unknown at county scale in China, being the most appropriate spatial unit for nutrient management advice. Here, we examined changes in P flows in the food chain in a typical agricultural county (Quzhou) during 1980–2017, using substance flow analyses. Our results show that external P inputs to the county by feed import and fertilizer were 7 times greater in 2017 than in 1980, resulting in a 7-fold increase in P losses to the environment in the last 3 decades, with the biggest source being animal production. Phosphorus use efficiency decreased from 51% to 30% in crop production (PUEc) and from 32% to 11% in the whole food chain (PUEf), but increased from 4% to 7% in animal production (PUEa). A strong reduction in P inputs and thus increase in PUE can be achieved by balanced P fertilization, which is appropriate for Quzhou considering a current average adequate soil P status. Fertilizer P use can be reduced from 7276 tons yr−1 to 1765 tons yr−1 to equal P removal by crops. This change would increase P use efficiency for crops from 30% to 86% but it has a negligible effect on P losses to landfills and water bodies. Increasing the recycling of manure P from the current 43%–95% would reduce fertilizer P use by 17% and reduce P losses by 47%. A combination of reduced fertilizer P use and increased recycling of manure P would save fertilizer P by 93%, reduce P accumulation by 100% and P loss by 49%. The results indicate that increasing manure-recycling and decreasing fertilizer-application are key to achieving sustainable P use in the food chain, which can be achieved through coupling crop-livestock systems and crop-based nutrient management.