Food digestion may be regarded as a physiological interface between food and health. During digestion, the food matrix is broken down and the component nutrients and bioactive compounds are absorbed through a synergy of mechanical, chemical, and biochemical processes. The food matrix modulates the extent and kinetics to which nutrients and bioactive compounds make themselves available for absorption, hence regulating their concentration profile in the blood and their utilization in peripheral tissues. In this review, we discuss the structural and compositional aspects of food that modulate macronutrient digestibility in each step of digestion. We also discuss in silico modeling approaches to describe the effect of the food matrix on macronutrient digestion. The detailed knowledge of how the food matrix is digested can provide a mechanistic basis to elucidate the complex effect of food on human health and design food with improved functionality.