Nonadsorbing polymers are widely used as thickening agents for colloids. A quantitative description of the structure and dynamics of such colloid-polymer mixtures is crucial to reveal the mechanisms accounting for the desired mechanical properties. We use confocal microscopy to study colloids with three types of commonly used polymers with different architectures: linear, subgranular cross-linked, and branched microgels. All three thickeners give rise to heterogeneous colloidal dynamics, characterized by non-Gaussian displacement distributions. However, while the ensemble-averaged particle dynamics in these materials are very similar, the underlying individual particle dynamics are not. Linear polymers give rise to depletion attraction and the formation of colloidal gels, in which the majority of particles are immobilized, while a few weakly bound particles have much higher mobility. By contrast, the branched and cross-linked polymers thicken the continuous phase of the colloid, squeezing the particles into dense pockets, where the mobility is reduced and requires more cooperative rearrangements.