The Casimir-Polder force on a single atom in the presence of microscopic magneto-electric bodies is studied within the framework of macroscopic quantum electrodynamics. In the first part, time-independent perturbation theory is used show that the static Casimir-Polder force on a ground-state atom in the presence of an arbitrary arrangement of bodies can be expressed in terms of the atom's polarisability and the scattering part of the Green tensor characterising the bodies. This result is used to verify the origin of the force and to study various bodies of simple shapes. In the second part, a genuinely dynamical approach is developed by starting from the operator Lorentz force. It is found that the dynamics of the force on initially excited atoms is governed by irreversible spontaneous decay in the case of weak atom-field coupling, while Rabi oscillations may arise for strong atom-field coupling.