Our ability to recognize the actions of others is subserved by a complex network of brain areas, including the inferior frontal gyrus (IFG), inferior parietal lobe (IPL) and superior temporal sulcus (STS). An unresolved issue is whether the activity within these regions requires top-down control or whether it arises relatively automatically during passive action observation. Here we used fMRI to determine whether cortical activity associated with action observation is modulated by the strategic allocation of selective attention. Participants observed moving and stationary images of reach-to-grasp hand actions, while they performed an attentionally demanding task at the fovea. We first defined regions-of-interest (ROIs) in the IFG, IPL and STS which responded to the perception of these actions. We then probed these ROIs while participants observed the identical, but now task-irrelevant, actions and instead performed an easy (low attentional load) or difficult (high attentional load) visual discrimination task. Our data indicate that the activity of the left IFG was consistently attenuated under conditions of high attentional load, while the remaining action observation areas remained relatively unaffected by attentional manipulations. The suppression of the left IFG was unique to the observation of hand actions, and did not occur during the observation of non-biological control stimuli, in the form of coherent dot motion. We propose that the left IFG is the site at which descending inhibitory processes affect the processing of observed actions, and that the attentional modulation of this region is responsible for filtering task-irrelevant actions during ongoing behavior.