Recent studies using ERPs in face recognition revealed that face processing starts around 100 ms after stimulus onset, 70 ms earlier than suggested before. While the neural sources of the N 170 component have repeatedly been found to be localized in the gyrus fusiformis and the inferior occipital cortex, sources have not yet been investigated for the P100 component during face processing. Therefore, we measured the ERPs elicited by faces and control stimuli in 72 subjects in order to localize the neural sources of both the P100 and the N 170 component. We observed significantly higher P100 and N 170 amplitudes to faces compared to control stimuli. LORETA source localization revealed significantly higher brain activity in the left and right gyrus fusiformis for the N 170 component, with additional regions of increased brain activation in a parieto-temporal-occipital network. For the P100, faces activated the left and right gyrus fusiformis significantly stronger than control stimuli. This study reveals that the first step of face processing (about 100 ms after stimulus presentation) is localized in the gyrus fusiformis. The second step of face processing around 170 ms involves the gyrus fusiformis, with additional activation in a more distributed network, including the occipital cortex.