Many visual tasks can be decomposed into a sequence of simpler subtasks. Ullman suggested that such subtasks are carried out by elemental operations that are implemented by specialized processes in the visual brain [Ullman, S. (1984). Visual routines. Cognition (18), 97-159]. According to this hypothesis, there are a limited number of elemental operations that, since they can be applied sequentially, may nevertheless give rise to a large number of visual routines. Examples of such elemental operations are visual search, texture segregation and contour grouping. Here we attempt to delineate how such elemental operations are implemented in the visual brain. When an image appears, feedforward processing rapidly leads to an activity pattern that is distributed across many visual areas. Thereafter, elemental operations come into play, and these are implemented by the modulation of firing rates. Firing rate modulations effectuate grouping of neural responses into coherent object representations. Moreover, they permit transfer of information from one operator to the next, which allows flexibility in the sequencing of operations. We discuss how the elemental operations provide a tool to relate cortical physiology to psychophysics, and suggest a reclassification of pre-attentive and attentive processes.