Key points Just as a portrait painting can come from a collection of coarse and fine details, natural vision can be decomposed into coarse and fine components. Previous studies have shown that the early visual areas in the brain represent these components in a map‐like fashion. Other studies have shown that these same visual areas can be sensitive to how coarse and fine features line up in space. We found that the brain actually jointly represents both the scale of the feature (fine, medium, or coarse) and the alignment of these features in space. The results suggest that the visual cortex has an optimized representation particularly for the alignment of fine details, which are crucial in understanding the visual scene. Abstract Complex natural scenes can be decomposed into their oriented spatial frequency (SF) and phase relationships, both of which are represented locally at the earliest stages of cortical visual processing. The SF preference map in the human cortex, obtained using synthetic stimuli, is orderly and correlates strongly with eccentricity. In addition, early visual areas show sensitivity to the phase information that describes the relationship between SFs and thereby dictates the structure of the image. Taken together, two possibilities arise for the joint representation of SF and phase: either the entirety of the cortical SF map is uniformly sensitive to phase, or a particular set of SFs is selectively phase sensitive – for example, greater phase sensitivity for higher SFs that define fine‐scale edges in a complex scene. To test between these two possibilities, we constructed a novel continuous natural scene video whereby phase information was maintained in one SF band but scrambled elsewhere. By shifting the central frequency of the phase‐aligned band in time, we mapped the phase‐sensitive SF preference of the visual cortex. Using functional magnetic resonance imaging, we found that phase sensitivity in early visual areas is biased toward higher SFs. Compared to a SF map of the same scene obtained using linear‐filtered stimuli, a much larger patch of areas V1 and V2 is sensitive to the phase alignment of higher SFs. The results of early areas cannot be explained by attention. Our results suggest non‐uniform sensitivity to phase alignment in population‐level SF representations, with phase alignment being particularly important for fine‐scale edge representations of natural scenes.