The anatomical connectivity of the human cerebral cortex resembles a "small-world" architecture, which is characterized by the coexistence of structurally segregated and integrative connectivity patterns. However, organizational differences in networks among individuals remain largely unknown. Here, we utilize diffusion tensor imaging tractography and graph-theoretical approaches to investigate the effects of sex and brain size on the topological organization of human cortical anatomical network. Weighted cortical networks were constructed from 72 young healthy participants by measuring anatomical connection densities between 78 cortical regions. As expected, all participants showed a small-world topology (high local clustering and short paths between nodes), which suggests a highly efficient topological organization. Furthermore, we found that females had greater local efficiencies than males. Moreover, smaller brains showed higher local efficiency in females but not in males, suggesting an interaction between sex and brain size. Specifically, we show that several brain regions (e.g., the precuneus, precentral gyrus, and lingual gyrus) had significant associations between nodal centrality and sex or brain size. Our findings suggest that anatomical network organization in the human brain is associated with sex and brain size and provide insights into the understanding of the structural substrates that underlie individual differences in behavior and cognition.