[Objective] To elucidate the influence mechanisms of the physicochemical properties of soil aggregates with different particle sizes on cadmium (Cd) speciation, this study investigated the distribution characteristics of aggregates in typical paddy soils and the effects of different aggregate size fractions on Cd environmental behavior. [Methods] Two Cd-contaminated paddy soils were selected, including granitic sandy soil (developed from granite) and eel clayey soil (developed from shale and slate). Soil aggregates were separated into four size fractions using the wet-sieving method: macroaggregates (2–8 mm), intermediate aggregates (0.25–2 mm), microaggregates (0.053–0.25 mm), and silt-clay particles (<0.053 mm). The adsorption effects of different-sized aggregates on Cd were analyzed after removing organic matter and free iron oxides. [Results] Both soils were dominated by macroaggregates. The aggregate structure of the eel clayey soil was more stable than that of the granitic sandy soil, with higher mean weight diameter (MWD) and geometric mean diameter (GMD). The granitic sandy soil had a higher proportion of fine aggregates, with looser structure and lower stability, mainly due to the lower contents of organic matter and free iron oxides in its parent material. As aggregate size decreased, the contents of organic matter, free iron oxides, and cation exchange capacity (CEC) significantly increased. Cd adsorption capacity progressively enhanced with reduced aggregate size, reaching maximum adsorption in silt-clay particles. The removal of organic matter significantly reduced the Cd adsorption capacity, whereas the removal of free iron oxides had a relatively smaller effect. [Conclusion] The compositional and structural stability differences in aggregates from distinct soil types significantly affect Cd adsorption and speciation patterns. These findings provide scientific basis for deciphering the evolution of heavy metal pollution in paddy fields and formulating remediation strategies.