The soil-water characteristic curve (SWCC) is the most fundamental relationship in unsaturated soil mechanics, relating the amount of water in the soil to the corresponding matric suction. From experimental evidence, it is known that SWCC exhibits hysteresis (i.e. wetting/drying path dependence). Various factors have been proposed as contributors to SWCC hysteresis, including air entrapment, contact angle hysteresis, ink-bottle effect, and change of soil fabric due to swelling and shrinkage, however, the significance of their contribution is debated. From our pore-scale numerical simulations, using the multiphase lattice Boltzmann method, we see that even when controlling for all these factors SWCC hysteresis still occurs, indicating that there is some underlying source that is not accounted for in these factors. We find this underlying source by comparing the liquid/gas phase distributions for simulated wetting and drying experiments of 2D and 3D granular packings. We see that during wetting (i.e. pore filling) many liquid bridges expand simultaneously and join together to fill the pores from the smallest to the largest, allowing menisci with larger radii of curvature (lower matric suction). Whereas, during drying (i.e. pore emptying), only the limited existing gas clusters can expand, which become constrained by the size of the pore openings surrounding them and result in menisci with smaller radii of curvature (higher matric suction).