American men's clothing is surprisingly large

Earlier this year I mentioned being of average height and having a fairly optimal BMI as judged by the medical charts. When I visit the US I typically shop for clothes because they are easier to find and rather cheaper than in Britain; one can bring a reasonable amount back home without extra tax liability. Of considerable surprise to me was that, although casual shirt sizes may commonly range from small all the way up to 3XL or so, American small shirts from multiple brands (including Van Heusen) fitted me fine. I wonder where the men who are underweight or short shop, the children's section? I suppose that even when I was decidedly obese I could easily still wear large but as an unremarkably sized guy I should not be finding myself at an end of the usual sizing scale.