Your subjective experience as a caffeine addict is not really applicable. Caffeine definitely impairs me. I can't sleep, can't sit still, can't poop, can't focus on anything. Weed, on the other hand, keeps my thoughts focused and doesn't interfere with sleeping or bowel movements. See? Our personal experience is irrelevant to the discussion.
You are not using the same meaning of "addiction". I'm most definitely an addict, of various substances. Can easily put them down for months or years, that's not proof that I'm not an addict. The addiction is the part where I keep coming back even though I can quit.