Won’t they be more of a successful colony(s) if you didn’t check on them as much?
Not at all. Queens and young colonies are only really scared of vibrations and light because they aren't used to it. My colonies are very used to it, and I believe my largest Camponotus chromaiodes colony even associates them with food.