The purpose of any sealant is to form a waterproof barrier, and stay there for 5-15 years depending on the product, manufacturer's specifications and type of application. Let's take for instance asphalt sealant - this prevents water from leaking into the main body of asphalt and compromising the integrity of the bonding between the aggregate and cement. The major issue is not one or two rainstorms - it is the constant wear and tear of traffic and water exposure over the course of many years that warrant the use of a sealant to increase the longevity of the road.
Now your concern of "leeching chemicals" has two points. One is that the sealant will penetrate your grout and cause problems for ants: There needs to be some kind of bonding between the sealant and grout so yes, it's going to penetrate a bit to allow for a strong mechanical bond because that's what it is designed to do. However, once it cures/hardens it will absolutely stop moving around and it will definitely not infiltrate the grout like Venom from Spider Man. This is no problem for your ants because there is no way the sealant is going to penetrate all the way to the nesting area. Something like deep penetrating epoxy resin will, but not a sealant that is designed to seal only the exterior of a surface.
The next point is that some of these products have low amounts of VOCs in them - volatile organic compounds. Don't apply these products when your ants will be exposed to the fumes. Give adequate time for the product to cure and store in a well ventilated area that's not too cold. A lot of mythology and pop culture surround the topic of VOCs - many people just don't believe that a toxic product can transform into something non-toxic. It can and it will. It'll be safe once cured.
Given what you've told us about your formicarium, I'm going to have to assume a couple things about it and the Rustoleum Neverwet sealant seems like the wrong choice of product for this application. I can elaborate on this if you want. Why don't you just glue something like glass to the bottom and sides of your nest? Or better yet, create a box you can slip the grout mould in and out of. You already have something similar glued to the top of the nest, right? You have plenty of options and it seems like you haven't poured the main piece of your nest yet. Get a tupperware container and pour your grout in that, then glue a glass or acrylic sheet on top. Glass cutters are $2 and easy to use. Easy waterproofing.
If you don't want to bother with any of that, use regular sealants or putty. Cheaper than the Neverwet and does the same thing. You're not looking for a hydrophobic barrier, you're looking for a sealant.
From what I've seen on the forums, most people do not have a standalone grout nest. They combine it with things like an acrylic box, sanded grout on the bottom, a wooden frame(??), etc. Easiest/cheapest way is to go to a dollar store, Daiso, GoodWill or other cheap places to buy containers and to pour your grout in that. Or use something from your house. If you want it to look nice, anything is possible, but you gotta open your wallet a bit for those luxuries.