I use organic soil from home depot. The brand is "earth ____" i forgot the actual name. Black bag with an earth on it.
This soil i've used in a couple of my builds and it hold very very well. My panera cup farm for example uses that soil. It has been dropped, kicked, as well as the sides being compressed and the tunnels NEVER collapsed. It's also not sterilized, so there's added beneficial nutrients and fungus that help. It also never gunked the walls, so i was always able to see my ants.
Not saying that soil directly from the area isn't the best, but it could have other issues. Spent nutrients, mites or bugs that were closing in to the nest (if that's possible) or even the chance of old fertilizer or pesticides that could be in the dirt. I've never had any of these issues with soil from the area, but i also don't use that soil after the colony is moved. Just throwing that out there.