Hello FelixTheAnter;
The reason for adding clay to sand is to stick the sand particles together. Perlite isn't sticky, so I suspect it will collapse just as readily as sand. As ANTdrew mentioned, you may have clay or clay loam soil in your area that you can mix with the sand. I would dry the clay as much as possible, break it up as much as possible, (almost to powder) and then mix it with the sand. Put it in the formicarium when dry, then wet it and the clay should help stick the sand together. (I was writing this as you posted the information about your sandy soil. It's unfortunate that you can't easily access clay loam soil, but maybe this information will help someone else with a similar problem.)
Once I accidently used the sand that is sold to put between the cracks of paving stone to make a formicarium. It has an added adhesive that sticks the sand particles together when it is wetted and then dried. I was worried that the ants wouldn't be able to dig through it, but I was pleased to see that they were able make tunnels and chambers with no problem. The adhesive isn't that strong, but the remaining sand seems to hold together well enough that there have been no collapse problems. Unfortunately, this sand is usually sold in fairly large bags, but any landscaping contractor will have smaller amounts.
RPT
My father always said I had ants in my pants.