So I got curious as the topic peaked my interest as well. Though I think I'll have to ask again about whether to dip the outworld in there too...
I was told:
1. Hydrogen Peroxide as the preferred method. Just dip it in and leave it for several hours and use a brush or something and clean.
2. Second method is from what he heard other people use: Diluted vinegar, gs5248 mentioned it above.
In both cases, let it dry and air out for a long time and you can use a test ant(s) to see its reaction if unsure...before moving the whole colony in.
Please verify, but parroting back to you what I was told when asked. The formicarium seems to be durable and can handle some clean and reuse.