The most important element for macro is the lens. Dedicated macro lenses have better sharpness than regular lenses, but most importantly: at least a 1:1 magnification ratio.
The magnification ratio means that if you place your 10mm ant at the minimum distance the lens can focus at, the projected image of the ant on the sensor will also have a physical size of 10mm. If the lens and the camera are, for example, in the micro four thirds system, that means a sensor size of 18x13.5mm. Since the image of the ant on the sensor will be 10mm big, it'll take up more than half of the width of the 18mm sensor. If you know the pixel resolution of your camera, you can count how many pixels that is, but I would say the pixel count is much less important than a bunch of other factors, which determine sensor quality.
Your ant will be magnified 1:1 only if it's at the "working distance" from the lens (equivalent to being at the "minimum focus distance" from the sensor). If it's closer, you won't be able to focus on it. If it's further away, it'll be smaller than 1:1, but sometimes that's what you want, of course. Macro photographers don't discuss the working distance of a lens that much, since with inanimate objects or sometimes even insects you can often get so close, that the glass is touching the subject. The main issue for them is then getting in there with enough lighting and not spooking the subject. For us the issue is the formicarium walls getting in the way. The
Olympus 60mm macro lens has a working distance of about 108mm, while the
Panasonic 30mm macro lens - about 40mm. The latter will already pose trouble when photographing ants in their nest
and lighting them properly, while taking photos in the outworld will always mean a lower magnification, since you can't get close enough. For this reason longer focal lengths are preferred for convenience.
Another thing to consider is the depth of field. At these focusing distances and apertures, the dof size might be under a millimeter. That means you will not be able to have much of the ant in focus, even if you managed to focus on it. With just a sliver of sharpness cutting it across, it will still look out of focus. To increase the DoF size you can either move away a bit, or increase aperture. We don't like moving away too far, since it decreases magnification. Increasing aperture works well, but you have to light your subject better to compensate for many times less light coming in through the lens. Check out the
DoF calculator to get a feeling of how these settings affect the DoF. You can't just decrease shutter speed to capture more light over time, since your ant is moving, and it will motion blur to oblivion. This is where being able to light the subject comes in handy. Additionally, this is where we can start talking about better/bigger sensors.
Bigger sensors can typically capture more light easily. One reason is that they simply have a larger area to capture the light, but they also have more space for each pixel, so there's extra room for lots of funky technology to avoid noise and artifacts when pixels are packed too tightly. So a better quality and bigger sensor will be able to catch more light from the same scene with an equivalent lens and the result will be a less noisy image with fewer artifacts. So Micro Four Thirds is a standard, which in its spec defines the sensor size, among other things. Sony's APS-C would be another one, with a considerably bigger sensor, but also pricier bodies. "full frame" cameras have even larger sensors, and are super-expensive. Before choosing your standard, make sure there are lenses made for it, that fit your needs. It's usually completely fine to mix and match body and lens manufacturers (that's what the standards are for), but making a lens fit a body from a different standard ranges from "you need a connector" to "impossible".
One more note on DoF: not only the size of the area in-focus matters, but also how quickly the image explodes to blur outside of it. With very short focal length lenses, the background gets blurry slowly enough, that you might be able to show your ant in its habitat. If you can
touch it with the lens, that is, because that's the working distance of short focal length lenses
With long focal length, as soon as you get out of the focus area, everything blurs to oblivion, so you're getting your ant on a background of abstract gradients.
Disclaimer: I don't actually have any good macro shots in my portfolio to prove I know what I'm talking about
I am however in the market for a new camera and lens, and I do work with light and lenses in my day job, so there's that
I hope I at least signaled the things you need to take into account.
