I do it when DH is OOT and I make sure I wash my hands really well. Right now I am just enjoying having DH do it.
See, that's the thing - if I wait for DH to do it, the one finicky cat gets upset and pisses on my bathmat. Never the one in DH's bathroom, mind you.
I grew up on a farm that always had about 20 feral cats around, so I'm sure I've already been exposed.
I avoid it when possible DH does it. However, I had to do it once and used gloves, a mask and took a shower after.
Do you have an indoor cat?
Here's the deal. If you spent time around feral cats as a kid, chances are good you're immune. If you now have an indoor cat, chances are good that s/he won't get it because they aren't exposed to it unless they eat teensy outdoor creatures.
Since my cat isn't in the habit of eating teensy outdoor creatures AND I, too, spent time with feral cats as a child, I don't worry. I change the litter (no masks or gloves) and wash up afterward. Unless you're smearing outdoor cat feces on your hands and having a lick, you're probably safe.