I had a bad breather on a Sienna AWD, dealer wanted to replace axle seals for more than $1k. I'm thinking that if your breather is not opening at the right internal pressure (caused by heat and expansion), it might be spilling out oil if it only opens at an "elevated" pressure. If that is the case, all you need is to get a new one.
Not sure if driving without it to test it will cause even more oil to leak out.