In particular, one of the issues that often gets a lot of airtime in discussions of new voting systems is how to prevent vote-buying (or coercion) attacks, including those where the voter cooperates with the attacker. But in vote-by-mail scenarios, it's pretty easy for the attacker to have you give them a copy of the ballot and then fill it in and mail it themselves (especially if voters aren't allowed to change their vote after the ballot is submitted, but I doubt that even if you can revise your ballot most people will bother to in order to cheat.)
Oh, i should mention: if you plan to work the polls, they recommend you vote early or by mail because you probably will be assigned somewhere else than your own precinct and so won't have time to vote yourself on election day.
I agree--the amount of whining done about the security of voting machines, voter identification, voter registration and so on seems ludicrous given the deafening silence surrounding the spread of flagrantly insecure mail-in balloting.
There has been a significant increase in electoral fraud in the UK since postal voting became generally encouraged. It's supposed to increase participation but it doesn't seem effective enough to compensate for the downsides.