Batman is a great fighter, but against someone who can learn and counter those fighting skills in a matter of minutes he cannot win. Batman is a fantastic character, but I have to agree with @johnny_blaze: Thanks to the Nolan films giving him a new lease of life, Batman has become this over-estimated character who people assume would be able to defeat anyone for the simple reason that "he's Batman." But that's not true.
And that's not necessarily a bad thing either. It's good for a character to have weaknesses. I'm not saying Batman isn't great, but I think people need to calm down and realise that he - like everyone else - has a few flaws. Taskmaster would win with a library of weapons and fighting styles that easily rival that of Bruce. My vote is for Taskmaster.