In my opinion, I think it's Deathstroke. I know they're only faced each other a handful of times, but each time they fight, Batman can barely hold his own against him. Not to mention that Deathstroke has beaten Batman more than vice versa.
Which villain do you think gives Batman the toughest time?
It's gotta be ra's hands down.
The reason being, almost every other villain has been quickly defeated at some time or another. Even Joker was once defeated within four pages of a comic. With Ra's, it's always a long drawn out story and all his followers willingly kill themselves for him.
I'll agree with the majority that Ras al Ghul gives Batman the toughest time. My reason being, as Red_Jack said above, that Batman can never seem to defeat him with ease. Whenever Bruce goes up against Ras it is a intricate and drawn out battle for Bruce which is fought in a variety of ways often due to Ras' great many sources.
That being said, while I think that Ras gives Batman the hardest time, I think that Batman would rather go up against him 100 times than he would go up against the Joker once. Though more easily taken down, I feel like the Joker is the villain that makes Batman think to himself, "Why did it have to be him? Why couldn't it be anybody else?"
What about the Court of Owls? Sure they might be fairly new but from what we've seen they've plagued Bruce since he was a child. It took him years just to even prove they existed and once they captured him put him under intense psychologicl torture. During the night of owls he had to fiht an army of Talons in his own home and it took the entire bat family to help stop them all. Then there's also the fact that they were run by his 'brother' which is definetely had a huge impact on Bruce. And finally in the end he only ended up stopping a small group of the Court so they'll be plaguing him again in the future
This edit will also create new pages on Comic Vine for:Beware, you are proposing to add brand new pages to the wiki along with your edits. Make sure this is what you intended. This will likely increase the time it takes for your changes to go live.