Thanos defeated himself. He knew he was not worthy to possess such power, he even told Warlock so. It's what happens every time he gets anything of enormous power. 
 
Also, Thanos doesn't have the gems. Pet Avengers isn't in mainstream continuity. Mostly because Thanos is dead. I only know of him having them during the Infinity Gauntlet and having the reality gem while the Infinity Watch still existed. He only ever used it once.