He's not smarter than a Pillar Men, Joseph can't learn a launguage by hearing a few words, understanding tech in seconds despite having never seen anything like it, have essentially near perfect memory and recollection, deduce traps and things in seconds, etc.
Joseph outsmarted them in things that he excels at, but in general he's dumb as **** in some cases. That's mitgated by Part 3 and he becomes well versed on many things but he's not on the level of the Pillar Men, not even close although he's on par and capable of outsmarting in very specific categories like trickery and strategy. He's an expert strategist and opportunist, you could say he's extraordinary in that category but in the rest he's just gifted or above average.