We don't scale hax by Attack Potency here. Like Akreious said, we usually judge whether or not hax works on the target based on if they possess resistance to such hax. If they have resistance, then hax likely gets resisted. If they don't have resistance, then it would work on them. Though if one character has hax and the other has a hax resistance that could be overpowered, then we judge the potency of their hax and resistance on a case to case basis. We generally don't judge hax based on Attack Potency or Tiers, especially if the character that suffers the short-end of a hax has no resistance to it.
The only exceptions would be if the character with certain hax were to face an enemy with
Abstract Existence,
Higher-Dimensional Existence, or if the enemy would simply require
higher-dimensional power to be able to content with them due to the sheer tier difference.
That is when the character affecting their enemy with a certain hax would be considered NLF, and would need to have proof that their hax can affect entities such as that. Also, including Galactus in your argument about being killed by instant-death magic is quite flawed as Galactus is a character with Abstract Existence, Low-Godly Regenerationn, and Type 8 Immortality (all of which circumvents Death Manipulation, thus Galactus wouldn't even be defeated by instant death abilities like these in the first place unless they can affect concepts). And yes, Ainz was once pitted against Saitama in a thread, and the thread was closed due to Ainz haxstomping Saitama (you should know, since you've been there).
Heck,
this thread is a prime example due to
Yoshikage Kira (a tier 8-C character at best) defeating
Cell (a tier High 4-C character at least) through nothing but hax, where the latter's lack of resistance to the former's hax was one of the main reasons that led to his defeat.
Overall, we don't estimate a character' hax with AP or Tiers unless their enemy happens to exist in a higher dimension than them (The enemy being a 4D being over the character being a 3D being with 3D powers) or is an abstract existence (Type 1 or even possibly Type 2), which would be when the character requires proof/scans of affecting higher existences with their hax (the former requiring higher-dimensional level abilities to affect them, and the latter requiring conceptual-level abilities to affect and/or defeat them).