We have a similar problem where we just plonked cheap nameserver nodes in UK, USA and India, resulting in customers getting random ns in non-local geographies which can have higher than ideal latency, it also causes non-caching web performance monitoring (ttfb including dns and handshakes) to jump all over the place (hitting the non-local DNS can double the overall page load time/latency).
I was thinking anycast might be something we need to explore to route the traffic locally (rather than GeoIP) and give us greater availability if under attack, I was pleased to see https://community.enhance.com/d/792-use-of-external-anycast-dns-service suggested and accepted as a potential solution to this.