nqnoc this is pretty fine. I have servers in Asia and in Germany, there is a latency from up to 170 ms. The panel works still pretty fine.
You feel the latency at phpMyAdmin, there you will like to have better a phpMyAdmin at the server available, because it gets painful slow. Filemanager still works fine. Also the Webmail works still good.
But with 10-40ms you will not even reallize the latency. The speed of the server will makes a bigger difference.