It seems like you've already narrowed it down to the server that is running the application. What are the metrics for that device - ping, CPU, RAM, etc?
If all of those check out, then the actual application running on the server may be at fault. Without the application specifics it's hard to know where, but applications can be slow for a number of reasons even when all the hardware elements check out fine.