Ssh into your server and run: free -h
Check to see if your server is actually using that ram, or if that's just Linux caching things like normal.
What optimizations have you made on your server for the php, and database confs? And do your sites have caching installed? And did you enable opcode cache and Redis?
Do any sites have errors in error log? One website can easily consume 4gb ram if it has enough web traffic or an error from a plugin or theme conflict.
