My understanding is that processes generally don't give back memory until it's actually needed, so looking at memory usage in a low-usage state is often misleading. The real test is whether memory is quickly available when requested.
>My understanding is that processes generally don't give back memory until it's actually needed
No, programs that release memory when there's high memory pressure is the exception, not the rule. You'd have to specifically go out of your way to code that in. A basic program that allocates some memory, and then frees it is going to cause the system memory usage to drop immediately.