I see no reason to worry. About the half of the physical RAM is used for buffering and caching. That's the best the kernel can do with unused RAM. When it needs more for programs and data, it will be taken from the buffering/caching pool.
As long as you have a quarter of RAM free, I wouldn't do anything.
When it gets less than 100 MB, then install the zram-swap package.