What are the differences between policy memory and policy performance?
Obviously the former saves memory, but how much? How severe is the tradeoff?
atomic updates pump whole set to userspace, does update there and pumps set back to kernel.
I don't know the exact answer to this question, but generally the memory impact will be significant when memory used is in the same ballpark as system's memory capacity. For example, if you have 256MB of RAM and your set consumes more than e.g. 100MB then the policy will be significant. If you have 2GB of memory then the set will need to be e.g. 1GB for the policy to make a significant difference memory-wise.
As to performance, it's anyone's guess, and this should be very dependent on the set contents. Whether this will have a practical implication depends on how often that set is hit, and the CPU of the machine.
Bottom line, for something like data server which sees a lot of traffic, go with the performance policy and add RAM as needed. For something like a home router, go with the memory policy, unless the set is much smaller than the memory capacity.
The critical impact will be adding one record to million record set - it will be pumped (malloc) to userspace, adjusted, and pumped a second copy back to kernel.
Essentially for small scale it works at all the memory layout is indifferent.