I haven't seen any evidence of swapping. Just tons of CPU usage and long run times from nft commands when run in the presence of large sets (such as those created by banIP).
Again, note that my problems occur after the banIP sets have been created and populated, not while they are being populated.