Use SQM with diffserv4 profile. Put dscp CS1 on packets to or from this ipset. You will need to set up sqm on egress of LAN side since iptables runs after ingress queue. See this recent thread for a veth based method Ultimate SQM settings: Layer_cake + DSCP marks