Because for example your typical downstream TCP flows need to send ACK packets in the reverse direction (the receiver tells the sender what data was received), if the upstream is saturated or badly bloated, these ACK packets will be delayed, which in turn slows down your downloads.
Well, a number of users is quite happy with per-internal IP fairness, see https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details, Making cake sing and dance, on a tight rope without a safety net (aka advanced features) for details.