That isn't really true.
$ LANG= ls -gG
total 8072
-rw-r--r-- 1 4128772 Sep 8 00:35 openwrt-19.07.4-ar71xx-generic-tl-wdr4300-v1-squashfs-sysupgrade.bin
-rw-r--r-- 1 4129069 Sep 8 20:40 openwrt-19.07.4-ath79-generic-tplink_tl-wdr4300-v1-squashfs-sysupgrade.bin
so, in total, the ath79 image is 297 byte larger than the ar71xx one.
Looking at the kernel itself:
$ LANG= binwalk openwrt-19.07.4-ar71xx-generic-tl-wdr4300-v1-squashfs-sysupgrade.bin
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
512 0x200 LZMA compressed data, properties: 0x6D, dictionary size: 8388608 bytes, uncompressed size: 5059140 bytes
1588012 0x183B2C Squashfs filesystem, little endian, version 4.0, compression:xz, size: 2526322 bytes, 1156 inodes, blocksize: 262144 bytes, created: 2020-09-06 16:19:39
vs.
$ LANG= binwalk openwrt-19.07.4-ath79-generic-tplink_tl-wdr4300-v1-squashfs-sysupgrade.bin
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
512 0x200 LZMA compressed data, properties: 0x6D, dictionary size: 8388608 bytes, uncompressed size: 5019885 bytes
1597836 0x18618C Squashfs filesystem, little endian, version 4.0, compression:xz, size: 2504270 bytes, 1149 inodes, blocksize: 262144 bytes, created: 2020-09-06 16:19:39
ergo, we have 1'587'500 bytes (ar71xx) vs. 1'597'324 bytes (ath79), which results in the ath79 kernel being 9'824 bytes larger than the ar71xx one, or in other words a delta of about 9 KB. While, obviously, every byte counts on a 4/32 device, I wouldn't call a delta of 9 KB (kernel) or 297 bytes (total firmware size for 19.07.4) "quite a bit larger".
--
Disclaimer: the figures above include tiny variances due to different padding, relevant but not really considerable; the problem is 4/32, not ar71xx vs ath79.