I currently have a Netgear XR 500 that is running OpenWRT. When using SQM my speeds are much lower than I would like (240 down 30 up). Without SQM I am getting about 900 down and 40 up. I would like to revert back to dumaos since bufferbloat was not as bad as openwrt without SQM. The loss of speed is too much for me to keep using it. My question is: Can I just use luci to flash the dumaos firmware image?
No, but using the OEM TFTP recovery is easy.
It is well documented in the R7800 docs. (XR500 is almost identical younger sister).
The TFTP can be used to flash the OEM firmware.
You should also test other qdisc than cake (which I have disliked in R7800). For me the old simple.qos/fq_codel works best in SQM.
Note also that you have highly asymmetrical connection, 900/40 = 22:1 ratio. Your uplink will be the limiting factor also for download protocol traffic.