Which file to use for flashing is identical to stock openwrt, this is described on https://openwrt.org/docs/guide-user/installation/generic.flashing. The sysupgrade file is very likely not what you want.
The R7800 can be put into tftp (unbrick mode) very easily see Netgear R7800 exploration (IPQ8065, QCA9984) - #5 by hnyman
How to flash openwrt with tfpt can be found at https://openwrt.org/docs/guide-user/installation/generic.flashing.tftp