The rule with relayd is don't' use it unless there is absolutely no other way. And here since you're going to route everything into a VPN tunnel anyway, you just need a standard routed wifi client connection for the encrypted packets to reach the Internet. Remove relayd and configure OpenWrt as a routed client where the wan is a wifi STA on your home network, then configure the VPN.
How would I go about this? Is using one wireless band as a client and another as an AP good? How would I connect that network and run it through Wireguard?
Thank you for your post! Ignore my reply below I investigated more and figured out what you meant
For the future onlookers. If you're trying to set up your OpenWRT router as a wireless repeater (with or without a WireGuard VPN). You can use this method without relayd, which is better considering conditions where one has a fresh install without internet access to download that package. This method is from the OpenWRT docs:
Go in the Network → Interfaces page, click on edit lan interface,
Set LAN as static IPv4 address as 192.168.x.1 (with x different from the network to which you will connect via Wi-Fi),
Go in the Network → Wi-Fi, click on scan and choose the “network” link and click “Join Network”.
Enter the Wi-Fi password, leave the “name of new network” as “WWAN” and select WWAN (or WAN) firewall zone. Click Save,
Go in the Network → Interfaces page, click on edit wwan interface,
Move to the Firewall tab. Click on Save and Apply.
Go in the Network → Firewall, click edit in wan zone and check WAN and WWAN in “covered networks”, click save and apply,