https://openwrt.org/docs/guide-user/network/wifi/guestwifi/configuration_webinterface
https://openwrt.org/docs/guide-user/network/wifi/guestwifi/configuration_command_line_interface
Guest not quest - sorry typo
https://openwrt.org/docs/guide-user/services/downloading_and_filesharing/transmission
option bind_address_ipv4 '0.0.0.0'
option bind_address_ipv6 '::'
Replace these with your guest private address.
Then used the (now deprecated) vpn-policy-routing (but you can use pbr) for source address of the guest network.
EDIT:
or if the setup above is too complex for your needs you can just create a secondary lan interface see Correct way to have 2 IP addresses on a given interface - #2 by golialive
and setup transmission to use it (link above) and pbr to route it over the vpn