Configure VPN for transmission on the router
Guest not quest - sorry typo

option bind_address_ipv4 ''
        option bind_address_ipv6 '::'

Replace these with your guest private address.

Then used the (now deprecated) vpn-policy-routing (but you can use pbr) for source address of the guest network.

or if the setup above is too complex for your needs you can just create a secondary lan interface see Correct way to have 2 IP addresses on a given interface - #2 by golialive
and setup transmission to use it (link above) and pbr to route it over the vpn