The odhcpd documentation mentions that it "Supports rewriting of the announced DNS server addresses" in relay mode. However, it is not clear on how to achieve that.
After trying out a bunch of things, and ultimately going through the source of odhcpd I figured out that if dns option is specifically mentioned in the dhcp config for the interface (either ipv6 or ipv4), it overrides the upstream dns with preffered ipv6 dns server of the router. It would be helpful if this was mentioned in the documentation somewhere.
Changing behaviour based on if the dns is specified or not is also not intuitive. The default value of dns option is the local address, but specifically specifying the same local address changes the behavior in this case.
Link to the documentation I'm referring to: https://openwrt.org/docs/techref/odhcpd