I think the ipaddr configured as local endpoint is used both when calculating the IPv6 prefix and as source address of the packets. Which won't work if the address isn't assigned to a local interface.
If you do assign it to a local interface, and then add SNAT and DNAT rules to translate the proto 41 packets to your private IPv4 address then it could work.