Glad to know it generally works. But, it doesn't entirely surprise me that you're having trouble connecting to the devices that are connected via the relay...
relayd is basically a clever hack that involves routing between two interfaces such that the same subnet appears on each interface. In any normal situation, this cannot be routed, so that's why relayd is needed. My best guess is that there is routing ambiguity about how to reach devices on the downstream subnet since both the upstream and downstream are the same... so it just ends up pointing the packets to the upstream and that's it. I could be wrong about the reason, but fundamentally a relayd based device will have some nuances and quirks that don't conform to normal routing methods.