Collectd-mod-ping: Multiple devices or sourceaddresses

All,

Recently I have been having quiet bad internet, so I wanted to monitor my pings to well-known good hosts such as 8.8.8.8 or 1.1.1.1.
I actually have two internet provides, so I have a dual-WAN setup (using mwan3), where pppoe-wan is my primary interface and eth0.109 my secondary interface. Both interfaces have different IPs and I wanted to set up collectd-mod-ping according to the documentation and use either Device or SourceAddress to make sure pings traverse through the proper interface.
However, when I try to use Device I get the following error: collectd collectd[13804]: ping plugin: ping_send failed: No such device
Using source address on the other hand seems to work, but when I started to watch the traffic with tcpdump I can see that only the current default route is getting used nevertheless.
Following my complete /etc/collectd.conf:

BaseDir "/var/run/collectd"
Include "/etc/collectd/conf.d"
PIDFile "/var/run/collectd.pid"
PluginDir "/usr/lib/collectd"
TypesDB "/usr/share/collectd/types.db"
Interval 30
ReadThreads 2


LoadPlugin conntrack

LoadPlugin ping
<Plugin ping>
        TTL 127
        Interval 15
        AddressFamily ipv4
        Host "8.8.8.8"
        Host "8.8.4.4"
        SourceAddress "redacted"
</Plugin>

<Plugin ping>
        TTL 127
        Interval 15
        AddressFamily ipv4
        Host "1.1.1.1"
        Host "1.0.0.1"
        SourceAddress "redacted"
</Plugin>

LoadPlugin memory
<Plugin memory>
        ValuesPercentage false
        ValuesAbsolute true
</Plugin>

LoadPlugin cpu
<Plugin cpu>
        ValuesPercentage true
        ReportByCpu true
        ReportByState true
</Plugin>

LoadPlugin rrdtool
<Plugin rrdtool>
        DataDir "/data/rrd"
        RRARows 100
        RRASingle true
        RRATimespan 3600
        RRATimespan 86400
        RRATimespan 604800
        RRATimespan 2678400
        RRATimespan 31622400
</Plugin>

LoadPlugin load

LoadPlugin interface
<Plugin interface>
        IgnoreSelected false
        Interface "eth0.109"
        Interface "pppoe-wan"
</Plugin>

I found a GitHub issue which asks for pinging the same IP on two different interfaces, but I don't really mind if I have to two different IPs to see which of the interfaces is having paketloss and when.

Is there anything I am missing or is there a better solution to my problem? I am fully aware that I could solve this by using iptables to force the pakets with a specific destination through a specific interface, but unfortunately, that would mean I would lose "full failover capabilities", hence I am looking for a better solution.

Thanks a lot!