My modem's IP is 192.168.1.1 I have my router set to 192.168.1.2 so they don't conflict. Also I can see my modem's line stats this way.
Edit:
172.16.11.200 is definitely my gateway
I also chose to not use my "default gateway" in LEDE but then I couldn't access the internet sooo is it even possible to use a custom gateway address?
Aha, so I'm guessing LEDE.lan is actually your modem, and it has no problem responding to pings, but that 172.16.11.200 is in the fiber cabinet or beyond that... and the hop between your modem and this gateway is where things go wrong...
This implicates your modem, it may be buffering data even though it doesn't need to. Have you tried turning it off and on again ? 
Probably three times at this point. I'm actually gonna reset and unlock it again. It's the only thing I haven't done hardware wise tbh
What is the model of the modem device? Can you call your ISP and ask for a replacement modem? or buy one?
It's a huawei hg612. You can buy them second hand off ebay.
I doubt my ISP would give me a replacement unless I had a major fault like loss of service.
https://www.increasebroadbandspeed.co.uk/g.inp
discusses various issues related to VDSL, including things like packet interleaving, which they mention can increase latency substantially.
I suspect you're dealing with line noise, possibly caused by equipment you or neighbors have, or by wind, or etc etc. Your modem probably has negotiated a more reliable but higher latency operation mode... you might see info on its status page that would tell someone more expert than me about this sort of thing.
what sorts of stats do you have on your modem's status page? maybe take a screenshot and post?
I'm gonna reset my modem when I can and see if anything changes.
I know another forum that's really knowledgable about my modem in particular so I'll show them my stats and see if anyone spots anything unusual.
Huawei modem's and ONT's are crap, i have a friend have Huawei hg 8247 ONT, it's cause sometimes high ping, other times device freeze and should be power cycled to get it back to work!
also you can initiate a lot of connections or use many devices this is because devices have a small ram which will be filled very fast, if your modem support status page, you will see that ram usage is around 95%