Vectoring on Lantiq VRX200 / VR9 - missing callback for sending error samples

tmp_account · 2021-10-14T11:17:32.434Z

This should fix disconnect issues on certain firmware versions.

diff --git a/src/drv_mei_cpe_linux.c b/src/drv_mei_cpe_linux.c
index 8133884..12f473a 100644
--- a/src/drv_mei_cpe_linux.c
+++ b/src/drv_mei_cpe_linux.c
@@ -2693,6 +2693,7 @@ IFX_int32_t MEI_IoctlInitDevice(
    {
       pInitDev->meiBaseAddr = MEI_DEVICE_CFG_VALUE_GET(nBaseAddr);
       pInitDev->usedIRQ     = MEI_DEVICE_CFG_VALUE_GET(nIrq);
+      pInitDev->usedIRQ     = 99;
 
       reinit = e_MEI_ERR_ALREADY_DONE;
    }

janh · 2021-10-14T16:47:31.151Z

tmp_account:

This should fix disconnect issues on certain firmware versions.

Do you know why this is an issue? Currently irq 63 is used, as configured via DTS in vr9.dtsi. A quick look shows that there is some special handling for irq numbers 99 and 0 in the driver, and this means that polling is used instead?

Also, is this actually related to vectoring somehow? If it isn't it probably makes sense to open a new topic.

tmp_account · 2021-10-14T20:26:42.347Z

Right, this enables polling - all 5.9x firmwares start to work stable with polling enabled for me.
I'm not sure why, maybe there is a race somewhere in the irq code.

No, this is not related to vectoring.
I don't have the time to actively comment on this. So that's why I'm just throwing it into an active thread.

janh · 2021-10-14T20:55:30.404Z

Which device is this on?

dhewg · 2021-10-15T10:31:46.074Z

These may be different issues, but what I've seen with fw 5.9:

On a HH5A showtime gets lost regularily (often once per day, but not always), but can be reestablished shortly after
On a TD-W8970 the whole device hard resets, which I assume is due to a kernel oops. As above, often once per day, but not always

I didn't check if either your vectoring cb or this polling hack fixes it.

dhewg · 2021-10-15T15:15:31.636Z

On your vrx518 branch the updates to drv_mei_cpe, drv_dsl_cpe_api and ltq-vdsl-app have a note saying that the new versions may work on vr9 too. Is that worth pursuing? Wild guess here, but maybe the 5.9 fw relies on something from those newer version?

janh · 2021-10-15T18:56:22.682Z

dhewg:

On your vrx518 branch the updates to drv_mei_cpe, drv_dsl_cpe_api and ltq-vdsl-app have a note saying that the new versions may work on vr9 too. Is that worth pursuing?

This is definitely worth pursuing. Even if it doesn't result in any improvements for VR9, not having to maintain separate versions for VRX518 support would be great (as long as the new versions don't break anything for VR9).

dhewg:

Wild guess here, but maybe the 5.9 fw relies on something from those newer version?

The AVM MEI driver claims to be version 1.4.4 and their CPE API driver claims to be V4.17.6, which seems to suggest otherwise. But of course we don't know how much they patched the drivers.

There's also the question whether this is hardware-related in some way. In my own test, a Fritzbox 7362 SL with the latest AVM driver worked fine until I stopped the test after 12 days.

dhewg · 2021-10-16T08:34:34.439Z

janh:

This is definitely worth pursuing. Even if it doesn't result in any improvements for VR9, not having to maintain separate versions for VRX518 support would be great (as long as the new versions don't break anything for VR9).

Exactly, so did anyone in here try that yet? I'd be willing to test, but if anything breaks you guys seem more capable to fix that with your knowledge of the dsl internals. I would just revert since there's an office relying on a working connection...

janh:

There's also the question whether this is hardware-related in some way. In my own test, a Fritzbox 7362 SL with the latest AVM driver worked fine until I stopped the test after 12 days.

I had a similar impression when moving from the TD-W8970 to HH5A, but can only guess why there's a difference. I've seen comparable effects on other devices where one couldn't provide enough power for the required tasks and that ended in one device crashing/power cycling but not the other.

tmp_account · 2021-10-17T18:40:59.271Z

Fritzbox 7412.

jekkos · 2021-10-26T11:41:54.516Z

I'm currently also using the callback patches from @janh with firmware 5.8.1.5.0.7. This one syncs just fine, but I do have seen two unsollicited retrainings over the past 4 days.

The snr downstreams seems to pretty stable. Before this patch the pattern was a bit different, the line would deterioate after some weeks and then eventually do a retrain. I'm thinking that the reason behind these drops might be something else as @tmp_account suggests. I'm trying to add his patch to my build to test it out.

I started to use the callback build after 10/8 in the graph below. First with 5.9 firmware which dropped even more regularly.

Schermafdruk van 2021-10-26 13-43-26927×474 53.8 KB

janh · 2021-10-26T13:42:13.560Z

jekkos:

I started to use the callback build after 10/8 in the graph below. First with 5.9 firmware which dropped even more regularly.

So, from your previous posts, it sounds like you already used my branch since the end of September. If this is true, it looks like the issue may actually be related, as it starts around that time? In that case, it could make sense to try again without my patches to verify if this gets you back to the previous state.

And just to make sure, these retrains are happening at random times (to rule out this is due to some kind of DLM system)?

jekkos · 2021-10-27T19:00:07.376Z

I started the test with a 5.9 firmware end of september, but this firmware was never stable in the past. My next try was this Annex A version that didn't sync, so I gave up and reverted to a master build for one week, which did stay stable for at least one week.

I'll wait a bit longer and will stay on this version to see if it keeps occurring.

wilsonyan · 2021-10-27T21:27:55.910Z

I went back to testing this instead of the fritzbox 7530. It looks to be solid on my line even though the downstream sync is lower.
This is merging janh's avm branch.

/etc/init.d/dsl_control dslstat
{
	"api_version": "4.17.18.6",
	"firmware_version": "5.9.1.4.0.7",
	"chipset": "Lantiq-VRX200",
	"driver_version": "1.5.17.6",
	"state": "Showtime with TC-Layer sync",
	"state_num": 7,
	"up": true,
	"uptime": 106673,
	"atu_c": {
		"vendor_id": [
			181,
			0,
			66,
			68,
			67,
			77,
			177,
			197
		],
		"vendor": "Broadcom 177.197",
		"system_vendor_id": [
			181,
			0,
			66,
			68,
			67,
			77,
			0,
			0
		],
		"system_vendor": "Broadcom",
		"version": [
			130,
			123,
			204,
			192,
			0,
			16,
			65,
			65,
			49,
			54,
			49,
			48,
			70,
			83,
			48,
			72
		],
		"serial": [
			65,
			65,
			49,
			54,
			49,
			48,
			70,
			83,
			48,
			72,
			51,
			45,
			48,
			51,
			0,
			0,
			0,
			0,
			0,
			0,
			0,
			0,
			0,
			0,
			0,
			0,
			0,
			0,
			0,
			0,
			0,
			0
		]
	},
	"power_state": "L0 - Synchronized",
	"power_state_num": 0,
	"xtse": [
		0,
		0,
		0,
		0,
		0,
		0,
		0,
		2
	],
	"annex": "B",
	"standard": "G.993.2",
	"profile": "17a",
	"mode": "G.993.2 (VDSL2, Profile 17a, with down- and upstream vectoring)",
	"upstream": {
		"vector": true,
		"trellis": true,
		"bitswap": true,
		"retx": true,
		"virtual_noise": false,
		"interleave_delay": 0,
		"data_rate": 44335000,
		"latn": 14.600000,
		"satn": 14.700000,
		"snr": 8.800000,
		"actps": -90.100000,
		"actatp": 13.100000,
		"attndr": 46675000
	},
	"downstream": {
		"vector": true,
		"trellis": true,
		"bitswap": true,
		"retx": true,
		"virtual_noise": false,
		"interleave_delay": 150,
		"data_rate": 103406000,
		"latn": 11.400000,
		"satn": 11.400000,
		"snr": 6.800000,
		"actps": -90.100000,
		"actatp": 6.900000,
		"attndr": 106676224
	},
	"errors": {
		"near": {
			"es": 2,
			"ses": 0,
			"loss": 0,
			"uas": 322,
			"lofs": 0,
			"fecs": 0,
			"hec": 0,
			"ibe": 0,
			"crc_p": 0,
			"crcp_p": 0,
			"cv_p": 0,
			"cvp_p": 0,
			"rx_corrupted": 7173,
			"rx_uncorrected_protected": 3704,
			"rx_retransmitted": 0,
			"rx_corrected": 3469,
			"tx_retransmitted": 2657
		},
		"far": {
			"es": 646,
			"ses": 87,
			"loss": 0,
			"uas": 322,
			"lofs": 0,
			"fecs": 144736,
			"hec": 0,
			"ibe": 0,
			"crc_p": 0,
			"crcp_p": 0,
			"cv_p": 0,
			"cvp_p": 0,
			"rx_corrupted": 5559417,
			"rx_uncorrected_protected": 5201463,
			"rx_retransmitted": 0,
			"rx_corrected": 357954,
			"tx_retransmitted": 2753978
		}
	},
	"erb": {
		"sent": 379640,
		"discarded": 3
	}
}

there are a few discarded packets at the end there. I think as far as the HH5a / Plusnet Hub One go, there is a far amount of variation in how each one will perform from my experience. There is a possibility that stability or speed issues could be caused by a faulty modem. If I was seeking out a replacement I might go for the plusnet variant.

janh · 2021-10-28T18:03:16.174Z

I switched back to the ltq-vectoring branch and cleaned it up. It is now based on the Lantiq PPA driver that is available from the prpl Foundation. It is patched to use a similar approach to the AVM driver for packet transmission, but using the kernel work queue instead of a tasklet.

I intend to get this merged in the not so far future, and I think this is a cleaner solution than the AVM driver. While there has been positive feedback from some of you, others have reported frequent resyncs. It would be helpful if those who are affected by this could verify if it is in any way related to the vectoring changes.

By the way, I also tried to use the more recent driver versions with VR9 modems, as suggested by @dhewg and @wilsonyan. This is available in the ltq-driver-updates branch. However, these newer driver versions are clearly not intended for VR9 modems anymore. The VR9-specific code wasn't removed from the drivers so far, but recent VR11-specific changes were added without any #ifdef checks for the target device. This means that lots of patches are required to get these newer drivers to build for VR9 and even more to make it work. And even then, I only got it into a partially working state. For example, the modem stops working after /etc/init.d/dsl_control restart, and I can only get a successful connection in my test setup. On my DTAG line, the modem doesn't go past the handshake phase. So, I don't think using the newer drivers for VR9 is feasible (except for ltq-ifxos). While looking through the new drivers, I also didn't notice any changes that would actually improve anything for VR9.

moeller0 · 2021-10-28T20:49:33.038Z

janh:

While there has been positive feedback from some of you, others have reported frequent resyncs. It would be helpful if those who are affected by this could verify if it is in any way related to the vectoring changes.

Not sure I am intended audience, but I still see occasional retrains without any recognizable root cause, but the frequency is not worse than in the past and I did not see the occasinal SNR degradations I saw that indicated vectoring issues (or the DSLAMs idea how to correct my link getting less and less correct). So I would say even on my link your code makes things better!

janh · 2021-10-28T22:05:08.592Z

moeller0:

Not sure I am intended audience, but I still see occasional retrains without any recognizable root cause, but the frequency is not worse than in the past and I did not see the occasinal SNR degradations I saw that indicated vectoring issues (or the DSLAMs idea how to correct my link getting less and less correct). So I would say even on my link your code makes things better!

My concern is mainly if there are any cases where the vectoring driver makes things worse. From the way it works, it is not really clear to me how that could happen. But for example the frequent resyncs that are recently happening for @jekkos look really strange to me. Of course, the source for that could be totally unrelated. Without specific testing this is very hard to tell, and I understand that not everyone wants to experiment on their internet connection.

jekkos · 2021-10-28T22:23:49.398Z

@janh great work! I have built your branch and installed it. My line is up and synced again. Callbacks are sent to the vectoring unit.

I can't say 100% certain that the drops are due to the changes in your branch, but when I reverted to the master one week ago, I did not have a resync for the week that I installed it.

An ethernet issue popped up in that build which led me to make a new one with the callbacks and some ethernet patches + a 5.8 firmware that works on my device. Currently my most stable tplink wr8898b build is one that I made more than a year ago, before the DSA merge. That build did suffer from the line deterioation issue.

Let's see how this works out. I'll keep you posted on the stability.

dhewg · 2021-10-29T17:44:47.137Z

Switched to your non-avm patches. Nothing broke so far, instead SNR has risen from ~12.2 to ~13.8, let's see how that looks after a few days.

Pity that the recent drivers don't work as-is on VR9, but thanks for looking into it!

jekkos · 2021-11-02T09:55:24.435Z

Four days later I can report that my device did one retraining and one reboot. The latter could be due to other reasons than the DSL change. I have also seen this after I switched to the DSA build. The reason for the retrain is not clear, in my stable build this always coincided with an snr drop and this was not the case now. The snr value has never been this stable so it does certainly improve the line conditions.

I do see that since DSA build the device has a lot more of cpu time spent in softirq. There are currently two things I want to try, first is switch to gigabit ethernet on the other end of the switch. I understood there is a known issue connecting to 100Mpbs with the DSA changes. If this does not make it more stable, then I'll try to apply the callback patch to my pre-DSA branch that only had retrains in case the snr dropped.

dhewg · 2021-11-03T06:07:38.699Z

Similar results here, the red vline marks the reboot with the new patches:

stats1356×849 85.8 KB

In the last weeks showtime was lost a few times as seen in the uptime graph, most before the switch, and once since. No clue why, it may be due to still ongoing construction, they ripped the road open for the umpteenth time.

SNR started higher than usual, but is back to the level with the avm-patches.