Lantiq vrx200 xDSL firmware recommendation thread

Right, so obviously my firmware blob comes with bitswap enabled by default, while the one you use doesn't. Fair enough. I don't have an unmodified Fritzbox to compare, but I think it would be really good to get all the line features in sync.

Silly question though: Your patch also does g997racg, which is G997_RateAdaptationConfigGet ... What is the point of getting the configuration for setup (and have the output go into the void)? If anything, shouldn't that be g997racs, as in G997_RateAdaptationConfigSet?

Erm, cumbersome, how :wink: ? In all honesty, the only value of this was in figuring out the scaling of the different fields from the ITU standards... Also I end up looking at g997dsnrg, g997bang, as well as g997dhlogg and g997dqlng (the QLN data is only acquired during a retrain but will show the noise level in all relevant frequency bands and is helpful in finding RF ingress issues).

Eh, "cumbersome" as in, I'm unfamiliar with matlab code and you do a lot of postprocessing in a lot of places.

Here's a thought, should we document somewhere, somehow, in real-world speak how to scale/normalize/interpret the values delivered by the respective functions? If there is such a documentation somewhere I was unable to find it.

Funnily enough, those don't seem to have "short" variants, but instead deliver a "more readable" set, with 512 values (presumably those 8-tone "buckets" again) and "255" as decimal "unavailable" instead of leaving out the unavailable ones like e.g. g997sang does. So in a way, a third, "mixed" way of formatting data.

(Also, should we split into a new thread? I have more thoughts about all of this, but I feel we are veering way off the "firmware recommendation thread" now.)

1 Like

I also saw that one, indeed I committed the wrong command there. It should be the set instead of get.

I at least put the references to the ITU standard into the code with a copy of the relevant standards text. This is a bit unwieldy and not real-world speak, but at least it is precise :wink:

% some command return unscaled data, so scale according to T-REC-G.997.1-201902
if isfield(parsed_dsl_output_struct, 'Data')
	parsed_dsl_output_struct.Data_orig = parsed_dsl_output_struct.Data;
	parsed_dsl_output_struct.Data_name_orig = parsed_dsl_output_struct.Data_name;
	switch dsl_sub_cmd_string
		case {'g997gansg', 'g997gang'}
			% T-REC-G.997.1-201902: 7.5.1.29.3 Downstream gains allocation (GAINSpsds)
			% This parameter specifies the downstream gains allocation table per subcarrier. It is an array of
			% integer values in the 0 to 4 093 range for subcarriers 0 to NSds. The gain value is represented
			% as a multiple of 1/512 on linear scale. The same GAINSpsds format shall be applied to ITU-T G.992.3
			% and ITU-T G.992.5 Annex C FEXT GAINSpsds and NEXT GAINSpsds.
			% The reported gains of subcarriers out of the downstream MEDLEY set shall be set to 0.
			% This parameter shall be reported with the most recent values when read over the Q-interface.
			parsed_dsl_output_struct.Data = parsed_dsl_output_struct.Data / 512;
			parsed_dsl_output_struct.Data_name = parsed_dsl_output_struct.Data_name(1:end);
			
		case {'g997sansg', 'g997sang', 'g997dsnrg'}
			% mask out the FF/255 bins, as FF is the "no measurement could be done" marker
			parsed_dsl_output_struct.ignore_bin_marker = 255;
			parsed_dsl_output_struct.ignore_bin_idx = find(parsed_dsl_output_struct.Data == parsed_dsl_output_struct.ignore_bin_marker);
			
			% T-REC-G.997.1-201902: 7.5.1.28.3 Downstream SNR(f) (SNRpsds)
			% This parameter is an array of real values in decibels for downstream SNR(f). Each array entry represents
			% the SNR(f = i � SNRGds � ?f) value for a particular subcarrier group index i, ranging from 0 to MIN(NSds,511).
			% The SNR(f) is represented as (?32 + snr(i)/2), where snr(i) is an unsigned integer in the range from 0 to 254.
			% A special value indicates that no measurement could be done for this subcarrier group because it is out of the
			% passband or that the SNR is out of range to be represented. The same SNRpsds format shall be applied to ITU-T G.992.3
			% and ITU-T G.992.5 Annex C FEXT SNRpsds and NEXT SNRpsds.
			parsed_dsl_output_struct.Data = (parsed_dsl_output_struct.Data * 0.5) - 32;
			parsed_dsl_output_struct.Data(parsed_dsl_output_struct.ignore_bin_idx) = 0;
			parsed_dsl_output_struct.Data_name = [parsed_dsl_output_struct.Data_name(1:end), ' [dB]'];
			
		case 'g997dhlogg'
			parsed_dsl_output_struct.ignore_bin_marker = 1023;
			parsed_dsl_output_struct.ignore_bin_idx = find(parsed_dsl_output_struct.Data == parsed_dsl_output_struct.ignore_bin_marker);
			
			% T-REC-G.997.1-201902: 7.5.1.26.6 Downstream H(f) logarithmic representation (HLOGpsds)
			% This parameter is an array of real values in decibels for downstream Hlog(f). Each array entry represents
			% the real Hlog(f = i � HLOGGds � ?f) value for a particular subcarrier group subcarrier index i, ranging
			% from 0 to MIN(NSds,511). The real Hlog(f) value is represented as (6 ? m(i)/10), where m(i) is an
			% unsigned integer in the range from 0 to 1 022. A special value indicates that no measurement could be
			% done for this subcarrier group because it is out of the passband or that the attenuation is out of range to be represented.
			parsed_dsl_output_struct.Data = 6 - (parsed_dsl_output_struct.Data / 10);
			parsed_dsl_output_struct.Data(parsed_dsl_output_struct.ignore_bin_idx) = NaN;
			parsed_dsl_output_struct.Data_name = [parsed_dsl_output_struct.Data_name(1:end), ' [dB]'];
			
		case 'g997dqlng'
			parsed_dsl_output_struct.ignore_bin_marker = 255;
			parsed_dsl_output_struct.ignore_bin_idx = find(parsed_dsl_output_struct.Data == parsed_dsl_output_struct.ignore_bin_marker);
			
			% T-REC-G.997.1-201902: 7.5.1.27.3 Downstream QLN(f) (QLNpsds)
			% This parameter is an array of real values in decibels with reference to 1 mW per hertz for
			% downstream QLN(f). Each array entry represents the QLN(f = i � QLNGds � ?f) value for a particular
			% subcarrier group index i, ranging from 0 to MIN(NSds,511). The QLN(f) is represented as (?23 ? n(i)/2), where
			% n(i) is an unsigned integer in the range from 0 to 254. A special value indicates that no measurement could
			% be done for this subcarrier group because it is out of the passband or that the noise PSD is out of range to be represented.
			% The same QLNpsds format shall be applied to ITU-T G.992.3 and ITU-T G.992.5 Annex C FEXT QLNpsds and NEXT QLNpsds.
			parsed_dsl_output_struct.Data = -23 - (parsed_dsl_output_struct.Data / 2);
			parsed_dsl_output_struct.Data(parsed_dsl_output_struct.ignore_bin_idx) = NaN;
			parsed_dsl_output_struct.Data_name = [parsed_dsl_output_struct.Data_name(1:end), ' [dB, ref 1mW/Hz]'];
			
			
		case {'g997bang', 'g997bansg'}
			parsed_dsl_output_struct.Data_name = parsed_dsl_output_struct.Data_name(1:end);
			
		otherwise
			% nothing to do
	end
	% get the correct frequency center bins.
	switch dsl_sub_cmd_string
		case {'g997dsnrg', 'g997dhlogg', 'g997dqlng'}
			% adjust the xvec to the group center bins
			if isfield(parsed_dsl_output_struct, 'GroupSize')
				parsed_dsl_output_struct.Data_xvec_orig = parsed_dsl_output_struct.Data_xvec;
				parsed_dsl_output_struct.Data_xvec_groupcentered = (parsed_dsl_output_struct.Data_xvec * parsed_dsl_output_struct.GroupSize) + (parsed_dsl_output_struct.GroupSize * 0.5);
				parsed_dsl_output_struct.Data_xvec = parsed_dsl_output_struct.Data_xvec_groupcentered;
			end
	end
end

Honestly, I think I didn't see the forest for the trees (or, as it were, the comments for the code). That is indeed really great documentation nesting in there. And yes, my face is red :wink:

I am not entirely sure what to do with all of the output. If we take the FritzBoxen as a template (and I think we should), what they do for their "spectrum" tab is pretty obvious:


(not my image)

On top there's the output of g997sang (in downstream direction -- I read in a few places that it only works for downstream, but interestingly I also get sensible values for upstream from my modem?)

On the bottom that is also quite clearly the the output of g997bang (except AVM seems to do some post-processing/averaging there?)

But what would we do with g997dhlogg, g997dqlng?

Anyway, the next step would be probably be something something LUA that takes the output of the g997* commands and parses them into JSON to feed to a LuCI XHR call. I have been looking at luci-app-nlbwmon as a template (it does pretty much the same, get data using a system call, create JSON, feed to chart.js on the LuCI frontend), but like I said before, LuCI changed a lot between 19.07 and snapshot (compare 19.07 to snapshot code).

I think it's a waste of time to code against an outdated target, so I will either update my main router to snapshot or one of my other Lantiq routers, so I can get coding against the new LuCI codebase.

I won't live this down, will I? :slight_smile:

AVM probably used DeltSNRGet in the past, which only returns the downstream SNR data, but newer labor development versions also show the upstream SNR (in an ugly purple).
Also the modem reports more than one pilot, but AVM only shows one of them...

I assume they want the two plots to show similar smoothness and hence need to modify the high resolution bit allocation data (where the modem reports all bins, instead of the groups used for the SNR).

Hav a look at https://github.com/moeller0/lantiq_dsl_parser/wiki?

Oh that was not tongue in cheek, the standards text is rather dry and technical, not really a nice an convenient to read description. And the matlab code is crufty, with the comments mainly left for me to being able to understand the rationale for the scaling post-hoc, not so much as user friendly documentation :wink:

I thought so. The interesting thing is that g997racg already reports RA_MODE=3 for me:

# . /lib/functions/lantiq_dsl.sh ; dsl_cmd g997racg 1 0
nReturn=0 nDslMode=1 nDirection=0 RA_MODE=3
# . /lib/functions/lantiq_dsl.sh ; dsl_cmd g997racg 1 1
nReturn=0 nDslMode=1 nDirection=1 RA_MODE=3

So, again a case of "my firmware blob does have other defaults than yours"?

... and the snapshot version does not use LUA at all anymore. It relies on the nlbw binary delivering JSON that it can just pass on to the frontend JavaScript. So I'm in a bit of a pickle. I would actually really like to forego LUA and do the data collection/postprocessing in a simple shell script, but busybox ash has no usable concepts of arrays. And I'm not entirely certain if I want to go down the rabbit hole of "array workarounds" for ash. :confused:

So I am thinking about a shell script that collects data from g997* command (maybe crops it to the nData="..." payload), passes them on directly to JavaScript and do the collection/postprocessing entirely on the frontend. The only downside is that this would not allow for CSV export.

Edit/update: client side parsing and charting turns out to be quite feasible. This is basically already the equivalent of a FritzBox's spectrum display:

On my fritzbox I found that the RA_MODE in downstream was set to 3 indeed. On OpenWrt it was set to 2. So indeed for some reason the (exact same) blob is initialized differently. This difference in my case is probably coming from the dsl daemon itself.

I also noticed that in openwrt the XTSE bits are masked out. If I configure my fritzbox with the template settings for my ISP, XTSE bits are set to AnnexJ. I then also saw that the dsl daemon is started with different XTSE arguments. I have tried to swtich between both but have not noticed any improvement in stability. Maybe someone here know why this was done that way?

I'm not sure what direction the luci interface is heading, but I suppose more stuff will move to the browser (angular/react?) so your idea of generating json from ash certainly make sense. I'd try to move the rendering load to the browser as much as possible. The command output basically just needs a bit of text reformatting. I whipped up a simple ash function yesterday that sends the values to collectd daemon.

The only problem I'm facing now is that grafana does not support building decent scatter plots.

Just a couple of observations from experience using a FB7490 and having looked at the Netgear DM200 source package:

  1. some capabilities need to be enabled at both ends and their use negotiated (I know this applies to SRA and suspect that it applies to bitswap and a couple of others) so they need to be set before the link is brought up and then you check whether they're currently active (I've seen the FB take hours to report SRA being active on a network where it is a design rule to have it; I think I've also seen the FB change the bitswap status).
  2. the OpenWrt DSL up script seems to use a similar approach to enabling some capabilities as the Netgear DM200 DSL up script but I don't remember whether the capabilities included were the same (Netgear's xDSL firmware is probably configured differently to others anyway).

I don't currently use a Lantiq modem as my line is long and crusty and Broadcom modems perform much better on it :roll_eyes: but it would be nice to see better connection reporting for the DSL modems OpenWrt supports!

Neither. It's a tailor-made framework. It's not too complicated, but it takes a bit to wrap the head around, and a device I can test on. (Even if it looks and works the same, and 19.07 is already somewhat JS-based, LuCI has changed massively between 19.07 and snapshot.)

I'm pretty much there already. I got the ash script on the router that takes the g997* command outputs, cuts out the payloads, and puts them into a neat JSON object ready as an answer to the XHR request. And I got the client side that parses the output into something the chart JavaScript can display. I will probably move from chart.js (large and crusty and generates canvas) to chartist.js (way leaner and generates SVGs). But my task for the next few days is getting my stuff updated to snapshot to code against the new JavaScript-based frontend.

As far as I can see, your ash script has it easier ... it just pushes a long list of key/value pairs to collectd. One of these days I will move to an external monitoring, too, but I don't have anything "permanent" set up that does prometheus/influxdb & grafana right now.

You could try transmitting the null values inbetween? Have you tried to parse the short variant of the g997 commands? They deliver a full 512/4096 value string, you'd just have to count up the key (i.e. bin/tone number) value yourself -- and possibly convert from hex to dec, but judging from your ash script you do the SNR value parsing on the receiving end anyway?

Indeed I think that adding these settings in the Autoboot section of the /tmp/dsl.scr file actually takes care of that. At least I found the downstream snr margin setting already in there and added my tweaks (which seem to be applied once the line is in sync).

The settings you need to apply seem to be more dependent on the ISP and the infrastructure you are talking with. I guess that is why the FB config is exactly what I need to copy as my ISP has a 'certification' procedure setup with AVM where they double check the configurations before they whitelist the dsl driver. If the modem does not report a whitelisted version, it is set to a low speed (fallback) profile.

I'm not really sure whether this is something they did for technical or commercial reasons, but I think they should at least publish the technical requirements to pass such a certfication procedure (so you can use / choose your own hardware instead of them forcing you to buy theirs).

stable on plusnet UK
It's my brother's setup but I manage it remotely (no updates yet - but planned to latest stable on my next onsite visit)

BTHOMEHUBV5A, LEDE 17.01.6
ATU-C Vendor ID: Broadcom 192.28
Firmware Version: 5.8.1.8.1.6
Annex: B
Line Mode: G.993.2 (VDSL2)
Line Uptime: 2d 13h 30m 12s

Well the data is in influxdb but it's just non obvious to plot anything on an x-axis that is not time in grafana. The screencap I posted was using a workaround in which you create a separate series per tone.
I did find a reference to another plugin 'plotly' that seems to be able to do that, so I might give that a try.

I'm not sure but did you also check luci-app-statistics? I thought it was also using collectd as dataource. so that might be easy to extend? Or was this also rewritten in that latest version of luci?

So stray observation after a few days now: I switched from 5.7.9.5.1.7 back to 5.7.8.9.1.7. Both have all the configuration bits, including bitswap, enabled. But the latter syncs noticeably less aggressively and will recover from a dip in SNR much, much better.

It may be a coincidence, and I have no deeper insight, but it's noteworthy that the firmware that runs best on my device is the one that originally comes with my device. I have no evidence for it, but it is obvious that different firmware blobs are "tuned" differently, and it feels like some don't only match better with the far end, but also with the device they run on.

Replying to myself here. Using blob 5.7.8.12.0.7, a month later, I did get 1 or 2 disconnects (on stock firmware I had 6 months uptime), but more annoyingly, I was losing downstream speed gradually. I was able to get 90 Mbps iperf3 TCP performance about a month ago, but it dropped to 80 Mbps, even when switching to another blob, power off/on cycles, etc.
So for measure, I have re-installed the ZTE H369A ("Experiabox v10") monstrosity, and immediately it synced at much higher speeds:

Link Status Up
Showtime 50 h 45 min 17 s
Modulation Type VDSL
Actual Rate(Up/Down) 32313/107674 kbps
Attainable Rate(Up/Down) 32313/109699 kbps
Noise Margin(Up/Down) 5/5.8 dB
Line Attenuation(Up/Down) 1.6/4.9 dB
Output Power(Up/Down) -6.2/12.5 dBm
Data Path(Up/Down) Interleaved/Interleaved
Interleave Depth(Up/Down) 4/8
Interleave Delay(Up/Down) 0/0 ms
INP(Up/Down) 80/79 symbols

Which translates to 97+ Mbps iperf3 downstream even on double NAT.
To me this sadly confirms that OpenWRT VDSL just isn't working for me. I have found a Draytek Vigor 130 for cheap that comes with 8 different firmware/blob combinations that they have certified with ISPs in various countries. My plan is to use it in bridge mode in front of a NanoPI R2S running OpenWRT. I will report here in the future for reference.

I can highly recommend the Draytek Vigor 130 that I got for €25. With "modem 8" firmware I am getting even better speeds than the ISP-provided (KPN Wholesale Netherlands) ZTE H369A / Experiabox v10. Path mode is even "fast" instead of "interleaved" with the ZTE. But no OpenWRT obviously.

> vdsl status
vdsl status
  ---------------------- ATU-R Info (hw: annex A, f/w: annex A/B/C) -----------
   Running Mode            :      17A       State                : SHOWTIME
   DS Actual Rate          :108673000 bps   US Actual Rate       : 31385000 bps
   DS Attainable Rate      :109482500 bps   US Attainable Rate   : 31385000 bps
   DS Path Mode            :        Fast    US Path Mode         :        Fast
   DS Interleave Depth     :        1       US Interleave Depth  :        1
   NE Current Attenuation  :        9 dB    Cur SNR Margin       :        5  dB
   DS actual PSD           :    -5.-7 dB    US actual PSD        :    11. 8  dB
   NE CRC Count            :        0       FE CRC Count         :       79
   NE ES Count             :        0       FE  ES Count         :        8
   Xdsl Reset Times        :        0       Xdsl Link  Times     :        1
   ITU Version[0]          : fe004452       ITU Version[1]       : 41590000
   VDSL Firmware Version   : 05-07-09-0F-01-07   [with Vectoring support]
   Power Management Mode   : DSL_G997_PMS_L0
   Test Mode               : DISABLE
  -------------------------------- ATU-C Info ---------------------------------
   Far Current Attenuation :        8 dB    Far SNR Margin       :        5  dB
   CO ITU Version[0]       : b5004244       CO ITU Version[1]    : 434db1bf
   DSLAM CHIPSET VENDOR    : < BDCM >