openNDS upload rate issue

Installing and Using OpenWrt
1

I installed openNDS captive portal and it works as expected, with one exception:

When limiting the upload rate (option uploadrate '512') clients still can upload up to the full speed of my internet connection. However, limiting the download rate (option downloadrate '512') works perfectly.

First I thought this could be due a conflict with other packages, so I set up Openwrt from the scratch using Openwrt 23.05.5 on a WRT3200ACM router, just installing minimum packages to run openNDS. But the issue is still there: Upload cannot be restricted, while download actually can be restricted by openNDS configuration.

Is anybody using openNDS with upload rate restrictions and can confirm that it is working? I would greatly appreciate advice what further information I could provide for anybody knowledgable and willing to help.

I only changed the following configuration options:

option gatewayinterface 'br-wifi_public'
option gatewayname 'Wifi Home'
option enable_serial_number_suffix '0'
option downloadrate '512'
option uploadrate '512'

This is the opennds startup logread output when using debuglevel 2, dnsmasq-full installed:

daemon.notice opennds[17897]: openNDS Version 10.3.0 is in startup
daemon.info opennds[17897]: openNDS Version 10.3.0 is in startup - Please wait....
daemon.notice opennds[17897]: The name of this gateway is Wifi Home
daemon.notice opennds[17897]: Attempting to Bind to interface: br-wifi_public
daemon.notice opennds[17897]: Interface br-wifi_public is up
daemon.notice opennds[17897]: Interface br-wifi_public is at 10.2.0.10 (6238e0bf1e01)
daemon.notice opennds[17897]: MHD version is 0.9.75
daemon.notice opennds[17897]: Number of Upstream gateway(s) [ 0 ]
daemon.notice opennds[17897]: Preemptive authentication is enabled
daemon.notice opennds[17897]: Created web server on 10.2.0.10:2050
daemon.notice opennds[17897]: Maximum Html Page size is [ 10240 ] Bytes
daemon.notice opennds[17897]: Socket access at /tmp/ndsctl.sock
daemon.notice opennds[17897]: Click to Continue option is Enabled.
daemon.notice opennds[17897]: Preauth is Enabled - Overriding FAS configuration.
daemon.notice opennds[17897]: sha256sum provider is available
daemon.err opennds[17897]: Error returned from system call - Continuing
daemon.notice opennds[17897]: Forwarding Authentication is Enabled.
daemon.notice opennds[17897]: Binauth is Enabled.
daemon.notice opennds[17897]: Externals flagged with debuglevel 1.
daemon.notice opennds[17897]: Initializing firewall rules
daemon.err opennds[17897]: Dnsmasq reload failed!
daemon.info opennds[17897]: list users_to_router is [ allow%20udp%20port%2053 allow%20udp%20port%2067 allow%20tcp%20port%2022 allow%20tcp%20port%20443 ]
daemon.info opennds[17897]: list preauthenticated_users is [  ]
daemon.info opennds[17897]: list authenticated_users is [ allow%20all ]
daemon.err opennds[17897]: Dnsmasq reload failed!
daemon.notice opennds[17897]: openNDS is now running.

This is the opennds startup logread output when using debuglevel 2, dnsmasq (standard, not full) installed:

daemon.notice opennds[23969]: openNDS Version 10.3.0 is in startup
daemon.info opennds[23969]: openNDS Version 10.3.0 is in startup - Please wait....
daemon.notice opennds[23969]: The name of this gateway is Wifi Home
daemon.notice opennds[23969]: Attempting to Bind to interface: br-wifi_public
daemon.notice opennds[23969]: Interface br-wifi_public is up
daemon.notice opennds[23969]: Interface br-wifi_public is at 10.2.0.10 (6238e0bf1e01)
daemon.notice opennds[23969]: MHD version is 0.9.75
daemon.notice opennds[23969]: Number of Upstream gateway(s) [ 0 ]
daemon.notice opennds[23969]: Preemptive authentication is enabled
daemon.notice opennds[23969]: Created web server on 10.2.0.10:2050
daemon.notice opennds[23969]: Maximum Html Page size is [ 10240 ] Bytes
daemon.notice opennds[23969]: Socket access at /tmp/ndsctl.sock
daemon.notice opennds[23969]: Click to Continue option is Enabled.
daemon.notice opennds[23969]: Preauth is Enabled - Overriding FAS configuration.
daemon.notice opennds[23969]: sha256sum provider is available
daemon.err opennds[23969]: Error returned from system call - Continuing
daemon.notice opennds[23969]: Forwarding Authentication is Enabled.
daemon.notice opennds[23969]: Binauth is Enabled.
daemon.notice opennds[23969]: Externals flagged with debuglevel 1.
daemon.notice opennds[23969]: Initializing firewall rules
daemon.err opennds[23969]: Dnsmasq reload failed!
daemon.info opennds[23969]: list users_to_router is [ allow%20udp%20port%2053 allow%20udp%20port%2067 allow%20tcp%20port%2022 allow%20tcp%20port%20443 ]
daemon.info opennds[23969]: list preauthenticated_users is [  ]
daemon.info opennds[23969]: list authenticated_users is [ allow%20all ]
daemon.warn opennds[23969]: libopennds - [Warning: dnsmasq nftset complile option not available - Upgrade to dnsmasq-full version. Trying ipset option....]
daemon.warn opennds[23969]: libopennds - [Warning: dnsmasq ipset complile option not available -- Upgrade to dnsmasq-full version. Unable to configure walledgarden....]
daemon.warn opennds[23969]: libopennds - [Warning: dnsmasq nftset complile option not available - Upgrade to dnsmasq-full version. Trying ipset option....]
daemon.warn opennds[23969]: libopennds - [Warning: dnsmasq ipset complile option not available -- Upgrade to dnsmasq-full version. Unable to configure blocklist....]
daemon.warn opennds[23969]: libopennds - [Warning: dnsmasq nftset complile option not available - Upgrade to dnsmasq-full version. Trying ipset option....]
daemon.warn opennds[23969]: libopennds - [Warning: dnsmasq ipset complile option not available -- Upgrade to dnsmasq-full version. Unable to configure walledgarden....]
daemon.warn opennds[23969]: libopennds - [Warning: dnsmasq nftset complile option not available - Upgrade to dnsmasq-full version. Trying ipset option....]
daemon.warn opennds[23969]: libopennds - [Warning: dnsmasq ipset complile option not available -- Upgrade to dnsmasq-full version. Unable to configure blocklist....]
daemon.err opennds[23969]: Dnsmasq reload failed!
daemon.notice opennds[23969]: openNDS is now running.
daemon.err opennds[23969]: failure: /usr/lib/opennds/libopennds.sh "auth_restore" &

Any advice appreciated!

2

@alex77
The rate quotas are thresholds above which a client will be throttled, not a ceiling maximum allowed rate.
A client will have no restrictions until the moving average rate exceeds the configured value.

As normal traffic is dominated by download, very often the download limiting will kick in first, resulting in the upload limiting not reaching its threshold for some time, if ever.
You should expect upload limiting to begin quite quickly if, for example, you try to upload a large file.

Without going through them in detail, your logs look fine. The errors and warnings are mostly reporting that autonomous walled garden is not available, or one or two startup "waiting for dnsmasq to start" messages. Most of these are already muted in the next version (not released yet).

You can check the uprate limiting with the following command:
nft list chain nds_filter ndsULR

You should get something like the following:

table ip nds_filter {
	chain ndsULR {
		ip saddr 192.168.1.176 limit rate 6193/minute burst 250 packets counter packets 3825 bytes 574599 return
		ip saddr 192.168.1.176 counter packets 239 bytes 14993 drop
	}
}

Similarly, you can check the downrate limiting with this:
nft list chain nds_mangle ndsDLR

3

@bluewavenet

Thank you very much for clarifiying. Considering what you wrote, I did some further tests using large files of 100MBytes in size to make shure, the buckets are filled fast and throtteling should start quickly. Upload and download rates were both set to 512kBytes/s throughout the tests.

First I checked the upload and download rates on Openwrt 23.05.5, as you suggested. The outcome seems reasonable to me:

$ nft list chain nds_filter ndsULR
table ip nds_filter {
        chain ndsULR {
                ip saddr 10.2.0.138 limit rate 17022/minute burst 250 packets counter packets 90 bytes 11269 return
                ip saddr 10.2.0.138 counter packets 0 bytes 0 drop
        }
}

$ nft list chain nds_mangle ndsDLR
table ip nds_mangle {
        chain ndsDLR {
                ip daddr 10.2.0.138 limit rate 22341/minute burst 250 packets counter packets 127 bytes 28608 return
                ip daddr 10.2.0.138 counter packets 18 bytes 2693 drop
        }
}

Then I "downgraded" to Openwrt 21.02.4 (openNDS 9.8.0-2) and tested the rate quotas again (with files of 100MBytes in size). Obviously both, upload rate, and download rate, are throttled as one would expect.

This is Openwrt 21.02.4 when throtteling download rate:
After an initial burst the average download rate drops to 62kBytes/s which is very good considering download rate was set to 512 kbit/s.

21024download
21024download666×512 34.2 KB

This is Openwrt 21.02.4 when throtteling upload rate:
This also looks great. After a short burst the average upload rate is throtteled to 62kByte/s, which is almost exactly 512kbit/s.

21024upload
21024upload673×536 34.5 KB

This is Openwrt 23.05.5 when throtteling download rate:
In 23.05.5 it takes considerably longer than in 21.02.4 for throtteling to take place. But after a while average download rate goes down to 56kbytes/s. But this time the download rate is not smooth. Alternating, there are bursts (which did not happen in 21.02.4), and very low download rates. But on average the rate is closed to 512kbit/s. But this throtteling seems to be much less elegant and less smooth as it appears to be in 21.02.4.
23055download

This is Openwrt 23.05.5 when throtteling upload rate:
The upload speed stays at 1728kBytes/s (this is the maximum upload speed of my internet connection) and is not throttled at all, it seems.
What really comes unexpected to me is, that the "Average Upload Rate This Session" on the status page is almost zero (I refreshed the status page several times during the upload). Instead the "Average Download Rate This Session" changed. But the "Average Upload Rate This Session" stayed at almost zero during all the 100MB upload. Could it be that there is no upload throtteling taking place, just because the "Average Upload Rate This Session" stays zero, never exceeding the upload rate threshold? But why does "Average Upload Rate This Session" stay at zero?
23055upload

So, I guess there might have been some changes to openNDS between 21.02.4 and 23 05.5, causing these symptoms. And there might be some openNDS-internal issue, keeping "Average Upload Rate This Session" too low in a upload scenario in 23.05.5.

As I am not very deep into openNDS, I might be wrong and my measurements may be too unsophisticated. And there might be some issue with my equipment, but I took all these tests on exactly the same hardware (WRT3200ACM). Nevertheless, I would like to use openNDS on a productive router and I am concerned that someone could grab all the upload-bandwidth when upload throtteling does not fully work.

I am greatful for any advice.

4

Note my previous post where upload throttling is clearly taking place:

There were many many changes over that period, with openNDS going from v9.0.0 up to the current v10.3.0.

One particular change was to significantly reduce the default ram and cpu requirements for the rate thresholds. Without this change, 128MB ram systems would oom very quickly with rate quotas enabled.

Your WRT3200ACM has 512MB? If so you can tune the options for rate limiting.
These options are "bucket ratio" and "max bucket size".
See:
https://opennds.readthedocs.io/en/stable/config.html#set-rate-quotas

Even with 512MB of ram, it is quite possible with high bucket ratio, high max bucket size and a large number of clients, to consume enough memory to get oom errors. Normally in such a state, openNDS v10+ will restart itself rather than crash like v9 and earlier did - but still.

It was pretty intensively tested before release and is running on numerous production system with no reported problems, so I think you should try tuning first.

Perhaps you have identified the symptoms of a "bug" of some sort, but for the moment, I think not.

5

@bluewavenet
It seems, openNDS runs with dnsmasq-full as well as with dnsmasq (not the full package). Would it be advisable to install dnsmasq-full anyways, or is dnsmasq (not full) sufficient?

6

Yes, dnsmsaq (not full) is sufficient. The full version is required if you want to define a walled garden. A walled garden is a list of "web sites" you want to allow access to before authentication. This walled garden list might include an Internet hosted FAS server that is on a CDN with a range of ip addresses.
https://opennds.readthedocs.io/en/stable/fas.html#using-a-cdn-content-delivery-network-hosted-server-for-a-remote-fas

For details about walled garden and block lists see:
https://opennds.readthedocs.io/en/stable/walledgarden.html#walled-garden-and-block-lists