I think the issue is that the traffic shaper really needs to have all traffic across an interface under its control, so if the shaper is running on different CPUs it still needs to coordinate to not exceed the global limit.... but I might be misunderstanding the core of your question.
My idea was for people with under-powered but say 2 or 4 core routers to set up say 2 or 3 interfaces and give each one a limit which is 1/2 or 1/3 of the true global limit. The true global limit is guaranteed not to be exceeded, but the aggregate amount might still be more than what is possible under a single CPU scenario.
1 Like
Yeah, that sucks raw eggs.... or put differently, I would be amazed if users are willing to make such a trade-off (unless each shaper would have a rate that allows to saturate the LAN interface...) So this would require a lot of work at "managing the users'" expectation such that they are sufficiently happy with the result.
Well, for whatever reason, the whole internet hasn't read my post on gigabit routers, and people still expect their $80 consumer all in one routers to SQM their new 500Mbps + connections but only get 250 or 300Mbps out of it.
Anyway, I agree that this hack is not the right solution, the right solution is to buy a wired router that handles well over a gigabit, and then move on. But some people are even crazily getting 10Gbps connections at $40/mo and then wondering why $80 all in ones can't handle those speeds...
2 Likes
Slackers! ... these kids today 
+1; or if 1 Gbps is cheap enough the willingness to simply ignore a big part of the potential throughput for better latency under load. Like when I shapedmy 100/36 link down to 49/35 because that was the limit my router could run cake with... (in all fairness my ISP has since days past considerably improved his bufferbloat game (cyclic ramping download BB from 30 to 60ms), still cake does mostly better (pretty flat at 30ms)). The coming 10Gbps links are going to make everything harder... (actually only traffic shaping, I think scheduler and AQM are cheap enough already).
MQPRIO with actual hardware offload is now being tested (see: mvebu mqprio testing).
I realise that in order to keep bufferbloat in check we want "no buffers" at all, and having hardware do MQPRIO or HTB or... it needs to buffer at least a few packets to re-order. BUT any NIC has some kind of queue right now already and I didn't see anywhere (yet) to actually reduce this queue to only 1 or 2 vs the standard 1000.
Thanks @dtaht for pointing this thread out to me.
I've been working on multicast at Akamai (in IETF and W3C) and we're getting some traction, though there's still a way to go.
I gotta say I agree almost completely with @dlakelan about the way we should be looking at it, and will also say this is not as hopeless as people usually think when they first hear about it. I do not think this ship has sailed. This is demonstrably useful, both for live video and for software download, plus perhaps some other cases. The main thing it solves that CDNs and p2p cannot solve is access network congestion, particularly in cable and gpon networks (and tho I agree with @moeller0 on DSL it won't make as much difference, but it can still reduce load on the DSLAM's uplink, which sometimes matters with frequency depending how oversold they are).
For those big-download days it would make a big difference to people, and there's a bunch of ISPs I've spoken to with some interest in making it happen.
For CeroWrt in particular, it would be awesome if the mcproxy opkg were on by default and enabled for v4 and v6 global ssm addresses (232/8 and ff3e::/96), so it would pass along global joins from the lan into the wan. This would make it so people with their own router would have the capability where the ISP provides multicast access. Note this functionality is enabled and in use by several European ISPs for their devices, where they have TV services with a mobile app. I don't know what all devices have it baked in or not, but I checked an out-of-the-box Fritz!Box, and it was there and on by default.
(Also great would be if DHCP would forward in the DOMAIN-SEARCH option if it arrives from wan, as this would enable DNS-SD for ISP-provided services, which would be potentially useful for many things and specifically useful for mnat...)
For a little more color on the status, we've got some specs (with prototypes) at various stages of completeness to fill in the important gaps we've found along the way:
https://www.rfc-editor.org/rfc/rfc8777.html
https://datatracker.ietf.org/doc/html/draft-ietf-mboned-ambi
https://datatracker.ietf.org/doc/html/draft-ietf-mboned-cbacc
https://datatracker.ietf.org/doc/html/draft-ietf-mboned-mnat
https://datatracker.ietf.org/doc/html/draft-krose-multicast-security
I don't want to hijack the thread completely, but I'll encourage anyone interested in global multicast to join the W3C community group (requires a free w3c account and agreeing to the w3c's cla), we meet on 1st Wednesdays of the month:
ooh, thanks for this, and I'd be very happy to pick your brain a little more about global multicast. Should we throw up a separate offtopic type thread here? maybe others are interested in the question, but I don't want to distract this thread too much.
With this we all agree, my question is however how prevalent live video still is compared to non-linear video that let's the user scrub through the timeline? I honestly have no idea, and personally stopped linear broadcast TV more than a decade ago.
Here I can see how this helps IFF delivery on somebody else's schedule is acceptable. And yes, I agree that it could help to lessen the impact of stuff like multi-gigabyte "urgent" patches for popular games which according to NANOG will show up as noticeable load spikes. Here I wonder more how this is going to work, have one multicast session continuously cycle through the full data and download clients that can start any time and will stop once they have the full file re-assembled? And how to deal with lost segments, truly force the resents on the full multi-cast or do fix-ups per uni-cast (or force clients tp simply wait for another iteration hoping that the second time around it is not going to be the same segment that are corrupted or missing).
Yeah, but that explains why ISPs might be interested in doing that end-users typically assume that their contracted speed is available any time.
Yes I can see this now, thanks.
Germany's incumbent Deutsche Telekom uses multicast for its TV offering (apparently on top of IPv4 in spite of offering full dual-stack since many years now), and AVM being based in Germany might effortlessly explain this Fritzbox feature for OpenWrt I believe recipes exist for igmpproxy. What would be the advantage of mcproxy over igmpproxy?
I think Dave is fine especially if the discussion is actually going in the direction of proposing things that need better testing/deployment, like apparently multicast capabilities.
How does japan do earthquake alerts?
I am not sure how many here ever saw the original project plan for cerowrt 1:
https://www.bufferbloat.net/projects/make-wifi-fast/wiki/Solving_the_Home_Router_Disaster_Annotated/
fountain coding. So yes, the sender just continuously sends an infinitely long stream of bytes, and when the receiver receives any N+2 blocks they reassemble the whole thing. The name "fountain coding" is based on the idea that the sender is like a "water fountain" and the receiver just holds out a cup until it's full, doesn't care which molecules get into the cup vs overshoot or spray off on the side.
Nothing stops a client from storing local video that allows traversing the timeline anywhere behind the broadcast time, of course.
But regardless, live video has some important cases, like the most popular sports and news channels, where most of the viewers want it while it's live, and it simply couldn't be done if you tried to scale them to pure-internet delivery with the same audience size.
Especially sports. Broadcast TV commonly carries free access ad-supported sports events that turn out to be surprisingly exorbitant to get via internet, largely because the costs don't scale right for spiky unicast load (even leaving aside the access network congestion piece).
But I don't think the utility is limited to the things that just don't work at scale. Time-shifting of popular stuff for a cheaper delivery of anything along the lines of a popular series you're subscribed to would still be hugely worthwhile if multicast delivery worked. Also, the most popular vod (think Squid Game, Game of Thrones, Dr. Who) has a lot of natural overlap for when people are watching (shortly after the latest episode is released), and could use multicast to great effect by just running a few carousels in parallel. 12 streams shifted by 5 minutes on a loop means you could ship an hour-long video that people can watch on demand and get an average of over 95% delivered by multicast (you just get unicast until you're sync'd enough with one of the multicast feeds, and you only have to buffer a max of 5 minutes at the client, plus however much you want to supoort for trick play without having to re-sync). And of course you can tune that if 5 minutes isn't right for some reason.
Re: software download:
There's some specs about how to do this (FCAST (RFC 6968), FLUTE (RFC 6726), or more generally anything on top of ALC (RFC 5775), plus NORM (RFC 5740) for some things would be ok, or there's a less-documented uftp implementation that seems to work, plus a few other unpublished things). In general, it's possible to run this kind of stuff with some different tradeoffs, and you can always fall back to unicast range requests for hole-filling to cover loss if you don't have anything better.
There's some complexity and some design choices to make here, and it gets further complicated by the state of the IPR on Raptor, so it's kind of a long topic to dig on and it's likely that not all deliveries will be done the same way (as you point out, there's some where it's ok to wait for a sync or where it doesn't have to support rushing the delivery, and others less so). But I have run some prototypes as part of those ISP discussions, and I'm confident this can make a major dent in the traffic load for popular downloads, even under the most challenging of requirements. It'll depend on the ISPs mostly, but things like good and well-tested multicast support in popular routers would ease their path somewhat.
Yeah, this is expensive to supply for real, and although I'm not privy to the details of how anyone in particular does things, I doubt there is anyone who supplies full 1-1 provisioning for home users at any scale. From what I hear, the really responsible high-end ones run at something like 3-8x. On the other side of the spectrum, I've seen anecdotal claims of at least one provider running at over 20,000x in at least one location:
It's one of those things that can't even be tested from the user side without coordinating between many users, so the more effectively the ISPs can squeeze on that number without getting busted for it, the more money they can make. The incentives are inherently broken here, so it's not too surprising to see this number start to get pushed on, even in places where it was good in the past. Multicast is of course more urgent for end user experience where these things are tending to the worse side, but I'd expect using it would be helpful on heavy traffic days even for the top 1% of well-provisioned consumer-grade service.
Yes, I think mcproxy also is there for openwrt, or at least it was a few years ago when I last looked. mcproxy has support for both v4 and v6, and at the time it seemed to me like sort of a newer and cleaner version of igmpproxy, plus I couldn't find a functioning mldproxy at the time.
But with that said, I don't know the history or latest status or have a complete view of the pros and cons, and I would be entirely satisfied with igmpproxy + mldproxy being on by default if anyone has a preference. I just want the functionality 
(That functionality being: downstream querier on lan + proxying to an upstream wan membership report, plus the snooping and the m2uc on the wifi nic, ideally for both igmpv3 and mldv2, and particularly including ssm support.)
A really good document on fountain codes was just given to me by mike luby, and it's on the starlink list.
Anyway, I wanted to share my own personal interest in multicast, and in extending the idea to home routers. I'm still, 30+ years later, trying to build "the jamaphone". Back in the 90s we had video/audio distributions boxes (analog!) that let us put scan-line cameras in each room of the studio, mix them all together, and let the band play as one, in each soundproofed room. This tech still exists, btw, but you kind of have to find it on ebay. The video quality wasn't all that hot, the audio was decent, and all i wanted, from starting to work on the internet since, was the ability to extend the concept across town.
With working multicast, and 240fps cameras (doing scan lines, damn it, no other encoding) (at i can't remember, 1024p?), it worked out to be about 250mbits per user,
for an overall perceptible latency of about 4-6ms including transport and encoding/decoding.
So I'd like to be able to multicast to 4 participants (either in a house or across town) for live, interactive, high quality music video and other forms of collaboration.
Dave that sounds like a really fun project but it's also probably possible now to do that with some kind of encoded video h264 or similar given the speed of processing we have now that should drop the total Network requirement to 5 megabits of send or something. Receive would be N times bigger.
The same basic principle applies to video games as well there's no reason why we shouldn't be able to join a collaborative video game with four or six or 10 users by simply having everyone join the same IPv6 multicast channel each player would send controls on a unicast channel to the world server and the game server would receive those and then the game server sends on a separate multicast channel the State updates and all the players immediately receive the game state without the server needing to send N world State updates. This would particularly be a big deal for massively multiplayer games where perhaps 1,000 players are in the world.
Yes, VR point-cloud video games would be pretty high volume, and if there are enough users sharing, this is one of the cases that might be worthwhile. I cite it sometimes as a future that will be easier to enable if we make generic multicast work, instead of the sort of 4g northbound interface handoff-to-the-isp approach to leveraging mbms (and other broadcast-capable media like cable/gpon). I mean yes, you can make that work for the 2 use cases they defined so far (streaming video or file download), and you could maybe extend the same functionality to non-cellular networks. But if you want to add more use cases, it takes a new northbound interface and corresponding client-side receivers that implement a standardized form for that specific use case, and that way lies madness. Making generic SSM work and leaving the use case to the application is a lot more flexible and a much better investment, is my opinion. (NB: not everyone agrees with me on this yet.)
Anyway, for stuff like realtime low latency music with low user counts, for the near (~8-12 year) future you're probably better off with a unicast mesh of peers, using hole-punching and some kind of low latency video and audio compression, plus hoping latency improves in the access networks. Multicast for that kind of low volume peer to peer communication is going to be harder to make happen than the broadcast-of-popular-content use cases that makes a dent in the traffic load for the ISPs, for sure.
But I do think a lot of people find that use case compelling and it might be a useful future direction, if the scaling issues for per-hop state requirements for routing get addressed well with more usage. Bier might ultimately help make it possible to enable some stuff on that front, but I think it's a longer journey on what's already a pretty long, hard, and uncertain road.
With that said, if you can make the home router into a BFIR that can discover the local bier domain's info and come up with a sane way (that doesn't need hop-by-hop state like PIM) to find the BFERs that are on or near the CMTS/ONT/DSLAM/etc. for your destinations, it might be possible to make it work pretty well without unaffordable overhead for the ISPs.
I think as things currently stand there are some substantial gaps for making that happen, some of which might be tricky. But it might be possible one day if ISPs start doing more with multicast and good vendor support for BIER gets deployed. It's definitely a nice dream to keep in mind. But my focus at the moment is about volumetric offload of shareable traffic, since that seems by far the easier win, tho I'm finding it much less trivial than I'd hoped.
which is heavily patented... so probably not an option for an open source implementation. Also I still have not figured out how much larger the data gets through this approach.
That would require to treat the whole file as one "block", I would have assumed that coding takes place in smaller blocks that then can be decoded and stored to disk in final form instead of having to store the encoded intermediary form first.
Yeah, I started looking into the few RFCs that address fountain codes, and that is clever, but my question is still how much larger does the data get by encoding.
Yeah, but scrubbing entails also jumping to arbitrary points in the future.... which for live events obviously is a non issue, but for time shifted programming it becomes an issue.
But that already works today... admittedly partly by cheating (using terrestrial or satelitte feeds of broadcast TV to feed the signal to different IPTV server locations which then feed the consumers via unicast or multicast, using infrastructure that already exists, but yeah, I see how multicast might offer ways to achieve similar effects in a pure-IP word, except as a TV station, I would certainly want to make sure these multicast streams get dedicated bandwidth, which I can not see ISPs being too hot on, unless they can monetize this, which then will need to be implemented in a way that is compatible with net neutrality rules in different jurisdictions).
As I said, I would have assumed that a competent broadcaster would use its existing broadcasting RF infrastructure to hop around the backbone issue and collect these feeds in multiple distributed data centers... as long as either terrestrial or satellites are still used to distribute that programming to viewers it can also be used as "feed-line" for multiple "conversion-to-IP" stations, no?
That said, my lack in interest in big sports events might color my evaluation here (and for things I do occasionally care for, like the recent chess WM I prefer commented replay videos that skip over all the long thinking pauses and that actually explain to amateurs like me why a specific move was clever or less so, but I digress).
Yeah that is what the local incumbent already does, resulting in funny failure modes where switching a channel appears to work at first (where unicast takes over) but fails a short while after if the switch to multicast goes pear-shaped, I guess these are typical teething/implementation issues that will get fixed though, so no show-stoppers (conceptually) even though the immediate show stops.
Within reason, your aggregate multicast traffic should stay well below what you would see with uni-cast to keep an incentive to use multicast, no?
Thanks for these pointers. Makes sense.
Ah, good to know.
Yeah, that is always a bit problematic, but I guess the potential to reduce traffic load should be enough incentive to get ISPs fully on-board.
Nobody does that, as it does not make any sense at all, given that the core of the internet is not scaled to handle the aggregate traffic of all of its leaf-nodes anyway. The "trick" is to scale things such that the probability of each user seeing its contracted speed is high enough.
Init7 is "special" in more than one regards.
And this is a case where, if the users do not notice this in observable slow-downs, it really is not a problem at all. My take on this is that this is an acceptable optimization for ISPS to make as long as they are willing to adjust the over-subscription rates per segment such that users continue to not notice
Germany just introduced regulations where customers can ether immedoately cancel their contracts (without additional payments) or can reduce their bills proportinally to any deviation of achievable rates compared to contracted rates (which needs to be documented using the official precudure and measurement system), so there are ways to counter the economic incentive for ISPs to overly increase the over-subscription levels.
It will require though to set aside some "bandwidth" for multicast streams given that they are not that tolerant to heavy losses?
Thanks for all that information and thoughtful responses, fun learning experience!
Funny how the same topic appeared at two places I visit at the same time
Mmmh, as single line (scanlines only really exist for rolling-shutter type cameras) will require
128038240 = 7,372,800 bits/second, at 1024 lines -> 1024128038240 = 7,549,747,200 we are talking 7 gigabit uncompressed.... a bit on the heavy side. Sure fur 38bit RGB is not the best format, but even with 8bit grayscale we only get down to 128018*240 = 2,457,600 ~ 2.5Gbps. Are there any compressors that do well on individual lines? And at 240Hz you either need bog sensors and lenses and/or loads of light.... Typically per frame MJPEG compression with acceptable latency should allow getting this down by a factor 10 which gets us to your 250Mbps target, but I do not see how that is going to work with your idea of sending line by line instead of full frames?
I am not sure whether any codec using temporal features is actually usable here, unless you restrict this to looking back in time?
Indeed one more example of a "real-time-sh" problem. I wonder however what currently limits the number of players per server instance the cost of sending individual packets or the actual processing of world state?
No, a block is say 16 bytes. The sender sends an infinite stream the receiver listens and grabs say 50 blocks in each packet (packets are about 800Bytes with a small overhead, let's say 820). if the file is N blocks you need N+2 usually. Packets have 50 blocks so already one extra packet is way more than enough. Let's say the file is 1GB then there are 1250001 packets needed to be received (instead of 1250000 without fountain coding)
Patents on the basic raptor codes expire in 2024 but we should get @Lynx on this. My understanding was that in the US "algorithms" were not patentable (let's face it in the US anything is patentable if you have enough money so the expiration date is probably the most important issue). There's also the possibility of inventing a new fountain code based on expired raptor parents.
Well let's say we want a really good experience. There are 1000 players. Each one wants to receive 120 packets per second and each packet is 500 bytes. Also the server should be hosted on one of the players networks (peer to peer rather than in a data center). That's 480Mbps of unicast uplink for the server hoster. Or with multicast, it's 480kbps. Now almost everyone on the internet can upload 480kbps but as of now only an elite few can upload 480Mbps.