ooh, thanks for this, and I'd be very happy to pick your brain a little more about global multicast. Should we throw up a separate offtopic type thread here? maybe others are interested in the question, but I don't want to distract this thread too much.
With this we all agree, my question is however how prevalent live video still is compared to non-linear video that let's the user scrub through the timeline? I honestly have no idea, and personally stopped linear broadcast TV more than a decade ago.
Here I can see how this helps IFF delivery on somebody else's schedule is acceptable. And yes, I agree that it could help to lessen the impact of stuff like multi-gigabyte "urgent" patches for popular games which according to NANOG will show up as noticeable load spikes. Here I wonder more how this is going to work, have one multicast session continuously cycle through the full data and download clients that can start any time and will stop once they have the full file re-assembled? And how to deal with lost segments, truly force the resents on the full multi-cast or do fix-ups per uni-cast (or force clients tp simply wait for another iteration hoping that the second time around it is not going to be the same segment that are corrupted or missing).
Yeah, but that explains why ISPs might be interested in doing that end-users typically assume that their contracted speed is available any time.
Yes I can see this now, thanks.
Germany's incumbent Deutsche Telekom uses multicast for its TV offering (apparently on top of IPv4 in spite of offering full dual-stack since many years now), and AVM being based in Germany might effortlessly explain this Fritzbox feature 
for OpenWrt I believe recipes exist for igmpproxy. What would be the advantage of mcproxy over igmpproxy?
I think Dave is fine especially if the discussion is actually going in the direction of proposing things that need better testing/deployment, like apparently multicast capabilities.
How does japan do earthquake alerts?
I am not sure how many here ever saw the original project plan for cerowrt 1:
https://www.bufferbloat.net/projects/make-wifi-fast/wiki/Solving_the_Home_Router_Disaster_Annotated/
fountain coding. So yes, the sender just continuously sends an infinitely long stream of bytes, and when the receiver receives any N+2 blocks they reassemble the whole thing. The name "fountain coding" is based on the idea that the sender is like a "water fountain" and the receiver just holds out a cup until it's full, doesn't care which molecules get into the cup vs overshoot or spray off on the side.
Nothing stops a client from storing local video that allows traversing the timeline anywhere behind the broadcast time, of course.
But regardless, live video has some important cases, like the most popular sports and news channels, where most of the viewers want it while it's live, and it simply couldn't be done if you tried to scale them to pure-internet delivery with the same audience size.
Especially sports. Broadcast TV commonly carries free access ad-supported sports events that turn out to be surprisingly exorbitant to get via internet, largely because the costs don't scale right for spiky unicast load (even leaving aside the access network congestion piece).
But I don't think the utility is limited to the things that just don't work at scale. Time-shifting of popular stuff for a cheaper delivery of anything along the lines of a popular series you're subscribed to would still be hugely worthwhile if multicast delivery worked. Also, the most popular vod (think Squid Game, Game of Thrones, Dr. Who) has a lot of natural overlap for when people are watching (shortly after the latest episode is released), and could use multicast to great effect by just running a few carousels in parallel. 12 streams shifted by 5 minutes on a loop means you could ship an hour-long video that people can watch on demand and get an average of over 95% delivered by multicast (you just get unicast until you're sync'd enough with one of the multicast feeds, and you only have to buffer a max of 5 minutes at the client, plus however much you want to supoort for trick play without having to re-sync). And of course you can tune that if 5 minutes isn't right for some reason.
Re: software download:
There's some specs about how to do this (FCAST (RFC 6968), FLUTE (RFC 6726), or more generally anything on top of ALC (RFC 5775), plus NORM (RFC 5740) for some things would be ok, or there's a less-documented uftp implementation that seems to work, plus a few other unpublished things). In general, it's possible to run this kind of stuff with some different tradeoffs, and you can always fall back to unicast range requests for hole-filling to cover loss if you don't have anything better.
There's some complexity and some design choices to make here, and it gets further complicated by the state of the IPR on Raptor, so it's kind of a long topic to dig on and it's likely that not all deliveries will be done the same way (as you point out, there's some where it's ok to wait for a sync or where it doesn't have to support rushing the delivery, and others less so). But I have run some prototypes as part of those ISP discussions, and I'm confident this can make a major dent in the traffic load for popular downloads, even under the most challenging of requirements. It'll depend on the ISPs mostly, but things like good and well-tested multicast support in popular routers would ease their path somewhat.
Yeah, this is expensive to supply for real, and although I'm not privy to the details of how anyone in particular does things, I doubt there is anyone who supplies full 1-1 provisioning for home users at any scale. From what I hear, the really responsible high-end ones run at something like 3-8x. On the other side of the spectrum, I've seen anecdotal claims of at least one provider running at over 20,000x in at least one location:
It's one of those things that can't even be tested from the user side without coordinating between many users, so the more effectively the ISPs can squeeze on that number without getting busted for it, the more money they can make. The incentives are inherently broken here, so it's not too surprising to see this number start to get pushed on, even in places where it was good in the past. Multicast is of course more urgent for end user experience where these things are tending to the worse side, but I'd expect using it would be helpful on heavy traffic days even for the top 1% of well-provisioned consumer-grade service.
Yes, I think mcproxy also is there for openwrt, or at least it was a few years ago when I last looked. mcproxy has support for both v4 and v6, and at the time it seemed to me like sort of a newer and cleaner version of igmpproxy, plus I couldn't find a functioning mldproxy at the time.
But with that said, I don't know the history or latest status or have a complete view of the pros and cons, and I would be entirely satisfied with igmpproxy + mldproxy being on by default if anyone has a preference. I just want the functionality 
(That functionality being: downstream querier on lan + proxying to an upstream wan membership report, plus the snooping and the m2uc on the wifi nic, ideally for both igmpv3 and mldv2, and particularly including ssm support.)
A really good document on fountain codes was just given to me by mike luby, and it's on the starlink list.
Anyway, I wanted to share my own personal interest in multicast, and in extending the idea to home routers. I'm still, 30+ years later, trying to build "the jamaphone". Back in the 90s we had video/audio distributions boxes (analog!) that let us put scan-line cameras in each room of the studio, mix them all together, and let the band play as one, in each soundproofed room. This tech still exists, btw, but you kind of have to find it on ebay. The video quality wasn't all that hot, the audio was decent, and all i wanted, from starting to work on the internet since, was the ability to extend the concept across town.
With working multicast, and 240fps cameras (doing scan lines, damn it, no other encoding) (at i can't remember, 1024p?), it worked out to be about 250mbits per user,
for an overall perceptible latency of about 4-6ms including transport and encoding/decoding.
So I'd like to be able to multicast to 4 participants (either in a house or across town) for live, interactive, high quality music video and other forms of collaboration.
Dave that sounds like a really fun project but it's also probably possible now to do that with some kind of encoded video h264 or similar given the speed of processing we have now that should drop the total Network requirement to 5 megabits of send or something. Receive would be N times bigger.
The same basic principle applies to video games as well there's no reason why we shouldn't be able to join a collaborative video game with four or six or 10 users by simply having everyone join the same IPv6 multicast channel each player would send controls on a unicast channel to the world server and the game server would receive those and then the game server sends on a separate multicast channel the State updates and all the players immediately receive the game state without the server needing to send N world State updates. This would particularly be a big deal for massively multiplayer games where perhaps 1,000 players are in the world.
Yes, VR point-cloud video games would be pretty high volume, and if there are enough users sharing, this is one of the cases that might be worthwhile. I cite it sometimes as a future that will be easier to enable if we make generic multicast work, instead of the sort of 4g northbound interface handoff-to-the-isp approach to leveraging mbms (and other broadcast-capable media like cable/gpon). I mean yes, you can make that work for the 2 use cases they defined so far (streaming video or file download), and you could maybe extend the same functionality to non-cellular networks. But if you want to add more use cases, it takes a new northbound interface and corresponding client-side receivers that implement a standardized form for that specific use case, and that way lies madness. Making generic SSM work and leaving the use case to the application is a lot more flexible and a much better investment, is my opinion. (NB: not everyone agrees with me on this yet.)
Anyway, for stuff like realtime low latency music with low user counts, for the near (~8-12 year) future you're probably better off with a unicast mesh of peers, using hole-punching and some kind of low latency video and audio compression, plus hoping latency improves in the access networks. Multicast for that kind of low volume peer to peer communication is going to be harder to make happen than the broadcast-of-popular-content use cases that makes a dent in the traffic load for the ISPs, for sure.
But I do think a lot of people find that use case compelling and it might be a useful future direction, if the scaling issues for per-hop state requirements for routing get addressed well with more usage. Bier might ultimately help make it possible to enable some stuff on that front, but I think it's a longer journey on what's already a pretty long, hard, and uncertain road.
With that said, if you can make the home router into a BFIR that can discover the local bier domain's info and come up with a sane way (that doesn't need hop-by-hop state like PIM) to find the BFERs that are on or near the CMTS/ONT/DSLAM/etc. for your destinations, it might be possible to make it work pretty well without unaffordable overhead for the ISPs.
I think as things currently stand there are some substantial gaps for making that happen, some of which might be tricky. But it might be possible one day if ISPs start doing more with multicast and good vendor support for BIER gets deployed. It's definitely a nice dream to keep in mind. But my focus at the moment is about volumetric offload of shareable traffic, since that seems by far the easier win, tho I'm finding it much less trivial than I'd hoped.
which is heavily patented... so probably not an option for an open source implementation. Also I still have not figured out how much larger the data gets through this approach.
That would require to treat the whole file as one "block", I would have assumed that coding takes place in smaller blocks that then can be decoded and stored to disk in final form instead of having to store the encoded intermediary form first.
Yeah, I started looking into the few RFCs that address fountain codes, and that is clever, but my question is still how much larger does the data get by encoding.
Yeah, but scrubbing entails also jumping to arbitrary points in the future.... which for live events obviously is a non issue, but for time shifted programming it becomes an issue.
But that already works today... admittedly partly by cheating (using terrestrial or satelitte feeds of broadcast TV to feed the signal to different IPTV server locations which then feed the consumers via unicast or multicast, using infrastructure that already exists, but yeah, I see how multicast might offer ways to achieve similar effects in a pure-IP word, except as a TV station, I would certainly want to make sure these multicast streams get dedicated bandwidth, which I can not see ISPs being too hot on, unless they can monetize this, which then will need to be implemented in a way that is compatible with net neutrality rules in different jurisdictions).
As I said, I would have assumed that a competent broadcaster would use its existing broadcasting RF infrastructure to hop around the backbone issue and collect these feeds in multiple distributed data centers... as long as either terrestrial or satellites are still used to distribute that programming to viewers it can also be used as "feed-line" for multiple "conversion-to-IP" stations, no?
That said, my lack in interest in big sports events might color my evaluation here (and for things I do occasionally care for, like the recent chess WM I prefer commented replay videos that skip over all the long thinking pauses and that actually explain to amateurs like me why a specific move was clever or less so, but I digress).
Yeah that is what the local incumbent already does, resulting in funny failure modes where switching a channel appears to work at first (where unicast takes over) but fails a short while after if the switch to multicast goes pear-shaped, I guess these are typical teething/implementation issues that will get fixed though, so no show-stoppers (conceptually) even though the immediate show stops.
Within reason, your aggregate multicast traffic should stay well below what you would see with uni-cast to keep an incentive to use multicast, no?
Thanks for these pointers. Makes sense.
Ah, good to know.
Yeah, that is always a bit problematic, but I guess the potential to reduce traffic load should be enough incentive to get ISPs fully on-board.
Nobody does that, as it does not make any sense at all, given that the core of the internet is not scaled to handle the aggregate traffic of all of its leaf-nodes anyway. The "trick" is to scale things such that the probability of each user seeing its contracted speed is high enough.
Init7 is "special" in more than one regards.
And this is a case where, if the users do not notice this in observable slow-downs, it really is not a problem at all. My take on this is that this is an acceptable optimization for ISPS to make as long as they are willing to adjust the over-subscription rates per segment such that users continue to not notice
Germany just introduced regulations where customers can ether immedoately cancel their contracts (without additional payments) or can reduce their bills proportinally to any deviation of achievable rates compared to contracted rates (which needs to be documented using the official precudure and measurement system), so there are ways to counter the economic incentive for ISPs to overly increase the over-subscription levels.
It will require though to set aside some "bandwidth" for multicast streams given that they are not that tolerant to heavy losses?
Thanks for all that information and thoughtful responses, fun learning experience!
Funny how the same topic appeared at two places I visit at the same time
Mmmh, as single line (scanlines only really exist for rolling-shutter type cameras) will require
128038240 = 7,372,800 bits/second, at 1024 lines -> 1024128038240 = 7,549,747,200 we are talking 7 gigabit uncompressed.... a bit on the heavy side. Sure fur 38bit RGB is not the best format, but even with 8bit grayscale we only get down to 128018*240 = 2,457,600 ~ 2.5Gbps. Are there any compressors that do well on individual lines? And at 240Hz you either need bog sensors and lenses and/or loads of light.... Typically per frame MJPEG compression with acceptable latency should allow getting this down by a factor 10 which gets us to your 250Mbps target, but I do not see how that is going to work with your idea of sending line by line instead of full frames?
I am not sure whether any codec using temporal features is actually usable here, unless you restrict this to looking back in time?
Indeed one more example of a "real-time-sh" problem. I wonder however what currently limits the number of players per server instance the cost of sending individual packets or the actual processing of world state?
No, a block is say 16 bytes. The sender sends an infinite stream the receiver listens and grabs say 50 blocks in each packet (packets are about 800Bytes with a small overhead, let's say 820). if the file is N blocks you need N+2 usually. Packets have 50 blocks so already one extra packet is way more than enough. Let's say the file is 1GB then there are 1250001 packets needed to be received (instead of 1250000 without fountain coding)
Patents on the basic raptor codes expire in 2024 but we should get @Lynx on this. My understanding was that in the US "algorithms" were not patentable (let's face it in the US anything is patentable if you have enough money so the expiration date is probably the most important issue). There's also the possibility of inventing a new fountain code based on expired raptor parents.
Well let's say we want a really good experience. There are 1000 players. Each one wants to receive 120 packets per second and each packet is 500 bytes. Also the server should be hosted on one of the players networks (peer to peer rather than in a data center). That's 480Mbps of unicast uplink for the server hoster. Or with multicast, it's 480kbps. Now almost everyone on the internet can upload 480kbps but as of now only an elite few can upload 480Mbps.
Algorithms are not patentable per se but that only means shifting what is claimed in a jurisdiction dependent way to, say, a computer readable medium having instructions that when executed by a processor or alternatively an apparatus comprising processor and memory, memory comprising instructions that when executed, etc.
See for example this pending US application:
A non-transitory machine readable, storage medium comprising machine executable instructions executable to assign a plurality of processes to a plurality of computational resources, each computational resource providing resource capacities in a plurality of processing dimensions, the instructions to assign comprising instructions executable to:
associate processing loads in each processing dimension with each process;
associate a loading metric with each process based on the processing loads in each processing dimension, and,
designate at least one undesignated computational resources from the plurality of computational resources for hosting unassigned processes from the plurality of processes;
assign in descending order of the loading metric one unassigned process from the plurality of processes to each one of the one or more designated computational resources; and
assign in ascending order of the loading metric any remaining unassigned processes from the plurality of processes to the one or more designated computational resources whilst there remains sufficient resource capacity in each of the plurality of processing dimensions.
Or this granted US patent:
An apparatus comprising:
a processor; and
a memory storing instructions executable by the processor to:
generate a first intrinsic mode function (IMF) by performing an empirical mode decomposition (EMD) on ambient light image data representative of ambient light conditions;
generate an ambient light color profile based on the first IMF, the ambient light color profile comprising an ambient light color coordinate in a color space of image data of an image to be displayed; and
color compensate for ambient light within the image to be displayed by subtracting a color component associated with the ambient light that is based on the generated ambient light color profile from a corresponding color component of the image data of the image to be displayed; and
cause the ambient light color compensated image to be displayed.
In many jurisdictions there are defences against infringement relating to private, non-commercial use. Are big companies going to care about a GitHub repository used by a few nerds? Probably not. Who to go after? But if that then goes into commercial products sold on mass - that is likely to attract more interest, at least against the company manufacturing and selling the infringing products.
1 Like
I can assure you it will not...
Well here is my question, are encoded symbols of the same size in bits as source symbols? So FEC traditionally requires redundant information that blows up the size a bit, where is the redundancy in fountain codes coming from?
I think that the patent holder will have made sure that that area of algorithms stays "mined"...
Could be, but how many of us have a 1000 friends they want to play with... I also wonder how resilient this whole thing could be... fountain codes are pretty much out as you do not have finite data, and more importantly if you can reconstruct a packet to far in the past it will not matter much anymore.
But the thing I am wondering about more in regards to multicast is, this requires routers and switches to play along, will operators of core routers and switches really allow folks like me to "reprogramm" part of their bid iron? This whole multicast discussion has been one great learning exercise for me, so I hope for another great explanation
My understanding is yes. Essentially there is a shared psuedorandom number generator. A packet has a little overhead in giving the seed, and then encoded blocks are some kind of xor of selected blocks from the source file plus optimizations for making things efficient.
If you get enough blocks you can reconstruct the whole file using math. But it doesn't matter which blocks, just enough of them.
Redundancy comes from the fact that you're sending an infinite bitstream to transmit a finite set of bytes. But you're doing that because you want anyone who starts and stops listening at any time to be able to reconstruct the whole thing.
Well, my kids play on Minecraft servers where 150000 people are online at any given time, of those they know... Probably 10 but they just play with whoever is in game now. You might argue this is data center stuff but a factor of 150000 reduction in bandwidth is interesting to data centers right?
Agreed that fountain codes solve a different problem. But I don't see how sending UDP multicast would be less resilient than sending 150000 UDP unicast?
Well I think carrot and stick would help. Carrot you get to offer vastly more services at vastly reduced traffic volumes. Stick is legislation requiring cooperation wrt multicast.
1 Like
If I only send one multicast stream, losses close to the root of the multicast tree will be much more devastating than if an occasional packet of an individual unicast stream is lost, no?
It's an interesting tradeoff I suppose. In some sense we are talking about enabling a whole host of things that just can't be done now. So in that sense we have 100% packet loss on those applications at the moment. Moving to a 1% packet loss would be a dramatic improvement 
1 Like
Yeah, in a sense for sure, but "playability" might suffer a bit... I would guess that big sports-ball kind of distribution events would require to somehow get bandwidth reservations from at least the lower branches of the tree otherwise it would be a bit risky.... Anyway, I am neither seeing myself distributing nation-wide sports event or news, nor running multi-dozen player real-time games, so I can relaxed sit on the side-lines and watch how this grows hoping for maybe a littles less congestion peering/transition points between ASs (something positive even for boring old unicast users 
).
This is one of the gaps we try to help address with CBACC. The idea is to give the ISP a way to know how much bandwidth each stream will take, so they can make an informed decision about whether to ingest and forward it and what to expect for its provisioning, and hopefully make the right decisions when coupled with the popularity within the ISP. But it's worth noting that it helps not only the stream being transported, but also the competing traffic because it reduces the network load. I think it's analogous to running public transport, a win for everyone by the traffic reduction it gives.
I do think there's some potential for network neutrality questions, but as long as it's not a content-based or provider-based decision and it operates transparently I don't see how it would fall afoul of net neutrality regulations. But with that said, I agree this is one point that worries me a bit if there is something I haven't caught here--sometimes if regulations are sloppy there can be unintended issues, and it's possible there is something here I don't know about, but as I understand it the European ISPs I spoke with did consider it as part of their due diligence, and seemed to think they expect the way we're proposing it (as a provider-agnostic standards-based ingest+forwarding decision based on improving overall network performance) to be ok, though this is all second-hand to me and I'm not sure I know all the considerations.
I should maybe note that the Init7 employee giving that presentation was the one reporting that he observed the high link-sharing setup in another (unnamed) provider's network at a shared colo of some sort, NOT the ones doing it themselves. In his presentation he claimed they didn't like to do business that way, and seemed to think it surprisingly and unreasonably high.
Of course your 100mbps internet subscription will never mean you get 100mbps for everything you try to fetch, as sharing of the internet path's forwarding resources among different users is expected behavior everywhere on the internet. But it's not so easy to pin down who is responsible when downloads or streaming is slow, and it does seem fair to say your ISP should have some kind of obligations here by taking your money. There's a big difference between a company that sells someone a 100mbps subscription and shares a 1gbps uplink among 20 people vs. one who does exactly the same thing but shares it among 1000 people. If they are selling these things at the same price, something is probably not right. Is the second abusing its customers? Is the first just running their business badly?
I guess it sounds like a noble effort to provide some regulatory relief against abuses here, but I'll start out as skeptical that it's going to work well, or that it will capture the over-sharing pressures effectively. The enforcement mechanism sounds kind of game-able, on both sides.
And on top of that, it will depend in an odd way on how much and how consistently your neighbors are using their internet. It's kind of odd if the answer to the question "is this a fair provisioning of the contracted service" changes if some app becomes popular that changes the usage pattern at scale (imagine for instance 60% of users start running something that does persistent off-peak downloading or p2p sharing of subscribed content, resulting in more overall usage). But something like that would impact the measurements you'd get. It sounds like a tricky problem, which is why I'd imagine it will have some trouble working well in practice. I guess what sounds better to me would be mandated transparency in your uplink sharing factors, so that as a consumer you have the ability to make an informed decision between available options if you care.
Interesting digression and interesting to hear about Germany's efforts here, thanks for pointing it out, I hadn't heard about it yet. I'll be interested to see how it goes.
Not really, but there are tradeoffs that make it a little complicated as to whether it's a good move, and it might be sometimes. It's pretty application-dependent, and different use cases with different protocols (and different operating profiles of the same protocols) also have different loss tolerance and different reactions to loss, so it's a little hard to generalize, but there are some considerations that can be articulated:
Unlike unicast, the sender would not typically back off in response to losses for most protocols, with some exceptions: NORM and RTP have some feedback channels here that might do so, but also are tunable from the sender side and need to avoid overreacting to individual receivers with bad connections. In most other protocols the receiver is supposed to react to bad enough losses by unsubscribing and using something else usually a lower-rate channel or an alternate unicast path (or in theory the gloriously broken approach that is described in RFC 3738), but a broken (or malicious) app might not do so, so you can't really rely on all subscribers being well-behaved either, and the network should therefore take steps to prevent having persistently oversubscribed links. So you might put a bandwidth cap on the amount of multicast that's lower than the link capacity, for instance.
But also, yes, for apps that repair loss in the multicast channel by using unicast, loss will result in a disproportionate impact that might be worth protecting against with reserved bandwidth (or even QoS prioritization of some sort) to improve the overall network performance, especially for the most popular streams. (The QoS observation actually made ISPs more worried about net neutrality in some cases--QoS prioritization seems like a touchy subject, with good cause I think. But although I do think it's probably a good approach for some cases, I also think it's optional, there are just consequences.)
Anyway, I don't think it's necessarily required to reserve bandwidth for multicast, but it might be a good idea for some cases, especially for sufficiently popular traffic.
1 Like