Netflix doesn't take it up, paying broadband customers are all requesting individual content that happens to be at Netflix due to the customer benefits.
Netflix just has some files on a server and streams content. Netflix isn't broadcasting but customers are pulling that content. The consumers/customers of Netflix take up 9.5% upstream, granted improvements can probably be made in their software.
There is a disparity how online entities are seen as bad due to taking too much interest from customers where TV numbers don't include bandwidth usage over broadband or spectrum usage and having x number of viewers is seen as good.
This just means Netflix is a popular and big channel or land of content that people spend lots of time in as paying broadband customers using their bandwidth and content customers paying Netflix for access to content. So this can't really be used to justify broadband companies lobbying.
There does need to be improvements to broadcasting on the web which was explored with multicast and maybe SCTP will be more interesting soon as that could help with the TCP problems.
> Netflix doesn't take it up, paying broadband customers are all requesting individual content that happens to be at Netflix due to the customer benefits.
Consider that "Netflix" can refer not only to the company (the interpretation you're using), but also to the service itself.
the newest report has Netflix at 9.48 percent of upstream and 34.89 percent of downstream [...] The average Internet subscribing household in North America uploaded 7.6GB of data per month in Sandvine's report in the first half of this year, increasing to 8.5GB in the latest report, a boost of 11.8 percent. The average household's monthly downloads increased from 43.8GB to 48.9GB, a boost of 11.6 percent.
So the average "Internet subscribing household in North America" downloads 48.9GB * 34.89% = 17.1 GB from Netflix, and uploads 8.5GB * 9.48% = 0.8 GB. If the upstream is all 40 byte ACKs, that's one ACK per 850 bytes downloaded, which is a bit more ACKing than I expected, but not completely unreasonable.
This seems high and would seem to amount to an ACK for almost every second downstream packet. Since I pcap ALL THE THINGS, I looked at some Netflix streaming sessions from a 3 different clients over a week.
Stream 1: 43MB down, 21KB up
Stream 2: 56MB down, 28KB up
Stream 3: 126MB down, 62KB up
That's using Wireshark to count the bytes in the TCP conversation.
Don't forget the retransmissions due to the under-provisioned links to Level3 networks by some companies who are currently (or have in the past) extorted money out of Netflix.
I'm sure you are aware of this, but the average here seems missleading. The median household has never Torrented anything, therefore I think that the median would be more representative of the average user's habits.
yeah but by averaging the upload and averaging the download and just finding the ratio of the two he's not really substantially affecting his argument vs. using the median, right?
The article lists the reason for the increase but stops short of putting 2+2 together.
The #1 source of upstream traffic - Bittorrent - has significantly decreased their peak-hour upstream (shifting it to other hours) as described in the article. This rather obviously means that you can go from 7.7% to 9.5% share without any significant changes to Netflix as such, simply because the total peak-hour upstream rate is now slightly lower.
All this statistic tells you is that upstream bandwidth is ridiculously constrained compared to downstream bandwidth, when TCP ACKs can fill it up to that degree.
Asymmetric bandwidth isn't particularly high on the list of priorities when it comes to the terrible state of the ISP industry in America. It's just "one more thing".
No, all this tells you is that there's not much upstream traffic in general. The amount of traffic could be constrained by available bandwidth, or it could be vastly overprovisioned and there's just no demand for it.
TCP deals with congestion and bandwidth allocation automatically. Decades of algorithms work in your favor if you use TCP.
A single missing packet will seriously mess up your video stream. Remember that long sequences of movie pictures are sent as a single whole image plus delta instructions to generate the rest of the images. A single bad packet can screw up a second of video.
There's a good explanation in http://www.dtc.umn.edu/~odlyzko/doc/net.neutrality.delusions... (2008) section 3 which explains "why faster-than-real-time progressive downloads of music or video are far preferable to real-time streaming. But first let us consider the strange situation in which this issue is not discussed publicly, and the advocates of each of the two types of video transmission mostly seem unaware there is a real alternative to their preferred solution, and that there is a serious decision that has to be made for each video service."
A few missing packets may not matter, you're right. MPEG has a lot of error checking and error handling built into the transport stream layer.
But I think the bigger reason Netflix uses TCP is because TCP has a much easier time with NAT traversal and not needing port forwarding. By looking and acting like normal web traffic it makes it easier to guarantee that things will just work rather than need any special accommodation.
UDP does not give any significant benefits to TCP when latency is not as important (e.g. playing previously recorded movies as opposed live streaming).
In fact UDP makes things much harder, for example unlike TCP with UDP you need to worry about the rate at which you're sending the data. Too slow and you're not utilizing the full bandwidth, too fast and packets are getting dropped because the client can't receive it fast enough.
If you don't have a real good reason for UDP you should stick with TCP otherwise be prepared to re-implement your own congestion/rate control, and unless you're familiar with the subject it might be a challenge.
IMO an good example of unnecessarily using UDP is Etsy's statsd. Their argument is that it's ok when data is lost so no need for TCP. Unfortunately with their approach when there's enough of traffic on the LAN to cause router to drop packets, there's not much traffic left for other data as well. If they instead would use TCP with (optionally) low send/receive buffer and use O_NONBLOCK socket. When sending data if send() returns EAGAIN, simply ignore it. This approach still causes data being dropped when there's too much of it, but at the same time it minimizes amount of data in the network and plays nice with other services sharing the same routers.
TCP is more efficient than UDP if you don't need low-latency delivery, because you can compress more data at once and waiting for retransmitted packets isn't a problem. This also ensures you get a high quality stream through the viewing experience unless there are major congestion problems.
A video chat will always prefer UDP transmission for the lowest latency possible, but quality may vary and bandwidth usage can technically be higher.
"TCP is more efficient than UDP if you don't need low-latency delivery, because you can compress more data at once and waiting for retransmitted packets isn't a problem."
Would you care to elaborate what you mean by any of those things? I happen to have some background in protocols but I honestly can't make sense of any part of that sentence.
If you know that the receiver is going to receive all of the data that you send (or at least a lot of it at once, for example if you have a 5-second buffer), then you can more efficiently compress all of the media being sent.
Typically, a UDP-based video stream will have less dependency among the packets, so that even if a packet is dropped, the next packet is still decodable. This means that there is more redundant information per-packet as a durability measure.
UDP-based streams also often have something called forward error correction (FEC). This is where you encode lower quality versions of media samples in subsequent packets. Again this is trading more bandwidth for realtime durability. If you miss a packet, then the next packet probably has the same media sample in a lower quality, and your 100ms jitter buffer gives you just enough time to make use of it. This is far more time-efficient than requiring the receiver to ask the sender to retransmit a missed packet.
To the point about retransmission: in UDP cases, it's often not worth bothering about. By the time you round-trip the request, you've probably already received more recent media.
I should say that this isn't about TCP vs UDP so much as it is about buffered vs realtime streaming.
"If you know that the receiver is going to receive all of the data that you send (or at least a lot of it at once, for example if you have a 5-second buffer), then you can more efficiently compress all of the media being sent."
That has nothing to do with the underlying transport.
"Typically, a UDP-based video stream will have...""UDP-based streams also often have something...""in UDP cases, it's often not worth bothering about... "
Here you're describing a bunch of possible properties a hypothetical protocol written on top of unreliable datagrams would/should have.
"I should say that this isn't about TCP vs UDP so much as..."
This part is correct. You should keep this part.
"..it is about buffered vs realtime streaming."
Right. Let's not conflate transport protocols with media streaming methodologies please.
Every site streaming HTML5 video is using TCP including YouTube. Traffic engineering appears to have reduced the argument for UDP for media in recent years, which is a good thing!
In addition to the fact that even a single missing packet is actually a "big deal" for compressed video streams, the assumption that the number of missing packets would be "only a few" is erroneous. I've seen situations where packetloss was as high as 60%, and you wouldn't want your video stream to turn to garbage or have your service and app spending all their efforts dealing with that rather than letting equipment just do its job.
Could this be a plan for Netflix to even out the upstream & downstream traffic? Upstream traffic is significantly more expensive for ISPs to absorb (esp at the last mile). ISPs are demanding payment because they claim they're absorbing more traffic than they're sending back [1]. Netflix could just update their client to saturate upstream traffic while maintaining downstream traffic.
From the article: "But for some reason, its share of uploads went up substantially in the latest measurement while downloads remained level."
[1] - http://arstechnica.com/information-technology/2014/07/how-co...Tier 1 networks have generally exchanged traffic with each other without payment if they send and receive roughly equal amounts of data. Schaeffer has claimed it's impossible for Comcast to be "in balance" with any other network because its own Internet service is designed for consumers to download far more than they upload
Could this be a plan for Netflix to even out the upstream traffic & downstream traffic?
No, because that would not help them.
A lot of ISPs are demanding payment because they claim they're absorbing more traffic than they're sending back.
This is just a misdirection tactic from ISPs who have no intention of offering free peering to Netflix under any circumstances. Backbone peering agreements are irrelevant since neither Netflix nor broadband ISPs are backbones.
So if Netflix makes up 9.48% of upstream and 32.39% of downstream, that means for each 10 packets sent, 3 ACK packets are sent back (I know those figures don't represent raw traffic). Does that seem a little odd? I'm not asking as a basis for some conspiracy, I'm more so wondering what that implies for data services I'm working on.
Your math would be meaningful if the total upstream volume would be the same as downstream volume, which is very much not the case. The percentages mostly depend on how much upstream traffic is (or isn't) generated by other sources.
That doesn't make sense; There isn't one pool of "upload" data and another of "download" data. Rather, there is one pool of data streaming from edges into the core (upload) and back out again (download). You can either attribute each chunk of data to its uploader or to its downloader, but in either case, the pool is the same size.
Edit: or these metrics don't mean what I think they mean, in which case, please tell me!
Youtube has far shorter videos (5-15 minutes on average), defaults to 480p or less most of the time and the visitor session length is much less than the average Netflix stream.
Netflix is doing 720p / 1080p defaults on most players, 1 hour to who-knows-how-long non-stop sessions (I personally have binge-watched like half a season of TV shows in one sitting occasionally) and gets daily recurring traffic from multiple members per household.
Right, what I meant though was YouTube let's anyone upload videos. That ingestion must account for a lot more than Netflix's TCP ACKs considering 100 hours of video are uploaded every minute to YouTube [1].
Anyone know what the equivalent upstream/downstream stats would be for all of cable/traditional tv companies? Most of these articles show how large Netflix is now - I'm curious how large they'll get.
With switched video [1], there is some upstream, as the set-top box requests a channel, and may need to send some sort of heartbeat. For video on demand, there's some amount of upstream for program selection and control as well. But that should be extremely low bandwidth, just control information, not acks for content (although, sending quality feedback to the headend could be very useful to solve delivery issues before customers notice)
It probably would have been clearer if you just said P2P. Considering Comcast's low upstream bandwidth, I don't think P2P is worthwhile. And of course Comcast would find a way to thwart it.
LOL Level 3 is not the problem here. In fact they are on the side of Netflix.
The regional ISPs are making big fuss, because until now they were sending about the same amount of traffic to Level3 as they were receiving, which allowed them to not charge each other for the traffic. As now there's more traffic coming from Level3 than goes the other way. As per their agreements would make them pay to Level 3 for the difference.
Because of that they prefer to simply throttle incoming traffic from L3, despite the fact that it is the traffic requested by their own users, they also refuse to peer directly with Netflix because Netflix also competes with their services as a cable companies.
I realize that Level3 is on the same side as Netflix. The question was can Netflix play the same game the regional ISPs are playing. Let's assume Netflix acquires Level3. Since Level 3 is the backbone for a large portion of the traffic in this country, some Comcast data would have to go through Level3 to reach their customers right? So if Comcast throttles Netflix traffic, then Level3(now Netflix) just blocks Comcast traffic at the "backbone level". The question is theoretical. I know this is really bad for the end consumer.
You can't do that. If you are an internet middle man your job is to deliver traffic from one network to another. If you're failing at it then you're failing at your job and quickly someone else will take your place and you get out of business.
On top of that there are peering agreements which might also mention against that.
What's happening here is that ISPs such as Comcast already purposefully hurt their own consumers by providing sub-par service. No one else would normally care about it but to Netflix that's large number of their own customers and they're willing to bend backwards to be able to stream to them.
The problem is that there's really no competition so either we should remove laws and allow once again for competition, provide competition for example by letting cities provide Internet as well or heavy regulation (starting with title 2/common carrier) of existing monopolies, similarly how we do this to other utilities such as electricity/water/gas etc.
I don't see how that would solve anything. Comcast was willing to hurt all Level 3, Cogent, and Tata customers just to extort Netflix. There is no "hack" to solve this problem.
Netflix just has some files on a server and streams content. Netflix isn't broadcasting but customers are pulling that content. The consumers/customers of Netflix take up 9.5% upstream, granted improvements can probably be made in their software.
There is a disparity how online entities are seen as bad due to taking too much interest from customers where TV numbers don't include bandwidth usage over broadband or spectrum usage and having x number of viewers is seen as good.
This just means Netflix is a popular and big channel or land of content that people spend lots of time in as paying broadband customers using their bandwidth and content customers paying Netflix for access to content. So this can't really be used to justify broadband companies lobbying.
There does need to be improvements to broadcasting on the web which was explored with multicast and maybe SCTP will be more interesting soon as that could help with the TCP problems.