Internet Congestion Collapse - Computerphile 

Great TCP "dynamic window" explanation with a history lesson and explanation not only how it works but why we needed it in the first place. Much, much better than in CCNA books.

I love the way Dr Clegg explains this and interacts with his audience. Great teaching.

TCP congestion control is one of my favorite topics in the entire world. Even today we're still trying to modernize it, there's been some great work with BBR

These days, with 32 kbps you can have a very fine phone call, but that's how far lossy audio compression has come. Opus codec for the win :-)

I remember reading - and re-reading - that paper in grad school. Truly a seminal paper and an EXCELLENT job of describing it!

I had the opportunity to work with Mike Karels (one of the coauthors of that paper) a few years back. He's a great role model... humble and eager to help out the less-experienced programmers (like me). He'd probably be happy to speak with the Computerphile team, if they wanted to connect.

Thank you for discussing this classic and important paper -- we need periodic reminders on these issues, as the problems keep re-appearing when the long-ago solutions are forgotten. Another I'd like to see (from the dark ages) is Denning's Working Set Model for Program Behaviour from the late 60's, when computers were still huge beasts that remembered using little donuts made of rust.

The early days of the internet were so fascinating! I was born in '83 and got my first computer around 1995, it had a Pentium CPU clocked at a whooping 133 MHz. I remember dialing into BBS boards and accessing the early internet via AOL and getting kicked off the internet because someone picked up the phone and buying my first 1GB 5.25" hard drive thinking there's no way I'd ever be able to fill that much space 😂

I've been learning about this in class this quarter, it's such a cool concept. Currently, our class is working on a project to make UDP reliable with congestion control too. So awesome to see a Computerphile video on it as well! Thank you Dr. Clegg!

Van had to have his arm twisted to publish because journals didn't want articles that cited an email. Now it is one of the most cited articles in internet research.

I remember Van Jacobson header compression in PPP configurations.

Wonderful explanation. I've wondered in the early dayes, how they cept IT working

Thanks for the nice explanation. Could you also do a video on quic and how that handles congestion differently?

Thanks for the Greta video! I have seen my network bandwidth throttle in task manager in a similar way to the curve described and I always wondered why.

What I find interesting is that in addition to a missing packet, sometimes there are errors in the received packet. Sometimes these errors can be corrected, and sometimes they cannot. When they cannot, the packet needs to be resent. This factor was not mentioned.

I like this guy, he's got a jazzy manner of communicating. There's pep to it.

Great video. Thanks.

Would love to learn a bit about BBR, a proposed replacement for this algorithm!

I really love how he talks and explains

My understanding is that there is a problem with slow start vis-a-vis web browsing. Slow start works great for connections which are open for a long time--the delay caused by slow start isn't noticeable. But web browsing usually involves lots of short connections for which slow start is a significant problem.

I've actually had worse speed tests when my only ISP option was Comcast. I would generally only able to get 37bps with 75% packet loss on top of that. And that was in 2012-2013 in the DC metro area!

In my bit of Norfolk, sometimes, 40bps would seem good.

Can you do a further video on the unintended effects of this congestion avoidance, such as TCP synchronisation, and how this can be avoided?

It's a very good video. Do a follow up with "silly window syndrome" that will be fun.

Can you please do a video about when Covid started a lot of areas ISPS in US and EU region hit prime time issues. Like between 5PM to 11PM areas bandwidth would drop in download speed. This was a huge issue but no one really talked about it.

8:07 that is how the old xmodem protocol worked, isn't it? yes, i did some BBS stuff back in those days, first using an acoustic coupler (300 bps, yawn), later using a series of increasingly fast modems... man, this brings back memories...

NIce - I had assumed that back off was in the original implementation. Have you done a video on the modern version of this problem -- buffer bloat?

Interesting, should do one on like CSMA/CD, CSMA/CA. I would love to see one on Radia perlamand and spanning tree.

Will be using in my college Data Communications class as supplement.

That was so good! I have a question: does these tricks are today implemented automatically by a network protocol or are those a set of practices that the ISP needs to configure manually?

Thanks for the video! Just wondering if collision avoidance is still in use today and if congestion is still a common problem which results in packet loss?

Can we have a description of ECN (Explicit Congestion Notification) protocol in one of future videos?

I'm glad he did correctly point out that the TCP window is based on bytes not packets as he kept saying but disappointed it took 15 minutes

I have a question for Dr Richard Clegg. What would cause packet loss on a network that increases and decreases in a almost prefect sign wave with peaks every 10 mins ( around 7% loss ). I believe its a buffer that is filling up on the main router causing it. The clients in turn reduces the TCP frame size to a point where the speed on the network drops and the router can then handle it. The frame size then slowly increases again and the problem repeats.

i think brady has osmosis'd a computer science degree at this point, his suggestions are almost always on point

Yesterday my internet just worked - now whatever I do I will know that the packet rate is modulated by congestion control. Well, I think that's worth knowing.

Love these vids and ths one inparticular. Try running it at 1.25 times speed for optimal effect !!! ;)

please more networking videos!!

Im so glad you cut away to those shots of yourself drinking tea and nodding /s

I'm wondering, why couldn't the intermediate nodes on the network look at the window header, compare against the current state of its own receiver buffer, and then modify the window to be the lesser of the two values? This assumes the bandwidth is limited by the overall bandwidth of all the receiver buffers in the network. That seems like a more deterministic approach than this kind of guess and check. Edit: I realize why this wouldn't work. The traffic speed would then be limited to the max speed of the slowest intermediate node.

1:04 Nobody would _want_ to make a phone-call through the Internet in 1995, they'd just use a normal phone. The Internet was still for transferring files, which were still relatively small at the time (though I distinctly remember staying up all night a few times, trying to download files 😕).

That graph with steady ups and sharp downs reminds me of the bitrate graph I get when I copy a large number of files from one location to another, especially if the destination lies on an external drive. Is it possible that the OS is using somewhat similar congestion contention algorithms to allow the use of shared IO resources by many processes in parallel at the maximum possible rate?

That iis bug around 2004 degraded the internet a lot too

love it 😀

So when I'm downloading a large file, the steady state is actually the sender is constantly ramping up then halving its send rate, "bouncing off the limiter" as it were? The rate looks so much more steady on my end..

This seems to rely on cooperation of all endpoints on the network to implement the same multiplicative decrease. Are there additional built in protections against a "bad actor" who tries to keep sending packets as close to the bandwidth limit as possible?

Is this window size calculated per remote host:port? If not, wouldn't firewalls that drop packets play havoc with the algorithm?

Can you do a video on how TCP/IP work please Dr.

I now wonder if this might be the fatal problem with the current philosophy that us ISPs have with their cost benefit analysis where the packet drops are more to do with bad route choice on a given bundle for a certain bandwidth and less to do with the total amount of bandwidth available. I feel like at some point that the faster you confirm receipt the less congestion you have and the more reliable the connections become

I guess that this only works if every TCP implementation honours it - what if one implementation chooses to be greedy? How do you protect against that?

The internet is represented as a cloud because it is nebulous; You know somewhat what's going on in there, but it's such a jumble it's next to impossible to map it. 😉 You can pull a cable out of a server, but the 'traffic controllers' won't know immediately that that server is no longer available, and that principle applies to those controllers between themselves as well, of course. So in a very real sense it's like looking into a cloud or mist, but the several waves to indicate a mist could be confused as 'somewhere between; 1~2', an unary programming symbol, a flow or a volume. Different eyes look differently at such symbols so a cloud makes the most sense.

Ack

what if it’s an ack that get lost? did i miss it?

Doesn't the "Cloud" come from the military pre-electronics. When a battle started, amongst all the smoke from poor gun powder a message was sent via carrier pigeon.

So does it mean if I wanna enjoy the bandwidth I was promised by ISP, I should go shut down neighbors' network for a larger congestion windows right? :)

I've seen my own internet drop as low as 40 bps, for hours at a time. But that's Australian broadband for you.

Why not just start with all packets and move straight into the halving after... assuming there was missing acknowledgements that is, as for the data itself rather than (if I remember rightly that is) resending everything in the event of a fail, just send those that were missed, it's easy to setup after all as you can just do something like this with linux based server: Step 1: launch a dedicated process for the connection Step 2: process creates folder ~/connections// Step 3: process create all packets expected to be sent in said folder naming them .packet (since process memory might not be enough) Step 4: process sets up an internal buffer of values to hold the acknowledgement state of each packet Step 5: process creates a pool of threads for as many packets as can be handled Step 6: each thread sends it's own packet & waits on acknowledgement, setting it's acknowledgement value to 1 if it does get acknowledged, leaving it if it doesn't Step 7: main thread determines if any packets need to be resent or still need to be sent and re-uses the pooled threads for those packets while doing the mentioned congestion control (in other words loop back to step 6 until all have been acknowledged or some abandon condition is triggered) Step 8: empty the folder of every *.packet file and exit Step 9: server re-uses the POOL_ID for the next connection it is handling With the above it is not necessary to re-generate the packets, just send what is not acknowledged over and over until the acknowledgement is retrieved or abandon it midway for whatever reason, it's the clients job to put them in the right order using the sequences after all

I came to this video because I've been researching ways to bring down data usage. In my country South Africa data is really expensive and the infrastructure is really poor.

Briliiant

Good thing all the implementations out there adhere to this, right? This pretty much relies on the cooperation of the network users. Luckily, I guess, if there’s only a small number of jerks on the network, things will continue to work.

Bandwith of the classic phone was about 3KHz, that is it needed 3kbps. 32 k would allow 10 phones calls simultaneously. That is why ADSL was working: phone only needed a very small portion of the bandwidth that could be carried on a copper twisted pair.

Don't look at my left ear the whole time 🤣

I watched at 125%

My internet feels like it collapses daily

At this point I'm starting to feel like most of the data on internet transfer is all kinds of protocols, keys, encryption and notes about the package rather than the actual data. Bernoulli's law would imply that the packages would flow faster in the bottleneck!

FECN and BECN.

You can see that graph in action on steam during the initial release of an AAA game

Syn Ack!!!!!

Yeah?

My internet speed right now isn't better than internet speed 40 years ago, suck.

It's faster than rfc 1149

faster details pleease

This is a low bandwidth presentation

What?

really not happy with the quality of this video -- any chance we can get another pass on this?

40 bps? Wow. I got way better than that on FidoNet years earlier over phone modems.

"fork handles"🕯🕯🕯🕯?

I'd tell you a TCP joke, I'm sure you'll get it. Are you ready to hear it?

The person who edited this video could've maybe put in some effort to remove the annoying background noise 😅

Best at 1.25 speed.

akash

1024th

Who's here from Hussein Nasser's channel?

omg! back to the stupid 132 column printer! Will you ever stop?

wornderful vid, if you want to use your full brain bandwidth play the video at 1.75 :)

25th

those laptops looks SUS

sixty ninth

First time ever I used the 1.5x speed button on RU-vid. He speaks so sloooooooooooooooooooooooooooooooow and seems to micro-nap between sentences losing orientation. His bandwidth certainly is below 40bps.

sixth

Second

CNN is reporting that Al Gore wrote the paper on TCP/IP and congestion.

Fifth

Not an uninteresting video but for my taste well too long winded.. basicly 16 min of pramble in a 20 minute video....

First

This could have been better explained in 5 mins with no protocol details

A functionally illiterate Doctor working at a University! How standards have avalanched. Sad.