So just how did a train end up hanging off the end of a viaduct in Millwall in 1987? Time for a closer look. The Island Gardens video: • Island Gardens and Nor... Patreon: / jagohazzard Ko-Fi: ko-fi.com/jago...
in Post Office engineering it was accepted nothing is foolproof and everything breaks eventually, today electronics fail more frequently, yet subscribers expectation is for greater longevity and reliability
I need to comment on the Microsoft Developer Channel. I had MS Exchange I think on my PC I was using to process my emails. It had an option to change the Inbox, etc , location, So I moved all the .?bx files to the new location, then told MS Exchange where the new location was. Instead of changing its internal pointers it created new (without checking) inboxes and so on, in the new drive/folder overwriting the data in the filenames (meaning a recovery program would not work ), with null new files, then , because I had moved the files beforehand, could not populate the new location with the original data (and yes , I had not made a back up, AND my ISP had managed to loose all the mailbox information too owing to an 'upgrade' the same weekend, AND my spare PC - at my in laws, developed a physical hardware fault on the disk drive so windows would not load (normally I would have brought the drive back by the in-laws took it in to be repaired - the company replaced the hard drive, but did not look to see it had been physically and logically partioned and did not copy over the DATA section of the disk , which was OK, and binned the disc.
Dear Customers! We are happy to announce that Fool V2.0 is now available for all users. Furthermore, we can also reveal at this point that we have begun working on Fool V2.5, as we are sure it will be a necessary upgrade in the near future.
I believe it was Douglas Adams who said “A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools.” I left the IT industry after 25 years: too many fools. The only problem now is to find someone who wants a 55 year old intern. 😀
If I miss anything about London life, it's the Driving the DLR game. The art is to combine nonchalance and apparent disinterest with extreme aggression and ruthlessness.
Reminds me of the time when I saw a dad with a small child pushing his way to the front seat. "Let's pretend we're driving", he said and then started making noises and moving invisible levers and controls with his hands. "Isn't this fun", he exclaimed to the completely disinterested child. It was obvious to all present that the child was merely a prop to let him indulge publically in his train driving fantasies without, in his mind at least, being regarded as an overgrown schoolkid.
Yep... If you are testing something that wasn't working you want it to work. If you're testing something was working you want it to fail. Admittedly that's stretching the definition of the word "want" way beyond any reasonable breaking point but it is effectively the whole point of testing... To be proven wrong if you are.
As someone whose day job is in software...as soon as you said the word "asynchronous" I immediately thought "oh no, it's going to be a race condition or timing bug isn't it?" and I wasn't far off. Awesome video my man. Always love your microhistories but this might be your best yet!
Reliant on three unsynchronized computers, and 0.2 second safety margin - it's like saying of course we can keep the marches in the same box as the fireworks - what can possibly go wrong...
And here is why you are a great youtuber: you are willing to correct yourself if the new information comes from a valid source. Great job on claryfing this.
I once had the keys after one of the staff dropped them "Just what I've always wanted, the keys to my own DLR train!" I exclaimed then gave them back.😂
Big respect to Jim for coming forward and lending you his expertise on the subject. It really is wonderful that your audience has within it, some very experienced people in the subjects that you will sometimes cover. I have to commend you for making this video. I know a lot of other people wouldn't have done the same thing. It's really interesting to see what really happened with it All very well done
A word in your shell-like: Be aware that "lighthouse keeping" is an entirely different job from "light housekeeping." I almost became a light housekeeper through that mistake.
After the crash: "Are we in trouble now? we did an unauthorized test and crashed one of the trains but we also found a huge safety issue... so are we heros or are we about to lose our jobs and possibly get sued?"
I know the SecOps team at my job would come over with a big bag of candy if anyone caught a safety issue like that. /or: they'd be amazed how anything we did could ever create a safety issue like that considering it's backend stuff for websites.
@@GustavSvard It's certainly possible depending on the data the site is processing, internet services are certainly used by other systems and personnel to make decisions that are safety-critical. An error in the information provided by those sorts of systems could lead to fatal errors.
As a software engineer of many years I can see exactly how this happens, great explanation. This would all be virtually modelled these days, a bit extreme to test an edge case in the real world though!
Or at least test it with plenty of safety track instead of the end of the line behind it. The test would be something like going at overspeed and checking how long it takes to stop the train. Unfortunately I wish I could say that people would understand that 1+1+1>2.7 and that a design like that is not good enough, but I expect that kind of problem exists in many newer systems as well.
The idea of a fixed *one second* loop seems heinously long to me here in the 21st century. But I’ve never had to design absolutely-cannot-fail systems like these, so maybe it’s less ridiculous than it sounds at first blush.
@@gormster Fixed timing loops are in fact fairly common in safety critical systems, since it makes it much easier to formally analyse the design for correctness and worst response time. Pity that it wasn't done here.
There is a school of thought which considers a test which achieves its normal output for normal inputs to be a failure, and only those from which you learn something new to be a success. I see where they're coming from, though I don't entirely agree with it as a position. Regression tests, for example, are meant to check for normal results given normal inputs even after some change has been made in the system.
@@clickrick There are many different types of tests, with different parameters. One is a big machine pulling at bolts to snap them. The bolt has a range of strength for its application. If it snaps well above the specified strength, then the test is a _success._ Open ended test(s) are different. Test something to see how it performs way above its specification or intended use, then analyse. In this train application, a test would be to saturate the control system to see how it performs. The tester would regard a _failure_ of the equipment as a _success._ He found a problem.
@@johnburns4017 And safety margins are very important parts of a test. Therefore the test was a failure. If they had performed the test one station back, then the outcome would have been a success. Same learnings, no embarrassment, no extra costs and no iconic images!
But was it wise to test a new train going at a faster than normal speed with a new computer system, on a line that actually did end 30 feet up in the air? Couldn't they have tested how well a train stopped on a piece of track that continued, just in case the train in fact didn't.
Well, the topology of the track is already inputted in the computer, tho. So the testing at the actual end of the track also tests whether the topology of the track is inputted correctly. Changing the end point removes this part of the equation. They indeed COULD have figured this exact bug on some other station tho, I must say.
Yes... though at some stage a "real" test would still have been a good idea. One where there is no artificial test harness, no possibility whatsoever that the test conditions didn't reflect reality for some unforeseen reason. Admittedly there might have been better choices of termini, for example one not built on a viaduct...
At some point testing like this would still be wise, for example, if the data at this station is incorrect or some other unique issue comes up as a result of it being the end of the line.
But surely, even if a test for Site A is undertaken at Site B, and passes with flying colours every time, it still won't prove that it'll pass at Site A until and unless it's tested at Site A. Apparently trifling variations in elevation, construction, angles and so on could make all the difference.
@@erejnion I think that after this testing failure they were able to re-use the whole crew & even, apparently, most of the train. I'm not sure that would have been true if they'd done the test somewhere where the train would have hit a station building if it overran the buffers... although maybe they _were_ testing in every location that an emergency automated stop might be required & it's just luck that the failure occurred where there was nothing solid to hit?
Boeing used to make trains. The Boeing LRV also called US Standard Light Rail Vehicle. They where so heavy, noisy, frequently derailed, broke down so often there the mean time between failures was so short that often only half of the rolling stock was operational. They where so bad that Boeing couldn't produce enough replacement parts for them. So in Boston they started to cannibalize some of the cars shortly after they went into operation just to get enough replacement parts to keep the remaining cars operational. But here is a funny quote from the Wikipedia Article: "Before they were delivered, Boeing claimed the US SLRV would be reliable and virtually maintenance-free." Only two Transit systems ever bought those car. The Boston MBTA and San Francisco SFMTA (Muni) Here it the Wikipedia link; en.wikipedia.org/wiki/US_Standard_Light_Rail_Vehicle?wprov=sfla1
Well if your going to have an accident it's best done during testing. I used to be a lighthouse keeper but I was fired for what they said was 'safety reasons.' They said I was unsafe just because I could not sleep with the light on.
That sitting up front was a thing on the old "heritage" BR DMUs. Some less sociable drivers would pull the blinds down to thwart the railbasher sitting behind him.. Thanks JH.
As a kid, I once overnighted (involuntarily) at Shrewsbury station. The station staff kept an avuncular eye on me and found out I wanted to go to Ludlow. Early in the morning, I was invited into the stationmaster’s office, given a mug of proper railwayman’s tea (hot, strong, milky and very sweet), asked if I had a ticket (which wasn’t checked) and put on the parcels train to Ludlow - in the cab with the driver! What a great journey! I can still remember the cows and horses in the fields swathed in pink mist, the oncoming trains making me jump every few minutes and the kindness of the entire railway staff over the course of that night and morning. A very happy day!
@@thomasm1964 Back in the 80s my wife and I used to take an early morning DMU from Melksham to Weymouth. It started from Cardiff Canton depot and delivered sets for other diagrams. The Cardiff drivers were really nice and we would sit at the front, only passengers on the train, while they would tell us about their railway, and sometimes personal lives. Real shame that RU-vid came along a little to late to record their experiences. Thanks for sharing. Nice.
@@thomasm1964 I wonder how often this happened. I had an uncle doing his National Service and returning to Ludlow for Xmas leave. With poor connections and late running he arrived at Hereford way after the last departure of the day. Enquiring of the sole staff member he could find what time the first train departed the next morning the chap arranged for the next mixed freight to be held at the platform and following conversation with train crew and guard he was accommodated in the brake van. Not the smoothest of rides, reportedly, but good company, digestives and hot tea from the billy atop the stove.
Happy memories of having the most-prized seat at the front of a magnificent 'heritage' class 105 DMU from North Woolwich to Camden Road in, I think, the summer of 1981. As a former B.R. South Suburban Electrics south-east Londoner for the first 20 years of my life, my joy was unconfined (some masterful understatement there) as I experienced the sounds of the rather aggressively- driven DMU on the service that had been introduced by the Greater London Council a few weeks earlier. Happy days!
Very interesting episode, nicely done...thanks Jago. You have given us a new phrase for the lexicon...”latency in message processing”. I will be using it the next time my wife instructs me to Hoover the flat before supper!
I have always quite fancied being a lighthouse keeper. Stormier the better. Just as long as I can still watch my football and have plenty of pizza making supplies.
When I was a kid, my primary school had an exercise where we each picked a lighthouse, and wrote a letter to the keeper. I got a nice reply! I decided then I wanted to be a lighthouse keeper when I grew up, but then Trinity House went and automated the lighthouses, and that job pretty much died out.
Yep, signaling testing is one of the major things taking time. I’d rather wait another year for it to open then get into an underground train crash due to a signaling error
Seems a strange computer interaction to me. In my experience an ATP computer controls the round train brake circuit directly: in the event of an issue anywhere on the train, e.g. a door is not properly closed, this circuit is opened (fail safe) and the brakes are applied or, if stationary, held on. On the Victoria Line the ATP consists of three identical computers running identical software: two of these computers must agree with each other at all times: it's called 'majority voting'. I spent several months in a factory testing the interface between the Victoria Line trains ATP system and the radio system that conveys the data messages between the lineside signal systems and the trains; then I went to site to commission said radio systems.
Computers in the 80s were less complex, which generally meant you needed more of them. It was the interactions between your multiple computers that added the complexity. Besides, making a single computer control everything is not foolproof either, just look up Therac 25 for a proper horror story.
Yep, safety testing is a good thing. The modern world seems to have completely forgotten that not all tests result in 100% success. Some result in a failure that you can learn something from, and make changes so it doesn't happen again. In many ways there are more valuable and contribute more to gained knowledge than the successes. When man was learning how to get into space 70 years ago there were a whole lot of tests that blew up or went corkscrewing sideways, and each one was a success in that they learned from it and the same thing didn't happen again. Now we expect all "tests" to merely be PR shows.
People are relearning that during SpaceX Starship tests now. And those tests were only "successfull" in the sense that they did not show any flaws in the actual tests.
It shouldn't take any testing at all to realise that three computers running asynchronously on one-second cycles can take up to three seconds to come to a decision, and that such a system therefore cannot guarantee a response within 2.7s. Having said that, I just can't believe the explanation in this video. If the train started braking 0.3s too late, it would have stopped 0.3s too late, and that shouldn't have resulted in anything worse than stopping very slightly closer to the buffers than normal. You don't go off the end of a viaduct by hitting the brakes a third of a second late.
@@davidioanhedges Oh. Duh. For some reason it didn't occur to me that braking 0.3s late means spending an extra 0.3s at whatever speed you were doing _before_ braking started. Duuuuuh. But, still, there must be multiple factors in play, here: unless they were Hail-Mary testing the emergency brakes from the last possible point they could be applied, the system should have had plenty of time to see it needed to brake harder.
What a blooming brilliant video. Clears up the obfuscation of ‘official sources’ and in a typically Jagoistic way. Don’t think of it as a correction; more of an expansion of the tale... from the DLR. 😃
Very interesting story. As computer programmer myself its nice to know that they did such thorough testing. It's actually worrying when you test and nothing goes wrong, you start wondering what you missed. :) I don't even live in London, but I love your videos
Except that the test that actually revealed the error wasn't part of the 'thorough testing', it was a very lucky unscheduled extra. If they'd stuck to the programme, the first time this problem would have been revealed would have been the day that a DLR train hit the buffers!
I can confirm I have never stood on any DLR station desperate to gain the front seats if the opportunity arises, on a train. I can confirm I have never shoved past some old dear in a desperate attempt to get said seats nor sat the rest of the journey fearful of turning around and seeing a sea of disapproving faces. Great video
@@mirzaahmed6589 Can you check if the lighthouse is working. No... Yes... No... Yes... No its off again, wait its back, no I was wrong, yes its OK, damn gone again.
If you don’t test your code properly, your customers will be the ones who find the problems, and customers always find the problems with enough use. Not a scenario I want on my head when the difference between life and death depends on my code performing within spec. Better 1,000,000 empty trains crashed than one innocent person lose their life because of an error in my program.
I don't see it as a software problem. The software in teh individual servers were behaving as required. They simply did not get the message to react in time. This was a failure of the low-level communications method employed to allow the three servers to talk to each other. OK for most environments but NOT OK if real-time reactions are needed. Imagine if that same method was employed with an aircraft. Then a split second might make all the difference. Things have changed a lot since then with professional GPS systems able to react in fractions of a second.
Always enjoyable to understand the real reason for stuff going wrong. Life if full of stuff not going to plan, and very happy to DLR looking after passenger interests. Thanks for the video.
@@highpath4776 or a known safety problem (it being too cold for flying or lumps of insulation falling off on every flight) being ignored by management...
Accidents that happen because of situations no one anticipated can be perplexing, to say the least. There were two of those when I worked in the Safety Department of the Chicago Transit Authority. One in 2013 which was referred to as the "ghost train"...which involved water getting in to control cables, causing shorts which overrode "deadman" circuitry and allowed the train to move without an operator. The other was the 2014 derailment at O'Hare terminal...which involved insufficient placement of emergency track trips to stop a train traveling at unanticipated speed as it entered the terminal. There were instances of human error and other factors leading up to the incident in both cases, but the ultimate results...a train running without an operator and another train crashing through the bumper at the end of the track were both from unanticipated causes.
As a software engineer, I really appreciate the technical explanation. It's always series of small details like these, lining up to create scenarios noone might've thought of. Whenever I find one of them in something I'm working on, and figure out what caused it, I feel a great sense of satisfaction, and I reckon I'm not alone in that :) We've just destroyed a testing database, or the undercarriage of a DLR unit, sure, but we get to fix it before going live, and have just saved ourselves a humongous headache later on.
This was a really interesting episode. It raises- in my mind at least- interesting moral questions of the interface between engineering and bureaucracy. Kudos to Jim for bringing it to light, and kudos to Jago for having the intestinal fortitude to break policy in order to get this story out there.
True, but a chap in my mechatronics class got a job with Trinity House, riding about in helicopters and changing light bulbs. I wouldn't have minded that job in the slightest.
Even in the early 1980s, most lighthouses were getting automated. I knew a chap who was a peripatetic engineer responsible for fixing faults. He said one of the tricks of trade was, when they got to a nice lighthouse, to diagnose the fault as requiring a part that had the longest supply time, so you could spend more time at that location before moving on to the next one.
Safety Critical Systems are a huge minefield that you can blunder into with tragic consequences if you make almost any assumptions at all. See Boeing 737 Max for a terrible example...
Geoff Marshall would’ve filled you in on a lot of things, I met him once filming at Victoria in London and his knowledge of facts about these types of lines is phenomenal.
for anyone whose curious, this ATP system is actually being applied to a large portion of the TFL rail network. geoff marshall has some good videos explaining how the system works!
Very good. Well explained. The incident was indeed a precious test in terms of Risk Analysis & QRA in a typical "Test & Trial" environment in the railway safety arena (authorised or not!).
This is why software has to be tested... and tested... and tested again every time something changes. I’m a bit shocked that the computers were running on a one second loop. Even in 1987, one second was an eternity - sufficient time for about one million instructions or 100 disk accesses.
Back many years ago, around 1971 a similar thing happened with the new San Francisco BART system, resulting in a new train driving past it's stop, through a fence and crashing into the parking lot. Software is funny stuff..
I'm minded to recall a similar accident on the Rotterdam metro (train going thru'buffers and ending up over the end of the viaduct) but I believe that this case was actually due to an operator error
2:20 people standing at the front of the station at bank station - done that MANY times as I used to have to take that route quite a bit for work :-) If you do want to sit at the front (not in pandemic times) then stratford intl is a good station as it's very quiet.
Jago, you can only work with the information at your disposal at the time. If that was provided by DLR, then there is nothing to apologise for. Let this video be an augmentation of the first video, rather than a correction. And yes, the fact that it was a "live test" meant the unfortunate incident made the DLR more reliable.
If it's any reassurance to those watching this, this sort of scenario is almost impossibly unlikely to arise again these days in safety critical systems. With intensive simulation capacity that can entertain billions of possible output scenarios, to mathamtical proving of safety critical systems (with things like Ada in most autopilot and train control systems for example), the chances of these sort of bugs making it into infrastructure is diminishingly unlikely. That's not to say it can't happen, see the jammed accelerators on a certain car brand a few years back, the MCAS debacle and so on, but we're (collectively) getting much better and incidents like this have helped pave that way.
Hi Jago. Well, once upon a time, many decades ago now, I did understand computer software. Well, a bit. (sorry!). To have a byte, sorry, bite, at this I'm amazed that the asynchronous polling time was anything like as long as one second, even back then. Literally a disaster waiting to happen and it did - fortunately during testing, which is one reason for doing testing ...... Maybe your source can throw some light on this and tell me why I am mistaken. To be clear, the software engineers may not have known the performance specs of the hardware their product was finally going to run on - it's one thing to test in a development environment, quite another to do it in the real world, and thank God that they did. BobUK.
Thanks to you (and Jim!) for another fascinating and educational video. It sounds like their rigorous testing was a "successful failure": something they could learn and improve from! And congrats on making it to 100K subscribers!
Excellent video and it reminded me of the Therac-25 accidents and how timing was everything in a safety critical system, unfortunately, there was more complacency from the supplier.
Nicely done! I suspect there's no need to set off in search of an unoccupied lighthouse to lead a life of solitude far from loved ones & things TfL anytime soon...
This is what the internet excels at! The free flow of information that used to just become known via the "I met a guy in the pub last night..." network. Please thank him for us Jago!
Very clearly explained. Well done. There is a similar communications method employed when severs within a Load Balancer Cluster Web site talk to each other using a round robin heart beat method. However, a few seconds of discrepancies between a web page from server to server, won't make that much difference. When you are trying to stop a train or even worse, change course of an aircraft however, then a half a second can mean the difference between life and death. Therefore, this communications method is NOT recommended and is not employed for real-time systems today.
One of the things I love about your clips Jago, other than the trains, models, sarcasm and humour, is that I know what I’m going to get and the repetition is part of that. I am of course referring to your use of the word video at the start. I’m know wondering how many times you could get away with saying the same word in one sentence as well as one clip? 😂😂😂
The old punctuation test: John, where James had had had had had had had had had had had the teacher's approval. Which becomes: "John, where James had had "had", had had "had had". "Had had" had had the teacher's approval.
While i agree that it was fortunate that the testing programme was comprehensive enough to catch the problem, I must say that it's a hanging matter that it wasn't caught much earlier in the design phase. If you have a hard real time constraint (we need to be able to apply the brakes in 2.7s, or else), you make damn sure that the worst case response time is 2.7s. It doesn't matter that the computers will be out of sync only rarely, because over the lifetime of the system, "only rarely" will have occurred many thousands of times. Admittedly it was a more naive time, but safety critical design was nothing new. (Edit: Some people in the comments are incredulous that fixed 1s loops where used. Such fixed timing loops are in fact quite common in safety critical software, since among other things it makes it easier to guarantee a maximum, worst case response time. *headdesk*)
In fact the only circumstance that would stop me from firing each and everyone into the sun is if the original requirement was for a response time greater than 3s, but the requirements changed very late in the project, because of increased speeds, different track layouts with shorter stopping distances etc. Then some, but not all, will survive my judgement.
Ah yes, the charge of the Bank Brigade! 🤣😂😅 Spot on. It was the same when we got on at the old Island Gardens. My girlie and I used to happily wait for the second train by going via the shop so by the time we arrived we knew we would have “pole position” for the following train as Zoe loved the trip especially the rollercoaster down to Bank! 🤪 As a driver of a big train, I totally concur with the importance of safety measures. Sadly we often have tragic accidents that cause changes: let’s be glad that in this case there were no fatalities or injuries involved 👍🍻
They still have a one second cycle on the inductive loop communication. You can accidently switch to "Emergency Shunt", which is easy to do, but should never be done without authorization. If you switch back quick enough (before the next cycle) it doesn't report it back to control and it just looks like the train dropped out of ATO.
Regarding how weird this connection was, with "computers out of sync", it was very interesting to hear about, and yes, a "routine test" discovering such an unlikely scenario, before it some day, resulted in an accident, must be considered extremely successful.
I know an expert witness who works on these rail accident cases. Pays well apparently, and he definitely knows his stuff. There arent that many people with the experience, knowledge, accreditations etc, that can give impartial analysis
It was an accident but it couldn't exactly be classed as a failure. This is what testing is for, to discover faults before you load your train with paying passengers.
And what people that complain about the Crossrail delays need to realise - better for it to be delayed so that they can make sure that everything works smoothly than try to rush things and risk something going wrong during operation!
People often seem to miss the main point of testing is that it is ok for things to go wrong (and often encouraged), so they don't after release of your product. In this case the momentum of a fully laden train may have caused the braking distance to increase and mean that the front carriage may have gone all the way over! Big thank you to Jim for explaining the process! and to you Jago for making fantastic videos!
"Hey boss, we carried out that safety test you ordered." "Great! Did it work?" "Yes! six times, no problems!" "But I said seven times." "Well yes. The seventh time... it's a bit, well..." "complicated?" "Yeah.. well no.. actually not at all. Just gravity." "Ah! Just gra...WHAT?!"
As the testing was to ensure the safety of the system, rather than to show that it could be run safely, on occasion, I would say the only effective, successful test was the last one. :)
@@Lee5p33dy Testing isn't for amateurs, especially on complicated, potentially devastating systems. The problem is usually finding the finance to do it properly. The Y2K bug could have been catastrophic, if the banking system, the National Grid or nationwide Telecommunications had crashed, with each of them hindering the repairs of everything. Diesel trains need electric signaling and those trains and cars and lorries need electric pumps to to fill up with fuel.
