As a much younger software engineer, I still remember this phrase I was taught regarding relational database development. The key, the whole key, and nothing but the key. So help me Codd.
As a pre-SQL programmer, SQL was a game changer. Using SQL gave me a logical, abstract view of the data structure, significantly reduced my design and coding time, and allowed changes to the database structure without having to break code or migrate data. I used several systems on PCs like DBase and Firefox which had SQL database at their core. Also need to remember that systems back then were so constrained by CPU performance, memory storage, and disk capacity that SQL would have been too large to run on some of the early computers. Great video as always!
@@zzbeasley Agreed. I first met dBase on an Osborne luggable with 5 1/4" floppy drives and no hard disk! Later used dbase on PC and I liked Superbase under DR DOS/Gem. I think Superbase used dbase for data storage with Superbase giving a GUI front end.
I once was interviewed for a senior development position with a major multinational bank with their database guru. He was the one man who knew how the bank managed all of its data. He revealed that they had just one database. It had 600 tables. There were no joins, no relationships, no structure; It was just a massive dumping ground. Trades, transactions, accounts, … everything. The goal was to take this unholy mess and redo something better, while keeping all the existing business units running without interruption. It was an Oracle database.
I recall working with such a DB dumping ground as a new hire at one of my previous jobs. Source data from a global company in the banking and payments space. It came with hundreds of mostly useless stored procedures and functions that were supposedly written around 2015, but contained zero proper joins in the code. My new colleagues were waiting for me to make snarky comments about it!
I'd argue that SQL is one of the most successful inventions of computer science, on the level of C or even more so. Nothing better has come up despite decades of attempts, and pretty much all complex datastores eventually support SQL as they grow.
Humankind is generating vast amounts of data. Making sense of it all is beyond what a relational database can do well. The next step is "big data" non-relational databases. There are no keys, there is no structure to the data. The goal is to find the relations, not order them in that way. It is where we are headed. A relational database is great for what we use it for and will be used that way for a long time to come, but big data will look information a new way for a new purpose.
"Humankind is generating vast amounts of data. Making sense of it all is beyond what a relational database can do well"... did you ask ChatGPT to write this bunch of vague and disconnected catch phrases just to oppose the OP ? What would be the practical purpose of "making sense" of all data we are producing? Make a real life Matrix ?
@@chpsilva A comment section is not the place to have a deep discussion about big data. I don't see that I opposed the OP, I presented what the next generation of databases are about and where the research is going. I have spent my professional life programming SQL relational databases, I do it for a living. If you are interested in the big data concepts I am talking about do a google search. Big data is useful and yes, despite your reservations, it is something that matters.
Database researcher and university professor here: Very good presentation as usual, though it of course ends with the state of things in the mid 1980s and a lot has happened since, not just in academia but also in industry. As early relational DMBS go, IBM's System R (which has been alluded to in the video but not named and whose DNA lives on to this day in IBM DB2) has had significantly higher impact and relevance than Ingres. Since you mentioned Boyce and Stonebraker, a number of others would have deserved mention at least as much, such as Jim Gray or Pat Selinger. Also, you added to my pain as an academic trying to attract young researchers into the field by making databases look real bland and boring (though important to business). Unfortunately, so many undergrad courses do the same. There is beautiful systems and theory research to be done, and there are interesting fundamental questions that arise here that are much cooler than anything mentioned in this video (though ultimately enabled by Codd).
For sure, a database is way more than an Excell spreadsheet. With column based storage, data compression, and parallel processing. We get things like Google, a keyword database of the entire internet. And Twitter a database that adds records at a rate of 100's of gigabytes a second, all instantly searchable.
Not the original poster but curious; a bachelor's degree in mathematics with no theoretical exposure to databases, and 6 months experience programming with SQL.
I had the privilege of digitising some Betamax tapes from the Australian Computer Society many years ago from the 10th Australian Computer Conference held in 1983. Chris Date, a relational database expert, gave a presentation on one of these tapes and made many claims about what the future would be for databases. "Like it or not, SQL is going to become a very important language. It might become an actual standard, and it almost certainly will become a de facto standard." The full speech is here ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-VnNbddUMZQI.html
@@MePeterNicholls I used one, a Key West Big Voodoo. It doesn't handle dropouts in the vertical blanking very well. The tapes were very degraded unfortunately.
@@MePeterNicholls The biggest problem were dropouts. A digital timebase corrector used with a domestic video recorder is not going to help with that. Sure they can on studio machines like Umatic, Betacam, MII etc., but domestic machines lack the required RF output signal. However it might be that the particular player could have usefully had the DOC (dropout compensator) sensitivity adjusted, I recently showed that with a Beta machine on RU-vid.
Really enjoyed that, but then I do have a 35 year SQL career behind me. Furthermore, almost all of it was new to me despite my extended immersion. I have to be honest, when I hear Codd's name, I usually switch off: it's usually in relation to Codd's Rules, academic rules of thumb that in practice are nothing more than common sense and second nature in this line of work. I watched a video on them that I happened to stumble across a few months ago, and yes, it is all common sense codified into a set of rules. So I congratulate you in managing to do a 20 minute video on SQL without ever referring to Codd's Rules!
for our management information systems program in university we barely do the SQL behind these academic concepts and it's really dumb, I am currently building a .NET interface for a access database just so I can practically learn this stuff.
As a newish developer, of all videos, this was the most exciting to see when I saw the thumbnail. SQL has become one the foundational blocks of software development and even today, when all the competing NoSQL paradigms (MongoDB, Firebase, Redis, etc) have claimed a space in database management, we are coming back to SQL with new ideas thanks to PostgreSQL and SQLite related projects like Turso and libSQL. And I really like the image in 14:50 while distant in time is so relatable to my knowledge. A testament on how SQL stood the test of time and won. Really thanks Jon for this video.
The problem is that most NoSQL databases (Mongo is fully ACID but records aren't usually normalizaed each document has all data so this doesn't mean much and so multi document transaction were only introduced in Mongo 4 mere 4 years ago) aren't fully ACID (Atomicity, Consistensy, Integrity and Durability). For things like banking transactions are a must: the money that gets out of one account must get in on another or the transaction must rollback. Unstructured or partially unstructured data are a case for NoSQL, many NoSQL databases are very simple and good in replication, but at a cost of using GUIDs as the primary key (as a surrogate key) this is not always desirable. When you get billions of records that need integrity and complex queries (Mongo performance with complex queries is abysmal) that must be done fast SQL is a must (good part of my life I passed tuning badly designed and maintained databases, and if I got a Dollar for each time I heard this can't be done faster or this can't be done in SQL I would have double what I earned).
@@agranero6 I agree with you, but my point really was that we are ending the era when NoSQL was trendy and "you have to use it to be cool" like many other bad ideas mostly in webdev (microservices for all, complex SPAs for everything, rewriting everything in the latest language, etc). Nowadays everybody is rediscovering that SQL in its many forms is a solid language with solid ideas. Even SQLite is becoming popular! 😜 For me, I'm in both sides: For small projects I'm happy with tools like Cloud Firestore, but for more complex projects nothing that good ol' SQL.
_Lots_ of people know where and how SQL got started. They just had to graduate college before the rise of Java. Heck, I've still got a Relational Databases textbook in a box somewhere.
I've been using SQL since September 1999. Database programmer has never been my formal title, but I have been one for 25 years this fall, in addition to my main programming job that's normally identified by the other programming language. It's been SQL the whole time, but the other language has gone from C to Perl to PHP to Java over the decades. What makes SQL different from other programming languages is that it's used by far more than just programmers. I've seen programmers write horrible SQL and non-programmers write terrific SQL. If you can wrap your mind around set calculus you'll become very good at SQL, and you don't need to be a programmer at all. Either way, practice helps. I was only what I would consider good at SQL after 15 years and I'm now fairly proficient at SQL such that I can jump right into any database get going.
Do professors tell you that physical structure of the RDBMS is as important as logical structure? As a contractor DBA I make much more money from physical structure issues than from logical ones.
@@kondybasAgreed to a point. Usually, the logical design isn’t that bad. My big problem is middleware… stuck threads, garbage collection, multi-threaded architecture and lack of commits.
I remember learning SQL in the early 2000's and dutifully memorizing that it was the unification of relational algebra and relational calculus. The fact that almost nobody would even bother to learn that today attests to how useful and intuitive SQL really is. The key was, I think, that they modeled it after regular English and the implicit set-theoretic operations in natural language - "select birthday from people where ...""
I have to profess my love of this channel and your content. I don't know how I was lured into this channel at first, I'm not Asian, and I'm just a mere MSP technician who grew up in the 80s and 90s. But you always seem to hit the right note, and Sunday nights haven't been the same since. You're doing it right, sir. 🤩 I eat these videos up, and wanted to say "thanks".
its only called Asianometry because there's a focus on Chinese, Taiwanese, Japanese, Indian technology sector! the channel is not actually about asian culture lol
@@Old_Jack_Ketchi agree. It is probably because the topics in the video's are not ancient history and still have relevance to this day. But even if you started studying or working in ICT you might have never heard of many contextual details in these vids but when you hear or see them you can often instantly relate them to personal experiences (at least i can) and that's entertaining. Greetings from ASML-city!
@@thinkingcitizen I get that. I meant in terms of the initial draw for me, what made me pick this content, based solely on the name of the channel - It speaks to how genuinely interesting the selection of topics are. 😉
I work in a social work/SUD treatment agency, and we somewhat recently began working with a company that helped us get backend data out of the cloud database. I haven't worked with SQL since college... The last few months have been a blast, and believe you me, I've been able to flex comp sci skills in a very not comp sci industry haha.
I like how You dig out filthy details: "Deliverability, Redundancy and Consistency of Relations stored in Large Data Banks" with the exact date on it +++
Gotta say I wish my dad was still around to appreciate this channel. As a historian and library scientist with most of his career transpiring before the internet explosion, he didn't really get to witness much refinement of video content. The variety and scope of the stories shared through Asianometry are just plain great... (Although I'm sure he would've griped about lack of formal reference summaries to back up details.)
I knew Charlie Bachman and can confirm he was a stand up guy. Kind, patient, a good teacher and a good boss. I worked for him for a few years when he was promoting his entity-relationship model which was more abstract than the Hierarchical (IMS), CODASYL (IDMS) and Relational (DB2) models and was useful for porting and reverse engineering.
Information exists outside of youtube, ya know. Pretending youtube is some super-special, locked down knowledge store that will never go away and is vital to maintaining a record of history... come on now. 😏
@@mercster Asianometry helps spread this information to a wider audience without them actively searching for it. Almost every person watching this video had it recommended to them via the RU-vid home page or through another recommendation method on the site. There are other sources for this information, but this video is a good way of pushing out that knowledge to a large number of people who would have not previously considered or cared about the topics discussed on the channel. Best regards.
@@broadestsmiler I'm not taking anything at all away from Asionometry, whose videos I very much enjoy. What I'm getting at is, when you lavishly heap praise in an overly-enthusiastic, hyperbolic manner that doesn't really accurately reflect reality, you diminish the genuine value.
@@mercsterNot everybody has the time to delve into historical documents. I see this channel as a low-treshhold gateway into the early history of the computer industry. Not overly theoretical yet very informative.
Hello Jon, In my opinion, this is one of your best video. As a side hustle to my job, I've been writing in SQL for more than a decade. I'd read about how it come to be, but the story you tell is at another level. Thank you Anthony
S-Q-L (originally SEQUEL) had to change its name to S-Q-L due to an existing language called SEQUEL and subsequent trademark dispute. I try my best to use S-Q-L on Monday, Wednesday, Friday, SEQUEL other days, and "Squirrel" when I have had too much to drink.
squirrel 😂 - I can laugh about that joke without an IQ of 10000. - I didn’t have to come up with it, I only needed to understand. Edit: on the other hand: it seems a bit unfair, as squirrels (allegedly) sometimes forget where they stored all of their seeds for the winter, thus enabling unintended growth of trees… (- „trees“ ?😂) Edit 2: maybe the 10000 wasn’t in the decimal system, but binary 😂
Great video as always! I would love future videos on more modern SQL since the 1980s -> e.g. the breakout of OLAP vs. OLTP databases, SQL in the cloud, Presto, etc.
I worked on databases on IBM mainframes. IMS was described as hierarchical, Total was a network database (you had two levels). I then came across early DB2 using Structured Query Language and of cause Codd’s relational database principles. I did a bit if work on Oracle and quite a lot using UDB (the djstributed version of DB2 for middle tier rather than mainframe systems. Interesting to see how other software sellers went about implementing Codd’s rules.
Later on in my life I am finding a lack of great videos that bridge the human gap between the great ideas we need to learn and how and there they came from. It's so easy to give up because everything comes down some arbitrary decisions than are then later generalized and that's what wins out. I am one of those people who needs that back story to calm down the "Why this way??" questions in my head with out mastering something enough to know why. Not much point to this for others. Just engaging the algorithm with something more than "Thanks for another good video. I enjoyed it."
Phenomenal video. The inoculation to being overwhelmed by a technology is to follow the simple ideas and history that built its complexity. You have a remarkable talent for leading audiences along those journeys. I hope that you'll take us on similar tours of these omnipresent, yet terrifying, tools that power our world. In the meantime, avoid dying of a brain aneurysm. Because evidently that's something I get to worry about now.
Big thumbs up and thank you , databases often overlooked are the backbone of most system design. One of the first high school jobs was digitizing the music collection the national radio station into a DBase IV system. I then used my database programming superpowers to make a movie rental database back in the days when stores rented VHS tapes using Novell Netware and Btrieve. A lot of money to be made from database design in the 1980/1990s! It was the HTML of the era.
Writing SQL is an incredibly therapeutic, zen experience. I cannot properly express the joys of having an end goal of what data you want to have rolled up and the fun, challenging path of writing SQL to get you there. I changed fields, in a way, and no longer get to use SQL. I miss working with it all the time!
I interviewed with Oracle and three other Bay Area companies before moving there in 1981. Oracle was my second choice. It was a prototypical startup - but at the time, of course, it looked like a young and untested outfit. What might have been… (I had plenty of time and read E. F. Codd's papers at the not-so-busy consulting firm I signed with instead, and got heavily into databases later.)
I appreciate the label ‘vicious’ applied to Oracle as that is how they were described by Michael Stonebraker and colleagues, developers of Ingres, Postgres, etc. In the 1990s. Oracle would do whatever it took to get the business. (I worked as a programmer in the CS Division at UC Berkeley in the 1990s)
My second cat, the last offspring of my first dilute Calico whose litter came while I was learning relational databases, was named SQL. A damned good cat, indeed.
Interesting, informative and well done, on a subject that I worked with for a number of years, but had not idea of 'the rest of the story'. Thank you for all the work and then sharing.
I've been needing a good overview on databases! You always put out videos addressing things I'm interested in :) Wait a minute... show us the database you use to predict our viewing desires!!!
I worked as a SW developer for 30 years. I started with IMS, IDMS then DB2. DB2 relational DB is the best for design, development and serving the business.
We usually prefer the term tree instead of network to avoid confusion with a computer network and because network is a wider concept (equivalent to a graph): networks don't have to be in tree form, they can for instance be semi-lattices for instance. Trees are good for seaching as we can traverse with simple algorithms and find a key in logarithmic time, if the tree is well balanced, and we can balance them for efficiency. In general graphs this can't be done. In your video you confuse related tables with trees: the trees in databases are assembled inside the indexes to the keys in the database. The relationship between tables doesn't need to be in the form of a tree and it usually is not.
Excellent, as always! The second word in Hawker Siddeley is usually pronounced as two syllables, sid-ley. Source: I was an avid 10-year-old aicraft spotter in London in 1972 when RDMS was getting going, and the HS Trident was my favourite airliner. (Though it started life as the de Havilland Trident.)
When I entered college in the fall of 1979 I was hired by its Educational Computing Service department which did contract programming for the university and government agencies. At the time CODASYL network databases were the most likely type a programmer would encounter and an absolute pain in the ass to deal with. Not because they were hard to navigate but because it was exceedingly difficult (read: expensive) to change the schema. I still have a copy of "An Introduction to Database Systems" by C. J. Date on my bookshelf from 1981 when relational databases were still controversial but it was clear to me would quickly become the standard.
Okay this was the first time for me there wasn't a fascinating video hidden behind a boring title. That is just me tho. Still watched it. Keep up the great work ❤
i had a professor in the early 80's who was a disciple of Ted Codd. he was always spending class time relating personal anecdotes about Codd instead of the class material. according to my Prof, IBM was not happy with Codd's research since they thought what he was doing was going to conflict with IMS and could cut into the profits IBM was making from it. my Prof respected that Codd was able to persist in spite of the corporate discouragement, and eventually created the System R project, which turned into DB2.
i am about to graduate with my buisness degree in Management Information Systems and im so excited! Im currently staring at a 3nf normalized ERD for a travel company. Its kind of like micromanagement games like openttd, you gotta make sure all your little guys are transporting what you need to each other.
While working on the Allen Brain Atlas I got to work a bit with Jim Gray ... I best remember talking over beer about both technology, sailing, and comparing Seattle to the Bay Area.
This was great. I hope you will follow up with the saga of the database wars of the late 1980s and early 1990s between Oracle, Ingres, Sybase, Postgres, and MS SQL Server.
You forgot to mention the predecessor from SQL (that I learned), it was dbase IV, and it had already SQL implemented in it. I programmed a lot in dbase III and IV mostly for bookkeeping purposes. At that time, this was very powerful, but I quit programming when I was conscripted for the army. After my 3 years duty, I lost interest, unfortunately. But SQL isn't much different from the other products. And later in my Carrère working in BPCS I used my knowledge to create my own query's and forms to extract data out of the mass database. My boss didn't know much about databases and programming or making forms on that level. Enjoyed my work very much. Making things a lot more efficient. I remember saying to him, everything is there, you just have to know how to extract things that are relevant for you and the team. But I left the job because it became boring and other opportunity's that were better paid.
All your videos are good, but this one was amazing. As someone who first learned SQL in 2004 (and who still pronounces it Es kju ell), I have never learned so much history about any technology than in this video. By the way, do you think that a video on the Memex by Vannevar Bush would fall into your area of expertise? It seems that every other GUI, knowledge retrieval system, hypertext and note taking system seems to sooner or later imitate the Memex, an invention from the 1930s or 1940s (depending on which paper you count)
SurrealDB is the next step in the database paradigm starting like SQL but also having record links and graph links. Complex queries are easier still to read and write.
wow hearing about Codd! jeeeze that takes me back to the 80s when i was a simple college student & my computer sciences teacher was telling me all about this interesting language. he was of course an ALGOL 68 man & me a humble assembler / dbase III guy.
Data on a database is navigated through links as you show on the graphs. Links have a direction, they go from A to B, not from B to A. Those links are the branches of a tree, and the pieces of information are the leaves. If the links in your tree go from the top, the crown, down to the root, the only possibility to navigate is one-way to the root. If you wanted to reach any other leaf on such a tree you are out of luck because your links point always towards the root, no links point to any leaf. That is why, to clarify the analogy, it is often said that data is structured as an *inverted* tree , meaning that the graph of the data looks like an upside down tree. Then, starting from the root, you can reach any leaf.
Excellent video. I didn't know any of this and I've been writing SQL code since the late 90s- nothing too complex. Most of it involved pulling data from process control historians, doing some calculations and writing it to html files so operators and accountants could have real time access to inventories without having expensive process control client software. I find myself wishing the windows file system had SQL abilities on an almost daily basis.
sometimes being a visionary isn't all it's cracked up to be, imagine going around the whole world trying to convince people you know of a better way but they're all too dumb or arrogant to see it...has happened so many times throughout history
When I started in '97, Rapid Application Development was hot. Since those tools often used key-value pair "databases", I avoided SQL altogether. I dabbled in Clipper (dBase III) and VBA for office automation/workflows but eventually used Lotus Notes/Domino for 10+ years (distributed storage model for LAN and/or offline use, replication for minimizing network traffic, support for rich text and attachments, shared-nothing servers so you could cluster a VAX with a mainframe with a Solaris, PKI easily integrated in custom apps, ...)
Jon, have you ever considered packaging all your videos as a structured "history of computing" course? Or multiple courses (history of semiconductor manufacturing, history of hardware, history of software, etc)? I bet home-schoolers and others would be happy to pay for access to an experience that is more ordered and structured, like a miniature college-level history course. I think I'd pay for that. In any case, thanks for the videos. They're fantastic.
I started my professional career in GIS in the early 2000's I realise how critically essential relational db's are for applications like GIS. (table joins , primary key, etc for doing thematical maps and spatial analysis) Thnx for this great video! Because this kind of historical context was never part of the curriculum we had to study.
E F Codd's ideas area a bit more subtle than just applying to databases. They also apply to any type of software, where you should just update the software in one location.
I would love a video on Visual Basic. I have a belief that VB is awesomely important both for applications running today especially in business but also for being one of the first examples of a minimal learning curve way to create applications really cheaply, especially in contrast to next with objective c
It may be "elitist", but... While conquering C, decades ago, I couldn't get past the literal meaning of the name of my first language: BASIC The name stands for "Beginner's All-purpose Symbolic Instruction Code". Can't remain only a "Beginner" forever. Can't imagine life without a pointer... Pointless existence... 🙂
@@rustycherkas8229 I know personally of at least three (maybe four but hazy memory) companies selling applications built using VB in active development to this day which started life as one-man bands. People may scoff at that language and look down on their work because of it, and those one-man band folk will be crying into their champagne tonight in their private south of france villas.
I recall working on IDMS (briefly do forgive me if I'm wrong) but as i recall the main issue was that once you built the database it was essentially impossible to change because not only did you have to change the database, you likely had to change the applications that navigated it.