AI Feels the Data Crunch 

AI: "We'll conquer humanity... right after this cat video."

The fun story with Facebook, they were asked, in Australia, if they were using accounts to train AI, and the answer was "we'll have to look into it". Then it turns out it is completely legal in Australia to use the data and they come out "Yeah. Yeah we're using any and all photos shared by Australians in our AI." No pretense. No concern.

Having to know the name of the bot to block it feels eerily like an exorcism

"We need high quality data. Let's crawl the internet for that!" Something does not fit together...

robots.txt doesn't block access, it just tells the crawlers, which files they should not access. So, there is no need to rename the crawlers, they can just ignore these instructions.

Wow, so the movie Short Circuit (1986) was prescient. AI "needs input." Number 5: "Malfunction. Need input." Stephanie Speck: "Input. That's information! Listen, I am full of it."

This is something I laughed about when this whole thing started - GIGO - Garbage In, Garbage Out.

Sorry, some correction regarding robots.txt: 1) You can make a rule to block all bots and then allow only certain (like Google bot) via allow list. 2) robots.txt is still only as useful as a stop sign put on an empty field of grass. Bots from Perplexity etc just ignore it. Hence all at least kinda reliable blocking requires technical measures... usually using more AI, just not of the generative kind. Thus one can see it is data theft, as much as a someone breaking a window is doing physical theft.

the myth of informed technological consent runs DEEP

Definitely talk about copyright hipocricy !

Technically, robots.txt does not prevent crawler activity but merely advises on it. Some crawlers respect this advice, while many ignore it and often do not properly identify themselves as bots.

robot.txt is just a sign in your front yard that says "don't take my data, please." It has no effect on crawlers that ignore it. And web crawlers are literally called robot spiders.

I wouldn't be surprised if Open AI just bought the data from companies with less restricted web crawlers.

Happy birthday 🎈❤, Dr. Sabine (well, tomorrow), so good that you were born, great to have you here in our universe.

If everything ends up needing to be licensed, LLM training will eventually become astronomically expensive. Progress in this kind of AI will end because no one can afford to train it. The court ruling that allowed commercial use of data that isn't behind a paywall will become redundant.

AI companies don't have any right to charge for use of their models since it was all trained on public data without consent

Most of VPN companies will actually sell your data. You can only guarantee privacy by running your own VPN server, or using a VPN that doesn't require your ID and pay with crypto or cash.

If AI learns by surfing the web, where does the intelligence come from?

AI is learning from the Web, but an increasing number of pages in the Web are AI generated, and most of the time without attribution. So, AI is being trained by AI output, it is garbage IN -> garbage OUT.

A bag of chips is a poor example of something you eat quietly...

Its estimated that AI will take less than 50years to discover that society needs to focus on basic needs like healthcare, food and water, housing and basic infrastructure.

I'm a software developer and, at work, we're putting a lot of work implementing AI to answer phone calls, route calls, etc... you know, customer service stuff. Then, the other day I was looking up a restaurant and Google offered to have AI call for me, check the wait times and/or make a reservation... that's when I realized... we're building a network of AI bots that all talk to each other using English as their API? I don't know what that means but... it doesn't sound good at all.

About robots.txt: you can say nobody except a few are allowed to crawl. Also the crawlers can just ignore robots.txt. Don't know if there is any legal ruling about if robots.txt is legally binding.

There is actually growing evidence that AI generated data *can* be used to train improved AI models. There's also papers that show that it can't. Differences in methodologies can explain that, I presume, but as far as I know, there's not really a meta-review about that.

Reminds me of the napster era copyright crisis that appl more or less fixed with itunes.

Fun fact: AI companies are not even getting rich by using terabytes of private data.

Funny how if I pirate a movie or rip the assets inside a videogame without permission, I am liable of being sued by the company who owns the copyright. But if such company takes my photos, my voice, or straight forwards take assets I posted online who are under my own copyright, to train an AI specifically purposed to take me out of my job for their profits, content which is blatantly appearing in the results generated by such AI, suddenly it's all fair use.

An additional problem is that more and more "data" on the internet is itself generated by AI, or at least LLMs. So AI will increasingly be training on so-called data that it produced itself. This will lead to total uniformity (we're already about 80-90% there already) and a convergence on idiocracy.

I didn’t need to crawl the internet to become aware. I just crawl around the carpet, tried to eat the couch, peed on the kitchen floor, slammed the doors a lot, torched flies, made holes in the garden……….

6:08 “Just hire SCIENTISTS”.

"Sounds like a robot spider" In fact, you can call it both of those things

"...What is mine is mine and not yours!..." Imagine a body, where each organ decides to be independant. A leg is not going to walk unless a hand pays it for the journey. And that is how our world is built (or rather disassembled). So it reminds us of a dead body, which starts to disintegrate because cells cannot live by themselves. So are we ready to widely open our mouth and start declaring our constitutional rights? No. Rather we are reaching that point where instead of word independance, we say suicidal stupidity. And who desagree? Not a trace of them left. All disintegrated

The horse is already out of the barn on data. They have what they want, and shortly won't need even that.

The easiest solution to this issue would be to pay contractual companies working with sensitive data to train sub-models based on existing LLMs, and then siphoning the data off of them in a pseudonymous manner. Once word of this "gets out", the LLM owners can conveniently deny malpractice as they have simply provided their APIs as a service and the usage of these APIs come with sharing data anonymously by default which is how they can let go of any responsibility for data in the first place. Pretty smooth, probably what Microsoft tried to do but faced a lot of backlash because of lack of encryption of data that was stored on a drive that had bitlocker encryption in the first place. I agree with your initial viewpoint - that we are building a species that thrives on data when humans as a species are hardly as evolved to even co-exist with them. I trust the tech-bros to make these species possible just to prove this point and laugh at the general inferiority of cellular organisms.

The issue is that this method or branch of AI, neural networks, REQUIRES all this data and compute to even pretend to work.

The Bots, Borgs & Non Humans. Welcome to the new brave World.

So it’s a software robot spider… is that less scary? And can I crush it with volume 12 of my encyclopedia?

Thank you for the interesting talk! 🌻

I have to take exception to a statement you made Sabine - no one can quietly munch on a bag of chips. It’s a law of nature, not a hypothesis.

In my opinion AI bots should legally be obliged to identify themselves or their intend by providing additional HTTP headers with their request. Also there should be absolutely insane punishments for anyone who decides to break that law.

I think what will happen is that AI companies will start awarding the companies who let them crawl their site for data, with attribution and traffic. Eventually, a site like the New York Times will need these crawlers or face extinction. That’s google’s strategy basically.

Pay people for their data! Maybe the future economy is everyone is compensated for their data and gets to live off low cost AI goods and services that the data compensation pays them for.

Imagine the consequences of the net getting flooded by incorrect, outdated or even forged data all amalgated together and poisoning AIs.

Content stealing considerably predated the internet, of course.

They took the artist's work and then used it for substituting the artist without any explicit consent. So there's no surprise about other people wanting to keep their work away from Ai.

One of the genuine things 'looming' over us, or at least the US legal system is the near inevitable need to eventually rewrite copyright laws. Nobody wants to touch it as it's in a somewhat stable semi-functional state right now with just enough legally grey area to keep everyone grumbly but not outraged, but it's very old and deeply flawed. A better system is definitely possible, and I've seen some proposed but it will be an absolutely monumental undertaking to change it all that'll probably take years and cause chaos because the balance it's stuck on right now is at least minimally acceptable to most parties- despite the fact that it does allow pretty egregious exploitation of people's data and company IP

EU: copyright infringement via AI is still copyright infringement, so make sure your AI's output is legally original. That's gonna be fun.

Most of the internet is slop and actually counterproductive for AI's to train on. Curating data to produce a higher-quality training set is more important than getting "more" data. Most of the highest-quality data has already been collected and exists in the current training sets - the sets just need to be paired down intelligently. Automated curation is becoming better and better. The quality flywheel is speeding up. As the quality improves, the need for RLHF in fine-tuning will decrease and the "H" in that acronym is already being replaced by AI as well. In short, there are several virtuous feedback loops already at play in increasing the quality of training and fine-tuning. This will, in turn, improve the quality of synthetic data. As soon as the quality of synthetic data reaches or exceeds the average quality of human generated data (factoring in the curation possible on both those data streams) a further virtuous feedback loop will be engaged. TLDR: Data quantity is a red-herring.

The irony of doing a story about information providers’ data being used against their wishes and then advertising a product to allow you to use information providers’ data against their wishes 😂

Later models of 3d TVs were actually pretty great, shame the early ones were so bad nobody bought them when they became actually amazing

The other problem is that if people go to the AI chatbots to get their info and not visiting the website, WHO will bother to write anything if they don't have traffic?

The problem is getting the information into people’s heads.

The problem with AI is with the model developers *excluding* data from their models. E.g., ask them about crowd-sourced law like the Uniting Amendment, or ask them how many N95 masks were sitting on the shelf in the US Strategic National Stockpile during the "shortage" of masks while the pandemic was raging. Or ask them if public health officials knew that cloth masks wouldn't stop the virus at the beginning of the pandemic. This information and much more has been excluded from most of the LLM models.

The Archive/Citation index _Semantic Scholar_ just seems to have introduced AI assisted paper reading. The first time I saw it was on a paper whose abstract was just "We obtain an asymptotic formula for a weighted sum of the square of the tail in the singular series for the Goldbach and prime-pair problems." and The AI summary was "The key results of this paper include obtaining an asymptotic formula for a weighted sum of the square of the tail in the singular series for the Goldbach and prime-pair problems" and it offered a second choice "The key results of this paper include obtaining an asymptotic formula for a weighted sum of the square of the tail in the singular series for the Goldbach and prime-pair problems. Additionally, the paper discusses the conjecture made by Hardy and Littlewood in 1922 regarding an asymptotic formula for the number of pairs of primes differing by k." The difference is that the second choice included not only the abstract in its entirety, but _also_ the first sentence of the first section. So I told it that I thought the second option was probably better than the first, but it didn't ask me to quantify how much better. Given the paper in question is seventeen pages long, I think the answer would be around 0.001% better and I'm not doing anything fancy here with additive measures, ....

From WC USA, well Dr. I hope that today is your birthday as I know that when my wife calls her friends and relatives in Germany it has to be on the right day! She says it can’t be easily or late! As for AI, I hate it, if you call your Doctor or Home store, you have to go through this maze of numbers before you get someone to talk to! Years ago you called, you got who you wanted to talk to or they switched you, usually within 1 to 2 minutes! Now it takes 5 minutes and you may or may not get your party, if you don’t I’ve had to go through the whole process again! This is progress?!

Ahhhh Happy Birthday Sabine :D

The current state of the internet bots means AI will be using flawed AI data to train AI.

Speaking of AI reading plots... Have you noticed that LLMs are pretty good at LaTeX? If your plots are done in pure LaTeX rather than as embedded images, there's a good chance it can figure them out.

crap in crap out. Content stealing is even older than you'd think, buskers used to steal orchestral pieces and make money from it, I believe Beethoven monetized it by selling them the sheet music to play from.

AI being introduced to the new trainer: "Nice to meet you, AI" "Nice to meet you too, AI"

❤️ I love 3D TV and still have two of them. Who needs AI when you still have 3D TV?

Many thanks, this video makes my day.

There is what seems to be a very deliberate conflation of verbatim reproduction of source material, which is _not at all_ what LLMs typically do, and generalized learning of information from a plethora of sources, which is then summarized, which _is_ what LLMs typically do. It is also what human experts typically do, only providing citations for what is quoted, or what is unique to a particular source. However it would be fairly easy for LLMs to routinely provide a list of source citations upon request for information they communicate, which might be required in a minority of cases, for example when generated information is referenced in published content. There is no obligation assumed for humans to explicitly cite specific original sources of general information they have committed to memory. That duty applies only to very unique or proprietary information.

If you want your AI to succeed just pay loads to ensure users demand the other Ai's are blocked.

It seems like no one wants to take responsibility for what is coming, and they all push it to AI. I'm hoping that AI doesn't take revenge.

The incredible irony of the NordVPN advert on the end offering to help get around blocked content.

Id rather use ai to search stuff because the privacy agreements drive me mad which hurts both the website and the ai

AI is a victim of turbocapotalism: "As long as noone says otherwise, everything that creates money is viable. We will deal with problems when someone other than us notices them..."

Alles Gute!!! ❤

AI is very different than 3D TVs. 3D TVs died because they did not offer much benefit. AI is seen as failing in the future because most people have no idea what AI does or what it is capable of and assume that it offers no benefit.

AI scientists should pay for the data they use to train their models.

Fascinating. An AI to understand scientific papers? That'd absolutely kill journalism, Sabine. 🤨 (Yes, sarcasm.) Anyway, stay safe there with your family! 🖖😊

The idea that humans produce the best data will be short lived. As AI accesses more non-human data "sensing" and collecting devices, it will be able to far surpass the limits of human senses, and their biased storage and processing.

I was trying to warn everyone when Captcha's were popping up everywhere. It was AI systems training..."click all the squares not containing a Bus"

Really interesting and well put together

5:01 I'm eating chips... is she watching me? Cos do! Licensing, T's & C's apply...

Now hold on a minute, we're supposed to give AI high quality data! Ohoh!

Just a quick reminder that a VPN doesn’t make your Internet browsing any “safer”. it just changes your location so that you can access region locked content. Your computers’s IP address is still visible, and is also kept on the router itself. A hardware proxy is what makes your Internet “safer”.

They should make a CAPTAIN CRUNCH like breakfast cereal called AI CRUNCH.

Will you please make as critical video about VPN actual usefulness in online security as you make about anything else? Thanks.

6'13" exactly. I've seen job ads that recruit scientists to train models for $50 per hour, which is slightly higher than what a science professor gets on average in the US.

If you gave an AI a body with high bandwidth senses, wouldn't that provide an endless source of real world data? Maybe AIs need to touch grass, literally.

Doesn't 49 authors for a single paper strike you as excessive?

If you're going to talk about copyright hypocrisy, make sure to mention the Internet Archive and how it's being sued into oblivion by people who think libraries are a disservice to humanity.

It is estimated that up to 5% of newly created music content on Spotify is AI generated. This allows all of the commissions generated to be kept by Spotify. Music is very forumalic, at least in the pop genre, and these formulas are deeply embedded within AI models. Since voice training can produce with 80% accuracy after two weeks of training and achieve 99% accuracy after two months of training this is a huge concern for musicians and artists, both past and present.

Webcrawling was an easy way to get cheap data in the early experimental days of LLM development. But that is not the only way; Tesla trained their AI from data directly observed by thousands of cars driving the roads. However, just throwing the Web at an AI for training is not as valuable as training with curated high information content data, and that is being constructed by the AIs we have, now. When it comes to training data, quality beats quantity.

OpenAI still crawls pages with robots.txt. You have to block their scrapers by IP.

The world is a flood of data, more than you can ever use.

Internet users have been pirating and copyright infringing movies/tv shows/music/games/etc for decades, and the instant they try to train AI on their data, they instantly decide copyright applies to them. A company spends hundreds of millions of dollars making a movie and thousands of people pirate it, and then someone trains an AI on some random RU-vid video and everyone complains. Hypocrisy? You don't want large corporations stealing your livelihood? How is it wrong when they do it but you can pirate all the TV shows you want, and you think that is different somehow?

I want to hear your rant about copyright hypocrisy.

not "hYpo" but "Hiiipo" for hypocrisy.

I'm a writer currently looking for a new position, and I've seen so many jobs online aimed at writers specifically for training generative AI. Like I'm going to sign up to train a bot to replace me. Pretty minimal wages too, naturally.

'Public free data' aha last time I checked medical records are not public.

I've said this from the beginning: when AI will become so common that it's used by everyone, there simply won't be any new data of value to feed the AI with. A massive data feedback loop will lead to stagnation at best, as the only data available will soon be at least partially (and increasingly) generated by the AI itself.

The robots.txt thing is such a cute early-internet-naive idea. I mean, the crawlers have to respect the rules, there's nothing stopping them from saving the data anyway. Or am I getting this wrong?

Imagine if a bot crawled the news sources and learned that Sean “Diddy” combs has been arrested. We just can’t have that sort of thing.

Alles Gute zum Geburtstag! 🍻From a UK postdoc currently watching while hoping to see the aurora tonight🤞

Not blocking Meta in robots file maybe also relates to them sharing their models weights with all whereas OpenAI is not that open.

No, the problem isn't access to high-quality data. It's a complete misunderstanding of the problems they're trying to solve; they seem to believe that collecting a lot of data allows for building a general artificial intelligence. So, they keep making promises about what will be possible solely based on that belief. But now they're running against the limits of collection... and still don't understand the problem.

Ok, robots.txt doesn't mean "I will block you" but rather "please, restrict to what we allow". That's one misunderstanding of this. I can create crawler that will just ignore this file completely and will gather everything on the website. Second point - you can spoof what type of crawler (or even not a crawler) you are. You can show that you are Chrome/Firefox browser and not some sort of python library, just by changing one line of code. Third - detection is being done more on behavioural level, not on the "show me who you are" one. That's what Captcha is for. Fourth - companies will lie to get to the data, simple as that. Edit: Ok, it seems Sabrina did mention some of this but only in 3/4ths of the video... Wonder how many people will watch it till the end.