The creators of TikTok caused my website to shut down 

the biggest crime here is naming it ByteSpider and not SpiderByte

You probably want to block them in cloudflare rather than on your server, currently they’re still wasting your bandwidth (just in a much much more reduced form) by blocking them in cloudflare they shouldn’t end up wasting any of your bandwidth at all, they’ll never even touch your server. A simple page rule should do the trick, and even on the free tier you should get 3 page rules.

ByteSpider checking for updates on the Lego Island series every 100 milliseconds

I definitely appreciate the PSA, you literally just saved me from having to diagnose the same issue on one of my servers.

Looks to me like they're downloading any and all images they can find. Could be for training an AI model. Looks like you have a forum, so that's why there's tonnes of requests coming your way.

oddly enough, a chinese crawler completely tanked my college professor's homework submission website. it was extremely persistent too! i remember the entire system was down for a few days while they worked out exactly how to block it. from what i remember, they eventually just blocked every IP not from the USA lol

You'd think non-malicious inadvertent DDoS would be the easiest thing for Cloudflare to spot and block? Maybe it's whitelisted?

i dont know if its still a thing, but back in the day i implemented a spidertrap to all of my websites. easy thing. you need a 1x1 pixel transparent image on every site linked to your trap-script and in your robots.txt you declare the script as disallowed. so good spiders wont go there and bad ones will be blocked...

At least they had the “decency” of using a proper UA. They could (and hope that not will) just use the Chrome UA or worst, weighted random UA’s

I'd recommend you set up a simple email notification that the server would send to you if an arbitrary bandwidth threshold you'd consider too high was exceeded. This way you could resolve the issue before any downtime occurs.

I'm just sick of us being expected to foot the bill for something a large corporation does without your consent. These days our data makes us look like little more than dollar signs instead of people to a lot of those tech company execs.

Maybe the crawler gets into an infinite loop. Might be the classic 'detecting cycles in an undirected graph' problem.

OH MY GOD ARE YOU KIDDING ME... so THIS was the reason my site went down too??? I suddenly couldn't reach my website at around July 11th or something too, and a few minutes later my provider sent me a mail saying they temporarily disabled my site till the figure out what's going on, it also looked like a DDoS in the logs and everything... wow... Now that mystery is solved, thanks! :)

Sue them for the 10$

When exceeding bandwidth, VPS's should not be suspended. I believe they should just shut off all network access, so the KVM console would still be accessible for troubleshooting. Also, if the reason is a DDOS attack, it will stop reaching the server and you can check where the traffic is coming from.

Why do I get the feeling that Bytedance paid Cloudflare to look the other way and ignore their aggressive crawler shenanigans...

knowing how tiktok spies on and tracks phones like crazy, their web crawlers being extremely overkill just to scrape every last bit of data makes sense

Thank you, I have the exactly same problem with my website, I don't even host anything on that website besides the default WordPress website but I had a huge amount of "users" accessing my site

Since auto-regressive language models are so trendy these days, and there might be fears of export bans for using already collected corpus like CommonCrawl, they might be trying to build their own. Maybe some Snapchat-esque annoying "friend" for lonely teens.

5:46 Have you tried contacting their email address from the UA string? In case it is a legitimate issue, they might want to hear about it.

Now to find out if it was Tik or Tok who was responsible for this.

Would be nice is certain user agents, like web crawlers, had a default limit on how often they could access the site. While obvious malicious crawlers could get around this, reputable ones that wish to stay whitelisted by default would obey.

10 bucks for 100GB? Jeez, I'm paying 1€ per TB over here, that is just straight up robbery

I believe Bytedance wants to create a new Chinese web browser to compete with the blocked ones

The old adage applies here: Never attribute to malice what can be easily attributed to incompetence.

Assuming it really was Byte Dance, I expect this was not intended behaviour. It would cost them bandwidth too, though maybe so little in comparison that it just looks like regular background noise. An earlier alert would be been very useful here.

Good thing this is happening in the age of DDOS-prevention. Imagine getting fucked by a spiderbot back in the days when you'd host your website on your home network and your ISP charged you by data usage.

this is very useful information. I been thinking about putting together a website for some friends, so now I know that I might need to look out for this.

Luckily your site has a static limit with a fixed price. Imagine the costs if it were an uncapped pay-as-you-go service like most cloud services

Really interesting stuff. I've considered making a website before, but I wasn't aware of stuff like this. Thanks for the heads-up!

That's a wild ride for sure.

i thought for sure that it would be the same as a friend of mine had about 10 or more years ago. he kept a travel log website where he uploaded photos to because he was a nerd who liked to travel. he eventually visited the original Starbucks and uploaded a picture of the original logo. a more popular website used the photo but they didn't download and host the photo themselves. instead they just linked to the photo so when you loaded up the more popular website it would give your browser a link to where the photo was located on my buddies website so it could display it. his traffic skyrocketed. i believe it has a name for when people do this but i don't know it. he did find a fix for it where any website linking to any photo on his website like that would then be blocked.

Feel like cloud flare should detect this sort of activity and automatically block the user agent. At least temporarily too see if it stops or continues. Instead of suspending the client.

"and i thought charli d'amelio was the worst thing bytedance had done to me" The description is the best part of this video XD

This reminds me when i had a minecraft server running for me and my friends. Woke up one day and went to check on it to see the that the server command window was full or disconnected messages. Did some stuff like editing the hosts file to make it redirect the IP back to itself or simular (prob did nothing) but eventually just went with running malwarebytes since it blocks suspicious requests

I'm interested if you could get a response or not by emailing the support. Probably not, but it would be funny to see their reply.

It's a shame you can't invoice them for the bytef**king they did.

It's always a good day when Matt uploads a new video. And rants about a random company as well.

Surely as this becomes more common cloudflare may begin to implement a catch for similar overzealous crawlers?

Kinda irony that your sub is 404k now. Stay safe out there

“im not that popular” hits hard 😭

funny you use the spider-man 3 in that clip about bytespider because im fairly certain the font from the bytespider logo is that same one from the sam raimi spider-man films lmao

Worth noting that Twitter / X and lots of other sites have stopped bots from crawling them now, something todo with them training their AI with content from their sites.

I would find it strange for them to be making a new SE. I believe TouTiao would not really need a remake, saying as it already has over 100 million users daily

always happy to see you upload :)

Hi Matt! I love storytime with Matt! You're really fun to listen to.

cheers on the psa, the more people that share knowledge like this in unbiased ways like this the safer we can all be on the interwebs :)

truly a chinese moment

I've also noticed a ton of traffic from Singapore recently, and my domain just has the default parking page!

Not the type of MattKC video we expected, but the one we deserved. Always happy to see you have uploaded, no matter the content ❤

ur uncached traffic ratio is really low tbh, maybe also take a look at that to take more load from your webserver.

You KNOW it’s going to be a good day when MattKC posts a video. Keep up the good work man

Thank you, excellent analysis!

ByteSpider's logo using the Raimi Spider-Man font is incredibly silly

...And several years ago here I simply failed to see several of the classic cartoon blogs simply because I was detected using VPN. Tom Scott has warned us well. The internet is a cesspool of patchwork offense and defense, shady strategies and clumsy turf war.

Please more content like this. I host websites and services and this helps me keep up on new threats and how to deal with them

I work at a cloud provider. We have a wide band of customers and I'm afraid to say that we have seen all sorts of issues with search engine bots. Not just from fringe ones either. Even the large ones can cause weird issues. The problems we have observed include big spikes in PHP-FPM processes, tens gigabytes of cache being generated by weird access patterns and even extremely high database loads... Funny how that goes sometimes.

Old man yells at cloud

I hope they won't stop using that user agent string, or else you've got an issue. It's weird how they give you that string, but do everything else in their power to stop you from blocking their crawler.

3:11 who tf uses android 5.0

They’re almost certainly collecting mass training data for a new AI model

Handy to know - thanks for the heads-up!

The fact that the requests were coming from seemingly random Singapore IPs still suggests a botnet. I wonder if there's a Bytedance app doing ill-conceived distributed computing in the background. You wouldn't need any kind of app permissions to browse the web, nor would any one user notice it the way crypto leech apps tend to be noticed.

New video from Mr. LEGO Island

gotta love their use of the spider-man movie font

Friggin fascinating mate

That was a nice reminder to check my home server nginx access logs. Thank god I set it up correctly before opening up to the world.

fail2ban with BadBots Rule should also do the job ;) then it would already block the IP Address temporarily in iptables (or other Firewall thing that fail2ban was configured), which reduces more the Traffic they will produce

giving them the benefit of the doubt is like giving a serial killer a benefit of the doubt it's just moronic

it seems to be grabbing every single image file on your forum block that thing asap

How's the Lego island decompilation doing?

Openly showing this is the best thing you can do. Even if you can't prove this is malicious, you're still providing info for the Internet. Maybe more OCD users will get to it. Who knows?

Happy 404k subs! 😄

NEW MATTKC FINALLY

Hi Matt, love the content you make! Looking forward to this watch!

Another thing to note is that your site doesn't have a robots.txt file. I can't say if it matters though.

Its funny that he made the spiderman reference because it uses the Toby Maguire spiderman font

The reveal on stream was epic.

All the crazy tech corporations been in the news recently LTT, now Bytedance again its crazy and also keep up the good work MattKC

Didn't see the email contents on mobile but if your provider wouldn't spin the VPS up with the external IP blocked so you could access it through a virtual console id probably ditch them lol.

Matt KC is back fr

These guys are a pain in the ass to block, you can't even just blackhole the entire bytedance IP range because the gits operate the crawler from other AS numbers. They're constantly in my home servers logs.

New Matt KC video 🎉

Have you sent an email to the feedback email address in the user agent string?

Learned quite a lot of new things today. I'll have this in mind in case my website gets any run-ins with unwanted attention

My mail server logs were an absolute mess before. I then selected a bunch of countries that I don't care about and blocked them in my router. And suddenly? Silence. Peace and tranquility

I was a webmaster once. Then i realised that i want to get older than 30 and stopped.

So a tech company with way too many resources incompetently played with tech and now everyone else has to pay for it

4:03 Where i can apply?

I remember it happened to me years ago with another one of these Chinese crawlers, I think it was Yandex or something Those guys never learn

Yoo new mattkc viv

Never watched Tiktok. Never will. Definitely will not now.

not people commenting on the video before even watching it assuming this is about people who publish content on tiktok and immediately jumping the gun, noo, that could never happen to a video that's actually about a web crawler

NEW MATTKC UPLOAD!!!!!!! AND I FUCKING **MISSED** IT BC I WAS ASLEEP ALL DAY.... 😭

wow. being unable to view my own access logs because my website’s bandwidth allocation has run out would be a MASSIVE dealbreaker for me. that is ridiculous.

I love this shorter type of informational video! It took me a bit to notice the lack of music, which might be why something felt off to me. Also not seeing you in the video as much as usual felt different. Totally not against you experimenting with it though, cause I can imagine that it would save some time on making the video and for me I don't think the video really suffered too much from it.

i call this a dub

Had few run-ins with this as well, it even ignored robots.txt majority of the time as well so added few rules on CF to just drop it.

Yep, I had to block them last week. It's an evil little runt.

Oh hey it’s the Lego island guy