Page Ranking and Search Engines - Computerphile 

Hahahahaha

+Higgins2001 I'd say less people are writing about the history of sweaters, their manufacture, etc., and more retailers are selling them and have websites on which they may be sold.

+Higgins2001 I too miss the days when the Google search was comprehendible: I used to know that it would favor things where the words I typed were near one another in the text of the page, and if I put it in quotes it would ONLY show me quoted strings (e.g. an arbitrary choice of "toasty horse" returns "Toasty's horse rider" which is not what I typed.) Now I have to fight against it guessing what my habits, preferences, and location are. And especially that it will always point me to a new product to buy-often when I'm actually looking for information on an older model, for instance.

+Higgins2001 It could be, that a great portion of that is not google changing the algorythm, but rather more people buying stuff online on more online platforms that didn't exist back then. So while people in 2000 may have searched for "sweater" to learn about how they are made mostly, today most people might search for "sweater" because they want to buy one online. In 2000 they bought the swaeter in a RL shop. This might be a huge factor, it might be a tiny factor, I don't know, and im not a marketing data analyst.

+Jason Olshefsky Try DuckDuckGo, for the habits, preferences and location i mean. That search engine doesn't use these factors (AFAIK).

+Higgins2001 I use NoScript to make it hard for website owners to figure out who I am, so they can't tell google and have them to targeted adds. This doesn't just apply to Google. Google knows when I click on a link in their search engine and what videos I watch on RU-vid. That already gives them a lot of information.

+HexerPsy I guess by filename, description on website it was found, website it was found, text on website, Metadata of file. Maybe even Color (if you search for yellow car, pictures with much yellow in the center and named car.jpg).

Stoppi Hmmm... probably...

+HexerPsy I think they sort images based on the page the image is on, the text around it and the name of the image, they also store a bunch of metadata about it.

username_unavailable I agree

+HexerPsy further i think they search for "similar" pictures and look how often they were clicked in the search to categorize the picture and add their own meta this way to their DBs

+wlfbck20 The search engine probably first presents the best result it can now and makes the checks after you clicked a link in the results in background, like if it was a present and you clicked a link where you can order it, it evaluates the effectiveness of the results and the order of the results after the effect.

+wlfbck20 Most of it's computing is already done when you search for something. Google already knows where you are and if you are a CS. So the conclusion that the horse might be a present can be made really fast, because the complicated stuff (finding out wich sites are for horse presents or where you live and so on) is already computed. It's the same strategy as the "spiders". Google doesn't search the entire web everytime you search for something. They already did it, and know what to show you as soon as you search, because they can just look at their "handy lists".

+wlfbck20 To leverage caching, they probably already categorized your profile as one of many similar profiles ahead of time. From there, it just takes one person in that group (of millions?) to take the latency hit of caching it. In other words, when you search, they may not uniquely cross check with only your profile, but a representative profile that your profile contributed towards.

+wlfbck20 They have auto complete right :) so its processing the search before you click the button.

+Stormfox there were?

+Rob Tyler In the fake web page URL's featured in the nice animations.

Bungis Albondigas Oh thanks, I'll re-look at the video

+Bungis Albondigas haha saw them, they weren't even subtle!

Stormfox seaxyc

+RandomVidsFrapsUser Yes, that is measured too. If for example you click on a page and then return to the search results by pressing the "back" button or by searching for the same keywords again, they track how long you spent on the page that they sent you to, and based on the time spent they'll know whether the page they served you was satisfactory or not.

+Crozix That's a really good point. They probably do weight the Wikipedia domain as a whole because it's such an incredibly popular site. So it might be that for single word searches, or phrase-noun searches, that the domain's weight is increased on the assumption that information on the subject is being sought. For example "Horse grooming" probably won't return Wikipedia, while "Horse Breeds" might.

+Crozix If I understand you correctly, then yes, these static weights are exatly what they were explaining. It's the link and get linked to thing. Many sites link to Wikipedia, and it is in the top 5 most visited websites. That gives it an enormous static weight. Plus, many people who search for single words (nouns) want to go to wikipedia to learn about it. Many people know what wikipedia is, so if an unknown website comes up first, explaining what a horse is, then many users will probably still click on the wiki link, because they know wiki explains stuff. This process makes it so the unknown website on top falls below wikipedia, which is what more people want. Popular websites like Wikipedia and RU-vid often show up because they get clicked alot by searches and because of their big static weight. Althought that static weight of yours is probably always changing (in small amounts). Sorry for wall of text, thanks for reading anyway;)

Stoppi That makes sense. I think the original question was about the weighting of the entire domain. For example a very very obscure wikipedia page might not have a high page ranking; it might have more links to sources than there are links to itself. But because the wikipedia domain is popular the page is ranked higher. I think your explanation is good though, and definitely still applies.

gizmoguyar Ah, I see. Well that is probably taken into account, too, yes. But another thing is, a very obscure wiki page only shows up if the search is very obscure, and therefore might not have many results, and certanly no high ranked ones. So i guess if you search for underground stuff, it is probably even more likely to get a wiki result. Also because most people know what a horse is, but maybe not so much know what Epizootic lymphangitis is, so for that search term, more people who search it want to know what it is. (I totally searched google and wikipedia for that example :D)

+linkviii It does if you care about people using screen readers as well as developers' sanity.

+linkviii It makes it so much easier to read when you develop it

+Haniff Diiltiown

+vandos1 thankss I'm trying to learn for uni aswell want to make a search engine for a project lol

+cubedude76 Yes. But you are not going to search for "Google" on Google. If you search for "Google" on Bing, for example, Google will be at the top of the list because of the page rank for the specific term of "google", it has the most authority, namely the keyword in the URL and other people linking to it as Google.

+GuildOfCalamity Yes please

That's pretty much the pages Google doesn't index

+Nikolaj Lepka do you have any proof for that? As far as I know Yahoo was there before Bing came along..

+Nikolaj Lepka Hi, are you sure? Do you have any references about this subject? In my testes, yahoo and bing have different responses for the same search.

Do a quick google search about yahoo and bing. It's something about Yahoo making a deal with Microsoft a few years ago to use bing as its back-end search technology

+Jonas Heinrich Nice! Thanks. We developed an anti-plagiarism app and this might help to improve performance, since Bing and Yahoo has the same engine, in theory.

***** how would this information help performance?

DuckDuckGo primarily > google if I don't get the hits I want > bing if I manage to sift through every result google throws at me.

+Suvi-Tuuli Allan DuckDuckGo with a browser plugin that provides a "Search on Google" button ;)

+Suvi-Tuuli Allan DuckDuckGo & if that fails then the same search with !G at the end, DuckDuckGo's way of allowing you to do a search using Google, or other search engines using different letters after the '!'.

Neat, yeah. I hear that Bing is good for pr0n.

+pcfreak1992 he was just distracted thinking about horses

+Ben Rowe My Lovely Horse sounds a bit fishy...

+Anston Music i'll just fetch my stool.

+pcfreak1992 This guy's fantastic. He has the clearest expression of any of the computerphile guys.

+pcfreak1992 Typical Computer Science guy while not being infront of a Screen and having to talk to RL people

+MrRolnicek As far as I understand DNS and Google (not as much as you probably), their DNS is helping the spiders with locating websites and contents, but I can't really see how a DNS could improve the quality of search results. Because it doesn't read content nor why users (who use the DNS) are there. He sees WHAT they visit, but google sees that anyway, even if you don't use their DNS. Does this help?

Stoppi I guess, but how does google see what they visit "anyway"? I mean sure google knows what anyone who uses their DNS wants to visit, but unles they go there from actual google and you dont use their DNS then they wouldn't know?

MrRolnicek Oh, I meant when you visit after a google request, cause that's probably what mostly matters for the rating of sites. But apart from that, and maybe more for adwords (google ads) and less for the search, they track you over trackers (I don't know about that, but the ghosterything detects stuff), and over google analytics, which is a service on basicly every site, they know who visitid what when from where and so on.

+zatoichiMiyamoto you forgot about copper that makes most of our computers work :p

+zatoichiMiyamoto your earthquakes have pathetic ground acceleration.

+Troy Adams well, it can and some probably do, but considering how much money traffic from search engines generates it'd be counter-productive for most.

They can and sometimes do. Also you can add a "robots.txt" file that tells spiders not to look at some things. However most sites want spiders to index it. Indeed the robots.txt can be used to add info that makes the spiders job easer.

+? the Platypus I've never thought the spider as a anthropomorphic computer program. Until now...

+Troy Adams: Most of the time (if we're not talking about deep web) one want's very much to be well indexed by a crawler. Websites are meant to be visited, read and interacted upon by users and what better way to have them on your website, than to let people know about you by web-searching?

+Kevin Fontanari Six thousand nine hundred forty-third

+teekanne15 Computer guys aren't the most relaxed when we're trying to eloquently explain something. If I have to debug or refactor something by explaining to myself how it's supposed to work I have to get up and walk around. Maybe the brain is screaming for more oxygen.

+Ted Chirvasiu macarena

+Ted Chirvasiu lmaoooooo

+deldrinov ugh...

+VishnuM I know, but it was just 15s

+VishnuM if you would stop being annoyed by "first" sayers, they would go away. Just saying...

+David S. Look at this guy. :D

.