In my paper on Benford's Law, I described this phenomenon in a way that will explain to my grandmother (who's quite smart, but doesn't know about this thing): Benford's law is a consequence of our tendency to use linear systems (e.g., rulers) to measure an exponentially dynamic world.
“What about the fundamental constants of the universe? They don’t change at all, let alone along exponential curves” Nobody tell him about renormalization group flow
I'm not disagreeing indeed. I just wanted to point out that the numberphile video was more about how counterintuitive the principle is - regardless of the precise distribution. When you know how it works things switch of course and as you've said it becomes more interesting when the law actually doesn't apply. But no way one would have expected that before coming across Benford's law. I for one am a mathematician (not exactly familiar with number theory though) but I had no clue whatsoever :)
This was great video, I have watched quite a few RU-vid videos including the ones you mentioned, and the Khan Academy one, in which Benford's Law applied to eg Fibonacci series and powers of 2 is very easily explained. But I was struggling to to get my head around why it works for 'real world data'. In your video of course this makes sense! I hadn't managed to make the jump eg With the population example if as you say if a population of a given country spends 30.1% of time with first digit one, then at any given time 30.1% of countries will have 1st digit one. Thanks so much, I was missing that!
Yes, although *why* the log scale is more 'authoritative' or 'relevant' than a standard linear scale is still an interesting question, the expected probabilities for each digit under Benford's law do indeed come straight from reading off a log scale.
Haha! Sorry about that, it's certainly a strange recommendation given that this is a rather niche video, really intended for folks who are already somewhat familiar with Benford's law and want to know the details. I have seen an uptick in views recently, so I'm going to add some temporary disclaimers to point out how useless Benford's law would be for trying to prove/disprove the presence of any fraud in the US election. Feel free to ask if you have any questions, though!
TheHue's SciTech Probability of leading one. For example from 100 to 999. Let f(n) to be the probability of leading 1 with n=raffle size (so plot of f will be an wavy graph as in the numberphile’s video and in your video as well). I define the weighted average to be the sum of f(n)log(1+1/n) from n=100 to 999. And I get 0.3023 for this case.
What I don't understand about Benford's Law is that it applies to numbers that don't grow exponentially. It even applies to numbers that don't grow at all, like the area of all the countries in the world, whose boundaries are entirely arbitrary.
The thing is, if you believe numbers are generally linearly distributed, then you might expect leading digits to generally be uniformly distributed. But if you believe numbers are generally multiplicatively or exponentially distributed, then you'd expect Benford's law to emerge. To say that a particularly precise explanation is required would be to imply that things being linearly distributed is the default position, but if you think about it that's an absurd position to take; one that us humans might be biased towards only because of our counting system. Take country areas; there's only a few large countries (Russia, China, USA etc), but lots of Liechtensteins, Monacos and Belgiums. Spanning that many orders of magnitudes, it's pretty obvious that the distribution is more exponential that linear. Moving on to things like fundamental physical constants; that's a bit harder to give a clean explanation for but it remains fundamentally true that the jump from 9-10 is a lot smaller than the jump from 10-20, no matter how you look at it.
@@TheHuesSciTech I can understand that some data grows exponentially, like the population of a country. The more people there are, the more children those people can have. Other numbers don't have to grow exponentially. For example say the amount of rain that falls in an area over a given period of time. If it rains for example 99 mm at a steady rate it will take the same time to go from 10 mm to 20 mm as it will take from 80 to 90. And if it doesn't rain at a steady rate, it may rain more heavily at the beginning and go faster from 0 to 20 or it may rain more heavily in the middle, from 40 to 60. And neither possibility is more likely than the other. Besides, say it never rains anywhere more than 250 mm. From 0 to 250 there are more numbers with a leading 1 than with any other digit. Then there are more numbers with a leading 2 than with the others. But digits 3 to 9 have all the same frequency from 0 to 250. So even if 1 will be more likely, I don't see why each leading digit should be a little more likely than the next. As for country areas, it's true Russia is about 35 million times larger than the smallest independent state, that is the Vatican. But Russia didn't actually *grow* to be that big, starting off as a tiny country and going all the way up to its current size. At some point someone decided that the boundaries of Russia should be those, and the same goes for each country in the world. And I don't see why such arbitrary decisions should lead to a distribution of leading digits that follows Benford's law. And still it does!
@@Eduarodi You're missing the part where the 10mm of rain is need to go from 90mm to 100mm, yet a full 100mm is needed to get from 100mm to 200mm. This is my main point, you're taking a very everything-grows-linearly, Arabic-numeral biased approach.
@@TheHuesSciTech And a full 100 mm is needed from 200 to 300, just as from 300 to 400, etc. Eventually only 1/9 of the time it'll have rained a number of mm beginning with a 1, just as in the raffles on the Numberphile video. Or a bit more, depending on at what point it stops raining. But that doesn't account for each single digit showing up more often than the next. I don't mean to say that everything grows linearly. I mean that some things grow linearly, like the rain, and other things don't, so it surprises me that Benford's law applies to things that do grow linearly.
But the law still works for all the natural numbers, kind of. Whatever the exact percentage is this fact is amazing by itself, because one expects all digits to be uniformly presented.
No, 1 is still the most common. Same formulae apply, but you just need to use base 20 logs instead of base 10 logs (when I had "log" in the video, that was the base 10 log).
does this have anything at all to do with us using base 10 for math? what would happen in base 16? i expect there would be an exponential curve there too, when covering several orders of magnitude, but how many percent of numbers would start with 1?
The logarithms (logs) I'm using in this video are base 10 logs (sorry, maybe I should have been more explicit about that). I'm using base 10 logs because we're dealing with base 10 numbers. If you did the same in base 16, you'd want to use base 16 logs instead, everything else would be correspondingly the same. log16(2)-log16(1) = 0.25 (exactly one quarter!), so 25% of hex number would start with 1 iff they were drawn from a Benford-style distribution.
+ThomasHaberkorn I'm fortunate to be old enough that my father had a slide rule sitting in his drawer -- he never used it, but at least it was there for me to play around with. And yes, I think having that slide rule around helped me to grasp logarithms, even if I've only ever used a slide rule for the sake of it.
And a Circular Slide Rule is even better for the most common prime aspect of the 1D universe. But the straight one is a similar representation of continuous connection as in the 2D Pacman universe of linear relationships.
TheHue's SciTech yes but keep these videos coming and please don’t hold them up trying to please those that want to listen for mistakes. It should be enough to issue an erratum in the description or in the comments. I do appreciate Lau and his comment here. But having a brain that can get these abstract concepts and yet have a gift for explaining them to mere mortals like me is a rare gift. The slip of the tongue was easy to spot in my opinion
Hello! Thank you for this very insightful video that you have made! I am currently this Law for a mathematics assignment. May I ask the following questions? 1. How do you derive and apply the formula of Benford's law? 2. Would the log Scale diagram be of aid in this scenario? 3. Would Probability distributions apply in this law as well? Once again, thank you for this insightful study of the law and I hope to hear from you soon! :)
The idea, put simply, is that things end up evenly smeared on a log scale, not on a linear scale. That's just how things happen in a world where multiplication happens more often than addition. So to answer your question #2: Yes. Question #1: Since it's on a log scale, and we're asking how big the probability is between points on the log scale, the derivation is trivial: the probability of getting a number between 1.000... and 1.999... is the distance between those points on a log scale. Those points are located at log(1) and log(1.999...), so the distance is log(1.999...) - log(1), basically, log(2)-log(1). Same applies all the way up. Question #3 doesn't make any sense to me sorry.
Benford's Law is really quite simple and there's precisely zero controversy about how it naturally arises where it does. I agree that some explanations are poor.
Repeated convolution always tends towards Gaussians in any case (although haven't even looked at this video in 7 years, not sure if I'm missing something)
+Carter Allen Not that I know of, I just "made it up" (although I give justification for it later, so it's not like it's false or anything :-) ). Benford's law refers to the first digits of "naturally occurring numbers" (fairly vague and opened-ended), whereas the theorem you're pointing to is extremely specific (and contrived, even).
+TheHue's SciTech Right of course. Well thanks for the video, it gave me a good idea of where this phenomenon is coming from. I was directed to the paper by Roger S. Pinkman for further investigation and I'd recommend it for anyone looking for some rigorous discussion of Benford's Law
I don't know but I am willing to be my life that benfords law does not work on the duration of pop songs. I believe 100% that the top number would be 3. It definitely be 1 or 2 or any number above 4. Can you tell me why ? And how long each song stayed at number 1 would be interesting..
For Bedford's law to apply, the set of numbers you're looking at must cover several orders of magnitude; which is not true of pop song durations, people's heights, etc.
Benford's law involves exponential growth, right? So let's say you start with $1 and double it every hour, then you have $10 in 3.32 hours. The time it takes to get from $1 to $2 is 1/3.32 which = .301. So p(d) = the time from d to d+1 divided by the time it takes to get to 10. But then that cycle will repeat from 10 to 100, 100 to 1000, and so on, which is where your "orders of magnitude" come in. I think I understand it now.
That's a great explanation, but it does rely on your assumption that exponential growth is required. Some people would posit that the effect appears even in situations where exponential growth is nowhere to be seen; the later part of my video is an attempt to explain why exponential-ish distributions are frequently found in nature, even where there is no growth to be seen.
@@CorrectCrusader I'd say trying to apply Benford's law to swing states is flawed for two reasons: A) the difference between a win for team A and a win for team B is very small, in the order of at most 10,000 in states with well over 100,000 votes each way. So the probability is very slim that the leading digit is even *changed* by the little addition or subtraction of votes that it would take to flip the result, let alone changed in a way that is able to reveal the presence or absence of fraud . B) the number of sample points (states) is waaaaay too small to apply Benford's law in any case. I would suggest you try applying the same test to previous elections. If you collected all the US elections in history together, you'd probably start to see some Benford's law appearing, but if you picked any one random election swing states, it's just too small a sample for it to be anything but a random mess of digits.
@@CorrectCrusader libguides.princeton.edu/elections looks fairly promising. FYI, some folks have pointed out that looking at the *second* digit and looking at votes on the county level might is more promising approach, that has been used for similar reasons in the past. But still, I urge the highest level of caution if you're not experienced with statistics, p-values, the dangers of p-hacking, etc. This is a very dangerous field to work in if you're going in with preconceptions and are just looking for anything that looks a bit like a smoking barrel, and basing your whole view of your democracy on that basis alone (just on the off chance that that's what you're doing :-) ). Kinda like how people who don't quite fully understand basic physics start and spread flat earth + moon hoax stuff. If, on the other hand, you're just playing around with numbers, then by all means have fun! But again, to have even a slight chance of coming to a valid conclusion, remember to apply your techniques to a whole bunch of past years to make sure your approach doesn't call "fraud!" year after year!
Not an expert at math or Benford law but just curious about things. So, I just tried to observe a couple of scorecards of matches of cricket and they seem to closely followed Benford's law as well. I counted the leading digits on runs (1-250), wickets(1-10), balls(1-300). Now, why does that work? Just another question- can this somehow detect match-fixing in sports too just like it is used to detect bank frauds?
Well, the single-digit wickets number is a bit of a silly case -- that'd be more along the lines of a Poisson distribution than an example of Benford's law. As for runs, I think we can both agree that if a player gets 50 runs, you can more be more confident that they'll get to 100 than you would be in a player than has zero runs (i.e., any batter) getting to 50 runs. They're probably a batter rather than someone at the bottom of the order, they have their eye in, they're having a good day, the opponent bowlers are bad, etc. So it'd be unsurprising to see something vaguely resembling an exponential distribution of runs; and anything with such a distribution will be seen to obey Benford's law. You could also make up formulas for the expected number of runs: E(runs) = batter goodness * bowler badness * luck factor etc. As explained in my video, long runs of multiplication create Benford's law. Detection of fraud through Benford's law requires a lot of data, I doubt it could be meaningfully applied even to an entire team's year of operation.
@@TheHuesSciTech The data was mixed (runs, wickets etc) as I saw that it would apply to mixed datasets as well. Probability numbers- 0.46 and 0.34 for digit 1 on two separate data sets and visually it seems like a good fit. Dataset was also not a really large number set- n=100 for both. Didn't do any statistical tests as such. I don't think I have still got an absolutely clear idea of where the distribution would pop up and where it wouldn't. I guess I will need to read more on it to grasp it clearly. Nevertheless, it's a pretty cool video and idea. The population example in terms of area made good sense. But I am still a bit unclear about the idea of multiple multiplications. Thanks for the reply anyway :)
Hi. Yeah, I got here from watching numberphile and I'm still pretty confused... could you please do another video but for someone like me who had to take Algebra 4 times! I was really good at geometry but that's spacial.?. 🤔😥😨🤓
Hmm, a few things: A) Have a look at ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-vIsDjbhbADY.html , IIRC it's a good one on the subject. B) The main point of my video is to show that Benford's law isn't the mysterious, spooky, "unexplainable" phenomenon that it's often described as. It's showing that it's just an artifact of our number system. If it seems confusing to you that it's a worthwhile thing to understand, you may very well understand it better than most. C) I can't make another video if I don't have a specific question or omission to address.
US Election Note ------------------------ --- TL;DR: Watch ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-etx0k1nLn78.html It seems there's an uptick in views on this otherwise very niche video due to the use of Benford's law to detect fraud, and the (perhaps maybe unsubstantiated??) claims of fraud in the US presidential election. I have absolutely no interest into wading in to any side of that debate, but I do want to point out the version of Benford's law in my video (using the first digit) is completely useless for the task of detecting fraud in a single US election, regardless of whether you're trying to prove or disprove fraud. So please don't try to apply it for this task; doing so would be completely intellectually dishonest. Justification for these statements in the next paragraph. TL;DR: Watch ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-etx0k1nLn78.html I say this for two reasons: A) the *difference* between a win for team A and a win for team B in a swing state is very small, far below 10% of the total vote count. So the probability is very slim that the leading digit is even *changed* by the little addition or subtraction of votes that it would take to flip the result, let alone changed in a way that is able to reveal the presence or absence of fraud. B) the number of sample points (states) is waaaaay too small to apply Benford's law in any case. TL;DR: Watch ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-etx0k1nLn78.html That's all there is to it. I know it's tempting to think "I want to detect fraud, I've heard Benford's law can be used to detect fraud, so I'm going to jump to applying Benford's law". But the moment you either A) think through how Benford's law works and whether you would expect it to apply to the vote counts in 50 states (or especially a handful of swing states), or B) run little toy simulated elections in Excel/Google Sheets, perform "fraud" in half of them, and then try to (and inevitably fail to) detect which is which using Benford's law, you'd see how completely ineffective it is in this particular situation. TL;DR: Watch ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-etx0k1nLn78.html Some articles are apparently suggesting using second digit distribution on county vote tallies, which sounds more somewhat more feasible. But although that second digit is more likely to the changed, they will also tend to have a much flatter distribution, so a huge deal of care is required. In any case, I can hardly overemphasize the importance of sound statistical thinking; determining your belief in your democracy by fooling around in Excel if you haven't even seen a p-value or null hypothesis in years is a very dangerous prospect. TL;DR: Watch ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-etx0k1nLn78.html But if you're trying it out just for fun, I would suggest trying to apply the same test to previous elections. If you collect all the US elections in history together (www.icpsr.umich.edu/web/ICPSR/studies/08611), you'd probably start to see some Benford's law appearing, but if you picked any one random election's swing states, it becomes much more subtle and difficult. If you think you have an algorithm that makes a call on what probability of fraud there is in any given election, try applying to past elections. If your algorithm says that a majority of elections since 1788 have been fraudulent, then... you might want to rethink whether your algorithm is correct! And really, if your p-value cutoff is 0.05, then only 5% of the elections should be considered "fraudulent" (unless you actually think there has been fraud in the past as well). And if 1000 different people each have their own go at this, all of them making no mistakes (!!!), then 50 of them would see "statistically significant" evidence of "fraud", even if there was none! TL;DR: Watch ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-etx0k1nLn78.html [Sidenote: I can't help but point out the only *careless* fraudsters get caught using Benford's law. Remotely competent fraudsters know about Benford's law, choose their fraudulent numbers from an exponentiated distribution (or add/subtract in percentages rather than absolute numbers), and thereby make their fraudulent numbers looks legitimate as far as Benford's law is concerned. So that's something else to keep in mind... ] TL;DR: Watch ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-etx0k1nLn78.html
In the couple of articles I've read talking about this the suggestion is to use second digit distribution on county vote tallies rather than first digit distribution on states. Whether that makes a difference or not I don't know, but it sure is interesting to watch.
Benford's law implies the data spreading or augmentation over time has some constant to follow. Seems years ago Washington Post has cited an article of detecting election fraud in Iran by looking at Benford's law. There is a more to consider.
“Digital analysis using Benford’s Law was also used as evidence of voter fraud in the 2009 Iranian election. In fact, Benford’s Law is legally admissible as evidence in the US in criminal cases at the federal, state and local levels. This fact alone substantiates the potential usefulness of using Benford’s Law.” -www.isaca.org/resources/isaca-journal/past-issues/2011/understanding-and-applying-benfords-law
@@southafricanizationofsociety20 Yep, I had already updated my message accordingly. Read carefully, the only thing I'm ruling out is A) looking at the first digit of state vote tallies only, and B) attempts by amateur statisticians to wade into the subtle field.
