Variance and Standard Deviation: Why divide by n-1? 

i agree.....

@@ankurpriyadarshan my proffersor at IIT Bombay explained this ..if you find expectation of variance of sample (n-1) then it will come out same as expectation of variance of population

True, still, it makes more sence when you meet idea of biased/unbiased estimators.

Thanku

many things are waved away these days haha...they definitely assume we know the purpose of everything

hey good luck man! i have heard actuarial science is really tough, it was one of the majors i was considering for uni as i graduate from HS this year and i got a friend in SA who's also doing actuarial science how's it been going so far?

You're on... though please ship in winter lest it arrive as Golden Gaytime soup.

Note to self: Golden Gaytime Soup.

I prefer Weis Bars!! :D

melting...

A lower order of integrability would be required for L1 norms, which with power laws of some choice of parameters might exist as a first moment while the second moments such as variance would not exist.

Thank you so much. I have been trying all morning to research this and you are the first person I have found who has directly and clearly said that the squaring method isn't better than the absolute value approach, it's just something that people often find useful when they want to do other things with the data later on. Every other resource that I have found on this topic seemed to be implying that there was some unexplained other reason why the squaring method was *better* than just taking the absolute value.

​​@@galenseilis5971A lower order of integrability would be required for what exactly? I might be missing something, but taking a norm of data is just a kind of aggregation (summation). So, whether you take an aggregation of an L1 norm shouldn't prevent another aggregation that is an L2 norm (variance).

​@@chasemcintyre3528 I'm glad I could provide some comfort 😄 Most often if someone can't give you an answer to the "why" it's likely that they are just parroting what they've been taught or heard. Independent thought is the only way to fill those gaps in knowledge. Good on you for searching all morning despite the resistance.

@@gustavstreicher4867 Reviewing your comments and the video, you are apparently missing the distinction between a sample and a(n infinite) population. I'll spare a few minutes to give you a more detailed explanation. But before getting to your question, I want to point out something misleading in the video above. They present a handy-wavy explanation of why we use n-1 degrees of freedom instead of n degrees of freedom in the denominator of variance. Many people call the former the "sample variance" and the "population variance", but this is misleading because they're both sample statistics that can be used to estimate the population variance when it exists. The reason we often prefer using the variance estimator with n-1 degrees of freedom is because it is corrected for estimator bias at small sample sizes assuming the data are sampled from a normal distribution. Both estimators are consistent estimators for the variance of a normal distribution, meaning that they both eventually converge to the population variance. You have not said anything that makes me believe you fell for this misunderstanding, but I am offering the caution just in case. Now let's head in the direction of you question. As you describe, you can calculate either of the (sample) mean absolute deviation (MAD) or a sample variance on a finite collection of real numbers. And as you mentioned, the L1 and L2 norms are closely related to these sample statistics. The L1 and L2 norms induce the Taxicab and Euclidean metrics respectively. The MAD is a rescaling of the Taxicab distance from the arithmetic mean. The variance is a rescaling of the square of the Euclidean metric from the mean. There is not particular issue with doing this on a sample, but that wasn't the substance of my comment which concerns the population. Let's go over some population statistics now. In mathematical statistics the population mean is the expected value of the random variable, often denoted as E[X] for a random variable X. I don't mean that some value is to be expected in an intuitive sense per se, but rather that there is a mathematical operator called the "expected value" that can be applied to a random variable. A random variable is a measurable function (i.e. its preimage exists) of the outcome space of the probability space. Which is to say, you should think of random variables as mathematical tools rather than something that is intuitively "unpredictable". A random variable is a type of mathematical model of a part of your data. In special cases an expected value of a random variable is an arithmetic mean, but it is more general than that. The population variance is likewise defined as E[(X - E[X])^2], so the expected value is relevant to understanding both the population mean and the population variance. The population analog of MAD is the expected value of the absolute difference of the expected value subtracted from the random variable, denoted E[|X - E[X]|]. For continuous random variables, like a normal random variable, you'll see that the expected value is defined in terms of an integral which is just a convenient notation for referring to certain infinite series. Okay, that's an overview of the definitions. But what's the problem then? The problem is that these population quantities do not always exist. Fortunately they do exist for many distributions, including the normal distribution. One example where none of the population statistics we have discussed so far would exist is for a Cauchy distribution. I invite you to try computing the MAD and variance (either flavor) on samples of increasing sample size from a standard Cauchy distribution. You'll find that neither of these statistics will show convergence behaviour in long term. The sample quantities will exist, but they will not estimate any stable population quantity. Instead they will just jump around aimlessly. The wikipedia page on the Cauchy distribution currently has some information on this unstable behaviour for the mean. Let's consider that "order of integrability" part of my earlier comment now. There is a statistic which generalizes both the MAD and variance. Instead of considering an L1 or an L2 norm, we can consider an Lp norm. It induces a metric which we can take to a pth power to obtain the generalization. In terms of population statistics we can consider E[|X - E[X]|^p] to be the formal generalization. There is a downward closure property that if for two orders p > q then if E[|X - E[X]|^p] exists so will E[|X - E[X]|^q]. The smallest order p in which the functional (E[|X - E[X]|^p])^(1/p) exists is what I called the order of integrability. So the population MAD might exist even when the population variance doesn't, which was the point I was making in the first place. Why doesn't the population variance always exist for any distribution? Well, the quick handy-wavy answer is that some infinite series represented by these integrals don't converge. We already touched on that above that estimating something that doesn't exist isn't really meaningful or helpful. I mentioned before about power laws, e.g. the Pareto distribution, which are interesting cases in this regard because sometimes these population statistics exist and sometimes they do not depending on the parameters. But I won't labor that as this comment is getting long. If my explanation isn't clear, I suggest you go to a site more suited to discussions about math to get clarification. An example is Stack Exchange's Cross Validated community which have support for mathematical notation and have members who are familiar with this topic.

The real analytical reason that the variance is divided by n-1 is that it is the only way to scale the sum of squared deviations from the mean so that the sample variance is an unbiased estimator. In other words, the expected value of the statistic given by SSD/(n-1) is equal to the population variance (see the definition of biased estimators). If you want a proof, you can google “sample variance is an unbiased estimator”.

The "rule" you mention is true only for one of a zillion distributions i.e. the normal distribution. It is not true more generally. There are bunch of other ways to place bounds on outliers such as Markov's an Chebyshev's inequalities.

Hi akhil! I'm sure you could do it numerous ways, but I just needed a random selection of integer values, so I used floor.math to remove all of the decimal places that tag along with the rand() function.

ru-vid.com?event=video_description&redir_token=QUFFLUhqbGZ4Y1BLT0p4UUVNZXBPX05Wa3c3SXZ5SV9vZ3xBQ3Jtc0tsc2dQUHk1dUlTd0wzQ1plWk1vZnV3dV9oSnBSNEhHYWpiaE95MVBBU1hDdy03dVcwU3lxbGl0aGYzX2RhVjdnUGpoX0JoWW1SdWpsQWJxZlBUUzNkaWlQOWFVaHg2NWtJT0pEXzVKUWw0Z1dBakV3cw&q=http%3A%2F%2Fwww.zstatistics.com%2Fs%2FEmpirical-proof-of-standard-deviation.xlsx