Wait, so comparisons in floating point only just KINDA work? What DOES work? 

this is completely incomprehensible. you don't actually understand how to teach. you're rambling and scribbling things that have literally nothing to do with the data you're presenting. everything in a lesson should help to understand that lesson. this is like explaining something in a loud cafe on a napkin, except you've recorded it. your sheer incompetence at your chosen occupation is admirable.

Interesting! It sounds a lot like fixed point?

@@simondev758 Fixed point using separate memory locations to speed up things like "convert to an integer" where one just clears the fraction part memory location... no calculations required.

This sounds like "integer" format (with number of bits twice the number of bits in your word length) scaled by 2^(-n) where n is the word length. Why not use double-word integers?

@@bpark10001 We used two int64s. The use of two memory locations made many frequent operations (truncating numbers, etc) much faster.

​@@simondev758 'decimal' type in c#

When we designed a language for PLC use in 1984, the language didn't have "Compare Equal" for the REAL (floating point type), but a "Compare Tolerance", with an explicit tolerance argument provided (as described in video). Many customers were confused at first, until they realized that "measurements" are not exact and need to be treated as approximations everywhere. I was young and inexperienced at the time, but the boss were old school veteran in analog computers, sensor technology and much more, so he insisted "no compare equals for REALs. It is not possible!".

​@@niclashShould typically be two tolerances, one relative and one absolute. The fun part with subnormals is they have variable relative precision, but their absolute precision remains the minimum available, so with both tolerance checks they don't need special handling.

@@0LoneTech​​⁠could you explain how both of those numbers would be used?

There was that scandal when one of Intel's processors did the approximation incorrectly.

Super cool! I found a reference for it here, problem 3: rstudio-pubs-static.s3.amazonaws.com/13303_daf1916bee714161ac78d3318de808a9.html

Oh, that makes sense! A third is like the perfect middle step between powers of 2, so the mantissa is all ones. But 7/3 has a one greater exponent than 4/3, so it's "missing" a decimal digit that's presumably rounded up. The difference between them cancels out everything except 1 and the least significant digit of 4/3, making it 1 + ulp. What a cool trick.

@@volbla Thanks for the intuition! It makes a lot of sense when you explain it that way.

I remember that the old Windows 3.1 calculator had a bug where 3.11-3.1 (the two major Windows releases at the time) would equal 0.00. Good times.

It's called rounding. The rounding mode in IEEE defaults to round to nearest even. So your trick only works in some rounding modes. Meaning your condition of IEEE is incorrect. And not understanding the trick is in rounding is a blunder.

Absolutely. This is part of the reason developers do soak tests.

Incremental stuff should always use a char (byte), short, int, or long, and every once in a while it's fine to convert that millisecond or nanosecond figure into a float of seconds. Given that a double has more significant figures than a 32-bit int, if a timer goes far enough to lose this much resolution in a double, it's ticking stupidly far anyway and should be reset or redesigned.

@@coopergates9680 Yeah, switching to integers and using millisecond delta times in general is one of the proposals my boss had to fix this problem for good. Just tedious and dangerous to do in an already existing game, so it's something we'll likely be doing for future games.

how do you normally fix it?

Since the time interval is constant (1/60) you should use fix point instead of floating point: use integers and count the number of 1/60ths seconds, i.e. the least significant bit is interpreted as 1/60 of a second. With a 32 unsigned integer you can then run it for 828 days (add more bits if needed!)

@@piisfun

It's good never to rely on them being equal, but it doesn't solve all your problems. Like, 50 billion and one is bigger than 50 billion, but if X=50,000,000,000 and Y=50,000,000,001, then Y>X will return false.

@@dylangergutierrez Hence it's a percent error issue, it's more like abs(X/Y - 1.0) < 0.000001. We all know that bug in old Minecraft when the player is far from the origin lol

​@@dylangergutierrez 50B + 1 is larger than 50B, if the former can be stored. Otherwise the result of the addition is 50B, and you would be comparing 50B with 50B.

that is what unity does in the animation, floats cant have an equal comparisson, only integers can

Which incidentally is only of the few non pitfall way of using floats. Keeping an error counter and controlling the interval manually. It's a PITA doing it in C thou

...or do it in integer & know the "error" is zero. The problem with floating-point fuzzy scheme is that the error builds with the number of chained computations, which the math doesn't know about. Of course, if you stack irrational (such as trig) computations, this error appears no matter what the number representation scheme. Floating point disease was so bad because as soon as it was introduced, everybody "had to have it" & it was boasting point for computer manufacturers. It was so much that computers had ONLY that format, even when working with integers. Early desktop HP computer calculated 2^2 = 3 (it used logs to compute exponentials).

Interval arithmetic is also incredibly useful for anything scientific

@@bpark10001 "computers had ONLY that format, even when working with integers" *cough* JavaScript *cough*

@@coopergates9680 Yes they did. HP made a desktop computer in the 1970's that had ONLY floating point format. (When loop counters & other integers were needed, the computer internally TRUNCATED to integer. That's what caused the "2^3 = 7" problem. (I had to add a "+ 0.5" to any exponentiation calculation to get 2^n iterations of the loop.) I guess this "simplified" the machine as there was only ONE type of variable, of a fixed size. Remember in those days a lot of the math was done by dedicated HARDWARE. It is simpler to have fixed-size fields in memory. Most of the calculators also used this format, not changeable. "JavaScript" in 1970 had something to do with coffee & writing, & nothing else.

Yes and even then we're still making a lot of assumptions about the nature of atoms.

And at 0:36

@@kirbofn524 thanks for pointing that out! haha

Heh yeah I hate repeating myself and figure you can always just rewind.

@@simondev758 thats why you are so clear ;)

Oooh fixed point is awesome, I've been meaning to make a vid on that.

@@simondev758ooh, did you ever get anywhere with that?

@@simondev758 Maybe you can also discuss the Java BigDecimal class.

@@Islacrusez I have notes and stuff jotted down, but I kinda go with what I get excited about at any given time. I was happy to dive back into graphics a bit the last few months.

in Minecraft bedrock, there's also the stripe lands

eps(num) gives you the minimum value that can be added to num. Usually useful to do something like abs(a-b)

@@SomeStrangeMan All well and good and practical, but I hate when people use "epsilon" (like, from analysis) to mean "really small number". The point of epsilon in analysis is that it's the _arbitrarily_ small number.

@@smorrow in MATLAB the eps function returns the smallest number that may be added to the floating point number given to it as an argument.

@@somestrangescotsman Yeah, but the _name_ of it is obviously a mistaken reference to epsilon from analysis. And epsilon in analysis really means "the smallest number you can possibly imagine, except for zero", sort of like an inverse infinity. It doesn't have an "actual value" that you could in principle write down, whereas the Matlab eps' whole point is to be an actual value, so it's really inappropriate to name one after the other.

The smallest number you can imagine is the smallest number you can add to another. Within the rules of floating point numbers, that IS epsilon.

Yes, it really is the significand, but in adopting mathematical techniques into computer engineering, the word mantissa was used, and became the defined word.

Some say I'm a master of horror...

@@simondev758 All we know, is he's called The Stig.

The end of the gpu shortage was crazy, can't believe how it went. Wait, has that happened yet?

@@simondev758 no wait... There is a shortage TIME TO BUY SOME GPUS!!! LOL

@@simondev758 all hail Proof of Stake

The fundamental problem with FP arithmetic is that Real numbers are not natural fit for computers. It doesn't matter what base you're working in. 1/3 is unrepresentable in base 10, since it's 3.33... repeating. You will run into this issue at some point. You have a countably finite space in any case and you need to cram in an uncountable infinity. There are more reals than there are integers, to cover *any* subspace exactly is impossible. Even if you had to exactly represent the space [0.00001, 0.000011] you would undoubtedly have to use either FP or fixed point and in either case lose a lot of precision. What FP does do is provide acceptable precision in the vast majority of cases through the observation that small numbers we work with often have smaller differences between them.

It's fun to think that floating point units were so complex the first processors didn't even have them and you'd have to use a coprocessor that was often larger than the processor itself (like the intel 8087 that had almost double the amount of transistors the 8086 had) and today we have GPUs that have thousands of FPUs in a single die

​@@smlgdat some point we are going to have to ditch digital computers and use analog voltages. Just look how terribly inefficient is Machine learning on GPUs.

All machine learning engineers agree

real numbers are, by definition, not fit for COMPUTERS. the definition of computation and computational problems requires that ALL inputs must be representable with a finite sequence of symbols. otherwise, it literally is not computation. the real number set (or any continuous subset of it) is not entirely representable with finite symbols. however, nothing stops us from picking a few real numbers and sticking some labels on them. and that's what floats are. (smartly ordered) labels for a (smartly picked) finite subset of the real numbers. in case you're wondering "hey, but what if we could have infinite inputs?", that's called hypercomputation. good luck with that.

Never been a lecturer, but I've spent a lot of time as a mentor because apparently I'm good at that. These videos are a great way for me to work on collecting my thoughts into a more cohesive form and working on my presentation skills.

A basic principle of writing business application software, especially involving money...

Yep, but games historically had to put all their chips in the "performance" pile

You need quantum physics to understand semiconductors and that's the closest thing to magic.

dark magic, possibly made of bees.

Yeah that approach works super well, basically a simplified fixed point?

That's what KiCad (tool for designing PCBs etc) does. Uses 32 bit integers with nanometer increments and you get a reasonable upper limit of just over 2m for the PCB size.

@@simondev758 Yes, well, it *is* fixedpoint. That is all fixed point is: Choosing a minimal unit that is some specific fraction of your base unit and counting how many such fractions you have.

The issue is the propagation of error. That works if you have to do a few (in computer terms) operations, but if you have to do many, like in the simulation it's one of the worse approaches. Each multiplication, for example, produces a loss of precision "identical" to truncate. It's simply not viable for problems of modern scale.

@@jaimeduncan6167 It is actually exact, that is the whole point of using fixed point. Floating point has a lot of precision problems, but when you are counting a specific number of your minimal units, you are simply counting an exact number of those units. There is no error and thus no error propagation. You have to accept that whatever you are counting is quantized, if you are doing a flight sim, your planes will be snapped to a 1mm grid (or whatever minimal unit you decide to use). As long as that is fine, you have no error and no error propagation. With floating point you do get an issue of error propagation which makes many operations much more complicated. Like the video says, you can't directly compare two floating point numbers for equality. If you are adding an array of numbers, you have to sort them by exponent and add the smallest ones first, before adding larger ones. If you don't, adding a number with an exponent 53 higher than a smaller number will make the smaller number vanish with no effect (the whole mantissa is too small to have an impact). In a summation of many numbers, that small number could have made a contribution if it had been added to an only slightly larger number first and thus been propagated up to the big numbers. This means that the order of addition is important in floating point, you lose commutativity. Without the ability to swap the order of summation freely, a lot of algebra is lost as well, making many other things much harder.

10^-9 yes, a power-of-two exponent is only used for hexadecimal float literals (0x1p-9 would be 1*2^-9)

The capitalization of the "E" doesn't matter in floating-point literals. "1E-9" and "1e-9" both mean "one times ten to the minus ninth power" (though the capital version is preferred to avoid confusion). Euler's number is represented in an entirely different way, depending on the exact programing language in question (e.g. "M_E" for C and most of its descendants).

I was confused why he was using base e lol

If i am not mistaken 1.19*10⁻⁷ meters is actually 0.00012mm which is around the diameter of a single coronavirus according google.

Hah! Happy to have increased the amount of confusion! I didn't read into the decision process itself, but if I had to guess, to me floating point is a better tradeoff as a general purpose data type with it's massive range compared to fixed point.

The reason (as far as I can tell) is because floating point is insanely simple to implement in hardware, and is extremely fast. I'm not certain, I should probably use the internet to find the answer

Rational datatypes often suffer from representing the same value multiple times. If you're using two 16 bit integers to store the numerator and denominator, then you have 65,000 ways to have 0, 32,000 ways to have 1/2, etc. This can cause problems with comparisons, overflow, etc. So most implementations I've seen simplify rationals into their lowest unique value, which decreases performance and requires prime factorization after each calculation. But you're still left with massive holes in your datatype, and you've slowed down all your algorithms tremendously anyways. Floating point represents a much wider range of values, with higher precision for small values, and there are no duplicate values to deal with (with an asterisk for NaNs and on systems where subnormals are truncated to 0).

Research how it's implemented in the hardware. The hardware limitations give rise to software limitations. This key understanding of hardware is the difference between programmers and computer scientists. With that said, 1/10 and .1 are the same.

@@AnarchistEagle 1e+1 and 10e+0 is also the same and still no problem to understand. They simply get both converted to 0.1e+2 or 10. Something like that could also be done here by converting everything to an integer and a power integer. So 0.032445 would get converted to 00000032445 and -6, 134.31 to 00000013431 and -2, 12300000 to 123 and 5. But i think speed is the key. Maybe mathematical operations aren't that fast with this format.

Good point, wish I had thought of that for the video.

Hah, I mean it doesn't take long to go through the comments, there's not a million of them. If you take the time to write a comment, I'll definitely read it. re: music, that's super neat! I loved the Office when it aired!

I feel like this is one of those things where you could go years without ever coming near an issue.

Heh yeah, I hate including a bunch of unrelated info to pad the video out.

Only read about them, haven't had a chance to try them out though

That's interesting, the language specification has some overlap with fixed point. My english teacher never encouraged my swearing either :(

C23 has Decimal numbers too.

Woah, you've been around! I was just a kid playing Sierra games back then, would love to hear more about your experiences if you have a blog or something.

@@simondev758I haven't written my memoirs yet, but have had many discussion with other folks about the earlier days of CPU design. After the FP design I was the Design Manager for the P6 (Pentium Pro) and then GM/VP for Pentium II, Pentium III, Pentium 4 and the first Celeron. It was fun until it wasn't and then I left and started a company and then worked at SpaceX for a while. I'm not sure how to do a blog about so much of this since so many other people are intertwined in the history.

That's a big float

Try looking for functions to extract the parts of a float, as well as functions to reunite them. You get the exponent of whichever value you're treating as dominant, then pack together with epsilon (at least, I THINK it was epsilon, it's been a while since I did this), and that gets you the smallest possible step size for the context you're interested in... more or less. You may want to consider the scale above and below as well... It may also be that extracting the exponent gets you everything you care about, but I've never tried that, so I can't speak to the sanity of attempting it.

Bookmark

"where you do not have full control over the exact order in which some low-level stuff is computed" Doesn't sound very deterministic to me.

@@Kalumbatsch That is why I have written "deterministic" in quotation marks, LOL. I had naively assumed that some fancy optimized function (also involving some multi-processor stuff) would perform just like an ideal mathematical function, giving you the exact same output for the same input every single time. In floating point reality, not so much.

Yep, floating point isn't associative heh

If you know the format well, it probably won't tell you anything new, but yeah totally agree it's a bit of a love hate relationship heh