Multi-Dimensional Data (as used in Tensors) - Computerphile 

Me too! I'd like that

Seconded

Yes

I'm curious whether "swizzling" as it's called is used in ML. Or if it's just a graphics thing.

@@MooseBoys42 probably both, as ML and graphics have a lot of _parallels_

It's kinda yes and no. Shifting is faster than multiplying. But adding paddings make the access pattern more complex leading to more cache misses. Which are far more expensive. Also, unless you are JITing or compiling, you need your library containing the optimization and detecting at runtime. Which may not be the case.

That is kind of inefficient. Imagine the same done for 5d. Your matrix is suddenly more than 3 times larger.

@cas curse Or if you're writing something that does not need to be optimized. Spending ages trying to optimize some code that'll only be run once every hour or so can be rather pointless.

@cas curse As long as your code is not idioticly unoptimal, these kind of optimizations are loss of time. Instead of thinking about the problem at hand, you think about all the itsy bitsy low level trickery to make it, what, 5% faster.

TL;DR If performance is critical (you work in HPC, aviation, automobile, game, etc.. ) You design the performance into your code. You consider the performance implications in the first place. Sometimes even 1% of performance make or breaks stuff in these industries. On the other hand, if performance is something you can easily get by scaling out (Web/Mobile App, low volume) or you know your code only takes like 0.1% percent of time in the system anyway (Language Wrappers, Function interfaces, one-off program). You build the thing then maybe optimize it. Then buy more computers.

I too am a simple man

That's the one you needed a special keyboard for, right? That probably didn't help! (Yes, yes, overlays, just learn to touch type ...)

Zero dimensional boolean, I wish it was a whole dimension :-D

Actually it is 2 dimensional as the like button is an image ;)

Whether boolean is its own dimension or not, its a shame you're getting 1 dimensional comments from an n-dimensional language space 😂😂😂

@@_adi_dev_ Agreed :-D

Cool! This might be useful for something I have been struggling with. If you recall, let me know!

"Numerical Recipes" book, from which I studied numerical computing in university, uses arrays of pointers to arrays of pointers...to actual data. I think a few multiplications are a lot better on modern machines, where processors are many times faster than memory...

Wanted to ask this too. I think, that rgb should be the first dimension, since these will most likely be accessed together, so they can be loaded into the same cache line.

For 100 px by 100 px I think they will be on the same cache line, but I don't think this method scales up for bigger images

Tiling, essentially. Put a maximum tile size on each dimension such that the whole tile (or several) fits in cache, then essentially treat tile number as another dimension in this system.

@@serkanmuhcu1270 Although, your intuition is somewhat correct, the real answer is slightly more complicated. NHWC vs NCHW (where N is batch, H and W are width and height and C is channel) is a somewhat controversial question, so most frameworks offer both versions and allow the programmer to choose. Even better, many frameworks actually use abstractions, which hide the real layout of data and just measure the actual performance in different configuration and choose the best one before running your code. This is the best approach, since the efficiency of different approaches can depend in complicated, unpredictable ways on the network architecture, hardware used, the size of the data and even what other applications are currently running on the system. There is no "one-layout-fits-all" approach, so it's best to select the best one for each case automatically.

Yep. Thats more or less how its done

I say shear is the way to go

MichaelKingsfordGray If you don’t know the answer, then don’t bother commenting.

Mathematically, yes, that would be correct. However, why would you need to switch from 1d to multi-d ? It's always the other way around.

Hopefully you'll get an answer to this. :)

Great question! There will be some cache lookup benefits to having it all stored together, which is less likely with jagged arrays. Mainly though it comes down to when you're doing block copies and stuff. In the case where you're looking up a single pixel, the speed benefit will be negligible. When this approach will really shine is when you say "copy me all the data from dimension 4 indexes 4 to 7", where the whole block is contiguous. In this case there will be the single initial lookup, and then just read the whole thing at once. Itll also be a lot easier this way to take "slices" e.g. half of an image, but I didn't really say much on this yet due to time constraints!

Well if you're looping through a 2d array, the most obvious thing to do is a nested for loop, but from that you get extra overhead from loop control. Now if you imagine that scaled up to many dimensions, just have a 1d loop and a 1d array, and you do away with most of that overhead. I can't be too sure, as it depends on languages and compiler optimisation, but sometimes the handling of >1D and 1D arrays are different in memory and machine code, and sometimes the same. I think the video is more about my first point, where loop flattening can make a big difference.

At the lowest level of programming all arrays are long sections of memory. So 2D array as x y is a section of memory which is x*y long. The constants are used to calculate the offsets to get the right value. With more dimensions there are more constants to calculate the offsets. Using constants means that you don't need to calculate fixed values all the time and so is quicker than doing extra maths.

AFAIK he's mostly just talking about how computers, in general, handle multi-dimensional arrays. If you use a higher-level language to work with multi-dimensional arrays, this is what the compiler/interpreter will be doing behind the scenes, to make it appear to be a multi-dimensional array. However, there are some occasions where the programmer may be forced to do these sorts of things explicitly.

the second video on multi-dimensional data

I've still got three cases of blue-bar paper in my garage.

I think you could go back to university. Doing things you don't like will make you bitter in the long run...

Littleton

Growing new trees reduces carbon more than keeping trees alive. Uses less water, too.

Well for chess you would need two structures one where white is winning and one where black is winning

And then it is basicly just about set up the rules for the game and let it play, as i understand it alpha zero have no idea of the value of pieces to start with? At some point you remove futile paths "moves" for the players. Probably you do not let it play all moves from beginning, just 10 and then evaluate the positions. Next you play 20, 30 and so on until a full game played thru. Next you revise structure by play 5,,15,25 before evaluation and see if other patterns have emerged in evaluation.

@@JmanNo42 No an expert in AI, are you saying that all possible moves are coded in an conventional chess program? Whereas alpha does that dynamically?

@@kkgt6591 Well i do not know i assumed that alphago wrote to file dynamically, not allocated all possible positions for each move, i mean that would not be much depth/dimensions of such a tree?

My understanding of chessprogram is that they do not store that much except for in memory, maybe 5 and six moves and evaluate constantly? And then they have the opening book stored, whille i thought that alpha go stores succesful "winning" games. And then cut of branches that do not look successful. And that they hare stored in a "linear" searchable tree/file. structure much like a file allocation table.

The number of elements is still the same; the way you address the data doesn not magically change the amount of data. The only place you'd possibly see a difference in speed is in the complexity of the operations you use to address specific pixels, which only really becomes a concern on a per-language basis. Fortran for instance still sees use today thanks to its array-handling features. As far as image-processing goes, I know OpenCV stores image data in the way you've described it by default - an MxN matrix of arrays of length 3 that each represent a pixel, so I personally don't see much benefit in deviating from it since its library functions are designed around it. It's better to look into the specifics of the language and library you're using to get a better idea about what practices you should be inculcating. I'm not expert on the subject, mind you. I just started image-processing and CUDA last month so that's all I really feel confident in saying. Cheers!

On a two dimensional array, like an image, it is O(image pixel count x convolution brush pixel count) in general.

@Riccardo Orlando That is complexity of trivial algorithm, but I would like to know more about the lower bound.

@@woowooNeedsFaith From what I can tell, lower bound is n*log(n). At least that's the limit for linear convolution, because it can be implemented as multiplication in frequency domain ( O(n) ) and conversions from and to frequency domain can be done via FFT. FFT is O(n*log(n)) and is proven to be the fastest possible way to do it.

Real mathematicians dont call multidimensional arrays as tensors

okay boomer

@MichaelKingsfordGray Guess it depends on the implementation? I know for Javascript everything in an array is really just an object that is hashed, making for easy access to elements and potential savings on memory for sparse arrays at the sacrificed of speed...quite a bit of speed. I know more mathy libraries exist for things like python, but they are beyond my knowledge!

It was a pop quiz to see if you recognized inaccurate graphical explanations. I guess you didnt. Fortunately, actual ability and ability to condescend are inversely proportional

@MichaelKingsfordGray Lecture, go on. At worst I chastened, rather more an admonishment. And what sort of android-pretending-to-be-human-phrase is "real adult name "? I'm legitimately concerned you might be having a stroke right now.

@@Mr.Beauregarde "...rather more an astonishment." "I'm legitimately concerned you might be having a stroke right now."

@@theyaoster I'm astonished you'd admonish what's so clearly a #dyac moment.

Computerphile paper is all left over from years ago so any impact on the environment has already been done, might as well use it. Some even say the very reason this channel exists is so that they can get rid of all that paper...

brzrkr still would have prefered if in some cases they would use the whiteboard, like more complex ones like this video.

@@555Devil555 Yeah I agree. Sometimes extra space is needed.

Try pytorch. Its the way to go now

you're not alone

@@existenceisillusion6528 Cuppy (A cup that is a pet)

Yeah, I definitely read it as pee rather than pie.

@@existenceisillusion6528 I read that as Q-pee

@@danieljensen2626 KooPie

Well, to be fair, it gives a warning whenever you do that.

It is very much NOT just javid. Altough, he is an amazing programmer.

Zero Cool Fully understood and agreed, I just like using these little connections to toss the names of people I like in the comments so others will hopefully see them and go subscribe.

Some deep what?

r/iamverysmart

/sci/

so?