Groups, Depthwise, and Depthwise-Separable Convolution (Neural Networks) 

yes, a good visualization of transformers would be great

Also graphnet

I'm using Blender and relying heavily on the Geometry Nodes feature.

You could theoretically separate any kernel into a depthwise-separated one. But you'd need a lot more filters in the depthwise convolution, so the result would be about the same performance. The performance improvement comes from training the network to take advantage of depthwise-separated convolutions.

In practice, it probably doesn't make a huge difference where you increase the number of channels. You could increase the channels in the depthwise convolution as long as you wanted the output channel count to be a multiple of the input. EfficientNet actually increases the number of channels with an extra pointwise convolution before the depthwise convolution. Yes, it increases the storage required during training in TensorFlow and PyTorch. Post-training, you don't necessarily need to keep around all the intermediate feature maps. So whether or not it doubles the required storage is dependent on the library (if any) that you're using for deployment.

That's correct, they're the feature maps which are the inputs/outputs of the layers. The depth of a feature map is equal to the number of filters in the convolutional layer that created it. So the depth increases that you're seeing are simply layers that have more filters than the number of features in their input. Let me know if that isn't what you meant by your question. This video shows the depth staying the same in a depth-wise separable convolution, but you can still depth-wise separate a layer that increases the number of filters and get the performance benefits. You can just take the input depth and use twice (or some other multiple of) that value for the filter count in both the depth-wise and point-wise convolutions.

@@animatedai I see, I see. So if the input is an RGB image, and the first convolutional layer uses 5 filters, then the depth of the feature map will be 5. If that feature map goes to another convolutional layer with 5 filters, will the output contain a feature map with a depth of 25?

In that example, both outputs would have a depth of 5 because both layers have a filter count of 5. My video on filter count might help you visualize the relationship there: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-YSNLMNnlNw8.html These videos are both part of this playlist on convolution: ru-vid.com/group/PLZDCDMGmelH-pHt-Ij0nImVrOmj8DYKbB

@@animatedai Hi animatedAI! I'll check the link out. I hope I'll get it afterwards haha. Thanks for sharing!

I appreciate your feedback! I could rant for hours about how hard it is to pick colors :). I have two clarification questions: 1) Which part of the video did you pause to stare back and forth between blocks? 2) Which feature of Viz Palatte do you think would have helped pick colors for that instance?