Excellent Explanation. But I want to know the most important thing in this video, How did you create those cool animations like @1:58-@2:20 and @8:00-@8:05
Nice work! A small correction to what you said: "Semantic segmentation labels each pixel in the whole image. It is not restricted to only pixels in the background".
@@nicolascarion3111 Can we then say that: "Panoptic Segmentation= Instance Segmentation+Semantic Segmentation minus annotations of foreground classes" ?
This is an end to end neural network defined by functions which all have derivatives. In the R-CNN family of algorithms you have one procedure that produces a bunch of region proposals, then you crop out these regions and feed them to a classifier, and then you run another algorithm to prune out overlapping and low confidence predictions. Since there are multiple steps that have logical rather than mathematical implementations, you can't take derivatives all the way through to back propagate information through the whole system.
The transformer isn't aware of the 2D structure of the image, because 1) we flatten it and 2) permuting the inputs of a transformer simply permutes its outputs (permutation equivariance). That's why we add 2D positional encodings. This is similar to what is done in NLP, to retain the order of the sentence.
@@nicolascarion3111 Thanks for your explanation. I have another question: Right now DETR because of rectangle bboxes of COCO-dataset produces rectangle-bboxes outputs, if we had polygon bboxes (8 points), which parts of the architecture must be modified to output a polygon shape bboxes?
@@ZobeirRaisi Well you need to modify the regression head as well as the loss and matching function (GiOU may not make sense anymore, so you'll likely have to stick to L1). For this kind of questions, it's best to open an issue on our github. Thanks!
