Тёмный

Dynamic Partition Pruning in Apache Spark 

Learning Journal
Подписаться 77 тыс.
Просмотров 14 тыс.
50% 1

Опубликовано:

 

27 окт 2024

Поделиться:

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист
Посмотреть позже
Комментарии : 17   
@EugenePetrash
@EugenePetrash 2 года назад
Genious explanation. Not only on that certain topic, but all of the author's videos and articles are also totally clear. Thanks a lot. Subscribed!
@skywalker66ful
@skywalker66ful 3 года назад
Best Explanation I have found till date about Dynamic Partition Pruning and infact about Predicate Pushdown and Partition Pruning as well
@anikethdeshpande8336
@anikethdeshpande8336 Год назад
super explanation! simple to understand, thanks for showing the execution plans!
@andre__luiz__
@andre__luiz__ Год назад
Amazing explanation!!!
@ylchen5975
@ylchen5975 3 года назад
Very useful and expiation is pretty clear, thank you!
@hierfnhg
@hierfnhg 3 года назад
Very informative thanks for deep diving.
@babyscookbook2751
@babyscookbook2751 2 года назад
Hi sir., can we use apache kafka for sending emails? Please sir, I need it help
@akshaygupta013
@akshaygupta013 3 года назад
Nice explanation. I do have a doubts what's the need for broadcast if the filter condition is already being applied to dimensions table and if it is required than tables which are greater than broadcast threshold in those case will this technique not work or just join type will be different.
@feelings__flicks
@feelings__flicks 2 года назад
Same doubt brother. If u get the answer can u please share it.
@mohammedsafiahmed1639
@mohammedsafiahmed1639 2 года назад
from what I understand, filter condition, or predicate pushdown as Databricks calls it, works only when querying single table. When you join two tables, you need to 'broadcast' the filter to the other table being joined.
@artemvolkov5682
@artemvolkov5682 Год назад
What if I just add year and month to the 'ON' statement ? I believe partition pruning will work, but AQE should be enabled.
@lancequin5209
@lancequin5209 2 года назад
Him: Make Sense? Me: Nope Him: Great
@artemvolkov5682
@artemvolkov5682 Год назад
hahah, feel exactly the same
@bhomiktakhar8226
@bhomiktakhar8226 3 года назад
Nicely explained !....but how does the filter is transferred to order table..since where condition is on year and month , query on fact table would still have to figure out what is full_date (of dimension table)column values on for 2021 Feb...could be multiple full_dates for month,year .?
@kolketzz
@kolketzz 2 года назад
probably it would do date like '2021-02%'
@octo3010
@octo3010 2 года назад
Neat feature
@trainsam22
@trainsam22 2 года назад
Hi Prashant, you know your concepts. but stop saying : makes sense.. or simple etc.. that is too uncle like.. b cool
Далее
What is Dynamic Partition Pruning in Spark
13:18
Просмотров 4,2 тыс.
Бокс - Финты Дмитрия Бивола
00:31
Partitioning
14:32
Просмотров 21 тыс.