Тёмный
No video :(

70. Databricks| Pyspark| Input_File_Name: Identify Input File Name of Corrupt Record 

Raja's Data Engineering
Подписаться 24 тыс.
Просмотров 7 тыс.
50% 1

Azure Databricks Learning: Identify Input File Name of Corrupt Record
INPUT_FILE_NAME:
==============================
Big Data Interview Question: How to identify the input file name of a corrupt record in Spark programming?
What is input_file_name and what is the use of it?
input_file_name is one of the spark in-built function, which is used to identify the input file name of a corrupt record in a dataframe.
This function can be leveraged to perform troubleshooting in spark programming.
This video covers complete details about this function and use cases
#Input_File_Name,#Spark_Input_File_name, #Pyspark_Input_File_Name, #Databricks_Input_File_Name, #DatabricksTroubleShooting, #SparkTroubleShooting, #SparkDebug, #DatabricksDebug, #DatabricksCorruptRecord, #SparkCorruptRecord, #DatabricksRealtime, #SparkRealTime, #DatabricksInterviewQuestion, #DatabricksInterview, #SparkInterviewQuestion, #SparkInterview, #PysparkInterviewQuestion, #PysparkInterview, #BigdataInterviewQuestion, #BigdataInterviewQuestion, #BigDataInterview, #PysparkPerformanceTuning, #PysparkPerformanceOptimization, #PysparkPerformance, #PysparkOptimization, #PysparkTuning, #DatabricksTutorial, #AzureDatabricks, #Databricks, #Pyspark, #Spark, #AzureDatabricks, #AzureADF, #Databricks, #LearnPyspark, #LearnDataBRicks, #DataBricksTutorial, #azuredatabricks, #notebook, #Databricksforbeginners

Опубликовано:

 

21 авг 2024

Поделиться:

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист
Посмотреть позже
Комментарии : 21   
@manikanta-zq1yg
@manikanta-zq1yg Месяц назад
great work
@rajasdataengineering7585
@rajasdataengineering7585 Месяц назад
Thank you! Cheers!
@hyk-f4r
@hyk-f4r Год назад
Hey just want to let you know that you are doing amazing job here. I am learning a great deal from your videos. Thank you, subscribed 👍👌🏼
@rajasdataengineering7585
@rajasdataengineering7585 Год назад
Happy to hear that! Thank you
@sivahanuman4466
@sivahanuman4466 Год назад
Great sir
@lakshay2462
@lakshay2462 6 месяцев назад
👍
@rajasdataengineering7585
@rajasdataengineering7585 6 месяцев назад
👍
@dineshdeshpande6197
@dineshdeshpande6197 2 месяца назад
Great informative videos .. pl keep up doing same :)
@rajasdataengineering7585
@rajasdataengineering7585 2 месяца назад
Thank you, I will!
@sravankumar1767
@sravankumar1767 2 года назад
Nice explanation Raja 👌 👍 👏
@rajasdataengineering7585
@rajasdataengineering7585 2 года назад
Thanks Sravan!
@ndbweurt34485
@ndbweurt34485 Год назад
beautiful explaination really!
@rajasdataengineering7585
@rajasdataengineering7585 Год назад
Thank you 👍🏻
@volda2000
@volda2000 Год назад
Thanks! Very helpful video.
@rajasdataengineering7585
@rajasdataengineering7585 Год назад
Thank you
@vinodhkoneti4473
@vinodhkoneti4473 2 года назад
Thank you for sharing this.. can you help me how to compare two dataframes, i found one video for comparision but primary key got populated with Null..so we are unable identify which primary key got mismatches..
@rajasdataengineering7585
@rajasdataengineering7585 2 года назад
Minus, intersect and join with hash key - these are some functions can be used to compare 2 dataframes. In your case, if you have null values, you can go with third method. Join both dataframes and create hash value by concatenation of all columns and apply filter to compare both hash values
@sravankumar1767
@sravankumar1767 2 года назад
I have one doubt Raja, in my current project we r using Delta lakes. We are using Raw, Trusted, refined, provisioned,provisioned extract.except raw & prov-extracg all are Delta format. Those layers are in Databricks. Can we create we mount point for Delta tables. Once copied raw layer in between we have notebook activity
@rajasdataengineering7585
@rajasdataengineering7585 2 года назад
Hi Sravan, for delta tables, mount point is not needed. Because delta tables are directly accesible using SQL command
@sravankumar1767
@sravankumar1767 2 года назад
@@rajasdataengineering7585 Thank you very much
@harshadkhedekar836
@harshadkhedekar836 2 года назад
can you share those sample datasets?
Далее
💀СЛОМАЛ Айфон за 5 СЕКУНД😱
00:26
Мама приболела😂@kak__oska
00:16
Просмотров 411 тыс.
34. Databricks - Spark: Data Skew Optimization
15:03
Просмотров 25 тыс.
💀СЛОМАЛ Айфон за 5 СЕКУНД😱
00:26