51. Creating a Spark Job Definition and Submitting it in Azure Synapse 

Your videos are helping a lot.....I was GCP Data Engineer...Just started with Azure Projects

Great explanation, better than official doc, thank you

Much appreciated ! Thanks for covering this topic

been a great series well done, so helpful

Good, appreciate your work. If possible kindly continue POWER BI series.

Hi Maheer, Thanks for sharing your knowledge. I have a question - Why we are using abfss path -- instead of url... ? Please help me in this.

Great video thanks. How did you configure access to ADLS for spark job definition? I watched your video on configure access to ADLS (lesson 45). I've assigned blob data contributor role to the managed identity of synapse workspace, but the spark job definition run still failed with error :Access unauthorized on the provided ADLS Gen2 account. The strange part is that I could use the same code in spark job definition in a notebook, then run the notebook in a pipeline, which means the synapse workspace managed identity does have proper access to ADLS. Now I wonder - whether the spark job definition uses the same managed identity to access ADLS. Look forward to your reply. Thanks again.

Can you please make video on dynamic creation of external table through serverless sql pool

Hi mahi, will it be possible to schedule the spark jobs without using the pipelines activities.

Thank you. Any plan for azure devops playlist?

Hi , My spark job definition failed due to time out error when I debugged Can you please suggest a solution

1) There are 5 activities in sequential in adf pipeline. When anyone of the activity fails it should send email from activity 6 send mail web activity. All of them are connected in sequential order? 2) There are 5 csv files in source folder in blob. You have to copy only last modified date file to output folder in blob Please let me know exact answers for these two interview questions

If we have a Python file that mention other files

Hi Sir , your videos are very helpful . could you please create a video for working with XML file in Cosmos db with Azure Data bricks ?