Intro To Databricks - What Is Databricks

Подписаться 95 тыс.

Просмотров 222 тыс.

50% 1

What is databricks?
How is it different from Snowflake?
And why do people like using Databricks.
This video will act as an intro to databricks.
We will discuss what is a databricks table(delta table), a databricks job, etc.
research paper on RDDs - www.usenix.org/system/files/c...
0:00 Intro - What Is Databricks
1:02 Presentation
6:29 - Hands On Demo - What Is Databricks
If you're team needs help setting up Databricks, then set up a consultation with me today!
calendly.com/ben-rogojan/cons...
Also! If you enjoyed this video, check out some of my other top videos.
Top Courses To Become A Data Engineer In 2022
• Top Courses To Become ...
What Is The Modern Data Stack - Intro To Data Infrastructure Part 1
• What Is The Modern Dat...
If you would like to learn more about data engineering, then check out Googles GCP certificate
bit.ly/3NQVn7V
If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.
seattledataguy.substack.com/
Or check out my blog
www.theseattledataguy.com/
And if you want to support the channel, then you can become a paid member of my newsletter
seattledataguy.substack.com/s...
Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio
_____________________________________________________________
Subscribe: / @seattledataguy
_____________________________________________________________
About me:
I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consult on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.
*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you.

Развлечения

Опубликовано:

6 июл 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 86

@SeattleDataGuy Год назад

If you guys want to learn more about data engineering, then sign up for my newsletter here seattledataguy.substack.com/ or join the discord here discord.gg/2yRJq7Eg3k

@kerimsever6674 Год назад

Transitioning into using Databricks and this is a great introduction!

@anishninan8374 2 года назад

The best part about databricks is that is unifies batch and streaming workloads. Also provides a single source for structured, semi structured and unstructured data.

@SeattleDataGuy 2 года назад

Yes those are all a lot of reasons why I do like databricks. Especially their streaming functionality.

@rachidt2764 2 года назад

Would be cool to see a mini project in databricks where you could compare and highlight why you don't see it as a data engineering / BI first tool

@SeattleDataGuy 2 года назад

Yeah I will likely do a comparison between this and foundry

@severtone263 Год назад

You earned my sub. I am all in. Thank you for this, it was a great help!

@SeattleDataGuy Год назад

Glad you found this helpful!

@dahof2789 4 месяца назад

Wanted to watch but can't filter that goofy background noise.... Everyone does it and it adds negative value to the listener.

@Skandawin78 Месяц назад

after reading this i can't listen to his voice

@keviny9392 Год назад

Excellent content. Exactly what I needed to get started. Thanks

@NroShock 2 года назад

Great video! Would love to see another one on Databricks; moving raw data from blob storage, transforming and storing in databricks tables

@SeattleDataGuy 2 года назад

That would be a great video!

@jackgolding4235 Год назад

Had this on while I was working and it just clicked to me, thank you!

@SeattleDataGuy Год назад

Glad to hear that!

@tallalmoshrif6643 Год назад

Great content as always. Thanks for sharing.

@hughesadam87 6 месяцев назад

Super helpful - have been using databricks in another system without ever really understanding how much of that other system was simply databricks.

@SeattleDataGuy 5 месяцев назад

Glad it helped!

@edwinfokobo5680 6 месяцев назад

I love this an would love to learn more and understand what’s needed as a prerequisite before I can get a job

@TheAndrewjoynson Год назад

brilliant intro - well done and and surprisingly i understand a lot of this. (im not a dev nor a data engineer)😄

@akshaybaura 2 года назад

great starter !! it'd be interesting to dive a little deeper into delta lake file format and also compare it with iceberg or hudi formats i.e. where they are similar, different, which situations suit one best over the other.

@SeattleDataGuy 2 года назад

Sounds like a great video. Let me see if I can get Ryan Blue in the video.

@devencareer 7 месяцев назад

Your approach in this particular video, is simple and precise, not deep, just right enough. Thanks! Devendra

@SeattleDataGuy 7 месяцев назад

Thank you! You're too kind

@artandrock4all 2 года назад

really cool all around video, would love to see more videos on databricks where a more deep dive analysis would be given on each topic ;)

@SeattleDataGuy 2 года назад

I will add it to the list of future data engineering videos

@DatabricksPro 6 месяцев назад

This channel is great, but you may also check mine. Cheers.

@codestrap8031 2 года назад

Awesome breakdown Ben. I'm looking forward to a comparison with Foundry. My personal opinion is Databricks has the edge when it comes to their pay version of Spark and Delta. It's hard to do a direct head-to-head though because many of Foundry's overlapping capabilities are not documented. IMO Foundry has much better e2e capabilities with a built-in version control system, online IDE, CI/CD, and Data Apps/ML/AI tools. I'm a huge fan of both companies and really think these are the two players that will be left standing at the end of the Big Data OS wars.

@SeattleDataGuy 2 года назад

Glad you enjoyed it! I think I might finally be able to start filming next week for pltr

@MichaelStephenLau Год назад

@@SeattleDataGuy Please do help us understand better from a technical/professional perspective on how Palantir solutions stack up against others in the market (Databricks, Snowflake, AWS, Google, etc.).

@mikenashtech Год назад

Great vid Ben. Like the way, you give tips with commercial thinking behind it. Thanks Mike

@nujanai 2 года назад

Great overview. Thanks!

@SeattleDataGuy 2 года назад

You're welcome!

@wrburggraaf Год назад

How would you compare this to SAS Viya? It seems like this is more for building a data lakehouse whereas SAS is primarily for data analysis and analytics (so it might connect to data bricka to get the data). Could you also do data analysis and analyrics well in data bricks?

@ShashankData 2 года назад

🔥 video! Learning so much from your vids man. I’m not really getting the effective difference between DataBricks and Snowflake.

@SeattleDataGuy 2 года назад

Filming that video now :)

@ShashankData 2 года назад

@@SeattleDataGuy woohooo looking forward to it!

@aiautoglasscrm Год назад

Subscribed. When you say productionize your job, I thought I was going to see API endpoints to hit to get results for example a regression price prediction model that gives you a number after inputting variables

@horaciosoldman4481 2 года назад

Thanks for this informative video Ben 🙌

@SeattleDataGuy 2 года назад

Glad you found it informative!

@zonezero3290 2 месяца назад

Thank you for sharing this!

@SeattleDataGuy 10 дней назад

My pleasure!

@surfh3r0 9 месяцев назад

I'm new on the subject, well explained!

@SeattleDataGuy 9 месяцев назад

glad you found it helpful!

@mashagalitskaia8642 4 месяца назад

a really cool introduction, thanks a lot!

@SeattleDataGuy 10 дней назад

Thank you! Glad you liked it!

@gardnmi 2 года назад

I've used both. Snowflake is toast. Serverless Clusters will take away the pain of managing clusters and they are making really fast improvement on delta lake which will reduce a lot of the common pains of filter, joins, and updates in spark.

@SeattleDataGuy 2 года назад

Oh boy! I want the competition though. The users win in that world

@Practicalinvestments Год назад

@@SeattleDataGuy this is all foreign language but I am an investor who’s been very closely watching this company and it sounds top notch from what you say

@jaserogers997 6 месяцев назад

This didn't age well.

@raphaeldwain7834 2 года назад

Very useful. Thanks.

@SeattleDataGuy 2 года назад

You're welcome!

@manticomar1146 Год назад

Good work

@hotpeppermovie 2 года назад

Databricks is awesome. Its so easy to work with

@SeattleDataGuy 2 года назад

It really is!

@venkateshkothapalli 2 года назад

Love Databricks!

@SeattleDataGuy 2 года назад

Seems to be a decent amount of love for it!

@femaledeer Год назад

Video didn't explain what databrick does. When was table show being built ?

@BuyNLarge_ 3 месяца назад

🎯 Key Takeaways for quick navigation: 01:36 *🚀 Databricks offers managed Spark services along with other tools like Delta Lake and MLflow, providing options for data processing and model deployment.* 03:56 *🏠 Databricks and Snowflake both promote the concept of data lake houses, combining data warehouse and data lake functionalities, but with different emphases on use cases.* 05:37 *🛠️ Key components of Databricks include workspaces, notebooks, tables, clusters, jobs, and libraries, providing an integrated environment for data processing and analysis.* 09:33 *📊 Databricks simplifies the transition from notebooks to production by allowing users to create jobs directly from their notebooks, enabling seamless integration and scheduling.* 11:13 *🌟 Databricks facilitates easier productionization of data science workflows compared to alternatives like Snowflake, with integrated features like job creation and version control.* Made with HARPA AI

@SeattleDataGuy 3 месяца назад

thanks for this!

@marklambrecht Год назад

We need to have you do a video about SAS Viya too!

@stelluspereira 7 месяцев назад

Enjoyed our video, thx Do you know how to install databricks &PySpark LOCALLY( on laptop ) & code & test locally Perhaps a video will be appreciated by the community WITHOUT depending on AWS/Azure

@AliTwaij 2 года назад

Thank you

@SeattleDataGuy 2 года назад

You're welcome!

@DavidKoleckar 3 месяца назад

that's high quality vid, thx :]

@SeattleDataGuy 10 дней назад

Glad you liked it!

@nguyetdang111 3 месяца назад

Is Databricks different from Azure Databricks?

@zacharythatcher7328 Год назад

To me this just looks like a great way to encourage developers to deploy untested code. Are there any testing pipelines built in that prevent job deployment prior to passing tests?

@gautam3305 Год назад

Testing in data field is completely not like typical software engineering cicd unit testing

@ganonymous8448 2 года назад

Great product overall, but sucks that you can’t use Airflow on it.

@SeattleDataGuy 2 года назад

I could have sworn I saw a partnership between them and astronomer

@fenderbender28 2 года назад

Their Workflows orchestrator got super powerful and is easier than airflow imo

@VicusBass Год назад

Whenever it's Databricks there a Romanian around :)

@cerberus1321 9 месяцев назад

Its the same as Domino

@SeattleDataGuy 5 месяцев назад

there are some similar features but it is kind of an apples and oranges comparison

@mikipatel2434 9 месяцев назад

Great video ruined by the annoying background music. The background music was really distracting and annoying. But very good information. Thank you

@joshi1q2w3e 2 года назад

I hate Databricks, would rather use Snowflake.

@michaeld9682 2 года назад

Why?

@SeattleDataGuy 2 года назад

I would also love to know why!

@sevegarza 2 года назад

Agreed. Snowflake + Prefect = 😇

@joshi1q2w3e 2 года назад

@@SeattleDataGuy a couple reasons: 1. A lot of times when I’m building a data pipeline there’s a lot of SQL queries I need to write to just analyze the data before I start creating certain metrics and to just start understanding the data. Because of Databricks 1000 row limit this is harder to do. If those tables were in a RDBMS or Snowflake I wouldn’t feel as hindered with regards to this very common task. I know it may seem weird to some people but sometimes just being able to see more of your data and scroll through just helps; maybe this is a Junior Data Engineer thing idk. 2. Idk if this is easier in Snowflake but passing a Parameter from a widget into SparkSQL was a pain in the ass in Databricks, the only reason I figured it out was because a notebook written by someone else did the same thing. We use Azure Synapse notebooks and I like that much more than Databricks as well; it was easier to do some of the same things.

@gautam3305 Год назад

@@joshi1q2w3e I understand data discovery is key aspect before modeling, but that can be achieved by using groups by, limit, distinct, windowing etc, don't need to print million rows and export to excel for that.