"The Data Channel”, your go-to destination for unraveling the mysteries of the ever-expanding data universe. In an era where information reigns supreme, understanding the intricacies of data and its related technologies is not just a skill but a necessity. Whether you’re a seasoned data enthusiast, a budding analyst, or someone simply curious about the transformative power of information, this blog aims to be your compass in navigating the dynamic world of data.
Join us at “The Datapedia” as we navigate the exciting intersections of data engineering, data science, and the ever-evolving landscape of cloud platforms. From foundational knowledge to advanced techniques, our mission is to make the complexities of these technologies accessible to all. Embark on this data odyssey with us and discover the limitless possibilities within the data spectrum.
Hi hope you’re doing great 👍 ! After learning basics n understanding ( types of data, 6vs of data, etl vs elt, db vs dw vs dm vs dl, full vs incremental load, fact vs dimensions, dw schemas, scd type n storage layer types, oltp vs olap) in one day , i have moved to databricks instead of data analysis as i want to become data engineer so today i have started this playlist n done with below topics ( 1- data bricks -- simple open source and single seloid teck stack ( all stack under 1 unberla ) lakehouse Data lake + dw = lakehouse (delta lake ) as delta lake is heart of lake house n lakehouse is build on top of delta lake (Topic-2 free trail - 2 ways aws n direct db website register ( need credit card 14 days trail n set termination 120 or 15 minutes as if u unchecked it will consume storage ) 1- trail version single node ( default driver node ) Multi node has worker + driver Topic-3 ( db architecture n services) ( control plane n data plane Control plane :: db cloud admin control Data plane : customer cloud ( aws azure gcp ) all exution transformation compute memory storage physically present here ) High level cluster : 1- general purpose interactive n job cluster Topic-3 db workspace Db page overview ( left side personas ( ml de n sql will b available in premium version) rest create workspace repos compute data workflow will b same in all 3 pasonas ( ds ml sql ) -- right side top ( user settings , admin consle , logout many more ) Topic -4 create n manage clusters Workspace create or compute create new ( single node -) When u click on created cluster u will have many options logs alerts metrics many more where we can mintor all Topic-5 db notebooks 1- create new notebook name n select lang u want 2- execute cell print (“hello”) 3- magic command %sql %python to change language of note book 4- md 5- %run “path” of child notebook or second notebook 6- %run “./notbookname” for exiting folder 7- dbutilies.help 8- display dbutlies.fs.help to display in tabular format Topic-6 git version n repos Create account in git hub settings developer mode acess token create copy then go to db settings add token repos n configure then create a child branch n to avoid conflict ( first always pull latest version n then push) Topic -7 taking brake may b evening i will practice Thanks 😊 once again !!
Learn in one day all below topics n return what i learn thanks now starting this playlist Topic -1 Types of data 1- structured ( oltp transactional online banking booking erp ) relation between tables Eg : oracle sql server ibm db2 vertica 2-semi structured- json xml format Dw / data bricks / delta or data lake ( iot sensor data ) 3- unstructured ( images videos ) ( dw / delta lake ) N for statistics data ( Topic -2 6 vs of data 1- volume (gb tb peta bytes) 2- variety (semi structured unstructured) 3- velocity-( 4- value ( analysis dashboard) 5- veracity ( redundancy scalability) 6- variability( change in variable) Topic -3 etl vs elt Etl : extract transform load- specific use case Elt : extract load n transform- many use case Topic -4 Db - rdbms oltp ( relationship primary n foreign key) Dw - storage ( dw / db) fact n dimensional ) Dm - subset of dw for fast recent revision history Dl- large storage Topic -5 full vs incremental load Full: replace full load in target system ( useful when less volume n frequency is often Incremental load : first time full load n from next load it only takes updates or modified record( volume is high n modifications) Topic-6 fact vs dimensions Fact: Measures Transitional Dimensions: Not change statics Dim tables Topic-7 scd Scd -0 to 6(5) Scd -0 fixed dimensions Scd-1 no history Scd -2 Scd -3 Scd -4 Scd -6(5) Topic -8 dw schemas 1- star - straight forward centralised fact n many dim tables but dim tables will not have relation with other dim tables 2- snow flake - Complex query performance low One fact table have many dim tables n each dim tables have relation b/w them know as child tables 3- galaxy star schema
@@thedatachannel878 thanks for making’s such clear content video I request you to make video on real time projects for azure data engineering with step by step notes 📝 n data sets which can easily downloaded n use for practice thanks 😊
We covered almost all the topics as part of certification. But to attend to certification exam, suggest you to please take exam dumps and understand the pattern of questions
Hi i want to learn data engineering n get job as experience data engineer can you suggest playlist in alphabetical order for de and ru data engineering if yes can i know ur LinkedIn? Thanks 😊
In client mode de spark session is registered into you machine so every result come back to your machine ! So il client mode if your machine fail the computing is done! For debuging is coll to see what contaning the spark session but is not necessy in oder cases ! note in cluster mode spark session is contained by driver program!
We acknowledge the issue and unfortunately there is no way to remove background noise. But we have made sure to remove noise in all our upcoming videos
@jenyroy8203 In real world project, if there is need of retention period more than 30 days they you can persist data in Blob storage which is cheaper solution or may be in adls etc
Hi Data Channel, thank you for the tutorial! I'm interested in setting limits in Fabric. Can operations that exceed a cost limit be automatically shut down? For example, if a company accidentally leaves a process running over the weekend, resulting in charges of over 30,000 Euros, would implementing a kill-cost limit of, let's say, 1,000 Euros have prevented this? I'd appreciate any insights you can provide. Thank you!
Absolutely, that’s great questions. Currently I don’t see any supporting MS documentation to support setting limits. However, I am definitely sure that there will enterprise level and even subscription level limit can be set. Need to explore and if I come across supporting documentation will post it here Thank you. Happy Learning