I am Arpit Singh. I have been working in IT industry for about 10 years now, and 8+ years in Big Data. I provide useful content for Big Data aspirants, who need essential interview questions, and use case scenarios all built and explained from scratch. Please subscribe to my channel for all interesting videos..!!
For any queries write to me at- gkcodelabs@gmail.com
Thanks a lot, I think we can use row_number window function to get updated records by using partitions by emp_id and order by date desc. Finally can filter for row_number=1
Conversion to Pandas is only done on final, aggregated and normalized data where volume is brought down only for what is required for "Visualization" Dont get confused with Data "processing" and "Visualization" You never run huge Spark jobs when a visual report is requested on the fly, that approach becomes "Pointless" Reports are pulled from aggregated data. And Matplotib is one of the approach, there are many other tools and approaches. Thanks for bringing this up btw, this will help others as well, in case someone has similar doubt.
Hey, Arpit! Do u suggest going with databricks after learning pyspark, even though i want to make a career as an AWS data engineer, ..or would it be more apt for me to just go with aws glue instead?
No doubts in going with AWS.. But dont just stick to AWS Glue, its just one convinient way to spark loads. Explore all Data Analytics services.. PS: AWS Data Analytics is now the new term for Data Engineering services. Happy Learning 👍🏻
Hi Arpit! I have been practicing PySpark for a while now, studied and implemented most of the concepts you mentioned in the video as individual concepts. I am looking for projects/use cases where I can combine all the concepts building a meaningful end result. Please could you suggest some? Thanks!
When you do repartition and then partitionby already data is partitioned now based on partitionby column they why no of part file depend on repartition() again?
Olap cubes and data warehouse are Different(in diagram it is marked together ). Data warehouse warehouse persist data , in Bigdata it will be in data lake gold layer.
Sirrrrr.... you are back. I started my data engineering journey from here. I learnt basics of GCP but now in my job AWS will be used. I am happy you chose AWS.
