Hadoop vs Spark | Lec-3 | In depth explanation 

Sure

Are you a fellow data engineering aspirant ?

Will try

@@manish_kumar_1 thanks

Watch one of my titled " How I bagged 12 offers "

Free me hi padhata hu. Aap Mera 12 offer wala video dekh lijiye. Saare free resources mil jayenge

Pentab is required to write it on notebook or ppt. You can buy online. I have medium size one. You can find the link in description

@@manish_kumar_1 Is it the iPad pencil you're referring to? Will wacom one pen tablet work the same?

@@reachrishav yes but it won't have any screen. You will get a pad and stylus. You have write on pentab with stylus but what ever you are writing will be shown in laptop one note or ppt or any other software that you are using

@@manish_kumar_1 Thanks. I guess you are using wacom tablet/stylus for this video?

If we have many smaller size disk blocks, the seek time would be maximum (time spent to seek/look for an information). And also, having multiple small sized blocks is the burden on name node/master, as ultimately the name node stores metadata, so it has to save this disk block information.

@@manish_kumar_1 Thank you :)

Yes it should be map reduce vs spark. But the term Hadoop vs spark is more popular

@@manish_kumar_1 Dont stick with the popularity stick with the concept. to avoid confusions

If "process 1" fails in the DAG, the recovery would typically involve retrying or restarting "process 1" itself. The success of this recovery depends on whether "process 1" is independent or has dependencies. If it has dependencies, those may need to be reprocessed as well to ensure a consistent state in the workflow. Essentially, DAG recovery for a failed process involves identifying the failure point, addressing it, and potentially rerunning dependent processes to maintain the integrity of the workflow. Thanks ChatGPT Let me elaborate it: A Directed Acyclic Graph (DAG) in Spark represents a computational workflow where nodes denote tasks or operations, and directed edges illustrate dependencies between these tasks. In the context of fault tolerance, if a task like "process 1" fails, the DAG aids recovery by re-executing the failed task based on information collected from its dependencies, ensuring the computational flow continues. Consider a scenario where you apply five transformations to a DataFrame (DF). Each transformation creates a new DF as DFs are immutable. If, for instance, "transformation 4" fails during execution, Spark retrieves information from "transformation 3's" DF (its dependency) and then re-executes "transformation 4." Regarding your question about "process 1" failure, if it fails, recovery involves restarting "process 1." Given interdependencies between tasks, subsequent transformations won't proceed if the initial process fails. The DAG orchestrates this recovery process by ensuring the restarting of the failed task, allowing the entire workflow to progress seamlessly. If I am wrong then please someone let me know because I am also beginner in Data domain.

@@soumyaranjanrout2843 thanks for details explainantion. What is the meaning of "Given interdependencies between tasks, subsequent transformations won't proceed if the initial process fails."?

@@TheBest-yh1yj In simpler terms, if one step in a process fails, the following steps that depend on it also get stuck until the initial issue is resolved. If I will simplify it more then as we knew every tasks are interdependent so if task 1 got failed(as per your question) then the remaining tasks that rely on its output cannot continue until the initial task is successfully completed. Hope you understood it😊

Hadoop me hdfs padh lijiye and yarn. MapReduce ki zarurat nahi hai

@@manish_kumar_1 ha utna dekha hu bhaiya vo great learning se tabhi kaise slow hai mapreduce samjha mai

Spark

@@manish_kumar_1 kis liye hai yeh

​@@abidkhan.10 for Data Engineer roles

​@@manish_kumar_1 bro daily upload videos of this series

Bro iska lecture ka baki content sunno!! Interviewer voh puchega!!! Jo correct karva re voh nhi😅