Bro I have a question. In this video data was distributed to two nodes even before executors are created ( before program logic starts ryt) But in my program if I read data from another system (like s3) during program execution only data will be loaded to cluster. So program execution ( creation of executor) should start first ryt before distributing data?
Bro, when I run in standalone mode with 1 master and 2 workers (in laptop, I have made host as master and first worker... (and) 2nd worker in VM)... Only when I put the same input file in both the worker node in gives me correct output. For eg. I need to put the same 1gb input file in both workers. If I partition them into 512mb (I did it manually, but the location of file in both worker is same) , first half of the file in one worker and other half in second worker, it gave incorrect results. Why is it so?
When manually partitioning data in a Spark standalone cluster, ensure that each partition resides on a separate worker node and has unique file paths to avoid data duplication and incorrect results
Is it fine to use Spark Standalone for a POC project , or should I use with Hadoop itself? The requirement is basically to migrate Django Cron jobs what we run on Celery(with RabbitMQ) to Spark .
In Spark's standalone cluster mode, worker nodes provide resource isolation, fault tolerance, scalability, parallelism, and efficient resource management, even if data splitting is not required.
