Note: For Standalone clusters: the --num-executors parameter may not work always. So, to control the number of executors: 1. define number of cores per executors with --executor-cores parameter (spark.executor.cores) 2. control max number of cores for execution with --total-executor-cores parameter (spark.cores.max) If you need 3 executors with 2 cores (you don't need to use --num-executors) --executor-cores 2 --total-executor-cores 6 --num-executors parameter can be used to control number of executor for yarn resource manager. No need to worry as we will work more with spark cluster configuration if future sessions.
Hello Kunal & Satish, I have a 4 core, 8 processor machine. Docker utilizes hyperthreading to enable multi-processing with the same core. This is the reason you see 16 cores (2 threads each processor) available in cluster. And docker doesn't allocate complete resource from host machine to containers rather some percentage of it, which can be controlled using parameters. You can learn more about it in Docker documentations.
Hi, can you please tell how a data frame with 10 column gets partitioned into 11 parts with 2 executors having 8 cores i.e. total 16 cores processing it?
How Are you running this Spark stand alone cluster? You have installed Spark on you system separately and running or what? I am using with pip install pyspark right now. What I have to do to use this standalone cluster like you are doing?
Standalone cluster used in this tutorial is on docker. You can set it up yourself. For notebook - hub.docker.com/r/jupyter/pyspark-notebook You can use the below docker file to setup cluster github.com/subhamkharwal/docker-images/tree/master/spark-cluster-new
