Install Apache PySpark on Windows PC | Apache Spark Installation Guide

Подписаться 11 тыс.

Просмотров 109 тыс.

50% 1

In this lecture, we're going to setup Apache Spark (PySpark) on Windows PC where we have installed JDK, Python, Hadoop and Apache Spark. Please find the below installation links/steps:
PySpark installation steps on MAC: sparkbyexamples.com/pyspark/h...
Apache Spark Installation links:
1. Download JDK: www.oracle.com/in/java/techno...
2. Download Python: www.python.org/downloads/
3. Download Spark: spark.apache.org/downloads.html
Winutils repo link: github.com/steveloughran/winu...
Environment Variables:
HADOOP_HOME- C:\hadoop
JAVA_HOME- C:\java\jdk
SPARK_HOME- C:\spark\spark-3.3.1-bin-hadoop2
PYTHONPATH- %SPARK_HOME%\python;%SPARK_HOME%\python\lib\py4j-0.10.9-src;%PYTHONPATH%
Required Paths:
%SPARK_HOME%\bin
%HADOOP_HOME%\bin
%JAVA_HOME%\bin
Also check out our full Apache Hadoop course:
• Big Data Hadoop Full C...
----------------------------------------------------------------------------------------------------------------------
Apache Spark Installation links:
1. Download JDK: www.oracle.com/in/java/techno...
2. Download Python: www.python.org/downloads/
3. Download Spark: spark.apache.org/downloads.html
-------------------------------------------------------------------------------------------------------------
Also check out similar informative videos in the field of cloud computing:
What is Big Data: • What is Big Data? | Bi...
How Cloud Computing changed the world: • How Cloud Computing ch...
What is Cloud? • What is Cloud Computing?
Top 10 facts about Cloud Computing that will blow your mind! • Top 10 facts about Clo...
Audience
This tutorial has been prepared for professionals/students aspiring to learn deep knowledge of Big Data Analytics using Apache Spark and become a Spark Developer and Data Engineer roles. In addition, it would be useful for Analytics Professionals and ETL developers as well.
Prerequisites
Before proceeding with this full course, it is good to have prior exposure to Python programming, database concepts, and any of the Linux operating system flavors.
-----------------------------------------------------------------------------------------------------------------------
Check out our full course topic wise playlist on some of the most popular technologies:
SQL Full Course Playlist-
• SQL Full Course
PYTHON Full Course Playlist-
• Python Full Course
Data Warehouse Playlist-
• Data Warehouse Full Co...
Unix Shell Scripting Full Course Playlist-
• Unix Shell Scripting F...
-----------------------------------------------------------------------------------------------------------------------Don't forget to like and follow us on our social media accounts:
Facebook-
/ ampcode
Instagram-
/ ampcode_tutorials
Twitter-
/ ampcodetutorial
Tumblr-
ampcode.tumblr.com
-----------------------------------------------------------------------------------------------------------------------
Channel Description-
AmpCode provides you e-learning platform with a mission of making education accessible to every student. AmpCode will provide you tutorials, full courses of some of the best technologies in the world today. By subscribing to this channel, you will never miss out on high quality videos on trending topics in the areas of Big Data & Hadoop, DevOps, Machine Learning, Artificial Intelligence, Angular, Data Science, Apache Spark, Python, Selenium, Tableau, AWS , Digital Marketing and many more.
#pyspark #bigdata #datascience #dataanalytics #datascientist #spark #dataengineering #apachespark

Опубликовано:

6 фев 2023

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 358

@indianintrovert281 Месяц назад

Those who are facing problems like 'spark-shell' is not recognized as an internal or external command On command prompt write 'cd C:\Spark\spark-3.5.1-bin-hadoop3\bin' use your own spark filepath(include bin too) And then write spark-shell or pyspark (It finally worked for me, hope it works for you too) If it worked, like this so that more people benefit from this

@SharinH 26 дней назад

It worked .. Thank you

@jagjodhsingh2358 19 дней назад

It worked, thanks :)

@nature.knowledge 17 дней назад

Thank you 😊 so much it worked

@nature.knowledge 17 дней назад

Thank you 😊 so much it worked

@vishaltanwar2238 9 дней назад

why did we get this error?

@sisterkeys 8 месяцев назад

What I was doing in 2 days, you narrowed to 30 mins!! Thank you!!

@ampcode 4 месяца назад

Thank you so much! Subscribe for more content 😊

@yashusachdeva 4 месяца назад

It worked, my friend. The instructions were concise and straightforward.

@ipheiman3658 11 месяцев назад

This worked so well for me :-) The pace is great and your explanations are clear. I am so glad i came across this, thanks a million! 😄 I have subscribed to your channel!!

@susmayonzon9198 Год назад

Excellent! Thank you for making this helpful lecture! You relieved my headache, and I did not give up.

@ampcode 11 месяцев назад

Thank you so much!

@moathmtour1798 11 месяцев назад

hey , which version of hadoop did you install because the 2.7 wasn't available

@eloscarc5782 Месяц назад

Your video helped me understand it better than other videos, now the other videos make sense. This was not as convoluted as I thought.

@neeleshgaikwad6387 Год назад

Very helpful video. Just by following the steps you mentioned I could run the spark on my windows laptop. Thanks a lot for making this video!!

@ampcode Год назад

Thank you so much!😊

@iniyaninba489 6 месяцев назад

@@ampcode bro I followed every step you said, but in CMD when I gave "spark-shell", it displayed " 'spark-shell' is not recognized as an internal or external command, operable program or batch file." Do you know how to solve this?

@sssssshreyas 7 дней назад

@@iniyaninba489 add same path in User Variables Path also, just like how u added in System Variables Path

@cloudandsqlwithpython 10 месяцев назад

Great ! got SPARK working on Windows 10 -- Good work !

@ampcode 4 месяца назад

Thank you so much! Subscribe for more content 😊

@ragisatyasai2469 Год назад

Thank for sharing this. Beautifully explained.

@ampcode Год назад

Glad it was helpful!

@saswatarakshit9488 9 месяцев назад

Great Video, awesome comments for fixing issues

@ampcode 4 месяца назад

Thank you so much! Subscribe for more content 😊

@nedvy 11 месяцев назад

Great video! It helped me a lot. Thank you ❤

@ampcode 10 месяцев назад

Thank you so much!

@juanmiguelvargascortes9933 10 месяцев назад

Excellent video!!! Thanks for your help!!!

@ampcode 4 месяца назад

Thank you so much! Subscribe for more content 😊

@HamsiniRamesh-ig6ih Месяц назад

This video was great! Thanks a lot

@veerabadrappas3158 Год назад

Excellent Video.., Sincere Thank You

@ampcode Год назад

Thank you!

@user-tr9pz1je7g 10 месяцев назад

Very helpful, thanks!

@ampcode 4 месяца назад

Thank you so much! Subscribe for more content 😊

@davidk7212 Год назад

Very helpful, thank you.

@ampcode Год назад

Thank you so much!

@Adhikash015 Год назад

Bhai, bro, Brother, Thank you so much for this video

@ampcode Год назад

Thank you so much!

@chrominux5272 3 месяца назад

Very useful, thanks :D

@joshizic6917 6 месяцев назад

how is your spark shell running from your users directory? its not running for me

@somanathking4694 Месяц назад

This works as smooth as butter. Be patient that's it! Once set up done, no looking back.

@SUDARSANCHAKRADHARAkula Месяц назад

Bro, which version of spark & winutils you've downloaded. I took 3.5.1 and hadoop-3.0.0/bin/winutils but not worked

@meriemmouzai2147 Месяц назад

@@SUDARSANCHAKRADHARAkula same for me!

@pratikshyapriyadarshini4677 4 месяца назад

Very Helpful.. Thankyou

@metaviation Год назад

very clear one thank you

@ampcode Год назад

Thank you!

@user-vq4oz9oc5o 10 месяцев назад

Brilliant, Thanks a ton

@ampcode 4 месяца назад

Thank you so much! Subscribe for more content 😊

@pooja1899 Год назад

Thank you for sharing this video

@ampcode Год назад

Most welcome!

@sicelovilane5391 11 месяцев назад

You are the best. Thanks!

@adamamoussasamake5119 11 месяцев назад

hi, which hadoop version did you use?

@sicelovilane5391 11 месяцев назад

@@adamamoussasamake5119 It's 2.7.1

@ampcode 10 месяцев назад

Thank you!

@gosmart_always 8 месяцев назад

Every now and then we receive alert from Oracle to upgrade JDK. Do we need to upgrade our JDK version? If we upgrade, will it impact running of spark.

@sanchitabhattacharya353 3 месяца назад

while launching the spark-shell getting the following error, any idea?? WARN jline: Failed to load history java.nio.file.AccessDeniedException: C:\Users\sanch\.scala_history_jline3

@danieljosephs 3 месяца назад

Very helpful video

@saravanang8576 28 дней назад

is there any thing wrong with the latest version of the python and spark 3.3.1 ? i am still getting the error

@ramnisanthsimhadri3161 20 дней назад

I am not able to find the package type: pre-build for Apache Hadoop 2.7 in the drop-down. FYI - my spark release versions that i can see in the spark releases are 3.4.3 and 3.5.1.

@ed_oliveira 4 месяца назад

Thank you! 👍

@ampcode 4 месяца назад

Thank you so much! Subscribe for more content 😊

@Manapoker1 Год назад

the only tutorial that worked for me.....

@ampcode Год назад

Thank you so much!

@NileshKumar9975 Год назад

very helpful video

@ampcode Год назад

Thank you so much!

@user-vb7im1jb1b 9 месяцев назад

Thanks for this video. For learning purposes on my own computer, do I need to install apache.spark (spark-3.4.1-bin-hadoop3.tgz) to be able to run spark scripts/notebooks, or just pip install pyspark on my python environment?

@practicemail3227 13 дней назад

Hi, I'm in the same boat, can you tell me what did you do. I'm also learning currently and have no idea.

@shankarikarunamoorthy4391 9 дней назад

sir, spark version is available with Hadoop 3.0 only. Spark-shell not recognized as internal or external command. Please do help.

@arnoldochris5082 6 месяцев назад

Ok guys this is how to do it, incase you are having problems👇 1.) I used the latest version 3.5.0, (Pre-built for apache hadoop 3.3 or later) - downloaded it. 2.) Extracted the zip file just as done, the first time it gave me a file, not a folder but a .rar file which winrar could not unzip, so I used 7-zip and it finally extracted to a folder that had the bins and all the other files. 3.) In the system variables he forgot to edit the path variables and to add %SPARK_HOME%\bin. 4.) Downloaded winutils.exe for hadoop 3.0.0 form the link provided in the video. 5.) Added it the same way but c>Hadoop>bin>winutils.exe 6.) Then edit the user variables as done then do the same to the path %HADOOP_HOME%\bin Reply for any parts you might have failed to understand🙂

@MANALROGUI 6 месяцев назад

What do you mean for the 3rd step ?

@stay7485 5 месяцев назад

Thanks

@ampcode 4 месяца назад

Thank you so much 😊

@sarahq6497 8 дней назад

Hello, I had to use the latest version as well, but I'm not able to make it work, I followed the tutorial exactly :(

@rayudusunkavalli2318 4 месяца назад

i did every step you have said, but still spark is not working

@Kartik-vy1rh 11 месяцев назад

Video is very helpful. Thanks for sharing

@ampcode 10 месяцев назад

Thank you so much!

@vennilagunasekhar5460 10 месяцев назад

Thank you so much

@ampcode 4 месяца назад

Thank you so much! Subscribe for more content 😊

@user-uc7qf6uf5c 6 месяцев назад

Great thanks

@ampcode 4 месяца назад

Thank you so much! Subscribe for more content 😊

@nikhilupmanyu8804 3 месяца назад

Hi, Thanks for the steps. I am unable to see Web UI after installing pyspark. It gives This URL can't be reached. Kindly help

@juliocesarcabanillas2433 9 месяцев назад

Love you dude

@ampcode 4 месяца назад

Thank you so much! Subscribe for more content 😊

@jeremychaves2269 10 месяцев назад

thanks dude!

@ampcode 4 месяца назад

Thank you so much! Subscribe for more content 😊

@pulkitdikshit9474 2 месяца назад

hi i installed but when I restarted my pc it is no longer running from cmd? what might be the issue?

@amitkumarpatel7762 2 месяца назад

I have followed whole instruction but when I am running spark -shell is not recognised

@matheswaranp9574 День назад

Thanks a Lot.

@Cardinal_Seen 8 месяцев назад

Thank you. :D

@ampcode 4 месяца назад

Thank you so much! Subscribe for more content 😊

@edu_tech7594 10 месяцев назад

my Apache hadoop which i downloaded previously is version 3.3.4 eventhough i should choose pre-built for Apache Hadoop 2.7?

@sriram_L 9 месяцев назад

Same doubt bro. Did u install now

@akira.19.9 9 месяцев назад

muy util !!

@ampcode 4 месяца назад

Thank you so much! Subscribe for more content 😊

@gangadharg7 3 месяца назад

This is perfectly worked for me. Thank you very much.

@manasa3097 10 месяцев назад

This really worked for me...I have completed spark installation but when I'm trying to quit from the scala the cmd is not working and it's showing the error as 'not found'.. can you please help me on this...

@theeewebdev 6 месяцев назад

and when downloading the spark a set of files came to download not the tar file

@basanthaider3238 6 месяцев назад

I have an issue with the pyspark it's not working and it's related to java class I can't realy understant what is wrong ???

@theeewebdev 6 месяцев назад

i have fallowed all these steps and installed those 3 and created paths too, but when i go to check in the command prompt... its not working.. error came... can anyone help me please to correct this

@bramhanaskari3152 Год назад

you haven't give solution for that warn procfsMetricsGetter exception is there any solution for that ?

@ampcode Год назад

Sorry for late response. This could happen in windows only and can be safely ignored. Could you please confirm if you’re able to kick off spark-shell and pyspark?

@abhinavtiwari6186 Год назад

where is that git repository link? Its not there in the description box below

@ampcode Год назад

Extremely sorry for that. I have added it in the description as well as pasting it here. GitHUB: github.com/steveloughran/winutils Hope this is helpful! :)

@mdsamiulislam2522 Год назад

Thanks

@ampcode Год назад

Thank you so much!

@nagalakshmip8725 Месяц назад

I'm getting spark- shell is not recognised as an internal or external command, operable program or batch file

@James-br9cu 3 месяца назад

Nice

@user-oy8gu5cs9j Год назад

ERROR Shell: Failed to locate the winutils binary in the hadoop binary path I am getting above error while running spark or pyspark session. I have ensured that winutils file is present in C:\hadoop\bin

@ampcode Год назад

Could you please let me know if your all the env variables are set properly?

@ismailcute1584 3 месяца назад

Thank you so much for this video. Unfortunately, I couldn't complete this - getting this erros C:\Users\Ismahil>spark-shell 'cmd' is not recognized as an internal or external command, operable program or batch file. please help

@JesusSevillanoZamarreno-cu5hk 3 месяца назад

execute as admin

@prateektripathi3834 5 месяцев назад

Did Everything as per the video, still getting this error : The system cannot find the path specified. on using spark-shell

@srishtimadaan03 26 дней назад

On command prompt write 'cd C:\Spark\spark-3.5.1-bin-hadoop3\bin' use your own spark filepath(include bin too) And then write spark-shell or pyspark (It finally worked for me, hope it works for you too)

@harshithareddy5087 5 месяцев назад

I don't have the option for Hadoop 2.7 what to choose now???

@LLM_np 4 месяца назад

did you get any solution? please let me know

@Mralbersan Месяц назад

I can't see Pre-Built for Apache Hadoop 2.7 on the spark website

@meriemmouzai2147 Месяц назад

same problem for me! I tried the "3.3 and later" version with the "winutils/hadoop-3.0.0/bin", but it didn't work

@ruuuff Год назад

@ampcode Год назад

Thank you so much!

@moathmtour1798 11 месяцев назад

hello, which Hadoop Version should i install since the 2.7 is not available anymore ? thanks in advance

@ampcode 10 месяцев назад

You can go ahead and install the latest one as well. no issues!

@venkatramnagarajan2302 7 месяцев назад

@@ampcode Will the utils file still be 2.7 version ?

@Jerriehomie Год назад

Getthing this error: WARN ProcfsMetricsGetter: Exception when trying to compute pagesize, as a result reporting of ProcessTree metrics is stopped. People have mentioned to use python fodler path which I have as you have mentioned but still.

@bukunmiadebanjo9684 Год назад

I found a fix for this. Change your python path to that of anaconda(within the environment variable section of this video) and use your anaconda command prompt instead. No errors will pop up again.

@ampcode Год назад

Sorry for late response. Could you please let me know if you are still facing this issue and also confirm if you’re able to open spark-shell?

@shivalipurwar7205 Год назад

@@bukunmiadebanjo9684 Hi Adebanjo, my error got resolved with you solution. Thanks for your help!

@nftmobilegameshindi8392 2 месяца назад

spark shell not working

@rakeshd3250 5 месяцев назад

not working for me i set up everything except hadoop version came with 3.0

@nagarajgotur Год назад

spark-shell is working for me, pyspark is not working from home directory, getting error 'C:\Users\Sana>pyspark '#' is not recognized as an internal or external command, operable program or batch file.' But when I go to python path and run the cmd pyspark is working. I have setup the SPARK_HOME and PYSPARK_HOME environment variables. Could you please help me. Thanks

@ampcode Год назад

Sorry for late response. Could you please also set PYSPARK_HOME as well to your python.exe path. I hope this will solve the issue😅👍

@bintujose1981 Год назад

@@ampcode nope. Same error

@ganeshkalaivani6250 Год назад

can any one please help...last two days tried to install spark and give correct variable path but still getting system path not speicifed

@ampcode Год назад

Sorry for late reply. Could you please check if your spark-shell is running properly from the bin folder. If yes I guess there are some issues with your env variables only. Please let me know.

@saikrishnareddy3474 8 месяцев назад

I’m little confused on how to setup the PYTHONHOME environment variable

@thedataguyfromB 5 месяцев назад

Step by step ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-jO9wZGEsPRo.htmlsi=aaITbbN7ggnczQTc

@khushboojain3883 9 месяцев назад

Hi, I have installed Hadoop 3.3 (the lastest one) as 2.7 was not available. But while downloading winutils, we don't have for Hadoop 3.3 in repository. Where do i get it from?

@sriram_L 9 месяцев назад

Same here.Did u get it now?

@khushboojain3883 9 месяцев назад

@@sriram_L yes, u can directly get it from google by simply mention the Hadoop version for which u want winutils. I hope this helps.

@hritwikbhaumik5622 7 месяцев назад

@@sriram_L it still not working for me though

@anastariq1310 10 месяцев назад

After entering pyspark in cmd it shows "The system cannot find the path specified. Files\Python310\python.exe was unexpected at this time" please help me resolve it

@mahamudullah_yt 10 месяцев назад

i face the same problem. is there any solution

@varun9922946392 10 месяцев назад

installed smoothely like butter ...thanks !!

@ampcode 4 месяца назад

Thank you so much! Subscribe for more content 😊

@ankushv2642 5 месяцев назад

Did not work for me. At last when I typed the pyspark in command prompt, it did not worked.

@rakeshkandula2318 6 месяцев назад

Hi, i followed exact steps (installed spark 3.2.4 as that is the only version available for hadoop 2.7). Spark-shell command is working but pyspark is thrwing errors. if anyone has fix to this please help me. Thanks

@thedataguyfromB 5 месяцев назад

Step by step solution ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-jO9wZGEsPRo.htmlsi=aaITbbN7ggnczQTc

@badnaambalak364 5 месяцев назад

I followed the steps & Installed JDK 17, spark 3.5 and python 3.12 when I am trying to use map function I am getting an Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe Error please someone help me

@rajkumarsahoo6937 Месяц назад

same problem 😢

@riptideking 2 месяца назад

'pyspark' is not recognized as an internal or external command, operable program or batch file. getting this error and tried it for whole day and same issue.

@srishtimadaan03 26 дней назад

@sanketraut8462 4 месяца назад

how to set up com.jdbc.mysql.connector using jar file, actually I am getting the same error that its not found while working in pyspark.

@Karansingh-xw2ss 8 месяцев назад

i'm facing this issue can anyone help me to fix this 'spark-shell' is not recognized as an internal or external command, operable program or batch file'.

@nikhilupmanyu8804 3 месяца назад

Try to add direct path at System Environment. It will fix the issue

@itsshehri Год назад

hey pyspark isnt working at my pc. I did everything how you asked. Can you help please

@ampcode Год назад

Sorry for late response. Could you please also set PYSPARK_HOME env variable to the python.exe path. I guess this’ll do the trick😅👍

@sriramsivaraman4100 Год назад

Hello when I try to run the command spark_shell as a local user its not working (not recognized as an internal or external command) and it only works if I use it as an administratror. Can you please help me solve this? Thanks.

@ampcode Год назад

Sorry for late response. Could you please try once running the same command from the spark/bin directory and let me know. I guess there might be some issues with your environment vatiables🤔

@dishantgupta1489 Год назад

@@ampcode followed each and every step of video still getting not recognised as an internal or external command error

@ayonbanerjee1969 Год назад

@@dishantgupta1489 open fresh cmd prompt window and try after you save the environment variables

@obulureddy7519 Год назад

In Environment Variables you give the paths in Users variable Admin. NOT IN System variables

@syafiq3420 Год назад

how did you download the apache spark in zipped file? mine was downloaded as tgz file

@ampcode Год назад

Sorry for late response. You’ll get both options on their official website. Could you please check if you are using the right link?

@georgematies2521 10 месяцев назад

@@ampcode There is no way now to download the zip file, only tgz.

@ganeshkalaivani6250 Год назад

FileNotFoundError: [WinError 2] The system cannot find the file specified getting this error even i have installed all required intalliation

@ampcode Год назад

Sorry for late reply. I hope your issue is resolved. If not we can have a connect and discuss further on it!

@user-ef9vh7qz9h 11 месяцев назад

java.lang.IllegalAccessException: final field has no write access: I'm getting this error while running the code when I run the same code in another system it is getting executed. Any idea?

@Nathisri 6 месяцев назад

I have some issues in launching python & pyspark. I need some help. Can you pls help me?

@JesusSevillanoZamarreno-cu5hk 3 месяца назад

same, did you fix it? it worked for scala for me but not spark

@karthikeyinikarthikeyini380 7 месяцев назад

hadoop 2.7 tar file is not available in the link

@ankitgupta5446 5 месяцев назад

100 % working solution ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-jO9wZGEsPRo.htmlsi=lzXq4Ts7ywqG-vZg

@syamprasad8295 8 месяцев назад

while selecting a package type for spark, Hadoop 2.7 is not available now. Only Hadoop 3.3 and later is available. And winutils 3.3 is not available at the link provided at the git. What to do now? can I download Hadoop 3.3 version and can proceed with winutils2.7 ? Pls help.. Thanks In Advacnce

@ShivamRai-xo8fu 7 месяцев назад

I got same issue

@ankitgupta5446 5 месяцев назад

100 % working solution ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-jO9wZGEsPRo.htmlsi=lzXq4Ts7ywqG-vZg

@kchavan67 6 месяцев назад

Hi, following all the steps given in video, I am still getting error as "cannot recognize spark-shell as internal or external command" @Ampcode

@psychoticgoldphish5797 6 месяцев назад

I was having this issue as well, when I added the %SPARK_HOME%\bin, %HADOOP_HOME%\bin and %JAVA_HOME%\bin to the User variables (top box, in the video he shows doing system, bottom box) it worked. Good luck.

@thedataguyfromB 5 месяцев назад

Step by step spark + PySpark in pycharm solution video ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-jO9wZGEsPRo.htmlsi=aaITbbN7ggnczQTc

@prajakta-dh7fc 28 дней назад

'spark' is not recognized as an internal or external command, operable program or batch file. its not working for me i have follow all the steps but its still not working waiting for solution

@antonstsezhkin6578 11 месяцев назад

Excellent tutorial! I followed along and nothing worked in the end :) StackOverflow told me that "C:Windows\system32" is also required in the PATH variable for spark to work. I added it and spark started working.

@Manojprapagar 11 месяцев назад

helped

@antonstsezhkin6578 11 месяцев назад

@@Manojprapagar happy to hear it!

@ampcode 10 месяцев назад

Thank you so much!

@user-gc6ku9mp3d 11 месяцев назад

Hi, I completed the process step by step and everything else is working but when I run 'spark-shell' , it shows - 'spark-shell' is not recognized as an internal or external command, operable program or batch file. Do you know what went wrong?

@viniciusfigueiredo6740 11 месяцев назад

I'm having this same problem, the command only works if I run CMD as an administrator. Did you manage to solve it?

@hulkbaiyo8512 10 месяцев назад

@@viniciusfigueiredo6740 same as you, run as administrator works

@shivamsrivastava4337 9 месяцев назад

@@viniciusfigueiredo6740 same issue is happening with me

@RohitRajKodimala 9 месяцев назад

@@viniciusfigueiredo6740same issue for me did u fix it?

@santaw 6 месяцев назад

Anyone solved this?

@viniciusfigueiredo6740 11 месяцев назад

I followed the step by step and when I search for spark-shel at the command prompt I come across the message :( 'spark-shell' is not recognized as a built-in command or external, an operable program or a batch file). I installed windows on another HD and did everything right, there are more people with this problem, can you help us? I'm since January trying to use pyspark on windows

@letsexplorewithzak3614 10 месяцев назад

Need to edit bottom "add this to env var path" path >> C:\Spark\spark-3.3.1-bin-hadoop2\bin\

@kiranmore29 8 месяцев назад

@@letsexplorewithzak3614 Thanks worked for me

@nayanagrawal9878 6 месяцев назад

Do everything that he said but not in User Variables but in System variables. I was facing the same problem but then I did the same in system variables and my spark started running.

@jayakrishnayashwanth7358 6 месяцев назад

Even I'm facing the same issue ,can you tell in more detail like what to do add in system variables??As we already added Java , Hadoop, Spark and Pyspark_Home in the user varaibles as said in the video.@@nayanagrawal9878

@penninahgathu7956 4 месяца назад

@@nayanagrawal9878 thank you!!! I did this and it solved my problem

@laxman0457 8 месяцев назад

i have followed all your steps,still i'm facing an issue. 'spark2-shell' is not recognized as an internal or external command

@nayanagrawal9878 6 месяцев назад

Do everything that he said but not in User Variables but in System variables. I was facing the same problem but then I did the same in system variables and my spark started running.

@thedataguyfromB 5 месяцев назад

Step by step spark + PySpark in pycharm solution video ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-jO9wZGEsPRo.htmlsi=aaITbbN7ggnczQTc

@meditationmellowmelodies7901 Месяц назад

I followed all the setps but getting error 'spark-shell' is not recognized as an internal or external command, operable program or batch file.

@Mralbersan Месяц назад

the same happens to me

@indianintrovert281 Месяц назад

Facing same error, Did you find any solution for it?

@BasitAIi 8 месяцев назад

In cmd the comand spark-shell is running only under C:\Spark\spark-3.5.0-bin-hadoop3\bin directory not globally same for pyspark

@s_a_i5809 7 месяцев назад

yeah man , same for me.. did you found any fixes... if, let me know :)

@BasitAIi 7 месяцев назад

@@s_a_i5809 add your Environment variables under system variables not user variables.

@ankitgupta5446 5 месяцев назад

100 % working solution ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-jO9wZGEsPRo.htmlsi=lzXq4Ts7ywqG-vZg

@lucaswolff5504 2 месяца назад

I added C:\Program Files\spark\spark-3.5.1-bin-hadoop3\bin to the system variables and it worked

@BasitAIi 2 месяца назад

@@lucaswolff5504 yes

@shahrahul5872 Год назад

on apache spark's installation page, under choose a package type, the 2.7 version seem to not be any option anymore as on 04/28/2023. What to do?

@shahrahul5872 Год назад

I was able to get around this by copying manually the URL of the site you were opened up to after selecting the 2.7th version from the dropdown. Seems like they have archived it.

@ampcode Год назад

Sorry for late reply. I hope your issue is resolved. If not we can discuss further on it!

@nikhilchavan7741 7 месяцев назад

'spark-shell' is not recognized as an internal or external command, operable program or batch file.-- Getting this error

@nayanagrawal9878 6 месяцев назад

Do everything that he said but not in User Variables but in System variables. I was facing the same problem but then I did the same in system variables and my spark started running.