Tutorial 1-Pyspark With Python-Pyspark Introduction and Installation 

Everything about this series is perfect. The pace, the information, and the clarity of the descriptions are as good as it gets. I've watched about 4-5 pyspark tutorials, from various instructors, and they don't even come close to the greatness of these videos. Thank you for providing such top notch content and using a no-nonsense approach. I thoroughly enjoyed these and learned a lot.

I am 9 minutes into the first video and let me tell you it is already better than the last 10 I have tried. It's great for real beginners like me and challenging enough too. Thank you for posting these!!

Krish I am a big fan of yours. You are an amazing teacher and have taught me numerous concepts in Data Science. Thanks a lot for the social service you do!!

We can like these videos even before we see them cause we know they are bound to be extremely useful.

Amazing😂one thing about your channel is that I get confused whenever I get here,I wanted to learn Feature Engineering for the day here I am enjoying pyspark

please make a video how to install pyspark. We installed it but its not importing on jupyter notebook. On terminal, its importing fine

Sir ek hi dil hai, kitni baar jeetenge ! Once again hats-off to your efforts in uplifting the entire data science community across the globe.

Hi Sir, When I am trying to import sparksession and sparkcontext it is throwing an error. The error is module Java.base doesnot support sun.nio.ch to unnamed module. Could you please resolve this Thank you

Thanks for this, but i am getting error while running the spark and the error is "Exception: Java gateway process exited before sending its port number". Can you tell me how to resolve it.

after wasting 2 hours on youtube at last found someone telling from scratch and what i was looking for

Thanks for this .. kindly provide complete playlist

Timestamps: 6:45 Create new environment and install spark via pip install 7:13 importing pyspark 9:34 Import SparkSession 9:47 Create SparkSession ...

Getting below error after running spark=SparkSession.builder.appName('Practise').getOrCreate(), > Py4JError: org.apache.spark.api.python.PythonUtils.isEncryptionEnabled does not exist in the JVM

Much waited playlist!!

Hi Krish! Great tutorial! Thanks for this! One (probably stupid) question and I'm a novice here. How did you enable the auto-suggest functionality in your jupyter notebook? Mine doesn't work. Could you please help? Thank you!

So glad that you started this new series, Krish! Looking forward for new videos in this series. Any idea when you would be uploading? :)

Really When i am doing search in ur encyclopedia playlist,I miss this..Thank you for uploading sir

Sir ek hi toh dil hai kitni baar jitoge🙏

I am stuck while creating Spark Session getting this error PySparkRuntimeError: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number.. Please help

I am having some problems with thinking. Can you share how you tackle thinking and do fast thinking?

Hi Krish, After installing Spark When I run spark session, I m getting below error. RuntimeError Traceback (most recent call last) Cell In [11], line 1 ----> 1 spark = SparkSession.builder.appName('Practice').getOrCreate() File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\pyspark\sql\session.py:269, in SparkSession.Builder.getOrCreate(self) 267 sparkConf.set(key, value) 268 # This SparkContext may be an existing one. --> 269 sc = SparkContext.getOrCreate(sparkConf) 270 # Do not update `SparkConf` for existing `SparkContext`, as it's shared 271 # by all sessions. 272 session = SparkSession(sc, options=self._options) File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\pyspark\context.py:483, in SparkContext.getOrCreate(cls, conf) 481 with SparkContext._lock: 482 if SparkContext._active_spark_context is None: --> 483 SparkContext(conf=conf or SparkConf()) 484 assert SparkContext._active_spark_context is not None 485 return SparkContext._active_spark_context File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\pyspark\context.py:195, in SparkContext.__init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls, udf_profiler_cls) 189 if gateway is not None and gateway.gateway_parameters.auth_token is None: 190 raise ValueError( 191 "You are trying to pass an insecure Py4j gateway to Spark. This" 192 " is not allowed as it is a security risk." 193 ) --> 195 SparkContext._ensure_initialized(self, gateway=gateway, conf=conf) 196 try: 197 self._do_init( 198 master, 199 appName, (...) 208 udf_profiler_cls, 209 ) File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\pyspark\context.py:417, in SparkContext._ensure_initialized(cls, instance, gateway, conf) 415 with SparkContext._lock: 416 if not SparkContext._gateway: --> 417 SparkContext._gateway = gateway or launch_gateway(conf) 418 SparkContext._jvm = SparkContext._gateway.jvm 420 if instance: File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\pyspark\java_gateway.py:106, in launch_gateway(conf, popen_kwargs) 103 time.sleep(0.1) 105 if not os.path.isfile(conn_info_file): --> 106 raise RuntimeError("Java gateway process exited before sending its port number") 108 with open(conn_info_file, "rb") as info: 109 gateway_port = read_int(info) RuntimeError: Java gateway process exited before sending its port number

Hello sir i am not able to create Pyspark session, while i am generating session i am getting follwing error :: Py4JError: org.apache.spark.api.python.PythonUtils.getPythonAuthSocketTimeout does not exist in the JVM can you give me solution of this problem

Its 2024 Sir, and still your video contents are unmatchable. My bad luck is that the moment I joined your iNeuron course, you separated away from it, but my only reason joining the course was to learn from only you! SAD :(

Sir last two lines code are not working in my system it shows ---- AttributeError: 'NoneType' object has no attribute 'printSchema' everything is all right even i restarted kernel

I am getting some errors while creating spark session PySparkruntime error[Java gate way exited] Java gateway process exited before sending it's port number

Please advise RuntimeError: Java gateway process exited before sending its port number

Most awaited!!!!!!! it was music to my years when he said Mlib 0:40

sir waiting for new playlist from a longtime and here it came!!!!

The command "!pip install pyspark" got successfully run I got the following error after the command import pyspark "ModuleNotFoundError: No module named 'pyspark'" I had created a new environment in Anaconda and installed pyspark in it. The above error got resolved by running "pip install pyspark" command

I am getting an error " Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext. : java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.storage.StorageUtils$ "

spark session command : spark = SparkSession.builder.appName('Practiceee').getOrCreate() is taking infinite time. Not able to run code further as it kept on running. What is the solution for this?

Hi Krish, Thanks for your videos, I dont know why I get ("Non type ) after correcting the header for pyspark and dose not show me the Schema.

Thanks man, just wanted to start with pyspark.

when i am trying to create Spark Session getting this error "RuntimeError: Java gateway process exited before sending its port number". Help me in resolving this please

Intro soundtrack jabardast hai bhai..also content bhi

I didn't realise when those 16 minutes ended...interactive n smooth!!

Hi Krish I am getting a Py4j error can you upload the reuirements.txt file along with the python version so that I can start

Hi, I am getting an error while running the commands as: raise RuntimeError("Java gateway process exited before sending its port number") RuntimeError: Java gateway process exited before sending its port number No answers found. Can you please help?

Hi Krish, I have created new environment for pyspark !pip install pyspark import pyspark are successful but import pandas as pd give error as: No module named 'pandas' what needs to do ?

Hi Krish, RuntimeError: Java gateway process exited before sending its port number I was getting this error in SparkSession.builder.appName('practice').getOrCreate() Please help me

When I am adding SparkSession code, I am getting below error. Exception: Java gateway process exited before sending its port number How to fix this

It's been ages...... I had waited for this from you krishhhhhhh😭😭😭😭😭🤩...Thank you💥

i am getting this error: [WinError 10061] No connection could be made because the target machine actively refused it can you plz help me to solve?

Hi I am getting following error while executing spark = SparkSession.builder.appName('learning').getOrCreate() RuntimeError: Java gateway process exited before sending its port number

Sir I am having issue when calling SparkSession.builder on local machine. The cell runs forever and nothing happens. I created a new environment and repeated the process. Still the cell gets stuck and doesn't proceed. Sir please reply

Hi @krish, I am getting ' RuntimeError: Java gateway process exited before sending its port number ' this error while starting spark session. could you please help me to resolve this

Hello, I keep getting this error and can't move past it could you kindly suggest please File ~\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyspark\java_gateway.py:106, in launch_gateway(conf, popen_kwargs) 103 time.sleep(0.1) 105 if not os.path.isfile(conn_info_file): --> 106 raise RuntimeError("Java gateway process exited before sending its port number") 108 with open(conn_info_file, "rb") as info: 109 gateway_port = read_int(info) RuntimeError: Java gateway process exited before sending its port number

Java gateway process exited before sending its port number I am getting this error while setting spark session builder. Can anyone help me out in this?

much waited playlist

what can be done when // spark=SparkSession.builder.appName('Practise').getOrCreate() is keep on running // is keep on running and not getting executed.

AnalysisException: Path does not exist: file:/C:/Users/abhi/test.csv How to solve this issue ....even i keep my file in the same location

type(df_pyspark) is always showing nonetype for me. kindly help me how to rectify the error

Trying to run code in jupyter ,but always getting errors.Dont know how to access file from local in jupyter

facing this error while creating sparkSession: RuntimeError: Java gateway process exited before sending its port number Any suggestions on this?

I am getting "py4JJavaerror", how to proceed further?

Hello Sir, I got error as :- "Exception: Java gateway process exited before sending its port number" while executing line number 5. How can I resolve it ?

Was eagerly waiting for this playlist. Thank you so much Krish! 🙂

Hi Sir,this playlist is enough for learning pyspark

Omg!! I have been literally been waiting for this!! Krish u r the man!!!

Most awaited playlist

I am stuck here " spark=SparkSession.builder.appName("Practice").getOrCreat() I am stuck here. It is showing error : AttributeError: 'Builder' object has no attribute 'getOrCreat'

Hi Krish I got error when creating sparksession, how I connect with you

Most awaited video from u... Thanks for the starting this session

Hi Krish, any idea why my code same as yours is not getting executed. I installed jupyter notebook using anaconda. I wish I could have pasted the screenshot here.

I desperately needed this course ! Thanks a lot !

when I run spark = SparkSession.builder.appName('Practise').getOrCreate(). I am getting "Java gateway process exited before sending its port number" this error

Thanks for this video. For learning purposes on my own computer, do I need to install apache.spark (spark-3.4.1-bin-hadoop3.tgz) to be able to run spark scripts/notebooks, or just pip install pyspark on my python environment?

Hi Krish, Thanks for these amazing videos. I am getting error "RuntimeError: Java gateway process exited before sending its port number" during creation of session in jupyter.

When I was creating pyspark seeion it's through an erro like this java gateway process exited before sending it's port number

Sir, how are you getting the automatic suggestions in jupyter notebook.. please help me, so that i can do the same with my system. Thanks alot

how to resolve this "Exception: Java gateway process exited before sending its port number"

pyspark works only with java 8 version and not the latest java software i.e java 17

Hello Krish I am getting the following error - "Java gateway process exited before sending its port number" when performing this code "spark = SparkSession.builder.appName("Practice").getOrCreate()" I have installed jdk as well and still it gives the same error please let me know how to resolve it as I have searched in net and I am not getting much of help. Thanks.

installing pyspark and when I am running the code , I am getting an error-" Java gateway process exited before sending its port number". I have set the path of java in environment variable, still getting the same error. If you could assist that would be great , so that we can start testing other videos

Do i have to install linux with virtual box in order tp work with pyspark?

my pyspark session is still running for last 2 hours. what to do, i tried other method like update my pip,etc. Did anyone face the same problem? any solution is appreciated.

Bro thank you! I swear other videos made it so complicated!

Getting "PySparkRuntimeError: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number." error while creating spark session. tried many ways to fix but still getting the same problem

Can you please let us know the advantages of using pyspark dataframe over pandas dataframe

When I am trying it on Jupyter it's showing: " RuntimeError: Java gateway process exited before sending its port number". What should I do now? I have tried it on Python idle also, but same error. Help if anyone has any solution.

Hi @krish Naik, When i import pyspark, i get an error "Kernal died" can you suggest what to do ?

Hello sir i'm facing RuntimeError: Java gateway process exited before sending its port number. Please let me know the issue

I really appreciate your videos. One thing which is missing is that your tutorial starts with python implementation only. If you create a video on how to configure spark in a system and connect with python, that would be a great help

can you please make a single video merging all individual files as we have internet problem at our place ,(ISPs router is placed in a commercial area) , so downloading one video will help me. PLEASE

sparksession is taking lot of time to create. Is there any solution for this?

Have been searching for good PySpark tutorials and this turned up 👍 Thanks!

Sir, after installing pyspark and when I am running the code , I am getting an error-" Java gateway process exited before sending its port number". I have set the path of java in environment variable, still getting the same error

Hey Krish, this nullable=True means it's not a primary key as we are working with SQL dataframe?

What's the difference between python and pyspark sir? The work done is same in both cases. Then y do we have to use pyspark. Are there any advantages? (A small doubt)

Hi Krish i am getting error while accessing file from D drive i kept csv in D drive .

Session builder is taking lot of time in executing and still not executed in jupyter notebook

i get spark output like this further unable to read the dataset

What we can divide dataset into multiple chunks in pandas and train the model on it is this good practice or bad practice?

Hey, I am trying to follow this.. and I am not getting the autocomplete feature. Is there any library I need to install?

How to create a timeseries in pyspark. Say for example I have a column named start_date wit the format (YYYY-MM) for some event, but its not continuous, i.e. I have 2015-01, 2015-04, 2015-07. Then how do I fill the missing dates between them and assign the values to other columns as 0 in pyspark? It was easy in pandas where I could just set this column as index and then resample the dataframe.

It's very important playlist. One querry about clustering. Which I faced in interview. How can you know which cluster is good??

bhai jupyter kaise kara install .. ???? python kaise kiya install??

Hi Krish , I am facing this error RuntimeError: Java gateway process exited before sending its port number @krishnaik

Sir why you chose python instead of scala? Is scala is not preferred in industry?

Very much useful.. Thanks Krish.