Running PySpark jobs on Bitnami Docker images

Подписаться 127

Просмотров 14 тыс.

50% 1

CODE: github.com/dsy...
An overview of how to set up a an Apache Spark cluster using Bitnami Spark Docker images (github.com/bit...) and run your first PySpark job. This video is part of a series on setting up a development and experimentation environment for Apache Spark, Livy, and Airflow using Docker and docker-compose.

Опубликовано:

2 окт 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 12

@mateusleao7093 2 года назад

it would be good if we heard an answer, had the same problem as the guys, when pointing to the image tag it can't fint it, then when pointed to localhsot:7077 it says the worker has no resources.

@jaykiran15 2 года назад

Did you get this sorted? I've the same issue

@HariharApamarjane-yi1qo 7 месяцев назад

I am also facing the same issue. Please let us know if you folks find any solution.@jaykiran15 @mateusleao7093

@bensberg3968 Год назад

WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master spark-master:7077

@ojitha Год назад

spark-master:7077 is not working. localhost not working. Any solutions?

@senthilkumarashokkumar8153 Год назад

check docker-compose has port forwarding 7077 to access outside docker

@vinayakkgarg2263 2 года назад

How can we use docker to set up spark workers on a different node(server) ?

@tieduprightnowprcls Год назад

try to expose 7077 port in the spark-master container so that it can be accessed publicly or privately. then those workers on the different server have it set the SPARK_MASTER_URL to the spark master IP:7077