How to set Master address for Spark examples from command line

673 Asked by pallav_2461 in Devops , Asked on Jul 1, 2021

*lassification.scala:105)

I have also tried adding in the Spark Master url anyways (though the code seems NOT to support it ..)

  spark://10.213.39.125:17088   --algorithm LR --regType L2 --regParam 1.0
  data/mllib/sample_binary_classification_data.txt

and

--algorithm LR --regType L2 --regParam 1.0 spark://10.213.39.125:17088
data/mllib/sample_binary_classification_data.txt

Both do not work with error:

Error: Unknown argument 'data/mllib/sample_binary_classification_data.txt'

Answered by Aashish chaursiya

One approach to resolve your errors is that you can set the Spark master from the command-line by adding the JVM parameter:

    -Dspark.master=spark://myhost:7077

Also, if you want to get this done from code you can use .setMaster(...) when creating the SparkConf:

val conf = new SparkConf().setAppName("Simple Application")

                          .setMaster("spark://myhost:7077")

And for Spark 2.x + do:

val spark = SparkSession.builder()

                        .appName("app_name")

                        .getOrCreate()

How to set Master address for Spark examples from command line

Your Answer