How to install Spark on Mac & Ubunbu

spark • installation • software

21 Sep 2016

Install Spark is handy, here a quick guide on Spark installation on Mac and Ubuntu.

• Download Spark 2.0 from the official website

• Extract the contents: cat /Users/<yourname>/spark.tgz | tar -xz -C /Users/<yourname>/

• Create a soft link cd /Users/<yourname>/ ln -s spark-* spark

• Add shortcuts to your .bash_profile:

export SPARK_HOME=~/Users/<yourname>/spark export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH

• Source .bash_profile and run

BONUS: To have similar environment like ipython/ ipython notebook, I added thesee alias in my bash_profile:

alias ipyspark='$SPARK_HOME/bin/pyspark --packages com.databricks:spark-csv_2.10:1.4.0' alias ipynbspark='PYSPARK_DRIVER_PYTHON=ipython PYSPARK_DRIVER_PYTHON_OPTS="notebook --no-browser --port=7777" $SPARK_HOME/bin/pyspark --driver-memory 15g'