Install Spark on Mac OS - python

Trying to install Spark and associated programs on Mac but receiving error messages when testing installation.
/Users/somedirectory/apachespark/spark-2.3.0-bin-hadoop2.7/bin/pyspark
/Users/somedirectory/apachespark/spark-2.3.0-bin-hadoop2.7/bin/spark-class: line 71: /Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home//bin/java: No such file or directory
from my bash_profile entry...
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/
export SPARK_HOME=/Users/directory/apachespark/spark-2.3.0-bin-hadoop2.7
export SBT_HOME=/Users/directory/apachespark/sbt
export SCALA_HOME=/Users/directory/apachespark/scala-2.11.12
export PATH=$JAVA_HOME/bin:$SBT_HOME/bin:$SBT_HOME/lib:$SCALA_HOME/bin:$SCALA_HOME/lib:$PATH
export PATH=$JAVA_HOME/bin:$SPARK_HOME:$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH
export PYSPARK_PYTHON=python3
PATH="/Library/Frameworks/Python.framework/Versions/3.6/bin:${PATH}"
export PATH
correction suggestions? Thanks.

As shown in the reported error message:
/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home//bin/java: No such file or directory
your file path for the Java executable $JAVA_HOME/bin generates an extra / due to the trailing / in your JAVA_HOME:
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/
Removing the trailing / in JAVA_HOME should fix the problem. Better yet, setting JAVA_HOME as shown below would automatically point to the active JDK version on Mac OSX:
export JAVA_HOME=$(/usr/libexec/java_home)

Related

Receiving error when attempting to setup python within GIT Bash

I attempted to follow the instructions from this link https://prishitakapoor2.medium.com/configuring-git-bash-to-run-python-for-windows-a624aa4ae2c5 to setup python with git bash. However when I attempt to run source .bashrc I receive the following error
bash: export: python.exe export PATH=/mingw64/bin:/usr/bin:/c/Users/Grant/bin:/c/Program Files/Oculus/Support/oculus-runtime:/c/Program Files (x86)/Common Files/Oracle/Java/javapath:/c/Program Files (x86)/Common Files/Intel/Shared Libraries/redist/intel64/compiler:/c/ProgramData/Oracle/Java/javapath:/c/WINDOWS/system32:/c/WINDOWS:/c/WINDOWS/System32/Wbem:/c/WINDOWS/System32/WindowsPowerShell/v1.0:/c/Program Files (x86)/AMD/ATI.ACE/Core-Static:/c/Program Files (x86)/Calibre2:/c/Program Files (x86)/QuickTime/QTSystem:/c/WINDOWS/System32/OpenSSH:/c/Program Files (x86)/NVIDIA Corporation/PhysX/Common:/c/Program Files/NVIDIA Corporation/NVIDIA NvDLISR:/c/WINDOWS/system32:/c/WINDOWS:/c/WINDOWS/System32/Wbem:/c/WINDOWS/System32/WindowsPowerShell/v1.0:/c/WINDOWS/System32/OpenSSH:/c/Program Files/dotnet:/c/Program Files (x86)/Microsoft SQL Server/150/DTS/Binn:/c/Program Files/Azure Data Studio/bin:/cmd:/c/Udacity/Python:/c/Udacity/Python/Library/mingw-w64/bin:/c/Udacity/Python/Library/usr/bin:/c/Udacity/Python/Library/bin:/c/Udacity/Python/Scripts:/c/Users/Grant/AppData/Local/Microsoft/WindowsApps:/c/Program Files/Azure Data Studio/bin:/c/Users/Grant/AppData/Local/atom/bin:/c/Users/Grant/Anaconda3:/c/Users/Grant/Anaconda3/scripts alias python=winpty:/c/users/grant/anaconda3:/c/users/grant/anaconda3/Scripts:\C\Users\Grant\anaconda3:\C\Users\Grant\anaconda3/Scripts:\C\Users\Grant\Desktop\anaconda3:\C\Users\Grant\Desktop\anaconda3\Scripts:/c/Users/Grant/Anaconda3:/c/Users/Grant/Anaconda3/scripts': not a valid identifier bash: .bashrc: line 13: unexpected EOF while looking for matching "'
I have attempted to uninstall both batch and anaconda and reinstall both to fix the issue but I am unable to resolve the problem. Any tips? I am very new to this and trying to get python setup on my local PC for the first time.
I attempted to update the path by using the echo 'export PATH="$PATH:[YOUR_PATH]:[YOUR_PATH]/Scripts"' >> .bashrc again but this did not allow me to change the path and I am still receiving the same error.
As error tells, you have unmatched " in your .bashrc file. To more accurate answer,will be good if you attach .bashrc file

modifying the PYTHONPATH variable in zsh

I would like to add my site-packages directory to the PYTHONPATH variable, so I can import modules which are stored there to use them in scripts in interpreter. My default shell is zsh, and the OS is macOS Monterey, version 12.6.
I have tried to open the ~/.zprofile in nano editor from my terminal, adding the path I need, and restarting my terminal, but it did not help to solve the issue. After restarting the terminal, I tried to run following commands:
narynaa#Narynas-MacBook-Pro ~ % echo ~/.z.profile
/Users/narynaa/.z.profile
narynaa#Narynas-MacBook-Pro ~ % cat ~/.zprofile
# Set PATH, MANPATH, etc., for Homebrew.
eval "$(/opt/homebrew/bin/brew shellenv)"
export PYTHONPATH="/opt/homebrew/lib/python3.10/site-packages"
The path I need is stored inside the ~/.zprofile file now, but the problem is not solved.
Thank you for your help!

Adding variable to path so that a python os.environ works

I am trying to add this python library to my MAC. Once the installation is done, I have to
Create environment variables:
NEUROD_ROOT = NeuroDecode path
NEUROD_DATA = path to the desired data folder
NEUROD_SCRIPTS = path to the desired scripts folder
I edited the file /etc/paths with nano:
sudo nano /etc/paths
And the file now looks like:
/usr/local/bin
/usr/bin
/bin
/usr/sbin
/sbin
/Users/mathieu/Documents/NeuroDecode/scripts
NEUROD_ROOT=/Users/mathieu/Documents/NeuroDecode
NEUROD_DATA=/Users/mathieu/Documents/NeuroDecode/SubjectData
NEUROD_SCRIPTS=/Users/mathieu/Documents/NeuroDecode/SubjectScripts
Supposely, the file to start NeuroDecode is nd_gui.cmd. Not ideal for macOS I believe, and this file only includes start /B pythonw %NEUROD_ROOT%/neurodecode/gui/mainWindow.py. So I tried to shortcut it by adding an alias in my .zshrc file:
function runpy() {
python "$#"
}
alias neurodecode="runpy '/Users/mathieu/Documents/NeuroDecode/neurodecode/gui/mainWindow.py'"
Note: python is also alias and start python 3.8.5.
Sadly, it doesn't work. A KeyError is raised when the program tries to do os.environ['NEUROD_ROOT']. If I have a look at the elemnent os.environ._data, I have (incomplete screenshot):
Any idea what I did wrong and how I could properly set the path variables?

Alternative Windows/bash command 'workon [insert virtualenv name]' for Mac Terminal [duplicate]

I have installed virtualenv and the virtualwrapper via apt-get, I got to a point where I created a virtual enviroment but however later on during that same day when I used the workon command it was not found. I further on went and inspected my home directory and .virtualenvs dir and the virtualenv I created earlier were still there.
Solving this problem took two steps:
Add this to your .bashrc / .bash_profile / .zshrc:
# load virtualenvwrapper for python (after custom PATHs)
venvwrap="virtualenvwrapper.sh"
/usr/bin/which -s $venvwrap
if [ $? -eq 0 ]; then
venvwrap=`/usr/bin/which $venvwrap`
source $venvwrap
fi
Then use:
source .bash_profile
# or .bashrc / .zshrc
to reflect the changes.
Additionally, if the terminal still sometimes cant find workon, use source .bash_profile to reset and find it again.
type source .profile in home directory from terminal.
Read the readme in the top of which virtualenvwrapper.sh
You need to source it inside bashrc
open ~/.profile
cd ~
nano .profile
add at the end
#virtualenvwrapper setup
export WORKON_HOME=$HOME/envs
export PROJECT_HOME=$HOME/dev
source /usr/local/bin/virtualenvwrapper.sh
to load your .profile file you just edited:
$ . .profile
I ran in to this problem too and I simply needed to logout and log back in.
This read in the changes which the debian package manager made to my system at /etc/bash_completion.d/virtualenvwrapper

Where do I find the bashrc file on Mac?

Hello I am following this page.. I'm installing Python onto my mac so that I can set up a Django / Eclipse development environment. However I am not too sure how to go about executing this step:
The script will explain what changes it will make and prompt you
before the installation begins.
Once you’ve installed Homebrew,
insert the Homebrew directory at the top of your PATH environment variable.
You can do this by adding the following line at the bottom of your
~/.bashrc file
export PATH=/usr/local/bin:$PATH
Where do I find the bashrc file on my mac and where do I find the homebrew directory?
I am running a macbook pro with OS 10.8.5.
The .bashrc file is in your home directory.
So from command line do:
cd
ls -a
This will show all the hidden files in your home directory. "cd" will get you home and ls -a will "list all".
In general when you see ~/ the tilda slash refers to your home directory. So ~/.bashrc is your home directory with the .bashrc file.
And the standard path to homebrew is in /usr/local/ so if you:
cd /usr/local
ls | grep -i homebrew
you should see the homebrew directory (/usr/local/homebrew). Source
Yes sometimes you may have to create this file and the typical format of a .bashrc file is:
# .bashrc
# User specific aliases and functions
. .alias
alias ducks='du -cks * | sort -rn | head -15'
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
PATH=$PATH:/home/username/bin:/usr/local/homebrew
export PATH
If you create your own .bashrc file make sure that the following line is in your ~/.bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
I would think you should add it to ~/.bash_profile instead of .bashrc, (creating .bash_profile if it doesn't exist.) Then you don't have to add the extra step of checking for ~/.bashrc in your .bash_profile
Are you comfortable working and editing in a terminal? Just in case, ~/ means your home directory, so if you open a new terminal window that is where you will be "located". And the dot at the front makes the file invisible to normal ls command, unless you put -a or specify the file name.
Check this answer for more detail.
On your Terminal:
Type cd ~/ to go to your home folder.
Type touch .bash_profile to create your new file.
Edit .bash_profile with your code editor (or you can just type
open -e .bash_profile to open it in TextEdit).
Type . .bash_profile to reload .bash_profile and update any
functions you add.
In my macOS Monterey version, zsh is the default terminal shell. zsh executes ~/.zshrc every time the terminal is opened.
vi ~/.zshrc
#Add your path export to .zshrc
PATH=/usr/local/bin:$PATH
Now, when you open the terminal, the path will be set correctly.
On some system, instead of the .bashrc file, you can edit your profils' specific by editing:
sudo nano /etc/profile
~/.bashrc is already a path to .bashrc.
If you do echo ~ you'll see that it's a path to your home directory.
Homebrew directory is /usr/local/bin. Homebrew is installed inside it and everything installed by homebrew will be installed there.
For example, if you do brew install python Homebrew will put Python binary in /usr/local/bin.
Finally, to add Homebrew directory to your path you can run echo "export PATH=/usr/local/lib:$PATH" >> ~/.bashrc. It will create .bashrc file if it doesn't exist and then append the needed line to the end.
You can check the result by running tail ~/.bashrc.
The .bash_profile for macOS is found in the $HOME directory. You can create the file if it does not exit. Sublime Text 3 can help.
If you follow the instruction from OS X Command Line - Sublime Text to launch ST3 with subl then you can just do this
$ subl ~/.bash_profile
An easier method is to use open
$ open ~/.bash_profile -a "Sublime Text"
Use Command + Shift + . in Finder to view hidden files in your home directory.
Open Terminal and execute commands given below.
cd /etc
subl bashrc
subl denotes Sublime editor. You can replace subl with vi to open bashrc file in default editor. This will workout only if you have bashrc file, created earlier.

Categories

Resources