I need to perform some custom settings for the python interpreter: specifically related to pyspark: the "interpeter" will actually be the spark-submit (aka pyspark) shell script. The intent is to be able to run pyspark jobs within the python console. Running within a Run Configuration would also be just fine: this would be an alternate approach.
I use IJ Ultimate - which has good python support: except well maybe for this particular use case.
Let us compare to pycharm - and specifically an ability to customize the interpreter - including setting local, remote, or virtualenv:
The Intellij Ultimate seems to lack those options: instead it is pointed to the libraries for a python sdk. That will not be sufficent for the given use case:
Here is the dropdown: notice there is no way to add a custom python interpreter.
So is there a way in Intellij to set the interpreter path? I want to set it to $SPARK_HOME/bin/pyspark ?
PyCharm and IntelliJ have the exact same options to add and configure Python code.
PyCharm just makes it easier.
Those windows do not have anything for interpreters
Pretty sure it does... You add interpreters here.
bin/pyspark is not an interpreter, it is a shell script. You just set the regular Python interpreter.You also need to add the Pyspark libraries. (See below)
Then, you configure the environment variables here (Run Configurations) (see those below)
As far as PySpark libraries go, you have to add these (use the full path, not variables)
$SPARK_HOME/python/
$SPARK_HOME/python/lib/py4j-X.X-src.zip
You also need to set these variables in the Edit Configurations window shown
SPARK_HOME = path to spark
PYTHONPATH = path to py4j-X.X-src.zip (also need to append the path to the current python interpreter's directory, I believe)
Ref:
How to link PyCharm with PySpark?
https://github.com/ybenoit/pyspark-ide-starter (it's got some French in it, but it's still readable)
And here's a video of some code running
https://www.youtube.com/watch?v=u-P4keLaBzc
Related
I am trying to use Pycharm to develop Libreoffice python Macro with in the virtual environment with not a lot of success. You'll find below a few things I have tried with the differnt errors I have had so far. I welcome any answers on the errors below as well as alternate solutions.
first try : set up directly the venv environment
As when you work with Libreoffice, you need to use the embedded python integrator, the first attempt is when defining the python integrator in pycharm, I reference this embedded interpreter (located somewhere in Program Files\Libreoffice\program` on my windows machine, I click the new venv box in pycharm and get the following error :
Error: standard python 'venv' module not found
So it seems venv is not packaged with the Libreoffice embedded python environment. Is there a way to add it?
Second try : inject embedded environment inside a precreated venv
What I did here was to create the Venv in pycharm using a standard python version aligned with the one libreoffice uses (here 3.8.10). Once the venv is succesfully created, I manually change the path to the python based interpretor to the proper libreoffice one.
I also update the pythonpath in pycharm for this venv to include Files\Libreoffice\program\ so that we can find some preinstalled modules suitable for Libreoffice, in particular uno.py.
This looks a little better, I can launch a libreoffice instance, but when I actually want to do stuff inside this instance from python, I get the following error loading component library </mergedlo.dll> failed
This mergedlo.dll file is in a directory referenced in the python path for this venv.
What configuration did I miss here?
This's probably quite off-topic but maybe someone else share my experience... when I started coding KiCAD plugins and learning Python I decided to use VSCode (with Microsoft Python extension) because I already use it for a number of other things and works great.
I have two main issues mostly bound to Python extension itself other than VSCode:
VSCode try to use python 3.7 binary (I installed it on my
machine and I need it) instead kicad/python 2 (it's installed on a different location);
even if I setup VSCode to look for additional packages with a specific configuration file (settings.json):
"python.autoComplete.extraPaths": ["C:/Program Files/KiCad/lib/python2.7/site-packages"]
VSCode refuse to investigate pcbnew.py (located in that folder) and thus every Intellisense like function (inspection etc) doesn't work.
How can I fix that? ... I'm more interested in fixing the second issue other than the first.
To correctly enable per project configuration of VSCode / Python open folder and create a workspace configuration for the project itself (i.e. create a .vscode folder inside project folder and store a settings.json file inside it with you custom configuration). In my specific case I needed to specifiy a custom location for an older Python binary trough python.pythonPath directive.
I am trying to use Linter with atom, on some python files and I keep getting this error.
Console Error Output. I am not sure what to do here, Python is on the PATH and so is flake8.
Can anyone provide clarification here?
Also I am using Python 3.7.0
Let speak a bit about your environment. Unfortunately, I can't suggest what exactly you need to do as you provided very little information. Nevertheless, your question contains enough information for a generic answer.
The flake8 tool could be in your PATH in terminal (whatever you use), but not in PATH of the Atom editor. You could achieve your goal by one of the following actions or a combination of them.
Modifying global PATH, like user PATH in Windows Environment variables, or /etc/paths in macOS (I have no information if ~/.profile would work)
Trying to run tool directly by full path (for example, /opt/local/Library/… or C:\Python37\…)
Use script-wrapper, which will set required variables and enables Virtual Environment if needed.
I am looking for best practices for maintaining different python installations (or environments) simultaneously, specifically, with regards to achieving the following 3 objectives/requirements:
Note: I am using Anaconda installation for Python
Requirements
Requirement1 - Jupyter Notebook convenience
All python interpreters (either in different installations or environments) should be available in jupyter notebook in the list of kernels.
Requirement2 - Command line convenience
On the command line it should be easy to specify any python interpreter available on my machine to run a py script, something like: 'py2 test.py' or 'py3_env1 test.py'
Requirement3 - Automation convenience
Run the py scripts using the *.bat files, using any python interpreter available on the machine.
Choices
As implicit above, there are 2 broad choices here:
Choice1 - Single installation, multiple environments
...i.e. maintain a 'single' python installation folder but create multiple environments in it using 'conda'.
Choice2 - Mulitple installations, single environment each
... i.e. maintain different python interpreters in separate installation folders with only the root environment.
If I make any any of the two Choices above, I can satisfy some, but not all 3 requirements.
For instance, if I make Choice1, I can satisfy Requirement1 easily, but not 2 and 3; and with Choice2, I can satisfy Requirement2 and Requirement3 only. Details as below.
Satisfying requirements
Satisfying Requirement1 - Jupyter Notebook convenience
If I make Choice1 (single installation, multiple environments), I can satisfy Requirement1 using the solutions posted here. It suggests registering ipykernels in all of the environments.
With Choice2, is it possible for jupyter notebook to see all kernels spread across many python installations? I couldnt make it work.
Satisfing Requirement2 - Command Line convenience
If I make Choice2, then I can satisfy Requirement2 by using Python Launcher or creating symlinks as suggested here
With Choice1, it doesn't work since Python Launcher doesn't recognise different environments (yet!). So, is there any other way to make it possible? Can I, lets say, use some kind of command alias, something like a short-cut command 'py' that refers to the 'root environment' python.exe file, and another one 'py2' that refers to another environment's python.exe. And if I do that, will calling 'py2' be robust in the sense that it will call correct site-packages folder? (Note, we haven't activated the environment here, so the windows 'PATH' may not point to the correct folders). I would assume that activating the environment is necessary, but I am not sure.
Satisfing Requirement3 - Automation Convenience
If I make Choice2, I can probably satisfy Requirement3 using Python Launcher as well (just like above). Also, not to mention, my script can also use shebang operator to mention which python it wants to invoke. However, I don't want to hardcode the python version in the script and keep the flexibility in the command itself.
If I make Choice1, I thought that a *.bat file like below will work:
conda activate py3_env1
python test.py
deactivate
So, I intend to activate the environment first and then call the py file. However, this doesn't work. Calling this bat file executes just the first line, activates the environment and then nothing happens.
So, in short, I am looking to have a consistent setup that allows me to invoke different python versions from all mediums - notebook, command prompt and automated bat files - very conveniently.
Any guidance or recommendation around best practices are very appreciated.
Using the PyCharm IDE, when setting up an external tool, how can you set up the external tools with a path relative to use the current virtual env defaults.?
An example being pylint - where I'd want the virtual env version and not the system one to run.
Not sure about older versions, but in PyCharm 5 one can use $PyInterpreterDirectory$ macro. It's exactly that we want
There's JDKPath macro you can use which points to the executable of the interpreter set for a project. You can combine this with the fact that many tools can be run by directly running their module(s) using the -m option of interpreter. Also there's issue PY-2734 New useful macros in external tools everyone can vote on.
just found your post while looking for documentation about the "variables" that could bew used when setting parameters for external tools.
No documentation but you can see a list of all the available stuff after pressing thE "INSERT MACRO" button in the Edit Tool dialog.
I don't see any reference to the interpreter path there but I usually use the virtualenv as my project path. If you are doing that too you could infer the python interpreter path from there.
In Tool Settings, set Program: to $PyInterpreterDirectory$/pylint