On a Linux server, I have some Python scripts using the built-in sqlite3 module (+ some Sqlite extensions built from source, as detailed in Upgrade Python's sqlite3 on Debian).
For another Python script, I need a newer version of the Sqlite shared library than the one I already have on the system. Reason: I need Sqlite higher than 3.25.0 for Window Functions.
If I install it from source here and do make and make install, it will probably overwrite previous versions of this library on the server, and could potentially break other OS tools using it.
How do you handle the general problem of having multiple versions of the Sqlite shared library?
I don't think Python virtual environments can be used for this context, or would it be possible?
Note: pip3 install --upgrade sqlite3 does not exist: we cannot upgrade Python's built-in sqlite3 package like this. And by the way we probably should not, since it could break some OS tools using Python + sqlite3.
This is very tricky and will need a little code change in your scripts.
What to do:
First, check the sqlite3 library version included with python just in case:
python -c "import sqlite3; print(sqlite3.connect(':memory:').execute('SELECT sqlite_version();').fetchall())
In my computer (python 3.8, windows) the output is [('3.35.5',)] which means python has the sqlite 3.35.5 library. I have no sqlite installed in my system: this is the library that comes with python3.8.
IF your python sqlite3 library is not the one you need :-( you have an alternative: you can use the pysqlite3 instead of the sqlite3 standard library. In this case:
You'll need to build the pysqlite3 library by yourself using the Sqlite3 'amalgamation' that matches the version you want to use (more on later).
You'll need to install the library, and...
You will need to change your python script imports
import pysqlite3 as sqlite3 # instead of sqlite3
Ok, what is the 'amalgamation` and how to build pysqlite3?
The amalgamation is the whole sqlite3 library in just one .c file (with the sqlite3.h file). You can get it from the sqlite3 download page: sqlite3.36 amalgamation.
Once you have the amalgamation, follow the instructions to build statically pysqlite3, and install the package.
Now you can use pysqlite3 in your code.
If you want 2 different version of sqlite3 (python3) on 2 different environments, you can do that.
Since you mentioned that sqlite3 is part of the std library, it seems like you can try the pysqlite3 package instead.
If you can't run pip, run the following command first.
sudo apt install python3-pip
Then,
pip install virtualenv
python3 -m venv sqlitev1 #(whatever name you want)
source sqlitev1/bin/activate
pip install pysqlite3==0.4.4 #(this can be whatever version u want)
source deactivate
python3 -m venv sqlitev2 #(whatever name you want)
source sqlitev2/bin/activate
pip install pysqlite3==0.4.4 #(this can be whatever version u want)
source deactivate
Now you have 2 python environments, sqlitev1 and sqlitev2, with 2 different version of sqlite3.
It might be super hacky but you can make the new version of sqlite and then make sure that the path pointing to the new version is on the pythonpath environment before the built in one. Python will scan the python path from first to last to find an import, so the new version first in the python path for the processes that want the new version and then exclude that path with the old processes that need the built in one. You can accomplish this with a bash script that loads the env and then runs the python process for the new services.
Again this is super hacky so last resort.
If you want a different version of Sqlite than that installed with your distro, and a Python that uses that version, then you could
Compile sqlite to an alternative location
Compile Python to a different location, and point it to the custom Sqlite installation.
The "pointing" is covered in the accepted answer to this question. The question body itself shows how you might compile sqlite to a custom location.
The other answer to that question proposes setting the LD_LIBRARY_PATH environment variable to the directory containing the custom sqlite build to avoid having to compile Python. This might work with a virtualenv (it could be set in the preactive hook, for example).
See also
What is LD_LIBRARY_PATH and how to use it?
Set LD_LIBRARY_PATH before importing in python
Another approach would be to compile pysqlite3 in a virtualenv over the custom sqlite build. You can read about this in this blog post (I won't copy the details as it isn't clear what licence is used by the blog).
Related
I have CentOS 6 on my system and I'm trying to update SQLite for Python. I've installed it from source and executing sqlite --version returns version 3.33.0 as expected.
However, when I try to check the python SQLite version using import sqlite3; sqlite3.sqlite_version; I still get the previous SQLite version 3.6.20.
Software Locations:
Python 3.6.9 - /usr/bin/python3
Sqlite3 - /usr/bin/sqlite3
I've tried the solution here, this does not work at all, after updating LD_LIBRARY_PATH and checking the python SQLite version it still gives '3.6.20', and here, when I try sudo LD_RUN_PATH=, it gives me the error No such file or directory, but when I execute it without sudo LD_RUN_PATH=, it successfully compiles but still gives me SQLite '3.6.20' (Compiled python without uninstalling).
Note: I have multiple python3 versions.
What can I do to resolve this?
When I did it (specifically trying to find a way to update sqlite3 for a running python program; did not work...), I compiled sqlite and got libsqlite3.so.0.8.6, and then replaced the system-wide sqlite3 with that. For me on debian, that was in /usr/lib/x86_64-linux-gnu. I did see (though now I can't find where) that this way may cause issues when updating in the future. It did update python's sqlite3 for me though.
You can import specific versions:
__requires__= 'sqlite3==3.6.20'
import pkg_resources
pkg_resources.require("sqlite3==3.6.20")
import sqlite
Note that this only works on the first import. If sqlite gets imported before pkg_resources, it will take the latest version.
I have downloaded pyOpt from its website and installed it on python in ubuntu, using the instructions on the website.
Still, I cannot import and use it in my pycharm projects.
I had the same problem, but I did not dare to try Akin's solution for my research project (I think it is a good solution if anyone wants to stick to PyOpt package with python3).
I used Pyomo instead.
By the way, Laurent's answer actually points to another package. PyOpt and pyopt are two different packages.
I have been trying to figure this out too. It seems that the code for the pyOpt package is written in Python 2; when attempting to import anything once all system requirements have been met, the errors I received all point towards this. I will update this answer once I have figured out a way round this.
It should be feasible to simply have the script code converted to python 3 but I have not do this before.
Edit
You can use the package 2to3 to convert all the scripts to be compatible with Python 3.
Navigate to the correct folder through the command line in which the pyOpt files are.
Then use 2to3[pyOpt-1.2.0]-wand this will rewrite all the scripts to be compatible with Python 3. Note that you don't need the [] in the command. See 1 for further details.
1
Then you should be able to run python setup.py install and all should be golden.
PyOpt correctly works with Python 3. I have tried with Python 3.6.
I recommend you to use a virtualenv. You can consult the user guide online.
For instance, you can do:
python -m venv .venv
source .venv/bin/activate
pip install pyopt
If you use pew, you can do:
pew new -p python3.6 pyOpt -i pyOpt
This will create a new virtualenv for you and install pyopt.
To test pyopt, you can follow the documentation. You can try:
>>> import pyopt
>>> expose = pyopt.Exposer()
>>> #expose.args
... def regular_function(arg1:str, arg2:int):
... """Your help - the docstring"""
... print(repr(arg1), repr(arg2))
...
>>> expose.run()
Usage: arg1 arg2
To work with PyCharm, you can consult the article “Virtual environment”.
In short: Open the Preferences and search Add Python Interpreter. Clic on And Interpreter... and select Existing environment. That way, you can select the python executable which is in .env/bin directory.
IMHO you should use Click instead of PyOpt, really.
This question already has answers here:
Dealing with multiple Python versions and PIP?
(28 answers)
Closed 5 years ago.
I've installed django using the pip command (pip install Django), but i can't run it using the py command, as it can't find the module.
I can only make it works using 'python' command.
Here is a summary of the screenshot I have atttached
$ python --version
Python 3.6.1
$ py --version
Python 3.6.0
It also looks like django works only with 3.6.1.
Is there any way to set both commands to run newest version of python?
Screenshot:
You're using Python launcher for Windows when you executepy. You could be specific about which Python interpreter version that you want py to execute with this command:
> py -3.6
See this section from PEP 397:
Python Version Qualifiers
If no version qualifiers are found in a command, the environment
variable PY_PYTHON can be set to specify the default version qualifier
- the default value is "2". Note this value could specify just a major version (e.g. "2") or a major.minor qualifier (e.g. "2.6"), or even
major.minor-32.
If no minor version qualifiers are found, the environment variable
PY_PYTHON{major} (where {major} is the current major version qualifier
as determined above) can be set to specify the full version. If no
such option is found, the launcher will enumerate the installed Python
versions and use the latest minor release found for the major version,
which is likely, although not guaranteed, to be the most recently
installed version in that family.
In addition to environment variables, the same settings can be
configured in the .INI file used by the launcher. The section in the
INI file is called [defaults] and the key name will be the same as the
environment variables without the leading PY_ prefix (and note that
the key names in the INI file are case insensitive.) The contents of
an environment variable will override things specified in the INI
file.
Plus Python launcher isn't just limited to launching different Python versions, it also parses shebang #! in source code files, providing a functionality similar to that in *nix operating systems in Windows.
*Refer to Python Launcher for Windows documentation.
On windows, py is an executable stored in the C:\Windows folder. I honestly don't know what it contains, as I am used to where it is a symbolic link on linux, and my windows install shows the normal python executable as being a fraction of the size of py, despite my being quite sure that they point to the same installation. Regardless, you can fix your problem by deleting or renaming (python.bak, etc) the executable you don't want to keep using from the Windows folder, then copying the one you want in place and renaming it to the same name that you previously deleted or renamed. I can't imagine this is the official way to fix this problem, but this will work. Also, in the future, feel free to specify the version you are installing to with pip explicitly if you want to be sure of which installation you are using instead of just running whatever points to pip:
py -m pip install packagename
python -m pip install packagename
Running into problems with multiple python versions on the same system is quite common with Windows, so setting up a virtual environment may be beneficial. This is explained in the Django Windows install how-to.
After installing the Nest Neural Simulator, I keep getting the following error when trying to run any of the example python files that came in the installation. I've tried re-installing Nest, Python, and using Anaconda, but no go.
Python error:
ImportError: No module named nest
Suggestions?
At https://nest-simulator.org/documentation you now find many different install instructions and how to solve the "ImportError: no module named nest" depends on the way you installed NEST.
System Python
The problem with the nest python module not being found is usually, that NEST is installed for a specific Python version and you can not load it from another. So while many OS still use Python 2.7 you may need to explicitly run
$ python3
>>> import nest
Additionally, if you have multiple Python 3.x versions installed, modules may still be installed for a different version and you have to explicitly start python with python3.6 or python3.8, etc.
Conda package
As #nosratullah-mohammadi already mentioned, if you have a Conda flavour installed, using the pre-built package is a very quick solution. The link in his post is unfortunately broken; this one should work, then go to "Installation" in the side bar.
$ conda create --name nest -c conda-forge python3 nest-simulator
$ conda activate nest
$ python # this should load the Python from the conda env
>>> import nest # this loads nest which is installed explicitly for that Python
From Source
For every install from source, be sure to have Python and other prerequisites installed before building NEST. Then you can create your temporary build directory (can be deleted afterwards) and configure with the flags you need.
cd somewhere
mkdir nest-build
cd nest-build
cmake -DCMAKE_INSTALL_PREFIX:PATH=/install/path -Dwith-python=3 .../sources/of/nest-simulator
Replace somewhere, /install/path and .../sources/of/nest-simulator with the paths correct for your setup. (A popular choice when compiling from source in conjunction with Conda environments, for example, is to use -CMAKE_INSTALL_PREFIX=$CONDA_PREFIX, which installs NEST directly into the active environment. Conda is however in no way necessary for NEST.)
Add more -D... flags as you prefer. Possible flags you see with cmake -LA .../sources/of/nest-simulator, as pointed out here. You are probably interested in many of the with-xyz at the end. Check the aforementioned documentation for deatils.
Check that the paths and libraries reported in the Configuration Summary make sense (you may need to scroll up a bit to see). It could for example look something like this:
--------------------------------------------------------------------------------
NEST Configuration Summary
--------------------------------------------------------------------------------
[...]
Python bindings : Yes (Python 3.6.8: /home/yourname/miniconda3/envs/nest/bin/python3)
Includes : /home/yourname/miniconda3/envs/nest/include/python3.6m
Libraries : /home/yourname/miniconda3/envs/nest/lib/libpython3.6m.so
Cython bindings : Yes (Cython 0.27.3: /home/yourname/miniconda3/envs/nest/bin/cython)
[...]
--------------------------------------------------------------------------------
[...]
PyNEST will be installed to:
/home/yourname/miniconda3/envs/nest/lib/python3.6/site-packages
--------------------------------------------------------------------------------
In this example CMake configured everything for Python3.6 from my conda environment.
If you are satisfied with your settings and all the found Python versions match, run the usual
$ make # optionally with -j$(nproc)
$ make install
$ make installcheck
In case that works out fine you are done and can delete the build directory to free the space. Congratulations!
Also if things get too mixed up and it doesn't seem to do what you expect, it is sometimes useful to delete the build directory and start off clean.
there is a new method of installation added to other methods, wich is installing nest with conda package and it's in its beta version. but it works and it's really simple.
you can find the installation from here!
simply after install mini conda package run your terminal and type this :
conda create --name ENVNAME -c conda-forge nest-simulator python
then type :
conda activate ENVNAME
and you're good to go!
NEST now provide the solution to that problem and similar ones, by providing a script which automatically sets the relevant system variables:
If your operating system does not find the nest executable or Python
does not find the nest module, your path variables may not be set
correctly. This may also be the case if Python cannot load the nest
module due to missing or incompatible libraries. In this case, please
run
source </path/to/nest_install_dir>/bin/nest_vars.sh
to set the necessary environment variables. You may want to include
this line in your .bashrc file, so that the environment variables are
set automatically.
https://nest-simulator.readthedocs.io/en/latest/installation/linux_install.html
Turns out I needed to move the directory where I installed nest (Users/name/opt/nest) into a nest folder in the following directory in anaconda. Specifically, I moved the contents of the folder (from the nest installation):
/Users/name/opt/nest/lib/python2.7/site-packages/nest
Into this one:
/anaconda/lib/python2.7/site-packages/nest
Disclaimer: I could very well run into problems for not having copied all the contents of the Nest installation, but this little hack is helping me run example files now.
I'm just starting out with Python, and have found out that I can import various libraries. How do I find out what libraries exist on my Mac that I can import? How do I find out what functions they include?
I seem to remember using some web server type thing to browse through local help files, but I may have imagined that!
From the Python REPL (the command-line interpreter / Read-Eval-Print-Loop), type help("modules") to see a list of all your available libs.
Then to see functions within a module, do help("posix"), for example. If you haven't imported the library yet, you have to put quotes around the library's name.
For the web server, you can run the pydoc module that is included in the python distribution as a script:
python /path/to/pydoc.py -p 1234
where 1234 is the port you want the server to run at. You can then visit http://localhost:1234/ and browse the documentation.
Every standard python distribution has these libraries, which cover most of what you will need in a project.
In case you need to find out if a library exists at runtime, you do it like this
try:
import ObscureModule
except ImportError:
print "you need to install ObscureModule"
sys.exit(1) # or something like that
You can install another library: yolk.
yolk is a python package manager and will show you everything you have added via pypi. But it will also show you site-packages added through whatever local package manager you run.
just run the Python interpeter and type the command
import "lib_name"
if it gives an error, you don't have the lib installed...else you are good to go
On Leopard, depending on the python package you're using and the version number, the modules can be found in /Library/Python:
/Library/Python/2.5/site-packages
or in /Library/Frameworks
/Library/Frameworks/Python.framework/Versions/Current/lib/python2.6/site-packages
(it could also be 3.0 or whatever version)...
I guess it is quite the same with Tiger
Considering that in every operating system most of python's packages are installed using 'pip' (see pip documentation) you can also use the command 'pip freeze' on a terminal to print a list of all the packages you have installed through it.
Other tools like 'homebrew' for macOS (used when for some reason you can't install a package using pip) have similar commands, in this specific case 'brew list'.