I am installing pyspark in ubuntu wsl in windows 10. These are the commands I used after installing wsl from Microsoft Store.
#install Java runtime environment (JRE)
sudo apt-get install openjdk-8-jre-headless
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64/jre
#download spark, visit https://spark.apache.org/downloads.html if you want a different version
wget https://apache.osuosl.org/spark/spark-2.4.7/spark-2.4.7-bin-hadoop2.7.tgz
#untar and set a symlink
sudo tar -xvzf spark-2.4.7-bin-hadoop2.7.tgz -C /opt
sudo ln -s spark-2.4.7-bin-hadoop2.7 /opt/spark
ln -s /usr/bin/python3.8.5 python
/opt/spark/bin/pyspark
Error:
/mnt/c/Users/akash/.pyenv/pyenv-win/bin/pyenv: 3: cmd: not found
/mnt/c/Users/akash/.pyenv/pyenv-win/bin/pyenv: 3: cmd: not found
I am not able to understand why it is looking in C drive. How to solve this? Any help is appreciated.
Instead of creating a symbolic link, try moving the unpacked directory spark-3.0.1-bin-hadoop2.7 to the opt/spark directory:
sudo mv spark-3.0.1-bin-hadoop2.7 /opt/spark
Also, go for the latest version which is Spark 3.0.1 with Hadoop 2.7.
You can read thru this online article for additional details.
I just came across a pip package I want to use however I'm new to python and PIP and not sure - is it possible to run that directly from terminal/the command line. If so, I can't seem to find the syntac to run a pip package.
So I installed pip using:
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
then
python get-pip.py
I then checked installation, by the command python -m pip
I then installed the package like:
python -m pip install openapi-cli-tool
Anyhow, as per the docs of that package I though I could just do:
openapi-cli-tool bundle -t html file1.json file2.yaml` > ./specification.html
Which didn't work, neither did this:
python -p pip openapi-cli-tool bundle -t html file1.json file2.yaml` > ./specification.html
Any help in explaining how this works would be appreciated.
You have to install it with pip, run command below:
pip install openapi-cli-tool
Then openapi-cli-tool will be available in your terminal, so you shall be able to run command, the command below is with correction of parameters passing:
Make sure you have file1.json and file2.yaml in file system.
openapi-cli-tool bundle -t html file1.json file2.yaml` > ./specification.html
python -m pip install
Installs the package to the user's local directory (iirc)
So, you can access it from ~/.local/bin/, like this:
~/.local/bin/openapi-cli-tool bundle -t html file1.json file2.yaml` > ./specification.html
You can add ~/.local/bin to your path by
export PATH=$PATH:$HOME/.local/bin
and probably add that line to your .bashrc or it's equivalent.
Then, you can access it by just openapi-cli-tool bundle
I have the problem that when I run my code on a linux server I get:
ModuleNotFoundError: No module named '_sqlite3'
So after researching, I found out sqlite3 was supposed to have been installed when I installed python, however it didn't.
I think the problem comes from the way I installed python. Since I do not have sudo permissions, I installed python3.7 in a local directory using: This guide.
All solutions to this sqlite3 problem that I can find requires sudo commands.
Is there another way that I can install python3.7 together with sqlite3 in my local Linux directory without using any sudo commands?
I hope I have stated my question clearly and I would appreciate all the help I can get. Thank you!
While installing a python package in a Linux system without "sudo" privileges you can use
For Python 3
pip3 install --user pysqlite3
You can install any third party packages with the same method
pip3 install --user PACKAGE_NAME
The --user flag to pip install tells Pip to install packages in some specific directories within your home directory. For more information click here.
Hope it helps !
The solution is to first build sqlite3 into a user directory and then build python using that directory's libraries and include headers. In particular, #Ski has answered a similar question regarding python 2, which can be adopted to python 3:
$ mkdir -p ~/applications/src
$ cd ~/applications/src
$ # Download and build sqlite 3 (you might want to get a newer version)
$ wget http://www.sqlite.org/sqlite-autoconf-3070900.tar.gz
$ tar xvvf sqlite-autoconf-3070900.tar.gz
$ cd sqlite-autoconf-3070900
$ ./configure --prefix=~/applications
$ make
$ make install
$ # Now download and build python 2, same works for python 3
$ cd ~/applications/src
$ wget http://www.python.org/ftp/python/2.5.2/Python-2.5.2.tgz
$ tar xvvf Python-2.5.2.tgz
$ cd Python-2.5.2
$ ./configure --prefix=~/applications
$ make
$ make install
$ ~/applications/bin/python
Alternatively, if you already have to specify a different --prefix for some reason (this has happened to me with pyenv), use LDFLAGS and CPPFLAGS when configuring python build:
$ ./configure LDFLAGS=-L/home/user/applications/lib/ CPPFLAGS=-I/home/user/applications/include/
I've cloned a django project to a Centos 7 vps and I'm trying to run it now, but I get this error when trying to migrate:
$ python manage.py migrate
django.core.exceptions.ImproperlyConfigured: SQLite 3.8.3 or later is required (found 3.7.17).
When I checked the version for sqlite, it was 3.7.17, so I downloaded the newest version from sqlite website and replaced it with the old one, and now when I version it, it gives:
$ sqlite3 --version
3.27.2 2019-02-25 16:06:06 bd49a8271d650fa89e446b42e513b595a717b9212c91dd384aab871fc1d0f6d7
Still when I try to migrate the project, I get the exact same message as before which means the newer version is not found. I'm new to linux and would appreciate any help.
I got the same error in CentOS 7.6 and Python 3.7.3 versions. I think you are using Django 2.2.* some version. In latest of Django 2.2, they changed the SQLIte version, that cause of your problem.
This is the release notes of Django 2.2 about SQLite.
The minimum supported version of SQLite is increased from 3.7.15 to 3.8.3.
So I found 3 steps to solve this problem,
Downgrade Django Version
So you can install latest version of Django 2.1 by using this command, which mean you're going to downgrade your Django version.
pip install Django==2.1.*
Upgrading SQLite on CentOS to 3.8.3 or Later
or you can followup below steps as well to keep the latest version Django. I directly get the steps from Upgrading SQLite on CentOS to 3.8.3 or Later article.
You can download the latest sqlite version from here.
wget https://www.sqlite.org/2019/sqlite-autoconf-3280000.tar.gz
tar zxvf sqlite-autoconf-3280000.tar.gz
./configure
make
sudo make install
We've installed to the latest version, but the problem is same. Here,
>>> import sqlite3
>>> sqlite3.sqlite_version
'3.7.17'
In the article, they've mentioned about LD_RUN_PATH and LD_LIBRARY_PATH paths.
Then make sure to compile python again using the LD_RUN_PATH environment variable.
It is better to use this variable over LD_LIBRARY_PATH.
Using LD_LIBRARY_PATH - whenever python is run it will look for linked libraries with that path.
What we want is for the libraries to be cooked into python at link time - compile time.
So based on the article, we can do the similar thing,
cd /opt/Python-x.y.z
LD_RUN_PATH=/usr/local/lib ./configure
LD_RUN_PATH=/usr/local/lib make
LD_RUN_PATH=/usr/local/lib make altinstall
Then try again,
>>> import sqlite3
>>> sqlite3.sqlite_version
'3.31.1'
Here we go, one thing they've mentioned,
If you do not use LD_RUN_PATH, then you have to make sure that the LD_RUN_PATH environment variable is set to /usr/local/lib for every user that is going to run python - which can be really annoying to do.
testing a Django 2.2 website with SQLite3 on CentOS 7
This is same as the previous one and based on LD_LIBRARY_PATH approach. Here is the steps from the article,
$ wget https://www.sqlite.org/2018/sqlite-autoconf-3240000.tar.gz
$ tar zxvf sqlite-autoconf-3240000.tar.gz
$ ./configure --prefix=/usr/local
$ make
$ sudo make install
$
$ python3.6 -c "import sqlite3; print(sqlite3.sqlite_version)"
3.7.17
$
$ export LD_LIBRARY_PATH=/usr/local/lib
$ python3.6 -c "import sqlite3; print(sqlite3.sqlite_version)"
3.24.0
If the last two steps didn't work, please comment below with the error you got and I'll find another solution for you.
I solved a similar situation with the following patches of code. Follow these steps that I used on my own centos7 & everything should be alright.
Just remember to let your centos7 know that you are calling python3 not just python otherwise it will call the default python2 followed by a series of errors in your virtualenv.
Installing python3 (from source):
cd ~
wget https://www.python.org/ftp/python/3.7.3/Python-3.7.3.tar.xz
tar xJf Python-3.7.3.tar.xz
cd Python-3.7.3
./configure
make && make install
export PATH=$HOME/opt/python-3.7.3/bin:$PATH
Then run: source .bash_profile
Confirming by
python3 --version
Python 3.7.3
Installing your sqlite3 (from source):
$ cd ~
$ wget https://www.sqlite.org/2019/sqlite-autoconf-3290000.tar.gz
$ tar zxvf sqlite-autoconf-3290000.tar.gz
cd sqlite-autoconf-3290000
$./configure --prefix=$HOME/opt/sqlite
$ make && make install
Now this is what you should also remember to do for centos7 know where to look for your python3 and not defaulting to python2. On your .bash_profile copy & past this piece of code or edit the paths accordingly:
export PATH=$HOME/opt/sqlite/bin:$PATH
export LD_LIBRARY_PATH=$HOME/opt/sqlite/lib
export LD_RUN_PATH=$HOME/opt/sqlite/lib
Make it permanent by running: source .bash_profile
and you are done with sqlite3 version >= 3.8. Confirm it by:
sqlite3 --version
3.29.0 2019-07-10 17:32:03
And then you can continue to use python3 to install python3 modules like django-2.2.
python3.7 -m pip3 install virtualenv
(myvenv37)[me#test my_project]$ python3.7 -m pip3 install django
Successfully installed django-2.2.3 pytz-2019.1 sqlparse-0.3.0
Remember, it is
PYTHON3.7 -m pip3 install MODULE
(myvenv37)[me#test my_project]$ python3.7 manage.py runserver
and the server should be running.
So, to conclude, in the case above it was migrate, & should look like this:
(venv)[me#test my_project]$ python3.7 manage.py migrate
As this was about Centos7, you can use the Fedora package to upgrade the Centos sqlite package:
wget https://kojipkgs.fedoraproject.org//packages/sqlite/3.8.11/1.fc21/x86_64/sqlite-3.8.11-1.fc21.x86_64.rpm
sudo yum install sqlite-3.8.11-1.fc21.x86_64.rpm
(from: https://www.reddit.com/r/linuxadmin/comments/c9hy5w/trying_to_upgrade_sqlite_3717_to_version_38_on/ezrtbkm/?utm_source=reddit&utm_medium=web2x&context=3)
This seems to work, although I'm never sure if doing this is really an ideal solution to a problem or not. I guess if you're not actually using SQLite, then this at least passes the version check and so gets you working.
django 2.2 need sqlite version >= 3.8.3
so the solution is update your sqlite:
download from sqlite3, select source_code version
tar -zxvf sqlite-xxx.tar.gz && cd xx
./configure && make && make install
mv /usr/bin/sqlite3 /usr/bin/sqlite3.bak
mv xxx/sqlite3 /usr/bin/sqlite3
export LD_LIBRARY_PATH="/usr/local/lib" and write it into ~/.bashrc
test1 :
sqlite3 --version
should be your version
test2:
$python
>>> import sqlite3
>>> sqlite3.sqlite_version
should be your version
To check which version of SQLite Python is using:
$ python
Python 3.7.3 (default, Apr 12 2019, 16:23:13)
>>> import sqlite3
>>> sqlite3.sqlite_version
'3.27.2'
For me the new version of sqlite3 is in /usr/local/bin so I had to recompile Python, telling it to look there:
sudo LD_RUN_PATH=/usr/local/lib ./configure --enable-optimizations
sudo LD_RUN_PATH=/usr/local/lib make altinstall
I had the same issue and I struggled with it for a while. For me the best solution was to comment out DATABASES section in settings.py file.
As I don't want to use SQLite database then issue does not exist anymore. Later on you can update DATABASE information with the db that is valid for you.
I was having trouble with the above solutions on an Amazon Linux 2 instance and found that what was installed from my various attempts to update were causing version conflicts.
Initially I had this:
$ python3 --version
Python 3.7.9
$ python3
>>> import sqlite3
>>> sqlite3.sqlite_version
'3.7.17'
>>> exit()
And sudo yum install sqlite-3.8.11-1.fc21.x86_64.rpm failed to install with this message:
I resolved it by the following steps:
sudo yum list | grep sqlite
These two packages were preventing updates due to conflict.
sudo yum autoremove sqlite-devel To uninstall the extraneous package.
sudo yum install sqlite-3.8.11-1.fc21.x86_64.rpm To update the version.
sudo yum list | grep sqlite
New list of updated sqlite packages:
$ python3
>>> import sqlite3
>>> sqlite3.sqlite_version
'3.8.11'
>>> exit()
You will only get make it worse with errors like multilib and other incompatibilities if keeps forcing the installation of sqlite 3.8.3 or higher. Just as Theo commented, the best approach is to comment the line #66 on the sqlite3/base.py (file within django) that triggers the method named check_sqlite_version().
This should be just warning that you are using a minor patch instead of raise an Exception (ImproperlyConfigured).
PD: Just make sure to document it!
After few hours of searching I solved it using one command inside my virtual environment:
pip install pysqlite3-binary
You can check github here and article(a bit outdated) here.
I also found here that you should then place the following lines of code in your settings.py:
__import__('pysqlite3')
import sys
sys.modules['sqlite3'] = sys.modules.pop('pysqlite3')
but for me it worked without it.
another option is to use atomic repo
wget -O - http://updates.atomicorp.com/installers/atomic |sh
yum install atomic-sqlite
LD_LIBRARY_PATH='/opt/atomicorp/atomic/root/usr/lib64/' python3
>>> import sqlite3
>>> sqlite3.sqlite_version
'3.8.5'
I answered this question here: ImproperlyConfigured('SQLite 3.8.3 or later is required (found %s).' % Database.sqlite_version)
I solved this issue by upgrading my version of sqlite3 using this command:
cd ~ && wget https://www.sqlite.org/2020/sqlite-autoconf-3320100.tar.gz && tar xvfz sqlite-autoconf-3320100.tar.gz && cd sqlite-autoconf-3320100 && ./configure && make && make install
I am using ElasticBeanstalk for my setup, so I added a .config file to the .ebextensions folder and put this in it:
option_settings:
aws:elasticbeanstalk:application:environment:
LD_LIBRARY_PATH: "/usr/local/lib"
commands:
01_upgrade_sqlite:
command: "cd ~ && wget https://www.sqlite.org/2020/sqlite-autoconf-3320100.tar.gz && tar xvfz sqlite-autoconf-3320100.tar.gz && cd sqlite-autoconf-3320100 && ./configure && make && make install"
Many thanks to Bejür for adding the LD_LIBRARY_PATH environment variable here in order to get it to work.
I make a fixed downgrade Django
pip3 install Django==2.1.*
i had recently same problem
My solution was to change source code site-packages\django\db\backends\sqlite3\base.py line around 68 So far no side effects.
def check_sqlite_version():
if Database.sqlite_version_info < (3, 7, 3):
raise ImproperlyConfigured('SQLite 3.8.3 or later is required (found %s).' % Database.sqlite_version)
What's the best way to get the latest SHOGUN / Python modular interface (http://www.shogun-toolbox.org) installed on 12.04? I tried from source without much luck (happy to post errors); Is it possible to install the Trusty Tahr package on 12.04? https://launchpad.net/ubuntu/+source/shogun/3.1.1-1
(I am the debian maintainer of this package).
You could try to
apt-get -b source shogun
but it will give you only the core libshogun library at this very moment. There is a python package on the way (in debian) but not yet accepted and not yet in ubuntu.
https://ftp-master.debian.org/new/python-shogun_3.1.1-1.html
So you are really best of installing from source. Shogun has buildbots running on ubuntu creating a python package. So you can just copy the settings from there:
https://travis-ci.org/shogun-toolbox/shogun/jobs/18605663
Following this posts here and this instruction i did the following that worked for me to install shogun directly into my conda env. Its not the most elegant way but worked out so far.
! Take care to have swig: ie. installed apt-get install swig
! I used anaconda and the shogun python infterface - so my cmake prefix was /home/user/anaconda/
You need cmake to build shogun:
SET UP CMAKE FIRST (skip if you have cmake > 3.1)
cd /path to your installation directory
workdir=$(pwd) #i.e. your home
Download and install cmake into your home:
wget http://www.cmake.org/files/v3.1/cmake-3.1.3.tar.gz
tar xzf cmake-3.1.3.tar.gz
cd cmake-3.1.3
cmake_dir=$workdir/cmake 5 ./configure --prefix=$cmake_dir
make -j 2
make install
Export cmake to your PATH so you can use it for shogun: export
PATH=$cmake_dir/bin/:$PATH
Download and install shogun into anaconda env
wget ftp://shogun-toolbox.org/shogun/releases/3.1/sources/shogun-3.1.1.tar.bz2
tar jxf shogun-3.1.1.tar.bz2 3 cd shogun-3.1.1/
mkdir build
cd build
cmake -DPythonStatic=ON -DPythonModular=ON -DCMAKE_INSTALL_PREFIX=/home/myusername/anaconda/envs/p27/ ..
make -j2 all #four processes takes a while
make install