I am facing a strange problem which I could track down to the python logging package. Let me shortly explain what I want to do: the goal is to create html reports with python. I am using the R rmarkdown package which runs python code trough reticulate using a local virtualenv.
The Problem:
As soon as I install the python logging package rmarkdown runs into a problem when loading matplotlib. I have written a small test script to reproduce the example.
My system:
Ubuntu 18.04 bionic
Python 2.7.15rc1
Test "1" script (without logging):
Create a new virtualenv (venv).
Use venv/bin/pip to install matplotlib.
Run reticulate::import (at the end via rmarkdown::render).
Test "2" script (with logging):
Create a new virtualenv (venv).
In addition to the first test: install logging via venv/bin/pip.
Use venv/bin/pip to install matplotlib.
Run reticulate::import (at the end via rmarkdown::render).
The modules installed (virtualenv):
backports.functools-lru-cache 1.5
cycler 0.10.0
kiwisolver 1.0.1
logging 0.4.9.6 <- only for "test 2"
matplotlib 2.2.3
numpy 1.15.1
pip 18.0
pkg-resources 0.0.0
pyparsing 2.2.0
python-dateutil 2.7.3
pytz 2018.5
setuptools 40.2.0
six 1.11.0
subprocess32 3.5.2
wheel 0.31.1
The system site packages do have the same module version.
Results:
All tests from test 1 (without logging) work nicely.
The tests from test 2 (with loging) fail when using the virtualenv. When calling rmarkdown::render (see below), when using the system python installation (not virtualenv) they work nice as well.
There seem to be something strange with reticulate when logging is installed in a virtualenenv.
The output of the test script (see below):
The full output including the error:
----------- no logging package installed ------------
Module(matplotlib)
Module(matplotlib)
--------- with logging package installed ------------
Error in py_module_import(module, convert = convert) :
AttributeError: 'module' object has no attribute 'NullHandler'
Detailed traceback:
File "/home/retos/Downloads/venvtest/venv/lib/python2.7/site-packages/matplotlib/__init__.py", line 168, in <module>
_log.addHandler(logging.NullHandler())
Calls: <Anonymous> -> py_module_import -> .Call
Execution halted
Module(matplotlib)
The Module(matplotlib) output is the success message of loading the module via reticulate::import. As one can see only the one test fails where the virtualenv is used with installed logging python module.
Anyone having an idea what could case these problems? I spent quite some time to identify the source of the error, but I am kind of lost now ...
Test script to reproduce the error:
Here is a small bash/shell script to reproduce my tests.
#!/bin/bash
# New virtual environment and install matplotlib
echo " ----------- no logging package installed ------------"
if [ -d venv ] ; then rm -rf venv ; fi
virtualenv venv &>/dev/null > /dev/null
venv/bin/pip install matplotlib > /dev/null
# Print installed packages
Rscript -e "reticulate::use_python('venv/bin/python'); reticulate::import('matplotlib')"
Rscript -e "reticulate::import('matplotlib')"
# New virtual environment and install logging and matplotlib
echo " --------- with logging package installed ------------"
if [ -d venv ] ; then rm -rf venv ; fi
virtualenv venv > /dev/null
venv/bin/pip install logging > /dev/null
venv/bin/pip install matplotlib > /dev/null
# Print installed packages
Rscript -e "reticulate::use_python('venv/bin/python'); reticulate::import('matplotlib')"
Rscript -e "reticulate::import('matplotlib')"
I first thought it is related to the problem "ImportError: cannot import name cbook" but the solution there did not work.
May thanks in advance!
R
Logging became a standard module included in Python library in version 2.3. You must not install it from PyPI. Remove it ASAP:
pip uninstall logging
Related
I have created a package via Github's pip instance, it is successfully installed:
Running setup.py install for lsm-shared ... done
Successfully installed lsm-shared-0.1.0
% pip list | grep shared
lsm-shared 0.1.0
If I start python and manually import it then it works fine:
% python
Python 3.9.6 (default, Jul 5 2021, 01:04:38)
>>> import lsm_shared
If I try and use the package in a project I get:
% quart run -p 3030
File "Development/lsm/servie/service/application.py", line 8, in <module>
import lsm_shared
ModuleNotFoundError: No module named 'lsm_shared'
The only difference between running manually and in a project is that I set PYTHONPATH, I don't know if that makes any difference.
I have looked at other solutions, including this StackOverflow question, and done the pip install with --user to no avail.
ansible command always print warning below:
/usr/local/lib/python2.7/dist-packages/pkg_resources/py2_warn.py:22: UserWarning: Setuptools will stop working on Python 2
************************************************************
You are running Setuptools on Python 2, which is no longer
supported and
>>> SETUPTOOLS WILL STOP WORKING <<<
in a subsequent release (no sooner than 2020-04-20).
Please ensure you are installing
Setuptools using pip 9.x or later or pin to `setuptools<45`
in your environment.
If you have done those things and are still encountering
this message, please comment in
https://github.com/pypa/setuptools/issues/1458
about the steps that led to this unsupported combination.
************************************************************
sys.version_info < (3,) and warnings.warn(pre + "*" * 60 + msg + "*" * 60)
system info: ubuntu 16.04
install method: apt-get install ansible
How can I get rid of this warnings?
TIPS: ansible seems to use /usr/bin/python which is python2.7 default in ubuntu environment.
Quick and dirty way:
edit /usr/bin/ansible
add code:
import warnings
warnings.filterwarnings("ignore")
But is there a better way to make ansible use python3 environment?
Better way
use pyenv or other tools to install python3
make sure run /usr/bin/env python runs python3
edit /usr/bin/ansible
#!/usr/bin/python -> #!/usr/bin/env python
use pip install ansble to install missing package.
otherwise ansible module not found error will show.
I'm getting this error:
ImportError: cannot import name 'SUTime' from partially initialized module 'sutime' (most likely due to a circular import)
when importing the sutime module as:
from sutime import SUTime
as suggested in the sutime GitHub example: https://github.com/FraBle/python-sutime
Context: sutime is a Python library for parsing date/time from a natural language input, developed by the amazing team over at Stanford CoreNLP.
Note: I've already run the pre-req installs as well:
>> pip install setuptools_scm jpype1 # install pre-reqs
>> pip install sutime
>> # use package pom.xml to install all Java dependencies via Maven into ./jars
>> mvn dependency:copy-dependencies -DoutputDirectory=./jars
It's not true that a circular import is the most likely cause of your error. A failed, incomplete, or in some way strange installation is more likely. Try this:
pip uninstall sutime
pip uninstall jpype1
pip uninstall setuptools_scm
pip3 install setuptools_scm jpype1 # note: pip3
pip3 install sutime
Then, in the python-sutime directory, enter this command:
./test.sh
It should output lots of log lines, and the last line but one should be similar to this:
======================== 5 passed, 2 warnings in 13.06s ========================
In the same directory you can enter and run the Python script from the Example section of the README. It should output many log lines before the reported output. There may be a way to avoid that, but anyway it happens only once, when the script starts.
Clone the git repo. -
!git clone https://github.com/FraBle/python-sutime.git
Go to the python-sutime/sutime in the cloned repo. There is a pom.xml file. Open the terminal and issue the following command.
mvn dependency:copy-dependencies -DoutputDirectory=./jars -P english
Now you can simply import the sutime from the sutime.py script in the cloned repo folder.
If you want to use sutime from anywhere, install sutime using...
pip install sutime
and replace the /usr/local/lib/python3.6/dist-packages/sutime folder with the sutime folder you get after step 2.
I'm building MultiNEAT from source, on macOS Mojave, with Python3, and I'm hitting a snag when trying "python setup.py build_ext". I get a bunch of errors. Any help is greatly appreciated.
Here is a step by step of what I'm doing from scratch, if you want to follow along in your own terminal. I'm doing all of this so others can do it themselves too, as it has taken me a while to even get here. I've gotten bits and pieces of information here and there, but there is no straight up instructions on how to build this library.
# Change things accordingly for you.
# Define work dir. Should be empty at this point.
WDIR=/Users/luis/Documents/neat
cd $WDIR
# Setup Python virtual environment and requirements.
python3 -m venv venv
. venv/bin/activate
pip install --upgrade pip
pip install psutil numpy opencv-python
# Get Boost.
# Get the url from here: https://www.boost.org/users/download/
curl -L https://dl.bintray.com/boostorg/release/1.70.0/source/boost_1_70_0.tar.gz | tar -xz
cd boost_1_70_0/
# Get system Python include files with: python3-config --includes
# Put that path into this exported var.
export CPLUS_INCLUDE_PATH=/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m
# Build Boost! This will take a bit.
./bootstrap.sh --prefix=$WDIR/boost
./b2 install
# Get MultiNEAT.
cd $WDIR
git clone https://github.com/peter-ch/MultiNEAT.git
cd MultiNEAT/
# Setup build. (Is this incomplete?)
export MN_BUILD=boost
export PREFIX=$WDIR/boost
# Build MultiNEAT!
python setup.py build_ext
# Supposedly, I'd do "python setup.py install" after, but errors are happening now :(
This is only the top part of when errors start happening, but most are similar:
In file included from src/Innovation.cpp:34:
In file included from src/Innovation.h:37:
src/Genome.h:689:19: error: expected ':'
public void set_children()
^
:
src/Genome.h:691:49: error: indirection requires pointer operand ('std::__1::vector<double,
std::__1::allocator<double> >::size_type' (aka 'unsigned long') invalid)
for(unsigned int ix = 0; ix < 2**coord.size(); ix++){
^~~~~~~~~~~~~
Here is what works:
First install the dependencies you want to use:
Boost 1.49 and above with Boost.Python and Boost.Serialization (optional)
ProgressBar (Python package) (optional)
NumPy (Python package) (optional)
Matplotlib (Python package) (optional)
OpenCV 2.3 and above (with Python bindings) (optional)
Cython (if you want Python bindings)
Then clone this git(this one seems to be older but does not throw those errors you encountered)
And finally enter the source directory and run "sudo python setup.py install"
This worked even with Python 3.7.
I am having trouble installing the PyZMP dependency for iPython. I have tried a number of things such as using pip/brew, but ended up installing the package manually using this answer.
Now, pip list packages yields the following
pyzmq (14.2.0-dev)
pyzmq-static (2.2)
(I have also tried using pyzmq-static).
Also, my python eggs are clearly present in the proper location:
ls /usr/local/lib/python2.7/site-packages/ | grep "pyzmq"
pyzmq-14.2.0_dev-py2.7.egg-info
pyzmq_static-2.2-py2.7.egg-info
Furthermore:
which python
/usr/local/bin/python
echo $PYTHONPATH
/usr/local/lib/python:
However, when I try to import
Python 2.7.6 (default, Mar 20 2014, 17:55:13)
In [1]: import pyzmq
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-1-2faa556e5dc0> in <module>()
----> 1 import pyzmq
ImportError: No module named pyzmq
Any help troubleshooting would be much apprecaited!
Update: I am able to list the zmq modules in python, but not the pyzmq modules:
>>help('modules zmq')
Here is a list of matching modules. Enter any module name to get more help.
buildutils - utilities for building pyzmq.
buildutils.constants - script for generating files that involve repetitive
updates for zmq constants.
buildutils.detect - Detect zmq version
zmq - Python bindings for 0MQ.
...
zmqversion - A simply script to scrape zmq.h for the zeromq version.
IPython.consoleapp - A minimal application base mixin for all ZMQ based
IPython frontends.
IPython.html.base.zmqhandlers - Tornado handlers for WebSocket <-> ZMQ sockets.
...
IPython.utils.zmqrelated - Utilities for checking zmq versions.
You have to use:
import zmq
instead of
import pyzmq
have a look at the official Examples at https://github.com/zeromq/pyzmq/blob/master/examples/
I also struggled getting IPython (Jupyter) installed, especially with the pyzmq dependency. In the end, I found a method that worked for me (in Ubuntu) with much less hassle:
$ sudo apt-get install libtool pkg-config build-essential autoconf automake python-dev
$ sudo apt-get build-dep matplotlib
$ sudo pip install matplotlib
$ sudo pip install "ipython[all]"