Develop Reference

ImportError while using AWS Lambda in Python (via zappa serverless); Numpy

I have an ImportError issue with my AWS Lambda function
I can run this code locally without ImportError
I use zappa serverless for uploading my code to AWS Lambda--which requires a virtual environment and is an end-to-end solution for creating and updating Lambda functions
I previously had this code working on AWS Lambda
I'm not sure of the change I made (#3) that made this code go from working to ImportError. The log output is as follows.
[ERROR] Runtime.ImportModuleError: Unable to import module 'dailycore': Unable to import required dependencies:
numpy:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.
We have compiled some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html
Please note and check the following:
* The Python version is: Python3.8 from "/var/lang/bin/python3.8"
* The NumPy version is: "1.23.3"
and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.
Original error was: No module named 'numpy.core._multiarray_umath'
I'm unclear if this is a Numpy issue, a Python issue, or a Zappa issue. Two changes I made that may have had an impact:
I separated out some of my personal code, uploaded it to github, and installed it as packages within the virtual environment using pip
I uninstalled Python from my machine and reinstalled it--as I had two versions causing conflict. Upon my reinstallation (Python 3.8), I changed the installation directory from its default.
The code runs from VS Code locally and wholly contained from within the virtual environment (i.e. I don't have any packages installed via pip on my machine outside of virtual environments at this point). The suggestions above may or may apply to this error. I've tried adding a system PATH variable to my Python 3.8 directory. I've tried downgrading numpy to 1.23.0. I've tried uninstalling all dependencies within the virtual environment and reinstalling them.
UPDATE: I found this link and am looking into these comments: https://github.com/Miserlou/Zappa/issues/1222
UPDATE2: I deleted the virtual environment and re-created per a suggestion at the link above. This did not work.
UPDATE3: I think my issue relates to the private package dependencies I installed via pip from my personal github repos (#2-2). This link may be related. Something about dependencies not being at the top level. I am unsure how to fix this issue and/or repackage my private repo or install correctly. Any clarification is appreciated.
UPDATE4: I've isolated my issue to the private package dependencies and how they get called. A the solution may be related to Lambda Layers, tutorial here. Looking into this.

The ImportModuleError is related to this python package: alpha_vantage and not either of my initial theories. I am unsure as to why, though. It seems like it has something to do with Alpha Vantage's setup.py file; although I could be mistaken.
I have removed the reference to Alpha Vantage for now, but if anyone knows why installing this package would throw a numpy error, please elaborate. I think the only reference to numpy would be through the pandas package--which is only an optional install (i.e. extras_requires in setup.py)

'reportMissingModuleSource' and 'reportMissingImports' errors for imported modules pandas and alpha_vantage

I'm very new to python and trying one of my first projects. I have installed modules 'pandas' and 'alpha_vantage' and imported them at the top of my code but they show the errors 'reportMissingModuleSource' for pandas and 'reportMissingImports' for alpha_vantage. I installed them in Command prompt and it was successful but it seems python doesn't recognise them or cant find them.
Potentially useful info:
When I first installed python I had to go into my Path settings and create a new Path to python.
I use VSCode
I had issues installing pip I cant remember why because I've been at this for hours.
Python version is 3.10.5 64bit
Pip version is 22.1.2
I am on Windows
I don't know what python files need to go where so they are slightly scattered around my directory.
Do I need to create a Path in my settings to each module, I already have one going to my scripts folder. I am very stuck so any help is appreciated.
Thank you for anyone that helps!

How to make downloaded Python libs work on Windows?

I got this simple problem and I can't find the answer anywhere, I'm wasting a lot of time!
I did a Python programm on Linux (which works OK), but when I try to run it on Windows, there are too problems with libs...
I have installed the libs I need (dateutil, lxml, xmlrpclib...) in C:\Python34\Lib\site-packages. But then, they don't work as they do on Linux. For example:
from dateutil.tz import tzlocal
Gives me next error:
File "C:\Python34\lib\site-packages\dateutil\tz.py", line 9, in module
from six import string_types, PY3 ImportError: No module named 'six'
That is, they are not finding the other modules... why???

Have you try this ?
http://www.instructables.com/id/How-to-install-Python-packages-on-Windows-7/
Maybe it can help

It looks like you're using Python 3.4 which comes with pip. pip is a tool for installing packages and any dependencies they might have (like the srting_types module from your error message). I'd suggest learning how to use it because it resolves most of the packaging problems with you needing to moving things around yourself. See an answer from a different question to learn more about pip.
There are some packages that need to be compiled. This can be difficult on Windows 7 if you don't have the proper toolchain set up to compile packages. I'd recommend Christoph Gohlke's wonderful collection of installable packages for Windows. You just need to make sure to grab the right version. Since 3.4 is still relatively new, some packages may not be available, so be warned.

ImportError: Permission Denied while using LXML

I've been having a ton of trouble using LXML, after installing it from https://pypi.python.org/pypi/lxml/3.2.1 using Easy_Install-2.7. I installed it on Windows using cygwin, and at first the package seemed to be okay. However upon further testing I ran into problems.
When I run code with:
import lxml
it works completely fine. But as soon as I try:
import lxml.etree
I get this error:
Traceback (most recent call last):
File "D:\Nick_Code\NewsScraper\testdummy.py", line 7, in <module>
import lxml.etree
File "/usr/lib/python2.7/site-packages/lxml-3.2.0-py2.7-cygwin-1.7.20-i686.egg/lxml/etree.py", line 7, in <module>
__bootstrap__()
File "/usr/lib/python2.7/site-packages/lxml-3.2.0-py2.7-cygwin-1.7.20-i686.egg/lxml/etree.py", line 6, in __bootstrap__
imp.load_dynamic(__name__,__file__)
ImportError: Permission denied
I've been trying to find information/work arounds for quite a while but no success. Please let me know if you have any insight or need information.
Thanks!
Michael

This is not a solid answer. But I will highlight several of the problems involved for obtaining a solution. Most likely the problem above, is like a cancer caused by several factors acting catastrophically together.
I have the same exact problem as in the OP, when attempting to use the native Cygwin supplied Python packages on my Windows Vista machine. Being new to Python I have spent several days in trying to get this to work, and understand why it is not working. But all my Google-fu returned nothing but countless dead ends. So here's my take on this.
There are many reasons why Python could have trouble under Cygwin, some which you can do something about and some which are beyond most peoples control. What it boils down to, are the following key issues:
Windows is a complete mess when it comes to file permissions, and Cygwin cannot handle windows file permissions very well. So what you see in Cygwin is far from the whole story.
Windows is shamefully character case-independent which causes loads of trouble, especially when you need to (cross)compile anything that was originally developed under *nix based system (i.e. everything). In fact, if you attempt extracting any archive that contains files whose names differ only in capitalization. (I.e. "makefile" vs "Makefile" etc.) files under Windows or Cygwin, you loose all but one of the files. in case their You need to enable case-sensitivity to do anything more than "hello world" *nix compilations.
Windows handles symlinks completely different than Cygwin. And if your ZIP, TAR etc. archives contain any symlinks, they will be broken after extraction to Windows environment.
Sloppy code practices, where developer have not properly tested their creations on various environments, or carefully set proper file permissions to their *.tar.gz collections. Including correct dependency specifications, or mentioning whether or not binaries has been statically linked etc.
For the full gory details and further (Win-Cygwin) issues, look HERE.
At first I tried to use Cygwin's own Python without any additional packages, and nstalling lxml using PIP and easy_install. Then I tried to use Cygwin's own libxml2, libxslt and xml python packages, and I had the same problems.
At first, after installing the static windows binaries (as suggested elsewhere),
I got this error:
File "/usr/lib/python2.7/site-packages/lxml-3.2.4-py2.7-cygwin-1.7.24-i686.egg/lxml/etree.py", line 6, in __bootstrap__
imp.load_dynamic(__name__,__file__)
ImportError: Permission denied
Aborted (core dumped)
Then I investigated the file permissions and changed those with: chmod -R 755 /usr/lib/python2.7/
I got one step further to isolate problem to an apparently missing file.
And enabling verbose and diagnostic mode's didn't help much either.
File "/usr/lib/python2.7/site-packages/lxml-3.2.4-py2.7-cygwin-1.7.24-i686.egg/lxml/etree.py", line 6, in __bootstrap__
imp.load_dynamic(__name__,__file__)
ImportError: No such file or directory
Aborted (core dumped)
HERE is the exact statement specification:
Load and initialize a module implemented as a dynamically loadable shared
library and return its module object. If the module was already initialized, it
will be initialized again. Re-initialization involves copying the __dict__
attribute of the cached instance of the module over the value used in the module
cached in sys.modules. The pathname argument must point to the shared library.
The name argument is used to construct the name of the initialization function:
an external C function called initname() in the shared library is called. The
optional file argument is ignored. (Note: using shared libraries is highly
system dependent, and not all systems support it.)
So I started reading on the lxml website which clearly state lxml's dependencies on both libxml2 and libxslt, and unless they are statically linked, they also depend on iconv and zlib. So you're lead to believe you need to install all of these. Don't! Continue reading. But if you're going to build from sources (as easy_install may try to do) you'll need everything, including the development header libraries: libxml2-devel, libxslt-devel. Another place states that you also need Cython and install with:
easy_install lxml==dev
The dependencies are shown in this picture from HERE:
So you think you may get away with something like:
STATIC_DEPS=true pip install lxml
But that doesn't do it either. Probably because the libraries used to compile Cygwin's Python have to be the same as those for compiling lxml. But I don't know. Notice how the lxml package refers to Cygwin "1.7.24". My Cygwin is already "1.7.25" and you can check this with uname -a. Then you can check your static python executable with file and ldd. Then you understand that this also depend on the C-compiler used for building python/cygwin under Windows or *nix. Smelling a nightmare I decided that building my own was not the way to go. So next I tried to install the Python libraries (supplied as
executables) meant for Windows Python. This didn't work since I never had windows native Python installed, and I was greeted with an error that the installed could not find Python in my registry. I could of course just extract the executable, but I wouldn't know where to put the binaries without the installer. So I had another idea...
There are 3 possible solutions to getting this to work, as far as I can see.
The easy way of installing a Windows native Python interpreter. You loose some native Cygwin functionality, unless you install in correct place: /usr/lib/python2.7 and make sure Cygwin can find it and use it. This also uses a different file-permissions, case-sensitivity and character set (UTF-16LE) than Cygwin (UTF-8), potentially creating many other issues down the line! Difficulty: Easy
Continue hacking the Cygwin's Python, to make it work with the binary libraries used in (1). But this requires:
a) Uninstall and remove all Cygwin Python packages, except bare Python interpreter.
b) Remove all PIP and easy install traces.
c) Hacking the Windows registry to pretend to have Python27 installed:
HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.7\InstallPath C:\Python27\
HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.7\PythonPath C:\Python27\Lib;C:\Python27\DLLs;C:\Python27\Lib\lib-tk
HKEY_CLASSES_ROOT...
d) Install the Windows binary libraries.
e) All the rest should now hopefully work with PIP or easy_install. Difficulty: Medium!
Doing it properly by compiling Python and all libraries from scratch. Difficulty: Hard!
I successfully did (1), but I still think (2) is the smarter way of doing it, but I have not tested it, which is why I don't consider this as a good answer. BTW. One more quirk, I have to run the interpreter with: python.exe -E to avoid an annoying: "SyntaxError: invalid syntax" when hitting return!
Conclusion:
Apparently, you don't need the libxml2 and libxslt python packages to use lxml!
In my case I needed Scrapy, so I also had to install a few other packages.
$ pip.exe list
cssselect (0.9.1)
lxml (3.2.4)
pip (1.4.1)
pyOpenSSL (0.11)
pywin32 (218)
queuelib (1.1.1)
Scrapy (0.20.0)
setuptools (1.4.1)
six (1.4.1)
Twisted (13.2.0)
w3lib (1.5)
zope.interface (4.0.5)
$ll /cygdrive/c/Python27/Lib/site-packages/
adodbapi
cssselect
isapi
lxml
OpenSSL
pip
pythonwin
pywin32_system32
queuelib
scrapy
twisted
w3lib
win32
win32com
win32comext
zope
cssselect-0.9.1-py2.7.egg-info
lxml-3.2.4-py2.7.egg-info
pip-1.4.1-py2.7.egg-info
queuelib-1.1.1-py2.7.egg-info
Scrapy-0.20.0-py2.7.egg-info
six-1.4.1-py2.7.egg-info
Twisted-13.2.0-py2.7.egg-info
w3lib-1.5-py2.7.egg-info
zope.interface-4.0.5-py2.7.egg-info
PyWin32.chm
setuptools-1.4.1-py2.7.egg
pyOpenSSL-0.11-py2.7.egg-info
pywin32-218-py2.7.egg-info
easy-install.pth
pywin32.pth
setuptools.pth
zope.interface-4.0.5-py2.7-nspkg.pth
pythoncom.py
six.py
pythoncom.pyc
six.pyc
pythoncom.pyo
pywin32.version.txt
README.txt
Useful References:
HERE
HERE
HERE HERE HERE HERE
HERE

How do you correct Module already loaded UserWarnings in Python?

Getting the following kinds of warnings when running most python scripts in the command line:
/Library/Python/2.6/site-packages/virtualenvwrapper/hook_loader.py:16: UserWarning: Module
pkg_resources was already imported from /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.pyc, but /Library/Python/2.6/site-packages is being added to sys.path
import pkg_resources
/Library/Python/2.6/site-packages/virtualenvwrapper/hook_loader.py:16: UserWarning: Module site was already imported from /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site.pyc, but /Library/Python/2.6/site-packages is being added to sys.path
import pkg_resources
I think it has to do with a combination of using distribute and virtualenv, but wanted to check if anyone else has run in to this or would know how to go about fixing it.

Perhaps use the virtualenv option --no-site-packages so you won't see any system site-packages within your virtual environment. Having items installed both in your virtualenv and on the system root may be the cause of this issue.
Using --no-site-packages when creating your virtualenv prevents any conflict between system packages. I almost always use that option when creating a new virtualenv to prevent any conflicts. Though I may have several copies of libraries, at least they don't mess with each other.

The python equivalent of putting a bit of electrical tape over the check engine light would be to use the -W command line flag or by adding a warning filter.

In my case reinstalling of anything did not help. There were some orphaned .pyc files (specifically pkg_resources.pyc) left in /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python
sudo find . -type f -name "*.pyc" -delete
made it work. This link helped me to track down the problem.

I had this sort of Python packaging hell visit today too.
Running Python 2.7.3 on Ubuntu, using namespace packages and using zc.buildout.
Finally, updating system wide Distribute from older version 0.6.30 to latest version 0.6.35 resolved the problem.

If the warning shows in a program you are modifying, try it this way (examply with pytz):
try:
import pytz
except ImportError:
from pkg_resources import require
require('pytz')