Develop Reference

Numpy module not found when working with Azure Functions in VS Code and virtualenv

I'm new to working with azure functions and tried to work out a small example locally, using VS Code with the Azure Functions extension.
Example:
# First party libraries
import logging
# Third party libraries
import numpy as np
from azure.functions import HttpResponse, HttpRequest
def main(req: HttpRequest) -> HttpResponse:
seed = req.params.get('seed')
if not seed:
try:
body = req.get_json()
except ValueError:
pass
else:
seed = body.get('seed')
if seed:
np.random.seed(seed=int(seed))
r_int = np.random.randint(0, 100)
logging.info(r_int)
return HttpResponse(
"Random Number: " f"{str(r_int)}", status_code=200
)
else:
return HttpResponse(
"Insert seed to generate a number",
status_code=200
)
When numpy is installed globally this code works fine. If I install it only in the virtual environment, however, I get the following error:
*Worker failed to function id 1739ddcd-d6ad-421d-9470-327681ca1e69.
[15-Jul-20 1:31:39 PM] Result: Failure
Exception: ModuleNotFoundError: No module named 'numpy'. Troubleshooting Guide: https://aka.ms/functions-modulenotfound*
I checked multiple times that numpy is installed in the virtual environment, and the environment is also specified in the .vscode/settings.json file.
pip freeze of the virtualenv "worker_venv":
$ pip freeze
azure-functions==1.3.0
flake8==3.8.3
importlib-metadata==1.7.0
mccabe==0.6.1
numpy==1.19.0
pycodestyle==2.6.0
pyflakes==2.2.0
zipp==3.1.0
.vscode/settings.json file:
{
"azureFunctions.deploySubpath": ".",
"azureFunctions.scmDoBuildDuringDeployment": true,
"azureFunctions.pythonVenv": "worker_venv",
"azureFunctions.projectLanguage": "Python",
"azureFunctions.projectRuntime": "~2",
"debug.internalConsoleOptions": "neverOpen"
}
I tried to find something in the documentation, but found nothing specific regarding the virtual environment. I don't know if I'm missing something?
EDIT: I'm on a Windows 10 machine btw
EDIT: I included the folder structure of my project in the image below
EDIT: Added the content of the virtual environment Lib folder in the image below
EDIT: Added a screenshot of the terminal using the pip install numpy command below
EDIT: Created a new project with a new virtual env and reinstalled numpy, screenshot below, problem still persists.
EDIT: Added the launch.json code below
{
"version": "0.2.0",
"configurations": [
{
"name": "Attach to Python Functions",
"type": "python",
"request": "attach",
"port": 9091,
"preLaunchTask": "func: host start"
}
]
}
SOLVED
So the problem was neither with python, nor with VS Code. The problem was that the execution policy on my machine (new laptop) was set to restricted and therefore the .venv\Scripts\Activate.ps1 script could not be run.
To resolve this problem, just open powershell with admin rights and and run set-executionpolicy remotesigned. Restart VS Code and all should work fine
I didn't saw the error, due to the many logging in the terminal that happens
when you start azure. I'll mark the answer of #HuryShen as correct, because the comments got me to the solution. Thank all of you guys

For this problem, I'm not clear if you met the error when run it locally or on azure cloud. So provide both suggestions for these two situation.
1. If the error shows when you run the function on azure, you may not have installed the modules success. When deploy the function from local to azure, you need to add the module to requirements.txt(as Anatoli mentioned in comment). You can generate the requirements.txt automatically by the command below:
pip freeze > requirements.txt
After that, we can find the numpy==1.19.0 exist in requirements.txt.
Now, deploy the function from local to azure by the command below, it will install the modules success on azure and work fine on azure.
func azure functionapp publish <your function app name> --build remote
2. If the error shows when you run the function locally. Since you provided the modules installed in worker_venv, it seems you have installed numpy module success. I also test it in my side locally, install numpy and it works fine. So I think you can check if your virtual environment(worker_venv) exist in the correct location. Below is my function structure in local VS code, please check if your virtual environment locates in the same location with mine.
-----Update------
Run the command to to set execution policy and then activate the virtual environment:
set-executionpolicy remotesigned
.venv\Scripts\Activate.ps1

I could solve my issue uninstalling python3 (see here for a guide https://stackoverflow.com/a/60318668/11986067).
After starting the app functions via F5 or func start, the following output was shown:
This version was incorrect. I have chosen python 3.7.0 when creating the project in the Azure extension. After deleting this python3 version, the correct version was shown and the Import issue was solved:

Anaconda prompt: ModuleNotFoundError when running Python script

I am trying to run the following script "test.py" on Anaconda Prompt:
from tensorflow.keras.applications.resnet50 import ResNet50
...with the following command:
(conda_env) C:\dev>test.py
This results in the following error:
ModulNotFoundError: No module named 'tensorflow'
When I run the same script on Anaconda Prompt with the following command I don't get any errors:
(conda_env) C:\dev>python test.py
I have installed tensorflow in the Anaconda environment 'conda_env'
(conda_env) C:\dev\>conda env list
# conda environments:
#
base C:\Users\xx\Anaconda3
conda_env * C:\Users\xx\Anaconda3\envs\conda_env
keras_1 C:\Users\xx\Anaconda3\envs\keras_1
tf-gpu C:\Users\xx\Anaconda3\envs\tf-gpu
Why is this like that?

You won’t get errors if you do
(conda_env) C:\dev> python test.py
because then you’re following the correct syntax for running python scripts in a terminal. By adding python before the .py file, you initiate the Python interpreter that executes your script. Without it, the terminal won’t know what Python interpreter to use to execute your script and may end up using an interpreter that doesn't have the modules you require. There are ways to skip writing python before executing if that’s what you want.
For example, see: Calling a python script from command line without typing "python" first

Run .py script via sh import module error

This is a very basic question on how to code in python and run your script from a very beginner.
I'm writing a script using Xcode9.4.1 which is supposed to be for python3.6. I then have an sh script run.sh, in the same folder of the script (say "my_folder") which simply looks like
python my_script.py
The python script looks like
from tick.base import TimeFunction
import numpy as np
import matplotlib.pyplot as plt
v = np.arange(0., 10., 1.)
f_v = v + 1
u = TimeFunction((v, f_v))
plt.plot(v, u.value(v))
print('donne!\n')
But as I try to run my_script.sh from the terminal I get a "ImportError: No module named tick.base" error.
But the tick folder is actually present in "my_computer/anaconda3/lib/python3.6/site-packages" and up to last week I was using Spyder from anaconda navigator and everything was correctly working, so no "import error" occurred.
The question is quite trivial, in some sense it simply is "what's the typical procedure to code and run python script and how modules are supposed to be imported-downloaded when running on a given machine?"
I need it since my script is to be run on another machine through ssh and using my laptop to make some attempts. Up to last year I used to work in C and only need to move some folders with code and .h files.
Thank for help!
EDIT 1:
From the Spyder 3.2.7 setting, where the script was giving non problem, I printed the
import sys
print(sys.path)
The -manually- copied the content to the sys.path variable in my_script.py and rerun 'run.sh' and now getting a new (strange) error:
Traceback (most recent call last):
[...]
File "/Users/my_computer/anaconda3/lib/python3.6/site-packages/tick/array/build/array.py", line 106
def tick_double_array_to_file(_file: 'std::string', array: 'ArrayDouble const &') -> "void":
^
SyntaxError: invalid syntax

First, check the python which you are calling the script with is pointing to the anaconda python and it is of the same version you are expecting it to be. You can do "which python" command in Linux and Mac to which the path which points to python. It if is pointing to some different version or build of python than the one which you are expecting then add the needed path to the system environment PATH variable. In Linux and Mac this can be done by adding the following line in the .bashrc file at the /home/ folder:
export PATH=/your/python/path:$PATH
And then source the .bashrc file.
source .bashrc
If you are on a operating system like cent os ,breaking the default python path can break your yum so be careful before changing it.

I am running a script in PyCharm and under the Project Interpretor I have the path
C:\envs\conda\keras2\python.exe
When I try to run the script via ssh on the server I get a 'no module named' error. I get
/usr/bin/python as the ans to 'which python' on the server itself. Could you tell me which path I must add for the script to run properly?

Any pyinstaller detailed example about hidden import for psutil?

I want to compile my python code to binary by using pyinstaller, but the hidden import block me. For example, the following code import psutil and print the CPU count:
# example.py
import psutil
print psutil.cpu_count()
And I compile the code:
$ pyinstaller -F example.py --hidden-import=psutil
When I run the output under dist:
ImportError: cannot import name _psutil_linux
Then I tried:
$ pyinstaller -F example.py --hidden-import=_psutil_linux
Still the same error. I have read the pyinstall manual, but I still don't know how to use the hidden import. Is there a detailed example for this? Or at least a example to compile and run my example.py?
ENVs:
OS: Ubuntu 14.04
Python: 2.7.6
pyinstaller: 2.1

Hi hope you're still looking for an answer. Here is how I solved it:
add a file called hook-psutil.py
from PyInstaller.hooks.hookutils import (collect_data_files, collect_submodules)
datas = [('./venv/lib/python2.7/site-packages/psutil/_psutil_linux.so', 'psutil'),
('./venv/lib/python2.7/site-packages/psutil/_psutil_posix.so', 'psutil')]
hiddenimports = collect_submodules('psutil')
And then call pyinstaller --additional-hooks-dir=(the dir contain the above script) script.py

pyinstall is hard to configure, the cx_freeze maybe better, both support windows (you can download the exe directly) and linux. Provide the example.py, In windows, suppose you have install python in the default path (C:\\Python27):
$ python c:\\Python27\\Scripts\\cxfreeze example.py -s --target-dir some_path
the cxfreeze is a python script, you should run it with python, then the build files are under some_path (with a lot of xxx.pyd and xxx.dll).
In Linux, just run:
$ cxfreeze example.py -s --target-dir some_path
and also output a lot of files(xxx.so) under some_path.
The defect of cx_freeze is it would not wrap all libraries to target dir, this means you have to test your build under different environments. If any library missing, just copy them to target dir. A exception case is, for example, if your build your python under Centos 6, but when running under Centos 7, the missing of libc.so.6 will throw, you should compile your python both under Centos 7 and Centos 6.

What worked for me is as follows:
Install python-psutil: sudo apt-get install python-psutil. If you
have a previous installation of the psutil module from other
method, for example through source or easy_install, remove it first.
Run pyinstaller as you do, without the hidden-import option.

still facing the error
Implementation:
1.python program with modules like platform , os , shutil and psutil
when i run the script directly using python its working fine.
2.if i build a binary using pyinstaller. The binary is build successfully. But if i run the binary iam getting the No module named psutil found.I had tried several methods like adding the hidden import and other things. None is working. I trying it almost 2 to 3 days.
Error:
ModuleNotFoundError: No module named 'psutil'
Command used for the creating binary
pyinstaller --hidden-import=['_psutil_linux'] --onefile --clean serverHW.py
i tried --additional-hooks-dir= also not working. When i run the binary im getting module not found error.

Loading matplotlib on cygwin

To load matplotlib on cygwin, I have:
Loaded pre-requisites using cygwin 64-bit setup: pkg-config, freetype2, libfreetype-devel, libpng-devel, gtk2.0, libgtk2.0-devel
Downloaded the matplotlib tar file (http://sourceforge.net/projects/matplotlib/files/matplotlib/matplotlib-1.3.1/matplotlib-1.3.1.tar.gz) and changed the source code to get around the "_tri" error as advised here:
matplotlib error while installing pyspeckit
Then built and installed matplotlib:
$ python setup.py build
$ python setup.py install
And am down to what looks like a matplotlib backend error. Does anyone know how to get around this:
/usr/lib/python2.7/site-packages/gtk-2.0/gtk/init.py:57: GtkWarning: could not open display
...
cursors.MOVE : gdk.Cursor(gdk.FLEUR),
et n RuntimeError: could not create GdkCursor object
EDIT: I just finally got it and matplotlib is finally working on cygwin. To do this, I did:
From cygwin setup, loaded the X-Server tools:
xorg-server xinit
From cygwin setup, I also loaded these so that use telnet or ssh connections to run remote X clients:
inetutils openssh
I set my display:
DISPLAY=":0.0"
export DISPLAY
From the cygwin shell, I did:
$ startxwin
Then I ran my python scripts which use matplotlib in the X-window

I had an issue with "python setup.py build" not finding the ft2build.h which comes from the the freetype2 package. I installed the freetype2 development package and I can find it in /usr/include/freetype2/ft2build.h but the error is still there.
After digging into the setupext.py at the function check_include_file(include_dirs, filename, package), I noticed that the package="freetype2" was not concatenated into to the search path. Hence it could not find "/usr/include/ft2build.h" when it should be "/usr/include/freetype2/ft2build.h
Fixing this line #134
if not has_include_file(include_dirs, "%s/%s"%(package,filename)):