I have a Python script which has been created to pull data out of a database and present it in an excel spreadsheet. Because multiple people need this script, I have placed the script on a network drive. To run the script, I have also created a virtual environment in a folder on my workstation, which I then copied to the network location. I have a batch file which runs the script using the virtual environment version of Python.
When I run the batch file on my workstation (from the network drive), everything works fine. When another user, who does not have Python installed on their workstation, runs the batch file, they receive the following error:
ModuleNotFoundError: No module named 'contextlib'
The traceback for the error comes from:
virtualenv\lib\site.py
virtualenv\lib\importlib\util.py
Is there something that I am missing when creating the virtual environment? To create it, I used the following commands (on Windows):
mkdir virtualenv
virtualenv virtualenv
\virtualenv\Scripts\activate
pip install [packages required for script]]
\virtualenv\Scripts\deactivate
Any help is very much appreciated.
Thanks for your help. It turns out that a virtualenv does not do what I thought it did. A virtialenv is not portable to other machines/evironments.
I ended up going with pyinstaller to package the script into an application. This was very easy and quick.
To install pyinstaller and create the app, I simply followed the instructions here:
https://www.pyinstaller.org/
Related
I've put myself in a big ol' pickle. This morning, I unsynced some heavy folders from Box in order to save on power consumption and memory. What I didn't realize is that for some reason years ago I installed python within this synced drive that carries all my research. So a lot of dependencies broke after I unsynced a folder called .local. Since then I've re-synced the folder just to have it working for the day but it has 20,000 files in it and it's taking an eternity to patch things up. Little by little, I'm able to load more libraries, but it's just a bandaid and ultimately what I want to do is move all my python stuff from that synced drive to a local directory.
I'm using WSL. I've tried "uninstalling" and reinstalling python3 using the Windows installer, but whenever I try to run a .py program from the terminal, I get the same errors as before, such as:
cannot import name '_np_version_under1p14' from 'pandas.compat.numpy'
and a little while later (after some files synced)
AttributeError: module 'pandas' has no attribute 'read_csv'
So it appears to still reference the same place that is currently missing some files due to the sync problem. That means it's still referencing the directory within the synced drive. How can I tell the terminal to use the packages installed elsewhere? I see a directory in /mnt/usr/local/lib/python3.6/ that contains the site-packages I think I need. How can I use the terminal command python3 to look for files in this other directory by default?
It seems like you're using the global python interpreter to run your scripts which is usually not the best thing to do.
You will run into dependency issues or things breaking like you described.
Much better way is to create a virtual environment, install all your dependencies like pandas, and run your scripts from there.
Create new virtual environment:
python -m venv .venv
Activate it:
source .venv/bin/activate
Install dependencies:
pip install pandas
And then run your scripts.
If you have a lot dependencies you can use tools like python poetry to manage them. Also you'll get virtual environments helpers and locking for free.
https://python-poetry.org/
I have a Python app running on windows that has imports for the following packages:
requests
json
psycopg2
I copy the entire project (I used Pycharms to write the app and import the packages) to a new machine and expected it would work. The new machine is also windows and I'm trying to run my script from the command line (i.e. no Pycharm on the new machine).
Instead, I get an error saying "ModuleNotFoundError: No module named 'requests'"
If I look at the project, I have the directories:
venv
Lib
site-packages
requests
What am I missing/doing wrong?
You have a couple of options here but first the problem. You are exporting your code base to a new machine without the required Modules installed on that machine and/or within your Python project's environment. After you have python installed on your new machine, you need to be sure to point your PyCharm Project to the proper environment.
File > Default Preferences > Project Interpreter
The window that appears on the right will contain a drop down menu labeled Project Interpreter. If you click on the drop down, it should reveal a list of the available Python environments on your machine.
Based on your description of your site-packages directory I would assume you do not have your interpreter pointed the proper environment on your new machine. With that said, you would be better served creating a new virtual python environment on your machine and installing each relevant dependency within that environment.
Take a look at this post here for your first best option on re-creating your old python environment on your new machine.
EDIT: I apologize for not reading the question more thoroughly before answering the questions. If this is running on a Windows machine you will need to double check the environment path python is using. It is very easy to install python at a different PATH than the command line environment is checking on a Windows box. If for example your PATH is pointing to a different version of Python and PIP is installing packages somewhere else this issue can occur. Double check your System PATH for python and which version the command line is running.
On the new machine you must source venv/bin/activate so your path environment variables are set properly. In particular, which python should say venv/bin/python rather than /usr/bin/python. Also, take care to conda env update or pip install -r requirements.txt so you'll have suitable venv libraries on the new machine.
My goal is to run a Python script that uses Anaconda libraries (such as Pandas) on Azure WebJob but can't seem to figure out how to load the libraries.
I start out just by testing a simple Azure blob to blob file copy which works when run locally but hit into an error "ImportError: No module named 'azure'" when ran in WebJob.
example code:
from azure.storage.blob import BlockBlobService
blobAccountName = <name>
blobStorageKey = <key>
containerName = <containername>
blobService = BlockBlobService(account_name=blobAccountName,
account_key=blobStorageKey)
blobService.set_container_acl(containerName)
b = blobService.get_blob_to_bytes(containerName, 'file.csv')
blobService.create_blob_from_bytes(containerName, 'file.csv', b.content)
I can't even get Azure SDK libraries to run. let alone Anaconda's
How do I run a python script that requires external libraries such as Anaconda (and even Azure SDK). How do I "pip install" these stuff for a WebJob?
It seems like you've kown about deployment of Azure WebJobs, I offer the below steps for you to show how to load external libraries in python scripts.
Step 1 :
Use the virtualenv component to create an independent python runtime environment in your system.Please install it first with command pip install virtualenv if you don't have it.
If you installed it successfully ,you could see it in your python/Scripts file.
Step2 : Run the commad to create independent python runtime environment.
Step 3: Then go into the created directory's Scripts folder and activate it (this step is important , don't miss it)
Please don't close this command window and use pip install <your libraryname> to download external libraries in this command window.
Step 4：Keep the Sample.py uniformly compressed into a folder with the libs packages in the Libs/site-packages folder that you rely on.
Step 5:
Create webjob in Web app service and upload the zip file,then you could execute your Web Job and check the log
You could also refer to the SO thread :Options for running Python scripts in Azure
In addition, if you want to use the modules in Anaconda, please download them separately. There is no need to download the entire library.
Hope it helps you.
You can point your Azure WebJob to your main WebApp environment (and thus its real site packages). This allows you to use the newest fastest version of Python supported by the WebApp (right now mine is 364x64), much better than 3.4 or 2.7 in x86. Another huge benefit is then you don't have to maintain an additional set of packages that are statically held in a file somewhere (this gave me a lot of trouble with dynamic libraries with crazy dependencies such as psycopg2 and pandas).
HOW: In your WebJobs files, set up a .cmd file that runs your run.py, and in that .cmd file, you can just have one line of code like this:
D:\home\python364x64\python.exe run.py
That's it!
Azure WebJobs looks at .cmd files first, then run.py and others.
See this link for an official MS post on this method:
https://blogs.msdn.microsoft.com/azureossds/2016/12/09/running-python-webjob-on-azure-app-services-using-non-default-python-version/
I have my project stored on OneDrive. It sometimes works on my pc and laptop both of which have Windows 10. The project on both is in the same directory- C:/OneDrive/code/etc...
When I use virtualenv and download different packages, it works fine, but when I use my laptop nothing works at all (same applies the other way around). I get the following error:
Could not import runpy module ImportError:
No module named 'runpy'
What can I do to fix this problem on my laptop and PC? Anyone experiencing a similar issue?
Don't do this. OneDrive - and similar systems like Dropbox - are meant for sharing documents. They are not meant for code, and even less for installed libraries.
Store your code in a version control system like git, and push it up regularly to a host like Github. Then on each of your computers, clone the repo and install the dependencies locally inside a virtualenv.
I had a similar issue with a virtualenv synced with OneDrive ('pip' was no more recognized as a command for example).
I solved it by creating a symbolic link inside my OneDrive directory of a virtualenv I created outside. Like this, your drive provider can not modify/optimize/etc. your local files but they will be synced.
You can create a symlink with Windows cmd:
mklink /D "C:\...\OneDrive\...\target_dir\venv" "C:\...\source_dir\venv\"
I currently have an executable file that is running Python code inside a zipfile following this: https://blogs.gnome.org/jamesh/2012/05/21/python-zip-files/
The nice thing about this is that I release a single file containing the app. The problems arise in the dependencies. I have attempted to install files using pip in custom locations and when I embed them in the zip I always have import issues or issues that end up depending on host packages.
I then started looking into virtual environments as a way to ensure package dependencies. However, it seems that the typical workflow on the target machine is to source the activation script and run the code within the virtualenv. What I would like to do is have a single file containing a Python script and all its dependencies and for the user to just execute the file. Is this possible given that the Python interpreter is actually packaged with the virtualenv? Is it possible to invoke the Python interpreter from within the zip file? What is the recommended approach for this from a Python point of view?
You can create a bash script that creates the virtual env and runs the python scripts aswell.
!#/bin/bash
virtualenv .venv
.venv/bin/pip install <python packages>
.venv/bin/python script