Develop Reference

Cannot set pythonpath environment variable from python

I'm trying to debug a project that has a lot of additional libraries added to PYTHONPATH at runtime before launching the python file.
I was not able to add those commands with tasks.json file prior to debugging python file in Visual Studio code (see post Visual Studio Code unable to set env variable paths prior to debugging python file), so I'm just adding them via an os.system("..") command
I'm only showing 1 of the libraries added below:
# Standard library imports
import os
import sys
os.system("SET PYTHONPATH=D:\\project\\calibration\\pylibrary\\camera")
# Pylibrary imports
from camera import capture
When I debug, it fails on line from camera import capture with:
Exception has occurred: ModuleNotFoundError
No module named 'camera'
File "D:\project\main.py", line 12, in <module>
from camera.capture import capture
I also tried
os.environ['PYTHONPATH']="D:\\project\\pylibrary\\camera" and I still get the same error
Why is it not remembering the pythonpath while running the script?
How else can I define the pythonpath while running Visual Studio Code and debugging the project file?
I know I can add the pythonpath to env variables in windows, but it loads too many libraries and I want it to only remember the path while the python script is executed.
Thanks

Using os.system() won't work because it starts a new cmd.exe shell and sets the env var in that shell. That won't affect the env vars of the python process. Assigning to os.environ['PYTHONPATH'] won't work because at that point your python process has already cached the value, if any, of that env var in the sys.path variable. The solution is to
import sys
sys.path.append(r"D:\project\calibration\pylibrary\camera")

Blessed / Curses controls don't work with Pyinstaller. Missing vtwin10

I have a very simple Python program that uses 'Blessed'. It works fine with the Win10 Python interpreter, but reports an error when packaged with Pyinstaller, and terminal control codes are ignored. Here's the code:
from blessed import Terminal
t = Terminal()
print(t.bright_green('Hello world'))
The string 'Hello world' is supposed to display on the console in bright green. Pyinstaller completes with no errors, and when I run the .exe, I get the message:
terminal.py:222: UserWarning: Failed to setupterm(kind='vtwin10'): Could not find terminal vtwin10
and then 'Hello world' is displayed in default terminal color.
It looks like Pyinstaller isn't including something in the build that the interpreter finds without issue. I found a vtwin10.py file in my Anaconda3 installation folder at:
C:\Anaconda3\Lib\site-packages\jinxed\terminfo
I looked at the referenced error in the blessed library's terminal.py file. Here's the code:
try:
curses.setupterm(self._kind, self._init_descriptor)
except curses.error as err:
warnings.warn('Failed to setupterm(kind={0!r}): {1}'
.format(self._kind, err))
So it looks like self._kind is being set to 'vtwin10'. There is a conditional import in terminal.py that looks like this:
if platform.system() == 'Windows':
import jinxed as curses # pylint: disable=import-error
HAS_TTY = True
(I get the humor.) It looks like the jinxed package is being imported explicitly in the code, and replaces the curses package. But somehow the vtwin10 definition is missing.
I found setupterm() in jinxed and dug deeper to find where that error message is coming from. It's in this code:
try:
self.terminfo = importlib.import_module('jinxed.terminfo.%s' % term.replace('-', '_'))
except ImportError:
raise error('Could not find terminal %s' % term)
This is where I get stuck. It looks like this code is unable to find the vtwin10.py file in the jinxed library. Does anyone know how to force Pyinstaller to include the vtwin10 terminal definition for curses? I'm guessing this is the problem.
Many thanks.

For now you will only need to specify jinxed.terminfo.vtwin10 and jinxed.terminfo.ansicon on Windows, but if you want it to be more dynamic, pyinstaller spec files are executable Python, so you can just dynamically look up any terminfo modules.
import pkgutil
import jinxed.terminfo
hiddenimports = [mod.name for mod in pkgutil.iter_modules(jinxed.terminfo.__path__, 'jinxed.terminfo.')

Finally figured this out. In the jinxed library, the code line:
importlib.import_module('jinxed.terminfo.%s' % term.replace('-', '_'))
dynamically loads a library module. Pyinstaller can't package dynamically imported modules. So to fix this, I need to specify the module using the --hidden-import option. The syntax is as follows:
pyinstaller --hidden-import=jinxed.terminfo.vtwin10 --onefile test.py
Program works just like in the interpreter. It works, but I'm concerned this breaks any platform independence jinxed was supposed to have. I can force import the vtwin10.py module, and it will work on win10 platforms. But the way jinxed is written, it figures out the windows platform and then dynamically loads the required terminfo module. There are a number of them in the jinxed.terminfo directory. Wildcards for --hidden-import don't work, so the only option is to use --hidden-import for every file in the jinxed.terminfo folder.

Run all Python scripts in a folder - from Python

How can I write a Python program that runs all Python scripts in the current folder? The program should run in Linux, Windows and any other OS in which python is installed.
Here is what I tried:
import glob, importlib
for file in glob.iglob("*.py"):
importlib.import_module(file)
This returns an error: ModuleNotFoundError: No module named 'agents.py'; 'agents' is not a package
(here agents.py is one of the files in the folder; it is indeed not a package and not intended to be a package - it is just a script).
If I change the last line to:
importlib.import_module(file.replace(".py",""))
then I get no error, but also the scripts do not run.
Another attempt:
import glob, os
for file in glob.iglob("*.py"):
os.system(file)
This does not work on Windows - it tries to open each file in Notepad.

You need to specify that you are running the script through the command line. To do this you need to add python3 plus the name of the file that you are running. The following code should work
import os
import glob
for file in glob.iglob("*.py"):
os.system("python3 " + file)
If you are using a version other than python3, just change the argument from python3 to python

Maybe you can make use of the subprocess module; this question shows a few options.
Your code could look like this:
import os
import subprocess
base_path = os.getcwd()
print('base_path', base_path)
# TODO: this might need to be 'python3' in some cases
python_executable = 'python'
print('python_executable', python_executable)
py_file_list = []
for dir_path, _, file_name_list in os.walk(base_path):
for file_name in file_name_list:
if file_name.endswith('.csv'):
# add full path, not just file_name
py_file_list.append(
os.path.join(dir_path, file_name))
print('PY files that were found:')
for i, file_path in enumerate(py_file_list):
print(' {:3d} {}'.format(i, file_path))
# call script
subprocess.run([python_executable, file_path])
Does that work for you?
Note that the docs for os.system() even suggest using subprocess instead:
The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function.

If you have control over the content of the scripts, perhaps you might consider using a plugin technique, this would bring the problem more into the Python domain and thus makes it less platform dependent. Take a look at pyPlugin as an example.
This way you could run each "plugin" from within the original process, or using the Python::multiprocessing library you could still seamlessly use sub-processes.

import local python module in HTCondor

This concerns the importing of my own python modules in a HTCondor job.
Suppose 'mymodule.py' is the module I want to import, and is saved in directory called a XDIR.
In another directory called YDIR, I have written a file called xImport.py:
#!/usr/bin/env python
import os
import sys
print sys.path
import numpy
import mymodule
and a condor submit file:
executable = xImport.py
getenv = True
universe = Vanilla
output = xImport.out
error = xImport.error
log = xImport.log
queue 1
The result of submitting this is that, in xImport.out, the sys.path is printed out, showing XDIR. But in xImport.error, there is an ImporError saying 'No module named mymodule'. So it seems that the path to mymodule is in sys.path, but python does not find it. I'd also like to mention that error message says that the ImportError originates from the file
/mnt/novowhatsit/YDIR/xImport.py
and not YDIR/xImport.py.
How can I edit the above files to import mymodule.py?

When condor runs your process, it creates a directory on that machine (usually on a local hard drive). It sets that as the working directory. That's probably the issue you are seeing. If XDIR is local to the machine where you are running condor_submit, then it's contents don't exist on the remote machine where the xImport.py is running.
Try using the .submit feature transfer_input_files mechanism (see http://research.cs.wisc.edu/htcondor/manual/v7.6/2_5Submitting_Job.html) to copy the mymodule.py to the remote machines.

Where to put images folder in python exe?

I have converted a python game I designed into an exe. Running the exe itself causes it to flash and then close, meaning an error has occured. Running it from the Command Prompt causes the error as well, but documents it:
Cannot load image: Playfield.png
Couldn't open images\Playfield.png
This is telling me that the load_image block is failing. I have encountered this before when I did not have an images directory.
I attempted to move the images folder to the dist directory. This is the error that shows up:
Traceback (most recent call last):
File "Table_Wars.py", line 728, in <module>
File "Table_Wars.py", line 51, in main
File "Table_Wars.py", line 236, in __init__
File "pygame\__init__.pyc", line 70, in __getattr__
NotImplementedError: font module not available
(ImportError: DLL load failed: The specified module could not be found.)
This is my first time with py2exe, so I'm not really sure what is happening. The raw python file itself, Table_Wars.py, runs as expected.
If it helps, the location for the entire Table_Wars folder is inside a folder called Games, located on my Desktop (C:\Users\Oventoaster\Desktop\Games\Table_Wars). I am running Windows 7 32 bit OS.
On request, here is the output.txt I have generated:
Folder PATH listing for volume OS
Volume serial number is 7659-4C9C
C:\USERS\OVENTOASTER\DESKTOP\GAMES\TABLE_WARS
build
bdist.win32
winexe
bundle-2.7
collect-2.7
ctypes
distutils
email
mime
encodings
logging
multiprocessing
dummy
pygame
threads
unittest
xml
parsers
temp
dist
images
Here is the setup.py I used to convert the file:
from distutils.core import setup
import py2exe
setup(console=['Table_Wars.py'])
EDIT: I have attempted to use the full py2exe example. This will create the exe, but gives the same Cannot load image error. Attempting to put the images folder in the same folder as the exe creates a Runtime Error: The application requested the runtime to terminate it in an unusual way.
The shortened form of the code Slace Diamond suggested prevents py2exe from finding Table_Wars.py:
from cmd:
running py2exe
*** searching for required modules ***
error: Table_Wars.py: No such file or directory.
setup and Table_Wars are in the same directory. If it help, I input the full path to python.exe and setup.py.
EDIT: I seem to be getting closer. I put the images directory within self.extra_datas, and now I am getting this:
Fatal Python error: (segmentation fault)
This application has requested the runtime to terminate it in an unusual way. Please contact the application's suppourt team for more information

When you build a distributable package with py2exe (and py2app for that matter), part of the package environment is to point to a local resource location for files. In your plain unpackaged version, you are referring to a relative "images/" location. For the packaged version, you need to configure your setup.py to include the resources in its own location.
Refer to this doc for very specific info about how to set the data_files option of your package: http://www.py2exe.org/index.cgi/data_files
That page has multiple examples to show both very simple paths, and also a helper function for finding the data and building the data_files list for you.
Here is an example of the simple snippet:
from distutils.core import setup
import py2exe
Mydata_files = [('images', ['c:/path/to/image/image.png'])]
setup(
console=['trypyglet.py.py']
data_files = Mydata_files
options={
"py2exe":{
"unbuffered": True,
"optimize": 2,
"excludes": ["email"]
}
}
)
This closely matches what you are trying to achieve. It is saying that the "image.png" source file should be placed into the "images" directory at the root of the resources location inside the package. This resource root will be your current directory from your python scripts, so you can continue to refer to it as a relative sub directory.

It looks like you've already fixed the image problem by moving the folder into dist. The missing font module, on the other hand, is a known problem between pygame and py2exe. Py2exe doesn't copy some necessary DLLs, so you have to override py2exe's isSystemDLL method, forcing it to include audio and font related DLLs.
If Table_Wars.py is the only module in your project, try running this script with python setup.py py2exe:
from os.path import basename
from distutils.core import setup
import py2exe
origIsSystemDLL = py2exe.build_exe.isSystemDLL
def isSystemDLL(pathname):
if basename(pathname).lower() in ("libogg-0.dll", "sdl_ttf.dll"):
return 0
return origIsSystemDLL(pathname)
py2exe.build_exe.isSystemDLL = isSystemDLL
setup(windows=[{"script": "Table_Wars.py"}],
options={"py2exe": {"dist_dir": "dist"}})
You could also try the example py2exe setup file on the pygame wiki. If neither of them are working, please add the error messages to your question.
I tried running py2exe on a sample project, and it also breaks for me when I use the default pygame font. If you're using the default font, try putting a ttf file in the root of your project and also in the dist folder. You'll have to change the call to pygame.Font in your script as well:
font = pygame.font.Font("SomeFont.ttf", 28)