Develop Reference

Where can python cached script files be located, and how can I flush them?

I have a python project that periodically throws the following error:
Running jobs:
RollingRewalkTriggersBase True 2019-03-17 18:44:47.772462 2018-12-10 13:05:47.977925 Traceback (most recent call last):
File "runScheduler.py", line 93, in doCall
self._doCall()
File "runScheduler.py", line 110, in _doCall
instance._go()
File "/media/Storage/Scripts/ReadableWebProxy/WebMirror/TimedTriggers/TriggerBase.py", line 44, in _go
self.go()
File "/media/Storage/Scripts/ReadableWebProxy/WebMirror/TimedTriggers/RollingRewalkTriggers.py", line 175, in go
self.retrigger_other()
File "/media/Storage/Scripts/ReadableWebProxy/WebMirror/TimedTriggers/RollingRewalkTriggers.py", line 124, in retrigger_other
for chunk in pbar:
File "/usr/local/lib/python3.5/dist-packages/tqdm/_tqdm.py", line 991, in __iter__
self.close()
File "/usr/local/lib/python3.5/dist-packages/tqdm/_tqdm.py", line 1088, in close
self._decr_instances(self)
File "/usr/local/lib/python3.5/dist-packages/tqdm/_tqdm.py", line 446, in _decr_instances
if inst.pos > abs(instance.pos):
AttributeError: 'tqdm' object has no attribute 'pos'
The problem here is that:
/media/Storage/Scripts/ReadableWebProxy/ doesn't exist (it's been moved to /media/Scripts/ReadableWebProxy/)
/usr/local/lib/python3.5/ Doesn't exist (I've switched from ubuntu 16.04 to 18.04, which only has python 3.6).
I"m running on pypy3, not cpython.
The tqdm source code referenced in the error doesn't match the files on-disk (the library has been updated, and the line numbers point to code that doesn't match the error output).
I have no idea where the error is coming from. I've reinstalled the freaking OS, and it's still there.
Now, /media/NNNN is a network drive that wasn't reinstalled, so I tried to clean out any potential *.pyc files:
From /media/Scripts/ReadableWebProxy/:
find . -name "__pycache__" -exec rm -rv \{\} \;
find . -name "*.pyc" -delete
At this point, I flat out don't understand what's going on. I'm getting error messages from a previous version of the project, and I can't seem to fix them. I don't understand where the offending files could possibly be cached, considering that the entire OS has been reinstalled, and any *.pyc files have been deleted.
Given the above context, basically, the question is "Where can python stick cached compilation files"? Something, somewhere seems to be referring to a file that doesn't even exist any more, and I've been unable to force python to refresh it's compilation cache.

pylearn2 PATH variable

there is surprisingly little documentation or tutorials about this.
I want to run pylearn2 on my Mac OSX 10.11.1.
Acoording to the tutorial I should run at first this line:
cd pylearn/pylearn2/scripts/tutorials/grbm_smd/
python make_dataset.py
Yet the script fails with this exception:
Traceback (most recent call last):
File "/Users/username/python/pylearn2/pylearn2/utils/string_utils.py", line 53, in preprocess
else os.environ[varname])
File "/Users/username/anaconda/lib/python3.4/os.py", line 633, in __getitem__
raise KeyError(key) from None
KeyError: 'PYLEARN2_DATA_PATH'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "make_dataset.py", line 27, in <module>
train = cifar10.CIFAR10(which_set="train")
File "/Users/username/python/pylearn2/pylearn2/datasets/cifar10.py", line 71, in __init__
string_utils.preprocess('${PYLEARN2_DATA_PATH}'),
File "/Users/username/python/pylearn2/pylearn2/utils/string_utils.py", line 56, in preprocess
reraise_as(NoDataPathError())
File "/Users/username/python/pylearn2/pylearn2/utils/exc.py", line 90, in reraise_as
six.reraise(type(new_exc), new_exc, orig_exc_traceback)
File "/Users/username/anaconda/lib/python3.4/site-packages/theano/compat/six.py", line 321, in reraise
raise value.with_traceback(tb)
File "/Users/username/python/pylearn2/pylearn2/utils/string_utils.py", line 53, in preprocess
else os.environ[varname])
File "/Users/username/anaconda/lib/python3.4/os.py", line 633, in __getitem__
raise KeyError(key) from None
pylearn2.utils.exc.NoDataPathError: You need to define your PYLEARN2_DATA_PATH environment variable. If you are
using a computer at LISA, this should be set to /data/lisa/data.
Platform-specific instructions for setting environment variables:
Linux
=====
On most linux setups, you can define your environment variable by adding this
line to your ~/.bashrc file:
export PYLEARN2_VIEWER_COMMAND="eog --new-instance"
*** YOU MUST INCLUDE THE WORD "export". DO NOT JUST ASSIGN TO THE ENVIRONMENT VARIABLE ***
If you do not include the word "export", the environment variable will be set
in your bash shell, but will not be visible to processes that you launch from
it, like the python interpreter.
Don't forget that changes from your .bashrc file won't apply until you run
source ~/.bashrc
or open a new terminal window. If you're seeing this from an ipython notebook
you'll need to restart the ipython notebook, or maybe modify os.environ from
an ipython cell.
Mac OS X
========
Environment variables on Mac OS X work the same as in Linux, except you should
modify and run the "source" command on ~/.profile rather than ~/.bashrc.
Original exception:
KeyError: PYLEARN2_DATA_PATH
I inserted the following information from the exception into ~/.profile
export PYLEARN2_VIEWER_COMMAND="eog --new-instance"
and ran
source ~/.bashrc
As it still threw an exception I did some research and found out I must put the .profile file into the .bash_profile. So I added this line to .bash_profile:
#.profile
source ~/.profile
However the outcome is still the same :(
Additional information
According to the installation guide says I should add another information to the path envoirement, yet I cannot understand which in particular.
hidden files in my system
.bash_history
.bash_profile
.bash_profile-anaconda.bak
.bash_sessions
.config
.ipython
.local
.profile
.python_history
.theano
.vminfo

After going through the code myself I've found the solution. There is a bug in the sourcecode, which is missing the download_cifar10.sh script. Plus the tutorial is missing a PATH variable which is necessary.
Instruction
1.) set the PATH variable
export PYLEARN2_VIEWER_COMMAND="eog --new-instance"
export PYLEARN2_DATA_PATH=/YOURPATHTOHERE/pylearn2/datasets
2.) download cifar-10 (python version)
3.) unpack it
You will get a "cifar-10-batches-py" folder
4.) Wrap "cifar-10-batches-py" in a "cifar-10" folder
5.) Put the "cifar-10" folder into /pylearn2/datasets
The final path containing the cifar-10 files should be:
../pylearn2/datasets/cifar-10-batches-py/cifar-10
You're ready to go!

The download script exist:
https://github.com/lisa-lab/pylearn2/blob/master/pylearn2/scripts/datasets/download_cifar10.sh
Did you download Pylearn2 correctly? Did you delete files in it by mistake?

python - pyinstaller "RuntimeWarning: Parent module 'PyInstaller.hooks.hook-PIL' not found while handling absolute import" and "tcl" related errors

I get a warning message while trying to create exectable file using pyinstaller. This warning appeared after installing Pillow. Previously i nevre got any warnings and was able to make it through.
the warning i get by pyinstaller is:
7314 INFO: Analyzing main.py
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyInstaller-2.1.1dev_-py2.7.egg/PyInstaller/hooks/hook-PIL.Image.py:14: RuntimeWarning: Parent module 'PyInstaller.hooks.hook-PIL' not found while handling absolute import
from PyInstaller.hooks.shared_PIL_Image import *
Also when i tried to run the executable's exe/consol version of my code that lies inside the dist folder created by the pyinstaller (dist/main/main), these are displayed..
Traceback (most recent call last):
File "<string>", line 26, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyInstaller-2.1.1dev_-py2.7.egg/PyInstaller/loader/pyi_importers.py", line 276, in load_module
exec(bytecode, module.__dict__)
File "/Users/..../build/main/out00-PYZ.pyz/PIL.PngImagePlugin", line 40, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyInstaller-2.1.1dev_-py2.7.egg/PyInstaller/loader/pyi_importers.py", line 276, in load_module
exec(bytecode, module.__dict__)
File "/Users/..../build/main/out00-PYZ.pyz/PIL.Image", line 53, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyInstaller-2.1.1dev_-py2.7.egg/PyInstaller/loader/pyi_importers.py", line 276, in load_module
exec(bytecode, module.__dict__)
File "/Users/..../build/main/out00-PYZ.pyz/FixTk", line 74, in <module>
OSError: [Errno 20] Not a directory: '/Users/.../dist/main/tcl'
logout
[Process completed]
so, i tried by uninstalling pillow, installing tk tcl dev version. And then installed pillow. Even that didnt helped.
I also tried reinstalling pyinstaller,. didnt help too
Update 1:
It seems Pyinstaller.hooks.hook-PIL.py file was missing in the Pyinstaller/hooks directory. And it was missing on all platforms(Mac, windows and linux). This is the warning/error message that i get on windows, which is the same i got on mac and on linux.
Later i found a link which said, its just to need Python import machinery happy. so i created as said so. Then i dont get the same error on all platforms, But on mac i still get the PILImagePlugin,Image and FixTk errors

Solution for tcl:
I found what was going wrong,.. Every problem that i faced on OSX was the OS itself(exactly the macport). Python by default comes with the mac OS. And this version of python may be useful for just learning basic python, but is not suitable for Development purpose.
Installing brew's python helped. I followed this SO link. After doing these i was still getting errors. Later i had to change the paths on /etc/paths. Basically rearranging them should work. But still then i wasn't getting it right.
Then i had to change the .bash_profile, which worked for most users, But still i was getting mac's version of python and pip, not the brews version of python.
Finally i had to restart the machine for a couple of times and do the /etc/paths and .bash_profile steps repeatedly to get the system wide effect to accept brews version of python and pip
Solution for PIL:
just adding a file called hook-PIL.py with an empty content would serve the purpose. I found a link which was having the hook files content of pyinstaller.
The location to create
for mac : /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyInstaller-2.1.1dev_-py2.7.egg/PyInstaller/hooks/ Actually for mac this step wouldn’t be required. When we install python through brew and change the path, everything that you try to install later either through pip install or from source packages tend to choose a different path. And everything will be taken care of.
for windows:C:\Python27\lib\site-packages\PyInstaller-2.1.1.dev0-py2.7.egg\PyInstaller\hooks
**Please check if this is a valid path on your machine before creating the file and then create the file. And im not sure or i don't know if just adding an empty file is the right way. But it worked for me

PermissionError: [WinError 5] Access is denied python using moviepy to write gif

I'm using windows 8.1 64 bit
my code
import pdb
from moviepy.editor import *
clip = VideoFileClip(".\\a.mp4")
clip.write_gif('.\\aasda.gif')
the exception is at write_gif method
Traceback (most recent call last):
File "C:\abi\youtubetogif_project\test.py", line 5, in <module>
clip.write_gif('G:\\abi\\aasda.gif')
File "<string>", line 2, in write_gif
File "C:\Python34\lib\site-packages\moviepy-0.2.1.8.12-py3.4.egg\moviepy\decorators.py", line 49, in requires_duration
return f(clip, *a, **k)
File "C:\Python34\lib\site-packages\moviepy-0.2.1.8.12-py3.4.egg\moviepy\video\VideoClip.py", line 435, in write_gif
dispose= dispose, colors=colors)
File "<string>", line 2, in write_gif
File "C:\Python34\lib\site-packages\moviepy-0.2.1.8.12-py3.4.egg\moviepy\decorators.py", line 49, in requires_duration
return f(clip, *a, **k)
File "C:\Python34\lib\site-packages\moviepy-0.2.1.8.12-py3.4.egg\moviepy\video\io\gif_writers.py", line 186, in write_gif
stdout=sp.PIPE)
File "C:\Python34\lib\subprocess.py", line 848, in __init__
restore_signals, start_new_session)
File "C:\Python34\lib\subprocess.py", line 1104, in _execute_child
startupinfo)
PermissionError: [WinError 5] Access is denied
I moved the script to another folder and partition, running moviepy dependancies and python as admin, turning off UAC still gives me error

I've run into this as well, solution is usually to be sure to run the program as an administrator (right click, run as administrator.)

Sometimes it occurs when some installations are not completed correctly, the process is stuck, or a file is still opened. So, when you try to run the installation again and the installation requires deleting, you can see the aforementioned error. In my case, shutting down the python processes and command prompt utilization helped.

this resolved my problem
Click on the search button in the taskbar and type “cmd”. Right-click on the Command Prompt and select Run as Administrator
pip install pydirectory

Solution on windows : restarted docker
On windows I used --use-container option during sam build
So, in order to fix stuck process, I've restarted docker

I got the same error when an imported library was trying to create a directory at path "./logs/".
It turns out that the library was trying to create it at the wrong location, i.e. inside the folder of my python interpreter instead of the base project directory. I solved the issue by setting the "Working directory" path to my project folder inside the "Run Configurations" menu of PyCharm. If instead you're using the terminal to run your code, maybe you just need to move inside the project folder before running it.

If you're encountering this in Jupyter/Jupyerlab while trying to pip install foo, you can sometimes work around it by using !python -m pip install foo instead.

On Windows, for me, it looked like at some point I'd set the folder to read-only.
Not really sure when, possibly during some mount failure on my Linux boot, but recursively clearing that flag helped.

I know it is pretty old and a couple of fellows has given the abstract answer to it.
But this is how I solved this problem on my machine. (Thanks #DevLoverUmar and #Vladyslav Didenko)
pip install gym --user

This might happen when the working directory path is different from the where the file is present . For example while running files and importing them in Spyder3 I have to check the working directory .

Maybe you wrongly set permission on python3. For instance if for the file permission is set like
`os.chmod('spam.txt', 0777)` --> This will lead to SyntaxError
This syntax was used in Python2. Now if you change like:
os.chmod('spam.txt', 777) --> This is still worst!! Your permission will be set wrongly since are not on "octal" but on decimal.
Afterwards you will get permission Error if you try for instance to remove the file:
PermissionError: [WinError 5] Access is denied:
Solution for python3 is quite easy:
os.chmod('spam.txt', 0o777) --> The syntax is now ZERO and o "0o"

pyAA with py2exe

Is anyone able to get pyAA working with py2exe? pyAA can be downloaded here.
I have trying to do this for the last 2 days and I am unable to reach a solution till now. The example files are like given below:
hello.py
import pyAA
print "Hello, World"
setup.py
from distutils.core import setup
import py2exe
setup(console=['hello.py'],
options = {"py2exe": {"bundle_files": 1}})
Now, if we run:
python setup.py py2exe
the executable is created. But running it gives the following error:
Traceback (most recent call last):
File "hello.py", line 1, in ?
File "zipextimporter.pyc", line 82, in load_module
File "pyAA\__init__.pyc", line 1, in ?
File "zipextimporter.pyc", line 82, in load_module
File "pyAA\AA.pyc", line 8, in ?
File "zipextimporter.pyc", line 82, in load_module
File "pyAA\pyAAc.pyc", line 5, in ?
File "zipextimporter.pyc", line 98, in load_module
ImportError: MemoryLoadLibrary failed loading pyAA\_pyAAc.pyd
I tried using dependency walker but I was unable to make too much sense of the same. I tried to add/delete some DLLs but to no avail.
p.s -> Please note that without bundle_files, it is working perfectly. But one of the requirements of the projects mean that bundle_file is required.

This king of thing seems to be a common problem with py2exe. Maybe try using another installer such as PyInstaller.

When I install pyAA and run depends on _pyAAc.pyd, it tells me I'm missing IESHIMS.DLL, though that might be because I'm on Windows 7.

_pyAAc.pyd is a DLL, I think those have to be loaded directly from the file system (not from memory buffers or archives).
If a one-file-solution is needed you could do your own bundling of the working unbundled py2exe result that unpacks itself to TEMP and runs from there. I suspect some self extracting archive maker could do this trick for you without the need for any coding on your part.

Having fought with Py2Exe a few times myself, bundling DLLs with dependencies usually causes headaches. If my memory serves me correctly, try adding sys.exec_prefix to PATH environment variable. This should allow the program to find the IESHIMS.DLL
os.environ['PATH'] = os.sep.join([sys.exec_prefix, os.environ['PATH']])