Develop Reference

Python keeps getting corrupted and requiring reinstallation

I'm on windows 10 and using Miniconda 64bit version 4.9.2. For some reason my python installation keeps getting corrupted after running scripts, sometimes it will happen randomly, other times it seems like specific scripts cause it to become corrupt. This doesn't happen to my other teammates though who are running similar environments.
What will happen is after trying to run a specific script or command (jupyter lab for example) that worked previously that day, I will randomly get an error with something like the following:
ValueError: bad marshal data (unknown type code)
Usually it seems to involve a library I'm using in the stack trace, but I don't think it's a specific library because it will be different ones in the stack trace from time to time, I'm guessing whatever got corrupted at that moment. I've tried deleting the pycache files for the specific libraries that error out, but that shows me what I think is the root of the problem, files are getting corrupted and overwritten with random characters. See an example error stacktrace below after deleting .pyc files and attempting to run again:
Traceback (most recent call last):
File "c:\users\user\miniconda3\envs\environment\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:\users\user\miniconda3\envs\environment\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "c:\users\user\miniconda3\envs\environment\Scripts\jupyter.exe\__main__.py", line 4, in <module>
File "c:\users\user\miniconda3\envs\environment\lib\site-packages\jupyter_core\command.py", line 18, in <module>
from . import paths
File "c:\users\user\miniconda3\envs\environment\lib\site-packages\jupyter_core\paths.py", line 15, in <module>
import tempfile
File "c:\users\user\miniconda3\envs\environment\lib\tempfile.py", line 832
self._rmtree(ȕ&�n�gH�6�
^
SyntaxError: invalid character in identifier
When I look in the file after getting this error, it literally has been cut off and random characters like those seen above have been inserted. This never happens in the middle of scripts though, it's only after something has run and I try to run it again or I try to run something else afterwards. Not sure what's going on I'm not super versed in python. It's getting to the point where I have to reinstall multiple times a day.
Is it just my system? I'm using pycharm as my IDE, would that have anything to do with it? I'm gonna try different versions of miniconda but not sure if that will do much. I haven't found much support for this issue through my research.

I have the same problem using Miniforge (python 3.7.10, conda 4.10.3, windows 10).
Last line of the .py files of some modules gets randomly corrupted.
I noticed that in addition to lib content, also all the .py files in pkgs subfolders are already corrupted, but the compressed files in tar packages are fine.
I found this temporary workaround:
delete the cached files in lib\brokenmodule\__pycache__
delete the source folder in pkgs\brokenmodule-version-build\
run conda update brokenmodule --force-reinstall

VMTK custom script producing "No module named: vmtk.custom" error

I'm working with a module called VMTK to build scripts for analyzing vascular models. They have a tutorial for writing these scripts so they're easily identifiable by the system allowing you to pipe scripts together known as PypeS. I've followed the tutorial exactly as shown and end up with the same "No module named: vmtk.name_of_script" error.
What I've tried:
Putting the script in the same directory as all the preloaded scripts provided by the module so __init__.py could catch it
Copy-pasting the code from the website into my editor so I'm sure there are no errors in what I've written
Submitting the question directly onto the forum meant for questions about VMTK, with no responses for the past week and a half
Changing my Python interpreter to the one within the VMTK directory
The tutorial says you can put the custom script anywhere in your filesystem, but the issue arises no matter what. I'm currently using Python 3.6.10 through Anaconda and a binary install of VMTK (as opposed to building from source) on MacOS High Sierra.
I really don't want to have to build from source, as I only need the preloaded scripts and ability to write new scripts, as opposed to using the C++ files that the scripts are built from. I've been stuck for so long and am not at all sure what the issue is. The closest thing I found was an SO question addressing this problem, but the OP simply said the problem had been resolved and provided no other information.
Here's the aforementioned code:
#!/usr/bin/env python
import sys
from vmtk import pypes
from vmtk import vmtkscripts
customscript = 'customScript'
class customScript(pypes.pypeScript):
def __init__(self):
pypes.pypeScript.__init__(self)
def Execute(self):
pass
if __name__=='__main__':
main = pypes.pypeMain()
main.Arguments = sys.argv
main.Execute()
As per the tutorial, I've also run chmod u+x customscript.py to change the permissions of my file so I can execute it. I get stuck at exactly the point where the script should do the most basic of operations within VMTK. Any ideas?
Edit: Here's the exact error:
$ ./customscript.py
No module named 'vmtk.customscript'
Traceback (most recent call last):
File "/vmtk/lib/python3.6/site-packages/vmtk/pype.py", line 290, in Execute
module = importlib.import_module('vmtk.'+scriptName)
File "/vmtk/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 978, in _gcd_import
File "<frozen importlib._bootstrap>", line 961, in _find_and_load
File "<frozen importlib._bootstrap>", line 948, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'vmtk.customscript'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./customscript.py", line 21, in <module>
main.Execute()
File "/vmtk/lib/python3.6/site-packages/vmtk/pypescript.py", line 688, in Execute
pipe.Execute()
File "/vmtk/lib/python3.6/site-packages/vmtk/pype.py", line 298, in Execute
self.PrintError(str(e))
File "/vmtk/lib/python3.6/site-packages/vmtk/pype.py", line 102, in PrintError
raise RuntimeError(errorMessage)
RuntimeError: No module named 'vmtk.customscript'
I want to backtrace the issue but I don't think editing any of the preloaded files should be necessary.

The solution I found after a week of trying and searching:
git clone the vmtk repo onto your computer and follow the VMTK download instructions on how to build it from source. It will absolutely feel like a headache and be a long process; just wait it out.
If you don't have cmake installed on your computer, they offer a GUI version on their website. I'm not sure how it would work for Windows, but it definitely works for Mac.
Once you've built it, make sure that each terminal instance of VMTK you set source vmtk_env.sh from the path where that file is. This will allow you to use the VMTK environment.
After writing the custom script, making sure to follow all the directions in the tutorial (assuming this is the first time), make sure to save it in the "site-packages" folder in the build folder. The path for me is "vmtk-build/lib/python-2.7/site-packages/vmtk/". I'm currently looking to update the version of Python it's using, but that's not the point.
Once that's done, navigate to that folder from within terminal and run chmod u+x name_of_script.py (if you're in a Unix-based operating system). This will allow you to execute the file with ./name_of_script.py.
I believe it was due to the version of Python VMTK is using, but I had two "No module named" errors come up, each one a separate time: one for tkinter and one for joblib.
The tkinter issue is one that can be resolved by going into the file where the import is, I believe vmtkscripts.py, and changing all instances of tkinter to Tkinter (capital T). This is because they changed the name of the module from Python 2 to 3, so in newer versions of Python it would be tkinter. That fixed the first error.
When I attempted to fix the joblib issue, I saw that the other version of Python available on my system already had it, thus I couldn't install it through pip. Instead, you can do python-2.7 -m pip install joblib to install it for a specific version of Python. Although I did uninstall joblib as a whole from my computer before doing that.
That should fix everything. The next time I attempted to run my script, it ran without any errors.
Extra: I wasn't using any Anaconda environment for this. Might go back and try to set that up, but I'm just glad I got it to work. Additionally, I deleted the binary install of VMTK from my computer. No need to keep it around if you're just going to build it from source. You can run VMTK through the terminal after setting source vmtk_env.sh by running the vmtk command.

Pycharm Startup will not connect to console

I am trying to start up pycharm and am getting the following error while connecting to the console:
C:\Users\i_am_yohan\Anaconda3\pythonw.exe "C:\Program Files\JetBrains\PyCharm Community Edition 2019.3\plugins\python-ce\helpers\pydev\pydevconsole.py" --mode=client --port=52061
Failed to import the site module
Traceback (most recent call last):
File "C:\Users\i_am_yohan\Anaconda3\envs\Base\Lib\site.py", line 579, in <module>
main()
File "C:\Users\i_am_yohan\Anaconda3\envs\Base\Lib\site.py", line 566, in main
known_paths = addsitepackages(known_paths)
File "C:\Users\i_am_yohan\Anaconda3\envs\Base\Lib\site.py", line 349, in addsitepackages
addsitedir(sitedir, known_paths)
File "C:\Users\i_am_yohan\Anaconda3\envs\Base\Lib\site.py", line 207, in addsitedir
addpackage(sitedir, name, known_paths)
File "C:\Users\i_am_yohan\Anaconda3\envs\Base\Lib\site.py", line 159, in addpackage
f = open(fullname, "r")
File "C:\Users\i_am_yohan\Anaconda3\envs\Base\Lib\_bootlocale.py", line 12, in getpreferredencoding
if sys.flags.utf8_mode:
AttributeError: 'sys.flags' object has no attribute 'utf8_mode'
Process finished with exit code 1
I am making a transition from spyder to pycharm and am currently experiencing a lot of problems most notable this one. I have set up the correct project interpreter.

I get the same error, after I uninstalled Python 3.6, which I did after installing Python 3.7. I was already using Python 3.7 through Pycharm successfully for a while, but realized after some time that I had Python 3.6 still installed, without using it. Doing the following fixed this for me.
Before trying anything else, you might want to check that you are using the expected Python interpreter for what you are trying to execute.
I don't know what exactly has changed that fixed this, and I wouldn't necessarily recommend this solution, since my understanding of what really happened is very limited. But it doesn't seem to have broken my project, and so maybe someone is helped by this.
I cut and pasted my .idea file from my project root folder. Note that I cut and paste it, I didn't delete it. Removing this folder makes you lose a many, if not all, of your Pycharm project's settings. So don't delete this folder unless you want to lose your project settings, like your run/debug configurations.
Then I restarted Pycharm, and it automically started to create a new .idea folder (which can be very time consuming). I closed Pycharm again, since I realized I didn't want to go through with this, I had too many settings that I didn't want to lose, and so I removed the newly created .idea folder, and put back the original .idea folder.
After starting up Pycharm again, all the time in the same project folder, Pycharm did have to do some indexing, but then it worked. The project I am working on is a Django project, and I have a run configuration for Django. The error was thrown when I wanted to run the Django configuration (Maybe I should have checked the interpreter is was set to before doing all this). The only thing I was required to do to after the swapping back and forth of the .idea folder was change the Python interpreter for the Django configuration, it seemed to be reset.

I have figured this out. I should have connected to:
C:\Users\i_am_yohan\Anaconda3\python.exe
rather than:
C:\Users\i_am_yohan\Anaconda3\pythonw.exe
I got confused because I was moving from Spyder to Pycharm and Spyder is connected to pythonw.exe rather than python.exe

This happends when python console wants to use Python 3.7 interpreter to connect to a project build based on previous version of python. Here is a quick solution if you have installed Python 3.7 correctlly.
Go to your Pycharm, find 'Switch project interpreter' and change the interpreter into the Python 3.7 version.

python ModuleNotFoundError: No module named 'gplearn'

I created a project in Pycharm including a gp.py, copied the project folder to another machine and tried to run the gp.py script from command line using python gp.py. However I got a error
Traceback (most recent call last):
File "gp.py", line 2, in <module>
from gplearn.genetic import SymbolicRegressor
ModuleNotFoundError: No module named 'gplearn'
The thing is I could see this gplearn library I used in the project folder, like below. The screen shot is not from Pycharm but the folder I copied to another computer. I tried to place the gp.pyinside and outside venv folder but neither worked. Could you please help to point out what the problem is ? Many thanks

gplearn package is not installed in the new machine.
Go to cmd prompt/terminal in pycharm and execute below line:
pip install gplearn

Sound like you are seeking for a method to deploy your script to end user, in this case maybe you can give Pyinstaller a try.

Django 1.9a1 __init__.py is visible in eclipse/PyDev even though it should be deleted (Windows)

I'm having a strange behaviour here:
I just upgraded to Django 1.9.4 from Django 1.9a1 using pip. The installation went fine and Django was running smoothly, but during start up, it still showed the version number "1.9a1".
Just to make sure I uninstalled Django again using pip and confirmed that my django folders are no longer existing in C:\Program Files (x86)\Python 3.5\lib\site-packages (neither the "django" nor the "Django-1.9.4" folder).
After starting the server again, I was surprised to read the error message
from django.core.management import execute_from_command_line
File "C:\Program Files (x86)\Python 3.5\lib\site-packages\django\__init__.py", line 1, in <module>
from django.utils.version import get_version
ImportError: No module named 'django.utils'
I expected the error to be something like "package django not found" or similar.
Since eclipse provides the file name as link, I clicked on it an eclipse opened a file with content
from django.utils.version import get_version
VERSION = (1, 9, 0, 'alpha', 1)
__version__ = get_version(VERSION)
[...]
I double-checked that this file is not available via command line and via windows explorer.
Where is this file located and how can I get rid of it? Or is it generated automatically (and if yes, how can I correct this mechanism)
Or am I missing something completely obvious?
Additional info:
I re-installed Django 1.9.4 and init.py file located at C:\Program Files (x86)\Python 3.5\lib\site-packages\django__init__.py shows the correct version info. But still, if I run a Django project, the version info given is 1.9a1 .
Even when I open a cmd prompt and enter
python -c "import django; print(django.get_version())"
the answer is 1.9a1
Update 2016-03-13 17:57:
I uninstalled again and used the explicit ==1.9.4 suggested by alecxe. This solved part of the problem:
Now, if I open the command line,
python -c "import django; print(django.get_version())"
gives me the correct answer (1.9.4). But in Eclipse/PyDev, my project is no longer starting. The first error message is:
Unhandled exception in thread started by <function check_errors.<locals>.wrapper at 0x03F72108>
Traceback (most recent call last):
File "C:\Program Files (x86)\Python 3.5\lib\site-packages\django\apps\config.py", line 118, in create
cls = getattr(mod, cls_name)
AttributeError: module 'django.contrib' has no attribute 'admin'
Of course, the admin module is existing on the file system.
Just to check, I started a new project in the console using
django-admin startproject mysite
and the new project is sucessfully created.
When I use PyDev inside Eclipse to start a new project (using "new"/"PyDev Django Project"), the process fails! The project directory as well as manage.py are created, but the project directory is empty (no init.py, settings.py, urls.py, wsgi.py).
It seems like Eclipse/PyDev is not using the correct django installation, even though all file names given are the correct one. Is there maybe any caching involved? And yes, I re-started Eclipse after re-installation.
Additional Info 2016-03-14 17:44:
I just re-started my computer and surprisingly, my eclips was now able to run my project again - the error described above disappearde, BUT the django version number displayed is 1.9a1 again!
Now I fired up the command prompt and python -c "import django; print(django.get_version())" gave me 1.9a1 again, too! What I did next was to start the command prompt again as administrator and - tataa - I got a 1.9.4.
So the problem seems to be related to admin privileges which might be due to installing python in C:\Program Files (x86)\Python 3.5. I used the console window with admin rights for all pip installs / uninstalls described above. Is there some windows mechanism which could explain this behaviour?
Solution 2016-03-14 18:17:
I now searched my whole C: drive for folders containing "django", and I found one in
C:\Users\[my_username]\AppData\Local\VirtualStore\Program Files (x86)\Python 3.5\Lib\site-packages\django
and - surprise surprise - it contained the init.py file with the wrong version number. So it seems like I installed the 1.9a1 without admin privileges and eclipse always used this version since it is started without admin privileges, too. After some research about Windows VirtualStore I decided to manually delete the whole C:\Users\[my_username]\AppData\Local\VirtualStore\Program Files (x86)\Python 3.5 folder.
Immediately after, my cmd-window showed me version 1.9.4, independent of admin- or non-admin-mode.
And - most important - when I ran my django project in eclipse, it showed the correct version number, too.
Just to be sure, I created a new django project using the PyDev wizard, and all project files were created correctly.
So in summary, the problem was due to a previous install of django without admin rights, followed by a django installation with admin rights. The responsible mechanism was the Windows VirtualStore, not Eclipse or PyDev.
I will leave the whole description of my Odyssee as is, maybe it helps someone to find the topic. I will give a short summary answer below.

The long story is described above, the short answer is:
I had an additional installation of django in my C:\Users\[my_username]\AppData\Local\VirtualStore\Program Files (x86)\Python 3.5\Lib\site-packages\django folder. This was due to installing the 1.9a1 version without admin rights.
Since eclipse was accessing django without admin rights as well, it used the outdated version, and not the recent one which was installed usind pip with admin rigths.
I deleted the C:\Users\[my_username]\AppData\Local\VirtualStore\Program Files (x86)\Python 3.5 folder and after a restart of eclipse everything worked.

What I remember helped me in a similar case was uninstalling Django completely and installing it over again:
pip uninstall django
pip install django==1.9.4
You may also need to check and manually remove django directory and the egg file if it is left in the C:\Program Files (x86)\Python 3.5\lib\site-packages directory after uninstalling django via pip.