Develop Reference

How to make Python script developed in Spyder IDE executable

I wrote a script in Spyder IDE, but in order for clients to use it, I would like to make it easier to run it then opening Anaconda and then Spyder and running it from there.
The reason I used Spyder was because it allowed me to use many modules that otherwise wouldn't be allowed on company stations.
I researched a lot but could not find a way to do this.
I was thinking of enveloping the script in a batch file and running it. I tried updating the python script to look for modules in the Anaconda default directories:
sys.path.insert(1, C:\\ProgramData\\Anaconda3\\python37.zip)
sys.path.insert(1, C:\\ProgramData\\Anaconda3\\DLLs)
sys.path.insert(1, C:\\ProgramData\\Anaconda3\\lib)
sys.path.insert(1, C:\\ProgramData\\Anaconda3)
sys.path.insert(1, C:\\ProgramData\\Anaconda3\\lib\\site-packages)
sys.path.insert(1, C:\\ProgramData\\Anaconda3\\lib\\site-packages\\win32)
sys.path.insert(1, C:\\ProgramData\\Anaconda3\\lib\\site-packages\\win32\\lib)
sys.path.insert(1, C:\\ProgramData\\Anaconda3\\lib\\site-packages\\Pythonwin)
sys.path.insert(1, C:\\ProgramData\\Anaconda3\\lib\\site-packages\\IPython\\extensions)
When running the script in Windows cmd, I got these errors when importing pandas module, so there is more to using Spyder then just accessing those modules (I couldn't figure out what):
Traceback (most recent call last):
File "script.py", line 32, in <module>
import pandas as pd
File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\__init__.py", line 13, in <module>
__import__(dependency)
File "C:\ProgramData\Anaconda3\lib\site-packages\numpy\__init__.py", line 110, in <module>
import warnings
File "C:\ProgramData\Anaconda3\lib\warnings.py", line 494
f"coroutine '{coro.__qualname__}' was never awaited\n"
Another approach I tried was using pyinstaller, but that is not available for us and I cannot install it from the website.
Is there a way of running the python script and using all the modules offered by Anaconda without going through Spyder?

Have you tried py2exe? It's a great tool that does exactly what you're asking for.
Also look into cython. It's a lot easier to generate an executable from C code, and you'll get a performance boost as well.

Also try pyinstaller. It'll give you both an executable and settings for that, like icon or no-console. first go to the directory where your .py file is and run the below command:
pyinstaller script.py --onefile
Before this, you'll need to install that python module:
python* -m pip install pyinstaller
Where * is your specific python version, if you have multiple installed.
The first command will give you dist folder, here is where your executable is placed. and the script.spec file is your settings.

sp_execute_external_script can't find modules installed by setuptools

I am actively developing a Python module that I would like to deploy in SQL Server 2017 installed locally, so I deploy the module in c:\Program Files\Microsoft SQL Server\<Instance Name>\PYTHON_SERVICES\Lib\site-packagesusing setuptoolslike so:
"c:\Program Files\Microsoft SQL Server\<Instance_Name>\PYTHON_SERVICES\python" setup.py develop
This produces an .egg-info directory in my project root, and a .egg-link file in the site-packages directory mentioned above. The .egg-link file correctly points to the .egg-info directory in my project root, so it appears setuptools is working correctly.
Here's my setup.pyfor reference:
from setuptools import setup
setup(
setup_requires=['pbr'],
pbr=True,
)
And here's the corresponding setup.cfg file:
[metadata]
name = <module_name>
description = <Module Description>
description-file = README.md
description-content-type = text/markdown
[files]
package_root = py/src
Since I am just trying to make the plumbing work, I have a single python script called uploader.py in <project_root>/py/src:
#uploader.py
class Upload:
pass
With this deployment in place, I am hoping to simply import the module I just published through .egg-link into a sp_execute_external_script call like so:
execute sp_execute_external_script #language= N'Python', #script= N'from <module_name>.uploader import Upload';
However, executing this stored procedure from SSMS produces the following error message:
Msg 39004, Level 16, State 20, Line 10
A 'Python' script error occurred during execution of 'sp_execute_external_script' with HRESULT 0x80004004.
Msg 39019, Level 16, State 2, Line 10
An external script error occurred:
Error in execution. Check the output for more information.
Traceback (most recent call last):
File "<string>", line 5, in <module>
File "C:\SQL-MSSQLSERVER-ExtensibilityData-PY\MSSQLSERVER01\C08BB9A7-66B5-4B5E-AAFC-B0248EE64199\sqlindb.py", line 27, in transform
from <module_name>.uploader import Upload
ImportError: No module named '<module_name>'
SqlSatelliteCall error: Error in execution. Check the output for more information.
STDOUT message(s) from external script:
SqlSatelliteCall function failed. Please see the console output for more information.
Traceback (most recent call last):
File "C:\Program Files\Microsoft SQL Server\<Instance_Name>\PYTHON_SERVICES\lib\site-packages\revoscalepy\computecontext\RxInSqlServer.py", line 587, in rx_sql_satellite_call
rx_native_call("SqlSatelliteCall", params)
File "C:\Program Files\Microsoft SQL Server\<Instance_Name>\PYTHON_SERVICES\lib\site-packages\revoscalepy\RxSerializable.py", line 358, in rx_native_call
ret = px_call(functionname, params)
RuntimeError: revoscalepy function failed.
I have obviously redacted module_name and Instance_Name from the error message.
I tried using install command instead of develop just to make sure the .egg-link file is not a problem. install installs the .egg-info file in site-packages but I get the same error.
I also tried removing pbr from the mix, but got the same error.
Lastly, I tried adding my <project_root> to sys.path as suggested by How can I use an external python module with SQL 2017 sp_execute_external_script?, but that didn't help either.
So at this point, I don't have a clue what I might be doing wrong.
The python version is 3.5.2 and I don't think an __init__.py is needed in the project for it to qualify as a module. Inserting a blank __init__.py in py/src doesn't help either.
My pip version is 19.3.1 and setuptools version is 44.0.0 and pbr version is 5.4.4 and I have confirmed all modules are installed in the site-packages directory mentioned above.

Based on my extensive experimentation, it appears that sp_execute_external_script doesn't follow symlinks (i.e. through the.egg-link file). Therefore, development mode installations will not work, whether you use setuptools, pip, pbr or anything else.
I even tried symlinking <package_name> folder as an OS symlink. Since I am on Windows, I used mklink /D command on Command Prompt to symlink /py/src/<package_name> inside site-packages. While the command goes through correctly, and I can see the symlinked folder in File Explorer, sp_execute_external_script fails to detect the package. Which tells me that there is probably something in sp_execute_external_script code that avoids traversing symbolic links.
I wonder if there is a way to make it traverse symbolic links.
The only workable solution is to develop a package's code under its own directory, so, in my case /py/src/<package_name>. Then, before running exec sp_execute_external_script #language=N'python', #script=N'...' copy the <package_name> folder to the site-packages directory.
This is, sort of, equivalent to setup.py install, but bypasses the creation of intermediate files and directories. So I am going to stick with this simple--though odious--approach.
I am hoping somebody more knowledgeable would offer a better way to solve this problem.

python - pyinstaller "RuntimeWarning: Parent module 'PyInstaller.hooks.hook-PIL' not found while handling absolute import" and "tcl" related errors

I get a warning message while trying to create exectable file using pyinstaller. This warning appeared after installing Pillow. Previously i nevre got any warnings and was able to make it through.
the warning i get by pyinstaller is:
7314 INFO: Analyzing main.py
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyInstaller-2.1.1dev_-py2.7.egg/PyInstaller/hooks/hook-PIL.Image.py:14: RuntimeWarning: Parent module 'PyInstaller.hooks.hook-PIL' not found while handling absolute import
from PyInstaller.hooks.shared_PIL_Image import *
Also when i tried to run the executable's exe/consol version of my code that lies inside the dist folder created by the pyinstaller (dist/main/main), these are displayed..
Traceback (most recent call last):
File "<string>", line 26, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyInstaller-2.1.1dev_-py2.7.egg/PyInstaller/loader/pyi_importers.py", line 276, in load_module
exec(bytecode, module.__dict__)
File "/Users/..../build/main/out00-PYZ.pyz/PIL.PngImagePlugin", line 40, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyInstaller-2.1.1dev_-py2.7.egg/PyInstaller/loader/pyi_importers.py", line 276, in load_module
exec(bytecode, module.__dict__)
File "/Users/..../build/main/out00-PYZ.pyz/PIL.Image", line 53, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyInstaller-2.1.1dev_-py2.7.egg/PyInstaller/loader/pyi_importers.py", line 276, in load_module
exec(bytecode, module.__dict__)
File "/Users/..../build/main/out00-PYZ.pyz/FixTk", line 74, in <module>
OSError: [Errno 20] Not a directory: '/Users/.../dist/main/tcl'
logout
[Process completed]
so, i tried by uninstalling pillow, installing tk tcl dev version. And then installed pillow. Even that didnt helped.
I also tried reinstalling pyinstaller,. didnt help too
Update 1:
It seems Pyinstaller.hooks.hook-PIL.py file was missing in the Pyinstaller/hooks directory. And it was missing on all platforms(Mac, windows and linux). This is the warning/error message that i get on windows, which is the same i got on mac and on linux.
Later i found a link which said, its just to need Python import machinery happy. so i created as said so. Then i dont get the same error on all platforms, But on mac i still get the PILImagePlugin,Image and FixTk errors

Solution for tcl:
I found what was going wrong,.. Every problem that i faced on OSX was the OS itself(exactly the macport). Python by default comes with the mac OS. And this version of python may be useful for just learning basic python, but is not suitable for Development purpose.
Installing brew's python helped. I followed this SO link. After doing these i was still getting errors. Later i had to change the paths on /etc/paths. Basically rearranging them should work. But still then i wasn't getting it right.
Then i had to change the .bash_profile, which worked for most users, But still i was getting mac's version of python and pip, not the brews version of python.
Finally i had to restart the machine for a couple of times and do the /etc/paths and .bash_profile steps repeatedly to get the system wide effect to accept brews version of python and pip
Solution for PIL:
just adding a file called hook-PIL.py with an empty content would serve the purpose. I found a link which was having the hook files content of pyinstaller.
The location to create
for mac : /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyInstaller-2.1.1dev_-py2.7.egg/PyInstaller/hooks/ Actually for mac this step wouldn’t be required. When we install python through brew and change the path, everything that you try to install later either through pip install or from source packages tend to choose a different path. And everything will be taken care of.
for windows:C:\Python27\lib\site-packages\PyInstaller-2.1.1.dev0-py2.7.egg\PyInstaller\hooks
**Please check if this is a valid path on your machine before creating the file and then create the file. And im not sure or i don't know if just adding an empty file is the right way. But it worked for me

Difficulties installing SOAPpy module

I am very new to python and have been figuring out how to install modules.I have been trying to install the SOAPpy module and have only had success with the required modules fpconst, wstools, and setuptools(not mentioned as required, but still ended up being required).
I am getting an error when I try and install the SOAPpy module.
Traceback <most recent call last>:
File "...\setup.py", line 43, in module <module>
__version__ = load_version()
File "...\setup.py", line 35 in load_version
execfile(filename, d.__dict__)
IOError: [Errno 2] Unable to load the version number (no such file or directory):
'...\\src\\SOAPpy\\version.py'
Not sure what i should do to fix this.
Any help is greatly appreciated!

Looking at SOAPpy's setup.py file, it tries to import SOAPpy.version. This doesn't work if you're calling it from somewhere else. Try running it directly in the SOAPpy directory because Python adds your current working directory to sys.path. For example:
cd C:\Users\eclaird\Download\SOAPpy\ # The folder with setup.py
python.exe setup.py install

Thanks for the help.
I eventually found what the issue was. I had to get some other required modules for the install to work, also the tar file I got was missing directories, so I found the completed version on Github.

changing virtualenv folder on windows

after a computer fix my python projects dir (windows) changed (say from d: to f:).
now all my virtualenvs are broken. after activating the env the project inside the virtualenv can't find the dependencies and the custom scripts (from the env\scripts folder)won't work
tried running:
virtualenv --relocateble ENV_NAME (with the env name ..)
like in this stackoverflow question and it outputted a lot of lines like:
Script agent\Scripts\deactivate.bat cannot be made relative
and my virtualenv is still broken.
when I manually changed activate.bat set VIRTUAL_ENV to the new path. some scripts work again. but the relocate scripts still doesn't run and most of the scripts are still broken
even running the python interpeter fails with:
Traceback (most recent call last):
File "F:\Python27\learn\agent\agent\lib\site.py", line 677, in <module>
main()
File "F:\Python27\learn\agent\agent\lib\site.py", line 666, in main
aliasmbcs()
File "F:\Python27\learn\agent\agent\lib\site.py", line 506, in aliasmbcs
import locale, codecs
File "F:\Python27\learn\agent\agent\lib\locale.py", line 19, in <module>
import functools
ImportError: No module named functools
is there any way to fix this? HELP
Update: I also changed manually the shebang python interpeter line in all scripts in ENV\Scripts. now all fail with the same python failure as above
Another Update: to #udi the system python path is:
['', 'C:\\dev\\Python27\\lib\\site-packages\\distribute-0.6.37-py2.7.egg', 'C:\\
dev\\Python27\\lib\\site-packages\\pip-1.3.1-py2.7.egg', 'C:\\dev\\Python27\\lib
\\site-packages\\numpy-1.7.1-py2.7-win32.egg', 'C:\\dev\\Python27\\lib\\site-pac
kages\\pandas-0.11.0-py2.7-win32.egg', 'C:\\dev\\Python27\\lib\\site-packages\\p
ytz-2013b-py2.7.egg', 'C:\\dev\\Python27\\lib\\site-packages\\python_dateutil-2.
1-py2.7.egg', 'C:\\dev\\Python27\\lib\\site-packages\\six-1.3.0-py2.7.egg', 'C:\
\dev\\Python27\\lib\\site-packages\\tornado-3.0.1-py2.7.egg', 'C:\\dev\\Python27
\\lib\\site-packages\\pyzmq-13.1.0-py2.7-win32.egg', 'C:\\dev\\Python27\\lib\\si
te-packages\\pygments-1.6-py2.7.egg', 'C:\\Windows\\system32\\python27.zip', 'C:
\\dev\\Python27\\DLLs', 'C:\\dev\\Python27\\lib', 'C:\\dev\\Python27\\lib\\plat-
win', 'C:\\dev\\Python27\\lib\\lib-tk', 'C:\\dev\\Python27', 'C:\\dev\\Python27\
\lib\\site-packages', 'C:\\dev\\Python27\\lib\\site-packages\\setuptools-0.6c11-
py2.7.egg-info']
since I can't run python from the virtualenv, I can't print the python path from there

Correcting python directory path in ENV_FOLDER\Lib\orig-prefix.txt helped me

Seems like your system and local environments create a mix of libraries and binaries from different versions of python.
Chances are you would need to delete Lib, Scripts and Include and start again with virtualenv .. You might be able to save the site-packages folder, but if you have requirements.txt files, you should probably reinstall packages instead (see also: How do I install from a local cache with pip? ).
Anyway, I believe you can create a script that does all this in one step.

I installed both py2 and py3 on my windows 10. And got this error by create virtualenv by using virtualenv xxx directly. After purging folder xxx and reinstalling with virtualenv -p TARGET_PY_EXE xxx everything works smoothly.
Hope this will help multiple python windows users.
By the way, I simply create env variables as PY2 and PY3 instead of adding absolute paths into PATH.