I am actively developing a Python module that I would like to deploy in SQL Server 2017 installed locally, so I deploy the module in c:\Program Files\Microsoft SQL Server\<Instance Name>\PYTHON_SERVICES\Lib\site-packagesusing setuptoolslike so:
"c:\Program Files\Microsoft SQL Server\<Instance_Name>\PYTHON_SERVICES\python" setup.py develop
This produces an .egg-info directory in my project root, and a .egg-link file in the site-packages directory mentioned above. The .egg-link file correctly points to the .egg-info directory in my project root, so it appears setuptools is working correctly.
Here's my setup.pyfor reference:
from setuptools import setup
setup(
setup_requires=['pbr'],
pbr=True,
)
And here's the corresponding setup.cfg file:
[metadata]
name = <module_name>
description = <Module Description>
description-file = README.md
description-content-type = text/markdown
[files]
package_root = py/src
Since I am just trying to make the plumbing work, I have a single python script called uploader.py in <project_root>/py/src:
#uploader.py
class Upload:
pass
With this deployment in place, I am hoping to simply import the module I just published through .egg-link into a sp_execute_external_script call like so:
execute sp_execute_external_script #language= N'Python', #script= N'from <module_name>.uploader import Upload';
However, executing this stored procedure from SSMS produces the following error message:
Msg 39004, Level 16, State 20, Line 10
A 'Python' script error occurred during execution of 'sp_execute_external_script' with HRESULT 0x80004004.
Msg 39019, Level 16, State 2, Line 10
An external script error occurred:
Error in execution. Check the output for more information.
Traceback (most recent call last):
File "<string>", line 5, in <module>
File "C:\SQL-MSSQLSERVER-ExtensibilityData-PY\MSSQLSERVER01\C08BB9A7-66B5-4B5E-AAFC-B0248EE64199\sqlindb.py", line 27, in transform
from <module_name>.uploader import Upload
ImportError: No module named '<module_name>'
SqlSatelliteCall error: Error in execution. Check the output for more information.
STDOUT message(s) from external script:
SqlSatelliteCall function failed. Please see the console output for more information.
Traceback (most recent call last):
File "C:\Program Files\Microsoft SQL Server\<Instance_Name>\PYTHON_SERVICES\lib\site-packages\revoscalepy\computecontext\RxInSqlServer.py", line 587, in rx_sql_satellite_call
rx_native_call("SqlSatelliteCall", params)
File "C:\Program Files\Microsoft SQL Server\<Instance_Name>\PYTHON_SERVICES\lib\site-packages\revoscalepy\RxSerializable.py", line 358, in rx_native_call
ret = px_call(functionname, params)
RuntimeError: revoscalepy function failed.
I have obviously redacted module_name and Instance_Name from the error message.
I tried using install command instead of develop just to make sure the .egg-link file is not a problem. install installs the .egg-info file in site-packages but I get the same error.
I also tried removing pbr from the mix, but got the same error.
Lastly, I tried adding my <project_root> to sys.path as suggested by How can I use an external python module with SQL 2017 sp_execute_external_script?, but that didn't help either.
So at this point, I don't have a clue what I might be doing wrong.
The python version is 3.5.2 and I don't think an __init__.py is needed in the project for it to qualify as a module. Inserting a blank __init__.py in py/src doesn't help either.
My pip version is 19.3.1 and setuptools version is 44.0.0 and pbr version is 5.4.4 and I have confirmed all modules are installed in the site-packages directory mentioned above.
Based on my extensive experimentation, it appears that sp_execute_external_script doesn't follow symlinks (i.e. through the.egg-link file). Therefore, development mode installations will not work, whether you use setuptools, pip, pbr or anything else.
I even tried symlinking <package_name> folder as an OS symlink. Since I am on Windows, I used mklink /D command on Command Prompt to symlink /py/src/<package_name> inside site-packages. While the command goes through correctly, and I can see the symlinked folder in File Explorer, sp_execute_external_script fails to detect the package. Which tells me that there is probably something in sp_execute_external_script code that avoids traversing symbolic links.
I wonder if there is a way to make it traverse symbolic links.
The only workable solution is to develop a package's code under its own directory, so, in my case /py/src/<package_name>. Then, before running exec sp_execute_external_script #language=N'python', #script=N'...' copy the <package_name> folder to the site-packages directory.
This is, sort of, equivalent to setup.py install, but bypasses the creation of intermediate files and directories. So I am going to stick with this simple--though odious--approach.
I am hoping somebody more knowledgeable would offer a better way to solve this problem.
Related
I'm trying to use the feed-parser module for this project im working on. When I upload the files to App Engine and I run the script it comes back with the error that the there is no module named feed-parser.
So I'm wondering if and how can I install this module on App Engine, so that I can fix this error and use RSS.
Error:
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/base/data/home/apps/s~vlis-mannipulus-bot/1.391465315184045822/main.py", line 7, in <module>
import feedparser
ImportError: No module named feed parser
Development 1:
So I've tried installing the module in the lib directory i created(in this fail example i forgot the /lib at the --prefix=..). And i get PYTHONERROR as is shown in the shell. Ive done some research on python paths and the solutions i tried didn't work for me.
kevins-MacBook-Pro-2:~ KevinH$ cd /Users/KevinH/Downloads/feedparser -5.2.1
kevins-MacBook-Pro-2:feedparser-5.2.1 KevinH$ sudo python setup.py install --prefix=/Users/KevinH/Documents/Thalia\ VMbot/Thalia-VMbot/
Password:
running install
Checking .pth file support in /Users/KevinH/Documents/Thalia VMbot/Thalia-VMbot//lib/python2.7/site-packages/
/usr/bin/python -E -c pass
TEST FAILED: /Users/KevinH/Documents/Thalia VMbot/Thalia- VMbot//lib/python2.7/site-packages/ does NOT support .pth files
error: bad install directory or PYTHONPATH
You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from. The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:
/Users/KevinH/Documents/Thalia VMbot/Thalia-VMbot//lib/python2.7/site- packages/
and your PYTHONPATH environment variable currently contains:
''
Here are some of your options for correcting the problem:
* You can choose a different installation directory, i.e., one that is
on PYTHONPATH or supports .pth files
* You can add the installation directory to the PYTHONPATH environment
variable. (It must then also be on PYTHONPATH whenever you run
Python and want to use the package(s) you are installing.)
* You can set up the installation directory to support ".pth" files by
using one of the approaches described here:
https://pythonhosted.org/setuptools/easy_install.html#custom- installation-locations
Please make the appropriate changes for your system and try again.
Then i tried with the "pip" command but then i get this:
can't open file 'pip': [Errno 2] No such file or directory
According to what I have read "pip" should be a default program installed with python 2.7 and up. So to be sure i did install python3.5 and ran it with that and still get the same error. I typed this with both pythons:
kevins-MacBook-Pro-2:feedparser KevinH$ python3 pip -m install feedparse
--
Not sure if this would work, but via terminal i went to the default directory where feed parser has been installed on my system and copied it to the lib directory i made. Then I've created the config file with the following:
from google.appengine.ext import vendor
# Add any libraries installed in the "lib" folder.
vendor.add('lib')
Deployed it and im still getting the same error as above no module named feeedparser.
Apologies if im doing something stupidly wrong, im still in the learning process.
The App Engine documentation that explains how to add third party modules is here
In summary, you will need to add a folder, usually named 'lib', to the top level off your app, and then install feedparser into that folder using the commands described in the documentation. You will also need to create an appengine_config.py file as descibed in the documentation.
Note that not all third party packages can be uploaded to App Engine - those with C extensions are forbidden. Feedparser looks OK to me though.
EDIT: further comments based on edit "development1" to the question.
Your appengine_config.py looks good.
You "lib" folder should be your application folder, that is the same folder as your app.yaml and appengine_config.py files.
You need to install the feedparser package into the lib folder. The Google docs recommend that you do this by running the command
pip install -t lib feedparser
This command will install the feedparser package into your lib folder.
You need to install and run a version of pip that works with Python2.x - the Python3 version will create a version of feedparser that only runs under Python3.
If you can't install a pip for Python2 this question might provide the right solution, otherwise I'd suggest you ask a separate question about how to install feedparser into a custom install directory on a Mac.I don't have a Mac so I can't help with this.
Once you have feedparser installed in your lib folder, verify that your app works locally; in particular verify that it's using your lib/feedparser installation: try logging feedparser.__file__ after importing feedparser to check the location of the file being imported.
Once everything is working locally you should be able to upload to GAE.
So in summary, your appengine_config.py looks good, but you need to make sure that the right version of feedparser is installed into the lib folder in your app folder before uploading to App Engine.
I am trying to build a Python Script into a stand alone application. I am using GUI2Exe. My script uses selenium package. I have it installed.
Project compiles fine and runs on python command line directly but fails to build a stand alone because it is referring to folder:
ERROR: test_file_data_extract (__main__.FileDataExtract)
----------------------------------------------------------------------
Traceback (most recent call last):
File "File_data_extract.py", line 18, in setUp
File "selenium\webdriver\firefox\firefox_profile.pyc", line 63, in __init__
IOError: [Errno 2] No such file or directory: 'C:\\users\\username\\PycharmProjects\\Python_27_32bit\\file_data_extract\\dist\\File_data_extract.exe\\selenium\\webdriver\\firefox\\webdriver_prefs.json'
It is looking for selenium package is located at :
C:\Users\username\Anaconda2_Py27_32bit\Lib\site-packages\selenium-2.48.0-py2.7.egg\selenium\webdriver\firefox
where C:\Users\username\Anaconda2_Py27_32bit is where I installed Anaconda Python 2.7, 32 bit version. By default it is looking for in \dist\filename.exe folder.
I was able to build it using bbfreeze. It works great.
First I had to install bbfreezee via pip (one time only):
pip install bbfreeze
Create a build_package.py file as:
from bbfreeze import Freezer
f = Freezer("project_name", includes=("selenium","SendKeys",)) #list problem packages here to manually include
f.addScript("project_name_script.py")
f() # starts the freezing process
Build project:
python build_package.py bdist_bbfreezee
in folder project_name where project_name_script.py sits you find project_name_script.exe with all the include packages including selenium and sendkeys. When you distribute the package you need to distribute entire project_name because it contains all dependent library dlls (python .pyd).
More details refer official bbfreezee here:
https://pypi.python.org/pypi/bbfreeze/#downloads
I'm trying to release my first Python package in the wild and I was successful in setting it up on PyPi and able to do a pip install. When I try to run the package via the command line ($ python etlTest), I receive the following error:
/usr/bin/python: can't find '__main__' module in 'etlTest'
When I run the code directly from my IDE, it works without issue. I am using Python 2.7 and have __init__.py scripts where required. What do I need to do to get this working?
I can easily replicate your problem, actually even without using your package:
$ python empty
.env/bin/python: can't open file 'empty': [Errno 2] No such file or directory
$ mkdir empty
$ python empty
.env/bin/python: can't find '__main__' module in 'empty'
$ python Empty
.env/bin/python: can't find '__main__' module in 'Empty'
So you are not calling your library at all, you are just giving the Python interpreter a nonexistent script name, which in case there is a like-named directory (case-insensitive even) in the working directory it tries to execute it.
I was able to install your package from PyPi just fine and can import it alright, but there doesn't seem to be any entry point to it, nothing useful is exported at the top level and no executable scripts are added to Python bin:
>>> import etltest
>>> dir(etltest)
['__author__', '__builtins__', '__doc__', '__file__', '__name__', '__package__',
'__path__', '__version__']
Running the etlTest.py directly as suggested in Quickstart from your docs doesn't work either:
$ python .env/lib/python2.7/site-packages/etltest/etlTest.py
2014-07-13 17:19:56,831 - settings - DEBUG - Attempting to load .env/lib/python2.7/site-packages/.etltest-settings.yml
2014-07-13 17:19:56,832 - settings - WARNING - No such file or directory .env/lib/python2.7/site-packages/.etltest-settings.yml
2014-07-13 17:19:56,832 - settings - DEBUG - Attempting to load .env/lib/python2.7/site-packages/.etltest-settings.yml
2014-07-13 17:19:56,832 - settings - WARNING - No such file or directory .env/lib/python2.7/site-packages/.etltest-settings.yml
2014-07-13 17:19:56,832 - settings - WARNING - Could not find settings file in .env/lib/python2.7/site-packages/.etltest-settings.yml,.env/lib/python2.7/site-packages/.etltest-settings.yml. Using defaults where present.
Traceback (most recent call last):
File ".env/lib/python2.7/site-packages/etltest/etlTest.py", line 73, in <module>
main(sys.argv[1:])
File ".env/lib/python2.7/site-packages/etltest/etlTest.py", line 22, in main
SettingsManager().first_run_test()
File ".env/lib/python2.7/site-packages/etltest/utilities/settings_manager.py", line 29, in __init__
self.app_name = etltest_config['app_name']
KeyError: 'app_name'
I'd say your package is not apt for ditribution yet. You might want to clean up some dependencies on your development environment and read upon setuptools entry points to provide proper command line executables along with your package. Also it shouldn't run in debug mode by default if it's not some kind of a testing release.
I recently got the same issue and finaly find the solution by my self.
So as the error said, I added a file __main__.py at the same path that my __init__.py.
Inside __main__.py I added the following code :
from mypackage.mymodule import main
main()
main() was the main function of my code. And it's works now.
here my directory:
package
|__dirpackage
|_mypackage.py
|_ __init__.py
|_ __main__.py
|_setup.py`
Just change the name __init__.py file to __main__.py
I had the same problem and solved it by making sure I'm in the correct directory of the package you are trying to run.
For Windows, type dir in the console, while on Linux/macOS - ls to see your current directory
I'm currently trying to make cx_freeze to work on a Solaris workstation I have to work with, in order to make an executable from a Python script I have. Problem is, I'm not administrator of this machine, and installation of cx_freeze requests write to site-packages, which is read-only for me.
So, obviously, I get this error:
creating /usr/local/lib/python2.6/site-packages/cx_Freeze
error: could not create '/usr/local/lib/python2.6/site-packages/cx_Freeze': Read-only file system
And if I try to run it anyway, it fails:
bash-3.00$ python /home/xxxx/cx_freeze-4.2.3/cxfreeze --target-dir cx_dist src/p_tool.py
Traceback (most recent call last):
File "/home/xxxx/cx_freeze-4.2.3/cxfreeze", line 5, in <module>
main()
File "/home/xxxx/cx_freeze-4.2.3/cx_Freeze/main.py", line 187, in main
silent = options.silent)
File "/home/xxxx/cx_freeze-4.2.3/cx_Freeze/freezer.py", line 91, in __init__
self._VerifyConfiguration()
File "/home/xxxx/cx_freeze-4.2.3/cx_Freeze/freezer.py", line 371, in _VerifyConfiguration
self._GetInitScriptFileName()
File "/home/xxxx/cx_freeze-4.2.3/cx_Freeze/freezer.py", line 283, in _GetInitScriptFileName
raise ConfigError("no initscript named %s", name)
cx_Freeze.freezer.ConfigError: no initscript named Console
Obviously, this is linked to the failed installation. So, here's my question:
Without installation of virtualenv, could I avoid the writing to site-packages, and make cx_freeze to execute from my home folder?
EDIT I had a look at site.py documentation, and PYTHONPATH filling should be equivalent to use of site-packages. So my question is now more something like: what is the path to be added to PYTHONPATH, so that cx_freeze could be executed from any location?
Notes:
I would like to avoid to deal with virtualenv, as I'm already struggling to understand the executable tools...
I saw this question, but this still requires access to site-packages folder, plus it's not user-specific;
I tried adding the following path to PYTHONPATH, but this does not work: /home/xxxx/cx_freeze-4.2.3/build/lib.solaris-2.10-sun4v-2.6;
I'm also trying to use PyInstaller but have dependency problems (and the administrator is not really helping me).
This works like a charm for me :
$ python setup.py install --home=$HOME
Run in the source directory of cx_freeze found on the Sourceforge download page.
I need to run some code on a Linux machine with Python 2.3.4
pre-installed. I'm not on the sudoers list for that machine, so I
built Python 2.6.4 into (a subdirectory in) my home directory. Then I
attempted to use virtualenv (for the first time), but got:
$ Python-2.6.4/python virtualenv/virtualenv.py ENV
New python executable in ENV/bin/python
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Installing setuptools.........
Complete output from command /apps/users/dspitzer/ENV/bin/python -c "#!python
\"\"\"Bootstrap setuptoo...
" /apps/users/dspitzer/virtualen...6.egg:
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
'import site' failed; use -v for traceback
Traceback (most recent call last):
File "<string>", line 67, in <module>
ImportError: No module named md5
----------------------------------------
...Installing setuptools...done.
Traceback (most recent call last):
File "virtualenv/virtualenv.py", line 1488, in <module>
main()
File "virtualenv/virtualenv.py", line 529, in main
use_distribute=options.use_distribute)
File "virtualenv/virtualenv.py", line 619, in create_environment
install_setuptools(py_executable, unzip=unzip_setuptools)
File "virtualenv/virtualenv.py", line 361, in install_setuptools
_install_req(py_executable, unzip)
File "virtualenv/virtualenv.py", line 337, in _install_req
cwd=cwd)
File "virtualenv/virtualenv.py", line 590, in call_subprocess
% (cmd_desc, proc.returncode))
OSError: Command /apps/users/dspitzer/ENV/bin/python -c "#!python
\"\"\"Bootstrap setuptoo...
" /apps/users/dspitzer/virtualen...6.egg failed with error code 1
Should I be setting PYTHONHOME to some value? (I intentionally named
my ENV "ENV" for lack of a better name.)
Not knowing if I can ignore those errors, I tried installing nose
(0.11.1) into my ENV:
$ cd nose-0.11.1/
$ ls
AUTHORS doc/ lgpl.txt nose.egg-info/ selftest.py*
bin/ examples/ MANIFEST.in nosetests.1 setup.cfg
build/ functional_tests/ NEWS PKG-INFO setup.py
CHANGELOG install-rpm.sh* nose/ README.txt unit_tests/
$ ~/ENV/bin/python setup.py install
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Traceback (most recent call last):
File "setup.py", line 1, in <module>
from nose import __version__ as VERSION
File "/apps/users/dspitzer/nose-0.11.1/nose/__init__.py", line 1, in <module>
from nose.core import collector, main, run, run_exit, runmodule
File "/apps/users/dspitzer/nose-0.11.1/nose/core.py", line 3, in <module>
from __future__ import generators
ImportError: No module named __future__
Any advice?
Have you actually run "make install" on your custom python build? Usually you'll want to do something like
./configure --prefix=/path/to/installdir (other options)
make
make install
Note Prefix can be any directory you have full write-permissions to, for example I very often use $HOME/apps on shared-hosting environments.
Then run /path/to/installdir/bin/python, not the one from your build directory. This should create the correct variables, and after that you can install virtualenv. Might be best to install virtualenv using its setup.py:
cd virtualenv_source_dir
/path/to/installdir/bin/python setup.py install
This may require installing setuptools first, using the same method.
Then finally:
# Just to be safe
export PATH="/path/to/installdir/bin:$PATH"
virtualenv ~/ENV
~/ENV/bin/pip install somepackage # (and such)
In addition to Crast's suggestion of making sure you actually installed your custom compiled Python, you should also check that the custom Python can actually find its libraries. This is the hint you're getting with the message about PYTHONHOME. The import errors suggest you need to set in your .bashrc or appropriate shell configuration export PYTHONHOME=/path/to/python_installation.
Additionally, when you are trying to tell virtualenv to use a non-default version of python, you need to use the -p,--python flag, e.g.,
virtualenv --python=/path/to/python_installation/bin/python myenv
See also the related question, "Use different Python version with virtualenv".
I had the same error when trying to install on an existing directory that already had easy_install in lib/python2.6. I had to put a link from lib64/python2.6 to lib/python2.6.
I am not saying my fix is the right fix, rather, I'm pointing to another reason why you might get this error.
Now you can easily install Python as an unpriviledged user using Anaconda: http://continuum.io/downloads
It's similar to this question. Once of the answers details making a new environment, so you don't need to use virtual-env and avoid the occasional gotchas: Installing Anaconda into a Virtual Environment.
conda create -n myenv1 ipython scipy
I don't have enough rep to add this as a comment on #Crast's answer and this question is 4 years old, but this might be useful to someone. In Windows, you have to path out to python.exe, but it seems that in Linux/OS X you just path to the folder. Example:
Windows:
virtualenv -p <PATH TO YOUR DESIRED PYTHON.EXE> venv
Creates a virtual environment in subfolder "venv" in current directory.