Develop Reference

sp_execute_external_script can't find modules installed by setuptools

I am actively developing a Python module that I would like to deploy in SQL Server 2017 installed locally, so I deploy the module in c:\Program Files\Microsoft SQL Server\<Instance Name>\PYTHON_SERVICES\Lib\site-packagesusing setuptoolslike so:
"c:\Program Files\Microsoft SQL Server\<Instance_Name>\PYTHON_SERVICES\python" setup.py develop
This produces an .egg-info directory in my project root, and a .egg-link file in the site-packages directory mentioned above. The .egg-link file correctly points to the .egg-info directory in my project root, so it appears setuptools is working correctly.
Here's my setup.pyfor reference:
from setuptools import setup
setup(
setup_requires=['pbr'],
pbr=True,
)
And here's the corresponding setup.cfg file:
[metadata]
name = <module_name>
description = <Module Description>
description-file = README.md
description-content-type = text/markdown
[files]
package_root = py/src
Since I am just trying to make the plumbing work, I have a single python script called uploader.py in <project_root>/py/src:
#uploader.py
class Upload:
pass
With this deployment in place, I am hoping to simply import the module I just published through .egg-link into a sp_execute_external_script call like so:
execute sp_execute_external_script #language= N'Python', #script= N'from <module_name>.uploader import Upload';
However, executing this stored procedure from SSMS produces the following error message:
Msg 39004, Level 16, State 20, Line 10
A 'Python' script error occurred during execution of 'sp_execute_external_script' with HRESULT 0x80004004.
Msg 39019, Level 16, State 2, Line 10
An external script error occurred:
Error in execution. Check the output for more information.
Traceback (most recent call last):
File "<string>", line 5, in <module>
File "C:\SQL-MSSQLSERVER-ExtensibilityData-PY\MSSQLSERVER01\C08BB9A7-66B5-4B5E-AAFC-B0248EE64199\sqlindb.py", line 27, in transform
from <module_name>.uploader import Upload
ImportError: No module named '<module_name>'
SqlSatelliteCall error: Error in execution. Check the output for more information.
STDOUT message(s) from external script:
SqlSatelliteCall function failed. Please see the console output for more information.
Traceback (most recent call last):
File "C:\Program Files\Microsoft SQL Server\<Instance_Name>\PYTHON_SERVICES\lib\site-packages\revoscalepy\computecontext\RxInSqlServer.py", line 587, in rx_sql_satellite_call
rx_native_call("SqlSatelliteCall", params)
File "C:\Program Files\Microsoft SQL Server\<Instance_Name>\PYTHON_SERVICES\lib\site-packages\revoscalepy\RxSerializable.py", line 358, in rx_native_call
ret = px_call(functionname, params)
RuntimeError: revoscalepy function failed.
I have obviously redacted module_name and Instance_Name from the error message.
I tried using install command instead of develop just to make sure the .egg-link file is not a problem. install installs the .egg-info file in site-packages but I get the same error.
I also tried removing pbr from the mix, but got the same error.
Lastly, I tried adding my <project_root> to sys.path as suggested by How can I use an external python module with SQL 2017 sp_execute_external_script?, but that didn't help either.
So at this point, I don't have a clue what I might be doing wrong.
The python version is 3.5.2 and I don't think an __init__.py is needed in the project for it to qualify as a module. Inserting a blank __init__.py in py/src doesn't help either.
My pip version is 19.3.1 and setuptools version is 44.0.0 and pbr version is 5.4.4 and I have confirmed all modules are installed in the site-packages directory mentioned above.

Based on my extensive experimentation, it appears that sp_execute_external_script doesn't follow symlinks (i.e. through the.egg-link file). Therefore, development mode installations will not work, whether you use setuptools, pip, pbr or anything else.
I even tried symlinking <package_name> folder as an OS symlink. Since I am on Windows, I used mklink /D command on Command Prompt to symlink /py/src/<package_name> inside site-packages. While the command goes through correctly, and I can see the symlinked folder in File Explorer, sp_execute_external_script fails to detect the package. Which tells me that there is probably something in sp_execute_external_script code that avoids traversing symbolic links.
I wonder if there is a way to make it traverse symbolic links.
The only workable solution is to develop a package's code under its own directory, so, in my case /py/src/<package_name>. Then, before running exec sp_execute_external_script #language=N'python', #script=N'...' copy the <package_name> folder to the site-packages directory.
This is, sort of, equivalent to setup.py install, but bypasses the creation of intermediate files and directories. So I am going to stick with this simple--though odious--approach.
I am hoping somebody more knowledgeable would offer a better way to solve this problem.

ModuleNotFoundError error with PyCharm project folder recs

I am working on a project in PyCharm. The project has the following structure:
/projectRoot/
folder1/
somecode.py
utils/
__init__.py
myutils1.py
I'd want to know how I can do an import such that the import works when running the code in the pyCharm console in an interactive manner, as well as when running the code using the
python somecode.py
command in the terminal.
Currently I do:
from utils.myutils1.py import myClass
But command line I get the error:
File "somecode.py", line 10, in
from utils.myutils1 import myClass ModuleNotFoundError: No module named 'utils'
and on PyCharm:
Traceback (most recent call last): File
"/home/ubuntu/anaconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py",
line 2881, in run_code
exec(code_obj, self.user_global_ns, self.user_ns) File "", line 1, in
from utils.myutils1 import myClass ModuleNotFoundError: No module named 'utils'
Any recommendations on the proper folder structure for modules within a project, and how to import them properly?
Thanks!

To explain the answer I recreated that project structure you had
/projectRoot/
folder1/
somecode.py
utils/
__init__.py
myutils1.py
somecode.py
from utils.myutils1 import myclass
if __name__ == "__main__":
print(myclass)
myutils1.py
myclass="tarun"
Running them from pycharm works without any issues, but running them from terminal will produce below error
File "somecode.py", line XX, in <module>
from utils.myutils1 import myclass
ModuleNotFoundError: No module named 'utils'
The issue is that Pycharm does few things for you because which you don't realize why it is not working in the terminal. So before telling you what you need to, I will tell you two things that PyCharm does on its own.
Python Console
When you launch a Python Console from Pycharm, there is some code that gets executed, using preferences.
As you can see there are two options
[X] Add content roots to PYTHONPATH
[ ] Add source roots to PYTHONPATH
And then a starting script as well. So what this does is that it adds the root of your project to python's path. Which is controlled by two main ways sys.path and PYTHONPATH environment variable
If I run the below code in Python Console
>>> import sys
>>> sys.path
['/Applications/PyCharm.app/Contents/helpers/pydev',
'/Applications/PyCharm.app/Contents/helpers/pydev',
'/Users/tarun.lalwani/.virtualenvs/folderstructure27/lib/python27.zip',
'/Users/tarun.lalwani/.virtualenvs/folderstructure27/lib/python2.7', ....
'/Users/tarun.lalwani/.virtualenvs/folderstructure27/lib/python2.7/site-packages',
'/Users/tarun.lalwani/Desktop/payu/projects/folderstructure27']
As you can see '/Users/tarun.lalwani/Desktop/payu/projects/folderstructure27' is added to the Python terminal.
Python Configurations
When you configure to RUN in code using Pycharm, you have similar two options.
We can change the code of our somecode.py to below
import os
print (os.environ['PYTHONPATH'])
import sys
print (sys.path)
/Users/tarun.lalwani/Desktop/payu/projects/folderstructure27
['/Users/tarun.lalwani/Desktop/payu/projects/folderstructure27/folder1',
'/Users/tarun.lalwani/Desktop/payu/projects/folderstructure27', ....,
'/Users/tarun.lalwani/.virtualenvs/folderstructure27/lib/python2.7/site-packages']
From the output we can see that PYTHONPATH is set to current project folder.
Running from terminal
Now let's run the somecode.py from terminal with the modifications we made.
$ python somecode.py
Traceback (most recent call last):
File "somecode.py", line 2, in <module>
print (os.environ['PYTHONPATH'])
File "/Users/tarun.lalwani/.virtualenvs/folderstructure27/bin/../lib/python2.7/UserDict.py", line 40, in __getitem__
raise KeyError(key)
KeyError: 'PYTHONPATH'
So that indicates there is no PYTHONPATH when we ran it in terminal. Let us run it again by removing the print(os.environ['PYTHONPATH']) code. You will get the below output
['/Users/tarun.lalwani/Desktop/payu/projects/folderstructure27/folder1', ...
'/Users/tarun.lalwani/.virtualenvs/folderstructure27/lib/python2.7/site-packages']
Traceback (most recent call last):
File "somecode.py", line 7, in <module>
from utils.myutils1 import myclass
ImportError: No module named utils.myutils1
As you can see folder1 is added to sys.path because it is the folder containing somecode.py, but the root folder has not been added. The fix in terminal is simple, which is to set the root directory path in PYTHONPATH.
PYTHONPATH=`pwd`/.. python somcode.py
And now the code will work from terminal also.
But the way they work are different from Python Console.
IMPORTANT NOTE:
Python Console using PyCharm on remote interpreter.
If running the python console using the remote interpreter option pycharm will fail. This is because it will append the path of the local PC and not the path of the remote server.
In order to fix this problem one has to add a mapping between the local PC directory and the remote server path.

You can use utils package inside folder1 folder:
Then the code will work either way:
from utils.myutils1 import myClass

Similar error here and this appears to be working for me:
Make sure Project Interpreter is set to, for example: C:\Python36\python.exe (in my case) and not a copy somewhere or another.
'File > Settings > Project ____ > Project Interpreter'
Or, long story short, if that route isn't cooperating, can also try finding workspace.xml and manually change SDK_HOME before starting PyCharm:
<option name="SDK_HOME" value="C:\Python36\python.exe" />

Getting error "ImportError: No module named" on Heroku but not locally

I'm getting ImportError: No module named when I try to deploy my app to heroku but the app builds fine locally. Here are the logs from Heroku
Traceback (most recent call last):
File "mr_meeseeks.py", line 4, in <module>
import Helpers.Plugin_Handler as Plugin_Handler
File "/app/Helpers/Plugin_Handler.py", line 5, in <module>
from Utils.constants import Plugin_Type
ImportError: No module named 'Utils.constants'
Here is my file structure:
As far as I can tell, Utils/constants.py exists. In case it's relevant, this is a SlackBot. The rest of the code can be found here.

The Python Interpreter looks for modules under the $PYTHONPATH environment variable. It looks like you or your editor (my editor does this when I mark a directory as sources) root has added SlackBot/ to $PYTHONPATH.
I myself encountered this error when I marked a directory as sources root.
You have a few options:
Make the bot launch script append SlackBot/ to the $PYTHONPATH
Use relative imports - from ..Utils import constants
Dynamically add SlackBot/ to the sys.path variable
Also, a note on style: python classes should be CamelCase and python modules should be lowercase_with_underscores. If you have an editor like PyCharm, your editor can fix these issues automatically, and more.
PEP 8 is the official python style guide, although I recommend using a linter so these issues can be automatically detected and fixed.

In your Procfile, append Slackbot to PYTHONPATH, try adding
--pythonpath SlackBot
as a argument. This link address the exact same issue as well.

Import statement works on PyCharm but not from terminal

PyCharm 2016.2.3, Mac OS X 10.11.1, Python 3.5 (Homebrew);
I have this folder structure
project
/somepackage
/subpackage
__init__.py
bar.py
__init__.py
foo.py
foo.py:
import somepackage.subpackage.bar
print("foo")
bar.py:
print("bar")
So my expected output is
bar
foo
This works fine when run from PyCharm. However, when I run it from my terminal I get an ImportError:
$ pwd
$ /home/project (not the actual path; just omitting some personal stuff)
$ python3.5 somepackage/foo.py
File "foo.py", line 1, in <module>
import somepackage.subpackage.bar
ImportError: No module named 'somepackage'
I have found this question, which is about the same problem. However, none of the suggested solutions work for me, as I am indeed using the same Python interpreter as PyCharm does and I am currently in the folder that contains the /somepackage folder.
Does anyone have any other suggestions about how to solve this issue?

You are running foo.py like a script, but you are really using it like a module. So the proper solution is to run it as a module:
python3 -m somepackage.foo
For the record, another alternative is to edit your path like:
export PYTHONPATH=.
(Or you could put the absolute directory in there, and of course you should append any other directories that are already in your PYTHONPATH.) This is closer to what PyCharm does, but is less philosophically correct.

Setting PYTHONPATH is what makes it work, as noted above. I use the following VSCODE .env content so that it works for any project:
PYTHONPATH=${PROJ_DIR}:${PYTHONPATH}
This is essentially what PyCharm does when you check "Add Content Roots to PYTHONPATH" in your run/debug configuration. It's a helpful setting, but it spoils you because your code fails outside PyCharm.
Or, if you run in terminal, first export:
export PYTHONPATH=...
Took me days to work all this out.

i solved my problem by two steps on Linux:
first step
go to the root directory of your project and set:
export PYTHONPATH=$PATHONPATH:`pwd`
second step
run python3 -m somepackage.foo
remember Without '.py' suffix

I just had the same problem using scapy.layers.http Module,
this problem occurred on my Kali (linux-Debian) but run fine on Win-10 (after few modifications.)
packet was installed (scapy-http) correctly and the program was running in PyCharm but not as script (from terminal)
I tried solving it with reinstalling in main root, and messing with the sys.path but None have worked.
Troubleshoot & Solution
I found that it looks for the http module in:
/usr/local/lib/python3.7/dist-packages/scapy/layers/init.py
and got the ImportError:
from scapy.layers import http --->
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: cannot import name 'http' from 'scapy.layers' (/usr/local/lib/python3.7/dist-packages/scapy/layers/__init__.py)
So then I checked where scapy-http is really installed, which is module called http.py, so I just copied it to:
/usr/local/lib/python3.7/dist-packages/scapy/layers/
found the http.py file in :/usr/local/lib/python3.7/dist-packages/scapy_http/
And that did it :)!
I know its a bit glitchie but that worked!

Add a Python module to a Google App Engine project

I'm trying to use the feed-parser module for this project im working on. When I upload the files to App Engine and I run the script it comes back with the error that the there is no module named feed-parser.
So I'm wondering if and how can I install this module on App Engine, so that I can fix this error and use RSS.
Error:
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/base/data/home/apps/s~vlis-mannipulus-bot/1.391465315184045822/main.py", line 7, in <module>
import feedparser
ImportError: No module named feed parser
Development 1:
So I've tried installing the module in the lib directory i created(in this fail example i forgot the /lib at the --prefix=..). And i get PYTHONERROR as is shown in the shell. Ive done some research on python paths and the solutions i tried didn't work for me.
kevins-MacBook-Pro-2:~ KevinH$ cd /Users/KevinH/Downloads/feedparser -5.2.1
kevins-MacBook-Pro-2:feedparser-5.2.1 KevinH$ sudo python setup.py install --prefix=/Users/KevinH/Documents/Thalia\ VMbot/Thalia-VMbot/
Password:
running install
Checking .pth file support in /Users/KevinH/Documents/Thalia VMbot/Thalia-VMbot//lib/python2.7/site-packages/
/usr/bin/python -E -c pass
TEST FAILED: /Users/KevinH/Documents/Thalia VMbot/Thalia- VMbot//lib/python2.7/site-packages/ does NOT support .pth files
error: bad install directory or PYTHONPATH
You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from. The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:
/Users/KevinH/Documents/Thalia VMbot/Thalia-VMbot//lib/python2.7/site- packages/
and your PYTHONPATH environment variable currently contains:
''
Here are some of your options for correcting the problem:
* You can choose a different installation directory, i.e., one that is
on PYTHONPATH or supports .pth files
* You can add the installation directory to the PYTHONPATH environment
variable. (It must then also be on PYTHONPATH whenever you run
Python and want to use the package(s) you are installing.)
* You can set up the installation directory to support ".pth" files by
using one of the approaches described here:
https://pythonhosted.org/setuptools/easy_install.html#custom- installation-locations
Please make the appropriate changes for your system and try again.
Then i tried with the "pip" command but then i get this:
can't open file 'pip': [Errno 2] No such file or directory
According to what I have read "pip" should be a default program installed with python 2.7 and up. So to be sure i did install python3.5 and ran it with that and still get the same error. I typed this with both pythons:
kevins-MacBook-Pro-2:feedparser KevinH$ python3 pip -m install feedparse
--
Not sure if this would work, but via terminal i went to the default directory where feed parser has been installed on my system and copied it to the lib directory i made. Then I've created the config file with the following:
from google.appengine.ext import vendor
# Add any libraries installed in the "lib" folder.
vendor.add('lib')
Deployed it and im still getting the same error as above no module named feeedparser.
Apologies if im doing something stupidly wrong, im still in the learning process.

The App Engine documentation that explains how to add third party modules is here
In summary, you will need to add a folder, usually named 'lib', to the top level off your app, and then install feedparser into that folder using the commands described in the documentation. You will also need to create an appengine_config.py file as descibed in the documentation.
Note that not all third party packages can be uploaded to App Engine - those with C extensions are forbidden. Feedparser looks OK to me though.
EDIT: further comments based on edit "development1" to the question.
Your appengine_config.py looks good.
You "lib" folder should be your application folder, that is the same folder as your app.yaml and appengine_config.py files.
You need to install the feedparser package into the lib folder. The Google docs recommend that you do this by running the command
pip install -t lib feedparser
This command will install the feedparser package into your lib folder.
You need to install and run a version of pip that works with Python2.x - the Python3 version will create a version of feedparser that only runs under Python3.
If you can't install a pip for Python2 this question might provide the right solution, otherwise I'd suggest you ask a separate question about how to install feedparser into a custom install directory on a Mac.I don't have a Mac so I can't help with this.
Once you have feedparser installed in your lib folder, verify that your app works locally; in particular verify that it's using your lib/feedparser installation: try logging feedparser.__file__ after importing feedparser to check the location of the file being imported.
Once everything is working locally you should be able to upload to GAE.
So in summary, your appengine_config.py looks good, but you need to make sure that the right version of feedparser is installed into the lib folder in your app folder before uploading to App Engine.