With a SQL Server 2017 Server, using pip install we have sucess fully installed a python package. It has been running without issue for around 2 years. I am trying to get the same package installed on sql server 2019 using sqlmlutils
import sqlmlutils
connection = sqlmlutils.ConnectionInfo(server="server_name",database="database_name", uid="sa", pwd="XXXXXXXX")
sqlmlutils.SQLPackageManager(connection).install(package='PackageName',version='1.xx')
This installs the package and I can see that the package has been installed on the SQL Server.
However when I try and execute code using that package I get the following error;
Msg 39004, Level 16, State 20, Line 0
A 'Python' script error occurred during execution of 'sp_execute_external_script' with HRESULT 0x80004004.
Msg 39019, Level 16, State 2, Line 0
An external script error occurred:
Error in execution. Check the output for more information.
Traceback (most recent call last):
File "<string>", line 5, in <module>
File "C:\ProgramData\XXX\Temp-PY\Appcontainer1\3A5D9F0F-2471-41C2-919A-9D9AAB5D199D\sqlindb_0.py", line 49, in transform
import Q as q
File "C:/Program Files/Microsoft SQL Server/MSSQL15.XXX/MSSQL/ExternalLibraries/8/2/1\Q\__init__.py", line 21, in <module>
from .Q import *
File "C:/Program Files/Microsoft SQL Server/MSSQL15.XXX/MSSQL/ExternalLibraries/8/2/1\Q\Q.py", line 13, in <module>
from . import _Q
ImportError: cannot import name '_Q' from 'Q' (C:/Program Files/Microsoft SQL Server/MSSQL15.XXX/MSSQL/ExternalLibraries/8/2/1\Q\__init__.py)
SqlSatelliteCall error: Error in execution. Check the output for more information.
There seems to be something fundamentally different in the way sqlmlutils works to pip install.
I have tried to change the premissions to allow both the server and the launchapd permissions to the Temp-PY directories, but no change in behavour.
Another issue that might be related - In another python script, I am trying output a PDF file to the c:\temp\ directory. I have change the permissions of the directory to allow the launcpad and server access to the directory, but still get the error 'Permission denied: 'c:\\temp\\sql_server_chart.pdf'.
This is very frustrating as it is a clean install on a new system.
Any help would be appreciated.
UPDATE: I installed the library using pip install on the SQL Server 2019 server and the package works correctly. This is definatly and issue with sqlmlutils.
Related
I have a bunch of local Python scripts that I need to execute from a local MS SQL Server. There is no problem if I try to import standard Python libraries (for example, pandas). I get an error, when I try to import a Python file, called Simulator.py located in the C:/Users/amusaeva/PyCharmProjects/ARW/WorkforceModel folder.
EXEC sp_execute_external_script #language =N'Python',
#script=N'
import sys
sys.path.insert(0, "C:/Users/amusaeva/PyCharmProjects/ARW/WorkforceModel")
import Simulator
'
Here is the error message I get when running this SQL script:
Msg 39004, Level 16, State 20, Line 0
A 'Python' script error occurred during execution of 'sp_execute_external_script' with HRESULT 0x80004004.
Msg 39019, Level 16, State 2, Line 0
An external script error occurred:
Error in execution. Check the output for more information.
Traceback (most recent call last):
File "", line 5, in
File "C:\ProgramData\MSSQLSERVER\Temp-PY\Appcontainer1\D4294516-2993-475D-9F61-DF7C5AF4FE69\sqlindb_0.py", line 35, in transform
import Simulator
ModuleNotFoundError: No module named 'Simulator'
SqlSatelliteCall error: Error in execution. Check the output for more information.
STDOUT message(s) from external script:
SqlSatelliteCall function failed. Please see the console output for more information.
Traceback (most recent call last):
File "C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\PYTHON_SERVICES\lib\site-packages\revoscalepy\computecontext\RxInSqlServer.py", line 605, in rx_sql_satellite_call
rx_native_call("SqlSatelliteCall", params)
File "C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\PYTHON_SERVICES\lib\site-packages\revoscalepy\RxSerializable.py", line 375, in rx_native_call
ret = px_call(functionname, params)
RuntimeError: revoscalepy function failed.
I know there is definitely a Python script called Simulator.py in that WorkforceModel directory and I don't think I have any syntax mistakes because when I run the exact three lines in Python through a PS window, the module gets imported with no problem. Why does my stored procedure not see the Python script?
It turns out that I needed to sort out some permissions. I used this article as an inspiration: https://www.red-gate.com/simple-talk/sql/data-science-sql/sql-server-machine-learning-2019-working-with-security-changes/ and here are the exact steps I followed.
The application containers are created during the SQL Server MLS setup. They are objects inside the Windows local directory, and such as all the objects in the local directory, they have a unique SID to identify the application container.
The All Application Container object has a fixed SID in the local directory, which is S-1-15-2-1 . You can use the application icacls to grant permission.
Open a command prompt as an administrator.
Run the following command:
icacls C:\Users\amusaeva\PyCharmProjects\ARW\WorkforceModel /grant *S-1-15-2-1:(OI)(CI)F /t
Open the SQL Server Configuration Manager, select ‘SQL Server Services’. Find ‘SQL Server Launchpad’, right-click on it and select ‘Restart’.
After this I ran my code with no issues (the only thing I needed to change was the slashes - using C:\\Users\\amusaeva\\PyCharmProjects\\ARW\\WorkforceModel).
I am actively developing a Python module that I would like to deploy in SQL Server 2017 installed locally, so I deploy the module in c:\Program Files\Microsoft SQL Server\<Instance Name>\PYTHON_SERVICES\Lib\site-packagesusing setuptoolslike so:
"c:\Program Files\Microsoft SQL Server\<Instance_Name>\PYTHON_SERVICES\python" setup.py develop
This produces an .egg-info directory in my project root, and a .egg-link file in the site-packages directory mentioned above. The .egg-link file correctly points to the .egg-info directory in my project root, so it appears setuptools is working correctly.
Here's my setup.pyfor reference:
from setuptools import setup
setup(
setup_requires=['pbr'],
pbr=True,
)
And here's the corresponding setup.cfg file:
[metadata]
name = <module_name>
description = <Module Description>
description-file = README.md
description-content-type = text/markdown
[files]
package_root = py/src
Since I am just trying to make the plumbing work, I have a single python script called uploader.py in <project_root>/py/src:
#uploader.py
class Upload:
pass
With this deployment in place, I am hoping to simply import the module I just published through .egg-link into a sp_execute_external_script call like so:
execute sp_execute_external_script #language= N'Python', #script= N'from <module_name>.uploader import Upload';
However, executing this stored procedure from SSMS produces the following error message:
Msg 39004, Level 16, State 20, Line 10
A 'Python' script error occurred during execution of 'sp_execute_external_script' with HRESULT 0x80004004.
Msg 39019, Level 16, State 2, Line 10
An external script error occurred:
Error in execution. Check the output for more information.
Traceback (most recent call last):
File "<string>", line 5, in <module>
File "C:\SQL-MSSQLSERVER-ExtensibilityData-PY\MSSQLSERVER01\C08BB9A7-66B5-4B5E-AAFC-B0248EE64199\sqlindb.py", line 27, in transform
from <module_name>.uploader import Upload
ImportError: No module named '<module_name>'
SqlSatelliteCall error: Error in execution. Check the output for more information.
STDOUT message(s) from external script:
SqlSatelliteCall function failed. Please see the console output for more information.
Traceback (most recent call last):
File "C:\Program Files\Microsoft SQL Server\<Instance_Name>\PYTHON_SERVICES\lib\site-packages\revoscalepy\computecontext\RxInSqlServer.py", line 587, in rx_sql_satellite_call
rx_native_call("SqlSatelliteCall", params)
File "C:\Program Files\Microsoft SQL Server\<Instance_Name>\PYTHON_SERVICES\lib\site-packages\revoscalepy\RxSerializable.py", line 358, in rx_native_call
ret = px_call(functionname, params)
RuntimeError: revoscalepy function failed.
I have obviously redacted module_name and Instance_Name from the error message.
I tried using install command instead of develop just to make sure the .egg-link file is not a problem. install installs the .egg-info file in site-packages but I get the same error.
I also tried removing pbr from the mix, but got the same error.
Lastly, I tried adding my <project_root> to sys.path as suggested by How can I use an external python module with SQL 2017 sp_execute_external_script?, but that didn't help either.
So at this point, I don't have a clue what I might be doing wrong.
The python version is 3.5.2 and I don't think an __init__.py is needed in the project for it to qualify as a module. Inserting a blank __init__.py in py/src doesn't help either.
My pip version is 19.3.1 and setuptools version is 44.0.0 and pbr version is 5.4.4 and I have confirmed all modules are installed in the site-packages directory mentioned above.
Based on my extensive experimentation, it appears that sp_execute_external_script doesn't follow symlinks (i.e. through the.egg-link file). Therefore, development mode installations will not work, whether you use setuptools, pip, pbr or anything else.
I even tried symlinking <package_name> folder as an OS symlink. Since I am on Windows, I used mklink /D command on Command Prompt to symlink /py/src/<package_name> inside site-packages. While the command goes through correctly, and I can see the symlinked folder in File Explorer, sp_execute_external_script fails to detect the package. Which tells me that there is probably something in sp_execute_external_script code that avoids traversing symbolic links.
I wonder if there is a way to make it traverse symbolic links.
The only workable solution is to develop a package's code under its own directory, so, in my case /py/src/<package_name>. Then, before running exec sp_execute_external_script #language=N'python', #script=N'...' copy the <package_name> folder to the site-packages directory.
This is, sort of, equivalent to setup.py install, but bypasses the creation of intermediate files and directories. So I am going to stick with this simple--though odious--approach.
I am hoping somebody more knowledgeable would offer a better way to solve this problem.
I am attempting to install this https://github.com/jordens/pyflycapture2 python binding on my Windows machine. The readme only has instructions on how to do it for Linux systems, but I imagine the library should still work.
I am able to run "python setup.py install" and it seems to successfully complete, but when I try to run "python test_flycapture2.py" I get this error:
C:\Users\clinic\Desktop\pyflycapture2>python test_flycapture2.py Traceback (most recent call last):
File "test_flycapture2.py", line 20, in <module>
import flycapture2 as fc2
File "build\bdist.win-amd64\egg\flycapture2.py", line 7, in <module>
File "build\bdist.win-amd64\egg\flycapture2.py", line 6, in __bootstrap__
ImportError: DLL load failed: The specified module could not be found.
This seems to imply that flycapture2 wasn't installed correctly. When I instead just open a python session and do "import flycapture2" I get the following error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "build\bdist.win-amd64\egg\flycapture2.py", line 7, in <module>
File "build\bdist.win-amd64\egg\flycapture2.py", line 6, in __bootstrap__
ImportError: DLL load failed: The specified module could not be found.
Have I done something wrong with the installation process or does pyflycapture2 just not work with Windows?
A dumb solution that's worth a try. There are chances that the DLL are searched directly from where you're starting the python script. So if you have the dll somewhere on your computer, copy it along where you have your test_flycapture2.py.
But given the fact that the setup.py file has a whole bunch of absolute paths in there, I would not place my hopes too high. You can also try to install FlyCapture 2 at the exact same path, run python setup.py bdist_wheel (you will need to install wheel first with pip) in the pyflycapture2 folder, and see if that succeeds.
If it does, try installing the generated wheel (that will be in dist/ subfolder) by doing pip install dist/pyfly....wheel and test again.
Hope this helps
I'm answering this mostly because I found another post where the same question had been posted but the original problem was never answered.
How do I run an installed Python module on Windows?
In the comments, the original poster says that it suddenly started working. I found that the solution was simply to restart my computer. I have now attempted this on two computers and this worked for both of them.
I am having an issue which has been driving me crazy. I am trying to run a python script as a non root user but when I try to execute the script I get the following error.
Traceback (most recent call last):
File "/usr/local/lib/EdgarRenderer/src/EdgarRenderer.py", line 13, in <module>
from arelle import PythonUtil # define 2.x or 3.x string types
ImportError: cannot import name PythonUtil
Now if I execute it as the root user it runs with out a hitch. I have triple checked all permissions and all the scripts and folders access by the desired user are in fact owned by that user (with the exception of the /usr/bin/python3.3 file). This is the command I am trying to execute
su - tomcat -c '/usr/bin/python3.3 /usr/local/lib/EdgarRenderer/src/EdgarRenderer.py -c /usr/local/lib/EdgarRenderer/conf/RunEdgar.xml --xdgConfigHome=/usr/local/lib/re3/arelle'
I run the exact same script the exact same way on another server with out any errors. I am using CentOS 6.5
Thanks!
I am trying to install imgseek- the server version (http://www.imgseek.net/) to do image analysis.I am able to install all the dependencies successfully using:
sudo port install swig
sudo port install swig-python
sudo easy_install twisted
sudo port install imagemagick
sudo easy_install epydoc
Then I download isk-daemon from the downloads(isk-daemon-0.9.3.tar.gz) and build and install it. Everything runs succesfully.
But when I run iskdaemon.py from the command prompt, i get the following error:
sk-daemon : WARNING | no config file (isk-daemon.conf) found. Looked at local dir, home user dir and /etc/iskdaemon. Using defaults for everything.
root : ERROR Unable to load the C++ extension "_imgdb.so(pyd)" module.
root : ERROR See http://www.imgseek.net/isk-daemon/documents-1/compiling
Traceback (most recent call last):
File "/Users/gghanakota/anaconda/lib/python2.7/site-packages/isk_daemon-0.9.3-py2.7-macosx-10.5-x86_64.egg/imgSeekLib/ImageDB.py", line 35, in
import imgdb
File "/Users/gghanakota/anaconda/lib/python2.7/site-packages/isk_daemon-0.9.3-py2.7-macosx-10.5-x86_64.egg/imgSeekLib/imgdb.py", line 28, in
_imgdb = swig_import_helper()
File "/Users/gghanakota/anaconda/lib/python2.7/site-packages/isk_daemon-0.9.3-py2.7-macosx-10.5-x86_64.egg/imgSeekLib/imgdb.py", line 20, in swig_import_helper
import _imgdb
ImportError: dlopen(/Users/gghanakota/anaconda/lib/python2.7/site-packages/isk_daemon-0.9.3-py2.7-macosx-10.5-x86_64.egg/_imgdb.so, 2): Symbol not found: __ZNSs4_Rep20_S_empty_rep_storageE
Referenced from: /Users/gghanakota/anaconda/lib/python2.7/site-packages/isk_daemon-0.9.3-py2.7-macosx-10.5-x86_64.egg/_imgdb.so
Expected in: dynamic lookup
Please help!
I had the same problem when I was trying to install iskdaemon on my Mac (osx yosemite).
The problem in my case was that when I was building it, the c++ compiler threw two errors on using min function in imgdb.cpp
The error was because the types of the variables in the min function were not the same. Consequently the build failed and the imgdb module wasn't produced.
I fixed it by adding a simple typecast to the variables passed to the min function:
I changed: min(sz, numres) to min(sz, (long int)numres) on line 1003,
and min((V.size()/2), numres) to min((int)(V.size()/2), numres) on line 1327
I built again and it is working now.
Check if you get any errors when you build, maybe you are facing the same issue.