I have a bunch of local Python scripts that I need to execute from a local MS SQL Server. There is no problem if I try to import standard Python libraries (for example, pandas). I get an error, when I try to import a Python file, called Simulator.py located in the C:/Users/amusaeva/PyCharmProjects/ARW/WorkforceModel folder.
EXEC sp_execute_external_script #language =N'Python',
#script=N'
import sys
sys.path.insert(0, "C:/Users/amusaeva/PyCharmProjects/ARW/WorkforceModel")
import Simulator
'
Here is the error message I get when running this SQL script:
Msg 39004, Level 16, State 20, Line 0
A 'Python' script error occurred during execution of 'sp_execute_external_script' with HRESULT 0x80004004.
Msg 39019, Level 16, State 2, Line 0
An external script error occurred:
Error in execution. Check the output for more information.
Traceback (most recent call last):
File "", line 5, in
File "C:\ProgramData\MSSQLSERVER\Temp-PY\Appcontainer1\D4294516-2993-475D-9F61-DF7C5AF4FE69\sqlindb_0.py", line 35, in transform
import Simulator
ModuleNotFoundError: No module named 'Simulator'
SqlSatelliteCall error: Error in execution. Check the output for more information.
STDOUT message(s) from external script:
SqlSatelliteCall function failed. Please see the console output for more information.
Traceback (most recent call last):
File "C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\PYTHON_SERVICES\lib\site-packages\revoscalepy\computecontext\RxInSqlServer.py", line 605, in rx_sql_satellite_call
rx_native_call("SqlSatelliteCall", params)
File "C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\PYTHON_SERVICES\lib\site-packages\revoscalepy\RxSerializable.py", line 375, in rx_native_call
ret = px_call(functionname, params)
RuntimeError: revoscalepy function failed.
I know there is definitely a Python script called Simulator.py in that WorkforceModel directory and I don't think I have any syntax mistakes because when I run the exact three lines in Python through a PS window, the module gets imported with no problem. Why does my stored procedure not see the Python script?
It turns out that I needed to sort out some permissions. I used this article as an inspiration: https://www.red-gate.com/simple-talk/sql/data-science-sql/sql-server-machine-learning-2019-working-with-security-changes/ and here are the exact steps I followed.
The application containers are created during the SQL Server MLS setup. They are objects inside the Windows local directory, and such as all the objects in the local directory, they have a unique SID to identify the application container.
The All Application Container object has a fixed SID in the local directory, which is S-1-15-2-1 . You can use the application icacls to grant permission.
Open a command prompt as an administrator.
Run the following command:
icacls C:\Users\amusaeva\PyCharmProjects\ARW\WorkforceModel /grant *S-1-15-2-1:(OI)(CI)F /t
Open the SQL Server Configuration Manager, select ‘SQL Server Services’. Find ‘SQL Server Launchpad’, right-click on it and select ‘Restart’.
After this I ran my code with no issues (the only thing I needed to change was the slashes - using C:\\Users\\amusaeva\\PyCharmProjects\\ARW\\WorkforceModel).
Related
With a SQL Server 2017 Server, using pip install we have sucess fully installed a python package. It has been running without issue for around 2 years. I am trying to get the same package installed on sql server 2019 using sqlmlutils
import sqlmlutils
connection = sqlmlutils.ConnectionInfo(server="server_name",database="database_name", uid="sa", pwd="XXXXXXXX")
sqlmlutils.SQLPackageManager(connection).install(package='PackageName',version='1.xx')
This installs the package and I can see that the package has been installed on the SQL Server.
However when I try and execute code using that package I get the following error;
Msg 39004, Level 16, State 20, Line 0
A 'Python' script error occurred during execution of 'sp_execute_external_script' with HRESULT 0x80004004.
Msg 39019, Level 16, State 2, Line 0
An external script error occurred:
Error in execution. Check the output for more information.
Traceback (most recent call last):
File "<string>", line 5, in <module>
File "C:\ProgramData\XXX\Temp-PY\Appcontainer1\3A5D9F0F-2471-41C2-919A-9D9AAB5D199D\sqlindb_0.py", line 49, in transform
import Q as q
File "C:/Program Files/Microsoft SQL Server/MSSQL15.XXX/MSSQL/ExternalLibraries/8/2/1\Q\__init__.py", line 21, in <module>
from .Q import *
File "C:/Program Files/Microsoft SQL Server/MSSQL15.XXX/MSSQL/ExternalLibraries/8/2/1\Q\Q.py", line 13, in <module>
from . import _Q
ImportError: cannot import name '_Q' from 'Q' (C:/Program Files/Microsoft SQL Server/MSSQL15.XXX/MSSQL/ExternalLibraries/8/2/1\Q\__init__.py)
SqlSatelliteCall error: Error in execution. Check the output for more information.
There seems to be something fundamentally different in the way sqlmlutils works to pip install.
I have tried to change the premissions to allow both the server and the launchapd permissions to the Temp-PY directories, but no change in behavour.
Another issue that might be related - In another python script, I am trying output a PDF file to the c:\temp\ directory. I have change the permissions of the directory to allow the launcpad and server access to the directory, but still get the error 'Permission denied: 'c:\\temp\\sql_server_chart.pdf'.
This is very frustrating as it is a clean install on a new system.
Any help would be appreciated.
UPDATE: I installed the library using pip install on the SQL Server 2019 server and the package works correctly. This is definatly and issue with sqlmlutils.
I am actively developing a Python module that I would like to deploy in SQL Server 2017 installed locally, so I deploy the module in c:\Program Files\Microsoft SQL Server\<Instance Name>\PYTHON_SERVICES\Lib\site-packagesusing setuptoolslike so:
"c:\Program Files\Microsoft SQL Server\<Instance_Name>\PYTHON_SERVICES\python" setup.py develop
This produces an .egg-info directory in my project root, and a .egg-link file in the site-packages directory mentioned above. The .egg-link file correctly points to the .egg-info directory in my project root, so it appears setuptools is working correctly.
Here's my setup.pyfor reference:
from setuptools import setup
setup(
setup_requires=['pbr'],
pbr=True,
)
And here's the corresponding setup.cfg file:
[metadata]
name = <module_name>
description = <Module Description>
description-file = README.md
description-content-type = text/markdown
[files]
package_root = py/src
Since I am just trying to make the plumbing work, I have a single python script called uploader.py in <project_root>/py/src:
#uploader.py
class Upload:
pass
With this deployment in place, I am hoping to simply import the module I just published through .egg-link into a sp_execute_external_script call like so:
execute sp_execute_external_script #language= N'Python', #script= N'from <module_name>.uploader import Upload';
However, executing this stored procedure from SSMS produces the following error message:
Msg 39004, Level 16, State 20, Line 10
A 'Python' script error occurred during execution of 'sp_execute_external_script' with HRESULT 0x80004004.
Msg 39019, Level 16, State 2, Line 10
An external script error occurred:
Error in execution. Check the output for more information.
Traceback (most recent call last):
File "<string>", line 5, in <module>
File "C:\SQL-MSSQLSERVER-ExtensibilityData-PY\MSSQLSERVER01\C08BB9A7-66B5-4B5E-AAFC-B0248EE64199\sqlindb.py", line 27, in transform
from <module_name>.uploader import Upload
ImportError: No module named '<module_name>'
SqlSatelliteCall error: Error in execution. Check the output for more information.
STDOUT message(s) from external script:
SqlSatelliteCall function failed. Please see the console output for more information.
Traceback (most recent call last):
File "C:\Program Files\Microsoft SQL Server\<Instance_Name>\PYTHON_SERVICES\lib\site-packages\revoscalepy\computecontext\RxInSqlServer.py", line 587, in rx_sql_satellite_call
rx_native_call("SqlSatelliteCall", params)
File "C:\Program Files\Microsoft SQL Server\<Instance_Name>\PYTHON_SERVICES\lib\site-packages\revoscalepy\RxSerializable.py", line 358, in rx_native_call
ret = px_call(functionname, params)
RuntimeError: revoscalepy function failed.
I have obviously redacted module_name and Instance_Name from the error message.
I tried using install command instead of develop just to make sure the .egg-link file is not a problem. install installs the .egg-info file in site-packages but I get the same error.
I also tried removing pbr from the mix, but got the same error.
Lastly, I tried adding my <project_root> to sys.path as suggested by How can I use an external python module with SQL 2017 sp_execute_external_script?, but that didn't help either.
So at this point, I don't have a clue what I might be doing wrong.
The python version is 3.5.2 and I don't think an __init__.py is needed in the project for it to qualify as a module. Inserting a blank __init__.py in py/src doesn't help either.
My pip version is 19.3.1 and setuptools version is 44.0.0 and pbr version is 5.4.4 and I have confirmed all modules are installed in the site-packages directory mentioned above.
Based on my extensive experimentation, it appears that sp_execute_external_script doesn't follow symlinks (i.e. through the.egg-link file). Therefore, development mode installations will not work, whether you use setuptools, pip, pbr or anything else.
I even tried symlinking <package_name> folder as an OS symlink. Since I am on Windows, I used mklink /D command on Command Prompt to symlink /py/src/<package_name> inside site-packages. While the command goes through correctly, and I can see the symlinked folder in File Explorer, sp_execute_external_script fails to detect the package. Which tells me that there is probably something in sp_execute_external_script code that avoids traversing symbolic links.
I wonder if there is a way to make it traverse symbolic links.
The only workable solution is to develop a package's code under its own directory, so, in my case /py/src/<package_name>. Then, before running exec sp_execute_external_script #language=N'python', #script=N'...' copy the <package_name> folder to the site-packages directory.
This is, sort of, equivalent to setup.py install, but bypasses the creation of intermediate files and directories. So I am going to stick with this simple--though odious--approach.
I am hoping somebody more knowledgeable would offer a better way to solve this problem.
I have a newbie question about creating osquery extensions using osquery-python. I Created a small extension that gets some additional RPM info from my linux system. Following the instructions in the docs, I added the path to the extension in /etc/osquery/extensions.load to get it to autoload. I restarted osqueryd and I see the extension running using ps ax.
If I interactively run osqueryi, I can see the table and get data. It all works perfectly.
However, when I run an osqueryi command 'one-liner' such as :
osqueryi .tables
I get a bunch of the following errors with my output:
#INFO:thrift.transport.TSocket:Could not connect to /root/.osquery/shell.em
Traceback (most recent call last):
File "build/bdist.linux-x86_64/egg/thrift/transport/TSocket.py", line 104, in open
handle.connect(sockaddr)
File "/usr/lib64/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
error: [Errno 2] No such file or directory
ERROR:thrift.transport.TSocket:Could not connect to any of ['/root/.osquery/shell.em']
What have I done wrong?
Extensions are run in a separate process. You can see the socket errors, which indicate the extension process cannot communicate with osquery process. Make sure osqueryd or osqueryi is running. Link: osquery doc page for extensions.
I am having an issue which has been driving me crazy. I am trying to run a python script as a non root user but when I try to execute the script I get the following error.
Traceback (most recent call last):
File "/usr/local/lib/EdgarRenderer/src/EdgarRenderer.py", line 13, in <module>
from arelle import PythonUtil # define 2.x or 3.x string types
ImportError: cannot import name PythonUtil
Now if I execute it as the root user it runs with out a hitch. I have triple checked all permissions and all the scripts and folders access by the desired user are in fact owned by that user (with the exception of the /usr/bin/python3.3 file). This is the command I am trying to execute
su - tomcat -c '/usr/bin/python3.3 /usr/local/lib/EdgarRenderer/src/EdgarRenderer.py -c /usr/local/lib/EdgarRenderer/conf/RunEdgar.xml --xdgConfigHome=/usr/local/lib/re3/arelle'
I run the exact same script the exact same way on another server with out any errors. I am using CentOS 6.5
Thanks!
So I am having trouble compiling a very simple python script using JPype.
My code goes like:
from jpype import *
startJVM(getDefaultJVMPath(), "-ea")
java.lang.System.out.println("hello world")
shutdownJVM()
and when I run it I receive an error saying:
Traceback (most recent call last): File "test.py", line 2, in
<module>
startJVM(getDefaultJVMPath(), "-ea") File "/usr/lib/pymodules/python2.7/jpype/_core.py", line 44, in startJVM
_jpype.startup(jvm, tuple(args), True) RuntimeError: Unable to load DLL [/usr/java/jre1.5.0_05/lib/i386/client/libjvm.so], error =
/usr/java/jre1.5.0_05/lib/i386/client/libjvm.so: cannot open shared
object file: No such file or directory at
src/native/common/include/jp_platform_linux.h:45
I'm stuck and I really need help. Thanks!
I had the same problem
RuntimeError: Unable to load DLL [/usr/java/jre1.5.0_05/lib/i386/client/libjvm.so], error = /usr/java/jre1.5.0_05/lib/i386/client/libjvm.so: cannot open shared object file: No such file or directory at src/native/common/include/jp_platform_linux.h:45
In my case wrong JAVA_HOME path was set
/profile/etc
export JAVA_HOME
JAVA_HOME=/usr/lib/jvm/java-6-openjdk-amd64
PATH="$JAVA_HOME/bin:$PATH"
export PATH
The work around is to define the full path directly in the call to the JVM:
from jpype import *
startJVM('/Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/MacOS/libjli.dylib', "-ea", "-Djava.class.path=/tmp/Jpype/sample")
java.lang.System.out.println("Hello World!!")
shutdownJVM()
Original text:
Similar issues when trying to run JPype on MacOS El Capitan. I could
not figure out how to coax the _darwin.py code finding the correct JVM
location, despite the JAVA_HOME system variable being set properly.
Caveat cursor, trying to run the above code in the Spyder IPython console did not produce any output, but the normal Console would.