This is my first post on stackoverflow, so please bear with me if I doing something wrong.
I'm currently trying to achieve a Python script which reads data from a CSV file, transforms it into a JSON Object and stores it in an SQL Server table. Everything is working fine if I do this directly in Python, I have a fully working Python script which reads the CSV and stores the data via pyodbc on SQL Server.
Unfortunately, when I try to use a similar script in sp_execute_external_script I get an error that the connection could not get established.
My T-SQL code:
DECLARE #Python as nvarchar(max)
SET #Python = N'
import pyodbc
import datetime as datetime
conn_str = (
r''DRIVER={ODBC Driver 17 for SQL Server};''
r''SERVER=xxx.xxx.xxx.xxx;''
r''DATABASE=xxxx;''
r''UID=xxxxxx;''
r''PWD=xxxx;''
)
cnxn = pyodbc.connect(conn_str)
'
EXEC sp_execute_external_script
#language = N'Python',
#script = #Python ,
#input_data_1 = N'',
#input_data_1_name = N''
Error message
Meldung 39004, Ebene 16, Status 20, Zeile 2 Unerwarteter
"Python"-Skriptfehler beim Ausführen von "sp_execute_external_script"
mit HRESULT 0x80004004. Meldung 39019, Ebene 16, Status 2, Zeile 2
Externer Skriptfehler:
Error in execution. Check the output for more information. Traceback
(most recent call last): File "", line 5, in File
"E:\Program Files\Microsoft SQL
Server\MSSQL14.CWDEV\MSSQL\ExtensibilityData\CWDEV01\6F73A5E0-4F82-4FEA-A5DA-7A8E7D8778D2\sqlindb.py",
line 53, in transform
cnxn = pyodbc.connect(conn_str) pyodbc.Error: ('08001', '[08001] [Microsoft][ODBC Driver 17 for SQL Server]Named Pipes-Anbieter: Es
konnte keine Verbindung zu SQL Server hergestellt werden [1326].
(1326) (SQLDriverConnect)')
SqlSatelliteCall error: Error in execution. Check the output for more
information. STDOUT-Meldung(en) aus dem externen Skript:
SqlSatelliteCall function failed. Please see the console output for
more information. Traceback (most recent call last): File
"E:\Program Files\Microsoft SQL
Server\MSSQL14.CWDEV\PYTHON_SERVICES\lib\site-packages\revoscalepy\computecontext\RxInSqlServer.py",
line 406, in rx_sql_satellite_call
rx_native_call("SqlSatelliteCall", params) File "E:\Program Files\Microsoft SQL
Server\MSSQL14.CWDEV\PYTHON_SERVICES\lib\site-packages\revoscalepy\RxSerializable.py",
line 291, in rx_native_call
ret = px_call(functionname, params) RuntimeError: revoscalepy function failed.
At the moment I'm just trying to make a connection to the destination server. Btw, the code is not running on the destination server, it will be executed on a different server. My idea is to use sp_execute_external_script with Python on a particular SQL Server to migrate data out of flat files and to store it on different destination SQL Servers.
Any advice will highly appreciated.
Many thanks
I figured it out.
There was a outgoing rule in windows firewall which blocks the network access for pyodbc connection.
firewall outgoing rules
After disabling it, everything worklike a charm.
Firewall rules for machine learning services is described here:
https://learn.microsoft.com/de-de/sql/machine-learning/security/firewall-configuration?view=sql-server-2016
Regards,
Related
What I'm trying to do is very basic: connect to an Impala db using Python:
from impala.dbapi import connect
conn = connect(host='impala', port=21050, auth_mechanism='PLAIN')
I'm using Impyla package to do so. I got this error:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/thriftpy/transport/socket.py", line 96, in open
self.sock.connect(addr)
socket.gaierror: [Errno -3] Temporary failure in name resolution
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/alaaeddine/PycharmProjects/test/data_test.py", line 3, in <module>
conn = connect(host='impala', port=21050, auth_mechanism='PLAIN')
File "/usr/local/lib/python3.6/dist-packages/impala/dbapi.py", line 147, in connect
auth_mechanism=auth_mechanism)
File "/usr/local/lib/python3.6/dist-packages/impala/hiveserver2.py", line 758, in connect
transport.open()
File "/usr/local/lib/python3.6/dist-packages/thrift_sasl/__init__.py", line 61, in open
self._trans.open()
File "/usr/local/lib/python3.6/dist-packages/thriftpy/transport/socket.py", line 104, in open
message="Could not connect to %s" % str(addr))
thriftpy.transport.TTransportException: TTransportException(type=1, message="Could not connect to ('impala', 21050)")
Tried also the Ibis package but failed with the same thriftpy related error.
In Windows using Dbeaver, I could connect to the database using the official Cloudera JDBC connector. My questions are:
Should pass my JDBC connector as parameter in my connect code? I have made some search I could not find something pointing at this direction.
Should I try something else than Ibis and Impyla packages? I had experienced a lot of version related issues and dependencies when using them. If yes, what would you recommend as alternatives?
Thanks!
Solved:
I used pyhive package instead of Ibis/Impyla. Here's an example:
#import hive from pyhive
from pyhive import hive
#establish the connection to the db
conn = hive.Connection(host='host_IP_addr', port='conn_port', auth='auth_type', database='my_db')
#prepare the cursor for the queries
cursor = conn.cursor()
#execute a query
cursor.execute("SHOW TABLES")
#navigate and display the results
for table in cursor.fetchall():
print(table)
Your impala domain name must not be resolving. Are you able to do nslookup impala in command prompt? If you're using Docker, you need to have the docker service name in docker-compose as "impala" or have "extra_hosts" option. Or you can always add it to /etc/hosts (Windows/Drivers/etc/hosts) as impala 127.0.0.1
Also try 'NOSASL' instead of PLAIN sometimes that works better with security turned off.
This is the simple method, connecting impala through impala shell using python.
import commands
import re
query1 = "select * from table_name limit 10"
impalad = str('hostname')
port = str('21000')
database = str('database_name')
result_string = 'impala-shell -i "'+ impalad+':'+port +'" -k -B --delimited -q "'+query1+'"'
status, output = commands.getstatusoutput(result_string)
print output
if status == 0:
print output
else:
print "Error encountered while executing HiveQL queries."
I need to create a linked server in MS SQL Server for 2 AWS RDS servers,
from Python 2.7. For this I use the pymssql library. The SQL commands are executed with a SQL Client without problems, the linked servers are created, they can be consulted that exist and can be used.
The problem occurs when you execute a SQL creation in Python (from an instance AWS Linux, in the same AZ), then you get the following error message.
Simple SQL queries from python (such as a Select) work without problems using pymssql.
The same happens when a stored procedure is created in the database and it called from python, if the instructions to create the linked servers are included then the same error is received. If you create the procedure in the sql in python the result is similar.
Does anyone know how to do it? Thank you very much for the help.
Linkedserver Commands:
EXEC master.dbo.sp_addlinkedserver #server = N'linkserver1', #srvproduct=N'', #provider=N'SQLNCLI', #datasrc=N'aa.bb.cc.dd';
EXEC master.dbo.sp_dropserver #server = N'linkserver1';
EXEC sys.sp_linkedservers;
Python Code:
**import pymssql
def main():
conn = pymssql.connect(server, user, clave, base)
cursor = conn.cursor()
cursor.execute("EXEC master.dbo.sp_addlinkedserver #server = N'RDSPrivate01', #srvproduct=N'', #provider=N'SQLNCLI', #datasrc=N'win-01.nnnnnn.us-west-1.rds.amazonaws.com'")
if __name__== "__main__":
main()**
Message Error:
cursor.execute("EXEC master.dbo.sp_addlinkedserver #server =
N'RDSPrivate01', #srvproduct=N'', #provider=N'SQLNCLI',
#datasrc=N'win-01.nnnnnnnnnnn.us-west-1.rds.amazonaws.com'") File
"src/pymssql.pyx", line 468, in pymssql.Cursor.execute
pymssql.OperationalError: (15002, "The procedure
'sys.sp_addlinkedserver' cannot be executed within a transaction.
DB-Lib error message 20018, severity 16:\nGeneral SQL Server error:
Check messages from the SQL Server\n")
Links used:
https://aws.amazon.com/blogs/database/implement-linked-servers-with-amazon-rds-for-microsoft-sql-server/
http://www.pymssql.org/en/stable/pymssql_examples.html#important-note-about-cursors
I need to connect to the Teradata database using python. I have used the below code:
import pyodbc
import teradata
cnxn = pyodbc.connect('DRIVER={Teradata};SERVER=<*ServerName*>;DATABASE=<*Database Name*>;UID=<*User ID*>;PWD=<*Password*>',ansi=True, autocommit=True)
cur = cnxn.cursor()
But on executing, I am getting the error as :
Error: ('28000', '[28000] [Teradata][ODBC Teradata Driver] Not enough
information to log on (0) (SQLDriverConnect); [28000] [Teradata][ODBC
Teradata Driver] Not enough information to log on (0)')
What I am missing here ? What else needs to be included to set up the connection ?
Also, is there any other way to set up the connection. While looking, I have come across teradata.UdaExec(). Can this also be used?
The following works in CentOS Linux server.
create a file with the below contents in any file (say odbc.ini)
[ODBC Data Sources]
my_data_source=tdata.so
[my_data_source]
Driver=/path/to/teradata/drivers/tdata.so
DBCName=<td_hostname>
LastUser=<user_name>
Username=<user_name>
Password=<password>
Database=<default_database>
DefaultDatabase=<default_database>
TDMSTPortNumber=<teradata_port>
set ODBCINI variable to the path of the odbc file
export ODBCINI=/file/to/path/of/odbc.ini
note: you can skip the setting of ODBCINI env variable by creating the odbc.ini file in the home directory i.e. /home/user/.odbc.ini (note that the .odbc.ini is a hidden file with a dot prefix in the file name)
now to connect to Teradata use the below snippet.
import pyodbc
pyodbc.pooling = False
conn = pyodbc.connect('DSN=my_data_source',ansi=True, autocommit=True)
I have a Python script which runs successfully from my Windows workstation and I am trying to migrate it to a Unix server. The script connects to a Teradata database using pyodbc package and executes a bunch of queries. When it is execute from the server, it triggers the following error message:
Error: ('HY000', 'The driver did not supply an error!')
I am able to consistently reproduce the error with the following code snippet executed on the server:
import pyodbc
oConnexion = pyodbc.connect("Driver={Teradata};DBCNAME=myserver;UID=myuser;PWD=mypassword", autocommit=True)
print("Connected")
oCursor = oConnexion.cursor()
oCursor.execute("select 1")
print("Success")
Configuration:
Python 3.5.2
Pyodbc 3.1.2b2
UnixODBC Driver Manager
Teradata 15.10
After enabling ODBC logging and running a simple SELECT query, I have noticed the following Invalid cursor GeTypeInfo errors:
Data Type = SQL_VARCHAR
[ODBC][57920][1481847636.278776][SQLGetTypeInfo.c][190]Error: 24000
[ODBC][57920][1481847636.278815][SQLGetTypeInfo.c][168]
Entry:
Statement = 0x1bc69e0
Data Type = Unknown(-9)
[ODBC][57920][1481847636.278839][SQLGetTypeInfo.c][190]Error: 24000
[ODBC][57920][1481847636.278873][SQLGetTypeInfo.c][168]
Entry:
Statement = 0x1bc69e0
Data Type = SQL_BINARY
[ODBC][57920][1481847636.278896][SQLGetTypeInfo.c][190]Error: 24000
Also, trying to list the connection attributes using the following code:
for attr in vars(pyodbc):
print (attr)
value = oConnexion.getinfo(getattr(pyodbc, attr))
print('{:<40s} | {}'.format(attr, value))
Fails with:
SQL_DESCRIBE_PARAMETER
Traceback (most recent call last):
File "test.py", line 28, in <module>
value = oConnexion.getinfo(getattr(pyodbc, attr))
pyodbc.Error: ('IM001', '[IM001] [unixODBC][Driver Manager]Driver does not support this function (0) (SQLGetInfo)')
Upgrading to the last (unreleased) version of pyodbc (v4) solved the issue.
https://github.com/mkleehammer/pyodbc/tree/v4
I am trying to connect to a SQL Server database using python.I have followed,
http://blog.tryolabs.com/2012/06/25/connecting-sql-server-database-python-under-ubuntu/
I have used following Python code to connect with the Microsoft SQL Server Management Studio 2014 with above setting.
import pyodbc
user='sa'
password='PC#1234'
database='climate'
port='1433'
TDS_Version='8.0'
server='192.168.1.146'
con_string= 'UID=%s;PWD=%s;DATABASE=%s;PORT=%s;TDS=%s;SERVER=%s;' %
(user,password, database,port,TDS_Version,server)
cnxn=pyodbc.connect(con_string)
cursor=cnxn.cursor()
cursor.execute("select * from mytable")
row=cursor.fetchone()
print row
I got following error,
Traceback (most recent call last):
File "sql.py", line 15, in <module>
cnxn=pyodbc.connect(con_string)
pyodbc.Error: ('IM002', '[IM002] [unixODBC][Driver Manager]Data source
name not found, and no default driver specified (0) (SQLDriverConnect)')
I also have installed pymssql and tried to connect to SQL Server. For this I have used following python code,
import pymssql
connection=pymssql.connect(user='sa',password='PC#1234',
host='192.168.1.146',database='climate',as_dict=True)
cursor=connection.cursor()
cursor.execute('select * from mytable;')
rows=cursor.fetchall()
I have got following error,
connection=pymssql.connect(user='sa',password='PC#1234',
host='192.168.1.146',database='climate',as_dict=True)
File "/usr/lib/pymodules/python2.7/pymssql.py", line 607, in connect
raise OperationalError, e[0]
pymssql.OperationalError: DB-Lib error message 20009, severity 9:
Unable to connect: Adaptive Server is unavailable or does not exist
Net-Lib error during Operation now in progress Error 115
- Operation now in progress
what is the reason for showing data source name not found and adaptive server is not available?