Develop Reference

Connecting Python with Google SQL

I need to connect the Python script with google sql.
I have created the database in google sql and now want to connect it with python script.
I am trying to use this so far:
conn = pymysql.connect(host='ip.ip.ipi.ip',user='user', passwd = "passwd", db = 'mysql',
charset = 'utf8')
cur = conn.cursor()
cur.execute("USE Example_Database")
but getting this error:
OperationalError: (2003, "Can't connect to MySQL server on '35.246.235.61' (timed out)")
Also tried to use this but not working:
from google.cloud.sql.connector import connector
connector.connect(
"instance:instancename",
"mysql-connector",
host='ip.ip.ipi.ip',
user="user",
password="passwd",
db="Database"
)

It seems there is connectivity issue to Google SQL from where you run the code.
Where are you running your python code?
Can you check if it can connec to you Google SQL endpoint on port 3306 (or custom port you are using)?
telnet <Google_SQL_IP> 3306

Depending on where your python code is hosted you need to allow necessary access for your app server to cloud sql.
You can connect to a Cloud SQL instance using the following methods:
By using the proxy (Second Generation instances only)
By configuring access for one or more public IP addresses
By using the JDBC Socket Factory (for the Java programming language, Second Generation instances only)
By using the Cloud SQL Proxy library (for the Go programming language, Second Generation instances only)
Please review the following link which describes options for you to connect:
https://cloud.google.com/sql/docs/mysql/connect-external-app

Cannot connect to GearHost database

I am trying to connect to my GearHost Database in python, I followed the instructions here on GearHost. My python code looks like:
from mysql.connector import connection
server = 'den1.mssql6.gear.host'
db = 'testmenow'
user = 'testmenow'
psword = 'TEST!!'
cnxn = connection.MySQLConnection(host=server, user=usr, password=psword, database=db)
cursor = cnxn.cursor()
cnxn.close()
I get the following error:
mysql.connector.errors.InterfaceError: 2003: Can't connect to MySQL server on 'den1.mssql6.gear.host:3306' (10061 No connection could be made because the target machine actively refused it)
I have also tried to connect to GearHost through mysql workbench, as instructed in GearHost's instruction page, which also cannot connect to the database:
Connecting to MySQL server ...
Can't connect to MySQL server on 'den1.mssql6.gear.host' (10060)
This is what my GearHost shows:
Tried (unsuccessfully)
Messing with and without a port number, as mentioned here: Issue connecting to gearhost database
Trying connecting through MySQL workbench
Considering the conversation here: Cannot Connect to Database Server mysql workbench , but does not seem applicable to my circumstances.
Different SQL packages in python, including mysql-python and mysql-connector-python
Question
Am I missing something obvious that is preventing me from connecting to GearHost, either with python or mysql workbench?
UPDATE
as #SMor pointed out, I had mistaken MSSQL for MySQL - I was able to successfully connect to my database using:
import pyodbc
server = 'den1.mssql6.gear.host'
db = 'testmenow'
usr = 'testmenow'
psword = 'TEST!!'
cnxn = pyodbc.connect('DRIVER={ODBC Driver 13 for SQL Server};SERVER='+server+';'
'DATABASE='+db+';UID='+usr+';PWD=' + psword)
cursor = cnxn.cursor()

Error Can't use pyhive to connect to Hive Database

This is the code use to connect to our hive database , it ran fine a week back but now it seems to be be failing to even open a session and get a cursor to execute the queries. The issue was temporarily fixed when i explicitly added a cursor.close() method but now it's back again. I am unable to access the hive database using python
I have tried using pyhs2 and pyhive both the libraries fail to connect to the hive database. Nothing on the cluster has changed so far. what could the reason for this be ?
I know Hive is not a relational DB so the concept of cursors doesn't make sense but Is there any way the the Hive database remembers the cursors created using the pyhive library ?? if so how can i delete the currently unused cursors ??
Here is the code and exception it raises upon execution
from pyhive import hive
import contextlib
class Hive():
def __init__(self,host="[hostnamehere]",db="default",port="10000",auth="KERBEROS",kerberos_service_name="hive"):
self.host = host
self.db = db
self.port = port
self.auth = auth
self.kerberos_service_name = kerberos_service_name
def connect(self):
return(hive.connect(host=self.host, port=self.port, database=self.db, auth=self.auth, kerberos_service_name=self.kerberos_service_name))
def query_one(self,sql):
with contextlib.closing(self.connect()) as connection:
with contextlib.closing(connection.cursor()) as cursor:
cursor.execute(sql)
result = cursor.fetch_one()
cursor.close()
return(result)
if __name__ == "__main__":
connector = Hive()
print("running query")
print(connector.query_one("SELECT * FROM [tablenamehere]"))
raise OperationalError(response)
pyhive.exc.OperationalError: TOpenSessionResp(status=TStatus(statusCode=3, infoMessages=['*org.apache.hive.service.cli.HiveSQLException:Failed to open new session: java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient:13:12', 'org.apache.hive.service.cli.session.SessionManager:openSession:SessionManager.java:289', 'org.apache.hive.service.cli.CLIService:openSession:CLIService.java:199', 'org.apache.hive.service.cli.thrift.ThriftCLIService:getSessionHandle:ThriftCLIService.java:427', 'org.apache.hive.service.cli.thrift.ThriftCLIService:OpenSession:ThriftCLIService.java:319', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession:getResult:TCLIService.java:1257', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession:getResult:TCLIService.java:1242', 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39', 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39', 'org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor:process:HadoopThriftAuthBridge.java:562', 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:286', 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1149', 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:624', 'java.lang.Thread:run:Thread.java:748', '*java.lang.RuntimeException:java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient:15:2', 'org.apache.hadoop.hive.ql.session.SessionState:start:SessionState.java:547', 'org.apache.hive.service.cli.session.HiveSessionImpl:open:HiveSessionImpl.java:144', 'org.apache.hive.service.cli.session.SessionManager:openSession:SessionManager.java:281', '*java.lang.RuntimeException:Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient:21:6', 'org.apache.hadoop.hive.metastore.MetaStoreUtils:newInstance:MetaStoreUtils.java:1566', 'org.apache.hadoop.hive.metastore.RetryingMetaStoreClient::RetryingMetaStoreClient.java:92', 'org.apache.hadoop.hive.metastore.RetryingMetaStoreClient:getProxy:RetryingMetaStoreClient.java:138', 'org.apache.hadoop.hive.metastore.RetryingMetaStoreClient:getProxy:RetryingMetaStoreClient.java:110', 'org.apache.hadoop.hive.ql.metadata.Hive:createMetaStoreClient:Hive.java:3510', 'org.apache.hadoop.hive.ql.metadata.Hive:getMSC:Hive.java:3542', 'org.apache.hadoop.hive.ql.session.SessionState:start:SessionState.java:528', '*java.lang.reflect.InvocationTargetException:null:25:4', 'sun.reflect.NativeConstructorAccessorImpl:newInstance0:NativeConstructorAccessorImpl.java:-2', 'sun.reflect.NativeConstructorAccessorImpl:newInstance:NativeConstructorAccessorImpl.java:62', 'sun.reflect.DelegatingConstructorAccessorImpl:newInstance:DelegatingConstructorAccessorImpl.java:45', 'java.lang.reflect.Constructor:newInstance:Constructor.java:423', 'org.apache.hadoop.hive.metastore.MetaStoreUtils:newInstance:MetaStoreUtils.java:1564', '*org.apache.hadoop.hive.metastore.api.MetaException:GC overhead limit exceeded:30:4', 'org.apache.hadoop.hive.metastore.RetryingHMSHandler::RetryingHMSHandler.java:82', 'org.apache.hadoop.hive.metastore.RetryingHMSHandler:getProxy:RetryingHMSHandler.java:91', 'org.apache.hadoop.hive.metastore.HiveMetaStore:newRetryingHMSHandler:HiveMetaStore.java:6463', 'org.apache.hadoop.hive.metastore.HiveMetaStoreClient::HiveMetaStoreClient.java:206', 'org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient::SessionHiveMetaStoreClient.java:76', '*java.lang.OutOfMemoryError:GC overhead limit exceeded:0:-1'], sqlState=None, errorCode=0, errorMessage='Failed to open new session: java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient'), serverProtocolVersion=7, sessionHandle=None, configuration=None)

Connecting Hive remote server from Python

I have a trouble connecting to hive running on remote server through my python script.
I'm using the same script (With different server details, of course) to connect to hive running on my localhost & am able to connect.
I'm starting the server on local host from command line with a command:
hive —service hiveserver2
That start the server and I run the python script
Script to connect to Hive running on local host:
import pyhs2
conn = pyhs2.connect(host='localhost', port=10000, authMechanism='PLAIN', user='hive', password ='', database='default')
with conn.cursor() as cur:
cur.execute("show databases")
for i in cur.fetch():
print i
Using above code, am able to access db # Hive on local host.
I'm using below code to connect to remote server, here I'm not doing anything on command line to start the remote server.
Script to connect to Hive running on the remote server:
conn = pyhs2.connect(host='<my remote server Ip>', port=<port no>, authMechanism='PLAIN', user='<usernameToConnectToRemoteServer>', password ="<remoteServerPassword>" database='default')
with conn.cursor() as cur:
cur.execute("show databases")
for i in cur.fetch():
print i
and this returns me a message:
thrift.transport.TTransport.TTransportException: TSocket read 0 bytes.
I've tried to google & find the solution, as much as i can, but all I see are the examples to connect to local host. please help me resolve this.

Try doing SSH to your remote machine and then connect to hive like below-
import paramiko
import traceback
def hive_query_executor():
dns_name = ''
conn_obj = paramiko.SSHClient()
conn_obj.set_missing_host_key_policy(paramiko.AutoAddPolicy())
try:
conn_obj.connect(dns_name, username="hadoop",
key_filename='')# or password
Hive_query="select * from abc limit 10;"
query_execute_command = 'ihive -e "' + impala_query + '"'
std_in, std_out, std_err = conn_obj.exec_command(query_execute_command)
conn_obj.close()
except:
print "Error :" + str(traceback.format_exc())
exit(0)
hive_query_executor()

Connection to external MySQL instance using Python

I have tried the following code:
MySQLdb.connect(host='xxx', port=3306, user='yyy')
But I get the following error:
(2005, "Unknown MySQL server host ...
I have tried to remove all firewall restrictions on the external MySQL instance, as a test. I am able to connect to the instance from my developing machine.
I believe this should be possible now that the App Engine supports sockets, or am I wrong?

I think this connection is not allowed (no external socket support in MySQLdb) :
https://developers.google.com/appengine/docs/python/cloud-sql/?hl=en#Python_Using_a_local_MySQL_instance_during_development
You have to use localhost/127.0.0.1 or CloudSQL socket (unix_socket='/cloudsql/'):
if (os.getenv('SERVER_SOFTWARE') and
os.getenv('SERVER_SOFTWARE').startswith('Google App Engine/')):
db = MySQLdb.connect(unix_socket='/cloudsql/' + _INSTANCE_NAME, db='guestbook', user='root')
else:
db = MySQLdb.connect(host='127.0.0.1', port=3306, user='root')
# Alternately, connect to a Google Cloud SQL instance using:
# db = MySQLdb.connect(host='ip-address-of-google-cloud-sql-instance', port=3306, user='root')