Develop Reference

Increasing performance in connection to Teradata

I need to connect to Teradata using Python. The problem I am facing is the long time to connect to database.
I am trying with teradata, teradatasql and pyodbc packages in Python. The code I am using is something like:
import teradata
import teradatasql
import pyodbc
import time
udaExec = teradata.UdaExec (appName="HelloWorld", version="1.0", logConsole=False)
try:
host, username, password = 'teradata', 'xxxxx', 'xxxxx'
tic = time.perf_counter()
session = teradatasql.connect(host=host, user=username, password=password, logmech="LDAP")
toc = time.perf_counter()
print(f"TERADATASQL: {toc - tic:0.4f} seconds")
tic = time.perf_counter()
udaExec.connect(method="odbc", dsn="TD")
toc = time.perf_counter()
print(f"TERADATA: {toc - tic:0.4f} seconds")
tic = time.perf_counter()
connection = pyodbc.connect('DSN=TD')
toc = time.perf_counter()
print(f"PYODBC: {toc - tic:0.4f} seconds")
except Exception as e:
print(e)
The usual result is something around the following values:
TERADATASQL: 6.2150 seconds
TERADATA: 2.8512 seconds
PYODBC: 2.6051 seconds
The problem is that most part of time I´ll make very simple queries that costs much less than 1 second.
I developed (by myself) a scheme using a pool of 5 pre opened connections. But I think this is very rudimentar, since it is a django serving a REST API service with multiple users. Are there any way to increase the performance to open connections? Any other solutions?

to me it looks like you compare LDAP authentification (TERADATASQL) with the default -authentification TD2 (UDAExec & PyODBC). To compare it completely the ODBC.ini would be useful to know the details of DSN being used.
Beside this there is nothing I'm aware you can do to speed up the connection step, except eliminating the step by using some open connections. Although even 2 secs are quite slow to me, whereas I would check the networking / routing to optimize this.
As you mention it's done to include Teradata to REST Calls, did you try to connect directly to Teradata via REST? (Called "Teradata QueryService")

Google pubsub late acknowledgement

I have an app deployed on GKE, separated in different microservices.
One of the microservices, let's call it "worker", receives tasks to execute from pubsub messages.
The tasks can take up to 1 hour to be executed. The regular acknowledgement deadline for Google pubsub messages being pretty short, we renew the deadline every 10s before it ends. Here is the piece of code responsible for that:
def watchdog(businessDoneEvent, subscription, ack_deadline, message, ack_id):
'''
Prevents message from being republished as long as computation is
running
'''
while True:
# Wait (defaultDeadline - 10) seconds before renewing if defaultDeadline
# is > 5 seconds; renewed every second otherwise
sleepTime = ack_deadline - 10 if ack_deadline > 10 else 1
startTime = time.time()
while time.time() - startTime < sleepTime:
LOGGER.info('Sleeping time: {} - ack_deadline: {}'.format(time.time() - startTime, ack_deadline))
if businessDoneEvent.isSet():
LOGGER.info('Business done!')
return
time.sleep(1)
subscriber = SubscriberClient()
LOGGER.info('Modifying ack deadline for message ' +
str(message.data) + ' processing to ' +
str(ack_deadline))
subscriber.modify_ack_deadline(subscription, [ack_id],
ack_deadline)
Once the execution is over, we reach this piece of code:
def callbackWrapper(callback,
subscription,
message,
ack_id,
endpoint,
context,
subscriber,
postAcknowledgmentCallback=None):
'''
Pub/sub message acknowledgment if everything ran correctly
'''
try:
callback(message.data, endpoint, context, **message.attributes)
except Exception as e:
LOGGER.info(message.data)
LOGGER.error(traceback.format_exc())
raise e
else:
LOGGER.info("Trying to acknowledge...")
my_retry = Retry(predicate=if_exception_type(ServiceUnavailable), deadline=3600)
subscriber.acknowledge(subscription, [ack_id], retry=my_retry)
LOGGER.info(str(ack_id) + ' has been acknowledged')
if postAcknowledgmentCallback is not None:
postAcknowledgmentCallback(message.data,
**message.attributes)
Note that we also use this code in most of our microservices and it works just fine.
My problem is, even though I do not get any error from this code and it seems that the acknowledgement request is sent properly, it is actually acknowledged later. For example, according to the GCP console, right now I have 8 unacknowledged messages, but I should only have 3. It also said there are 12 when I should only have 5 for an hour:
I have a horizontal pod autoscaler using the pubsub metric. When the pods are done, they are not scaled down, or only 1 hour later or more. This creates some useless costs that I would like to avoid.
Does anyone have an idea about why this is happening?

python grpc deadline exceeded errors in large percentages

I'm getting a lot of deadline exceeded errors in python grpc client calling a scala grpc server.
I'm reporting metrics from both client as well as server and I have a large discrepency between server reported time vs client reported time which I don't think can be explained by network latency only (as the variance is big). The returned objects are of similar size, I would assume serialization time is negligable compared to network times.
I've set the timeout to 20ms
My client code is simple:
self.channel = grpc.insecure_channel(...)
self.stub = MyService_pb2_grpc.MyServiceStub(self.channel)
timeout = 0.02
try:
start_ms = time.time()
grpc_res = self.stub.getFoo(Request(...), timeout=timeout)
end_ms = time.time()
total_duration_ms = int((end_ms - start_ms) * 1000)
....
except Exception as e:
status_code = str(e.code()).split('.')[1]
logger.error('exception ....: %s', status_code) # around 20% deadline exceptions
My server code is reporting 5ms on average, the client code is reporting 7ms on average , but as mentioned , hitting 20% timeouts at 20ms
Is there a way to debug the root cause for this problem, i.e. lower level logging etc.?

You could try running under environment variables:
GRPC_VERBOSITY=DEBUG GRPC_TRACE=all
https://github.com/grpc/grpc/blob/master/doc/environment_variables.md

Sleep python script execution until process completes

I know this question has been asked prior, but I found no answer that addressed by particular problem.
I have a Python script to create kubernetes clusters and nodes in Azure, which takes anywhere between 5-10 mins. There's a function(get_cluster_end) to get the cluster endpoint, but this fails as the endpoint is not yet ready when this function is call. The func I wrote(wait_for_end)does not seem to be correct.
def wait_for_endpoint(timeout=None):
endpoint = None
start = time.time()
while not endpoint:
if timeout is not None and (time.time() - start > timeout):
break
endpoint = **get_cluster_end()**
time.sleep(5)
return endpoint
My main func:
def main():
create_cluster()
start = time.time()
job.set_progress("Waiting for cluster IP address...")
endpoint = wait_for_endpoint(timeout=TIMEOUT)
if not endpoint:
return ("FAILURE","No IP address returned after {} seconds".format(TIMEOUT),
"")
The script fails, because no endpoint has yet been created. How do I set the sleep after the cluster has been created and before the "wait_for_endpoint()" is called?

PyCurl request hangs infinitely on perform

I have written a script to fetch scan results from Qualys to be run each week for the purpose of metrics gathering.
The first part of this script involves fetching a list of references for each of the scans that were run in the past week for further processing.
The problem is that, while this will work perfectly sometimes, other times the script will hang on the c.perform() line. This is manageable when running the script manually as it can just be re-run until it works. However, I am looking to run this as a scheduled task each week without any manual interaction.
Is there a foolproof way that I can detect if a hang has occurred and resend the PyCurl request until it works?
I have tried setting the c.TIMEOUT and c.CONNECTTIMEOUT options but these don't seem to be effective. Also, as no exception is thrown, simply putting it in a try-except block also won't fly.
The function in question is below:
# Retrieve a list of all scans conducted in the past week
# Save this to refs_raw.txt
def getScanRefs(usr, pwd):
print("getting scan references...")
with open('refs_raw.txt','wb') as refsraw:
today = DT.date.today()
week_ago = today - DT.timedelta(days=7)
strtoday = str(today)
strweek_ago = str(week_ago)
c = pycurl.Curl()
c.setopt(c.URL, 'https://qualysapi.qualys.eu/api/2.0/fo/scan/?action=list&launched_after_datetime=' + strweek_ago + '&launched_before_datetime=' + strtoday)
c.setopt(c.HTTPHEADER, ['X-Requested-With: pycurl', 'Content-Type: text/xml'])
c.setopt(c.USERPWD, usr + ':' + pwd)
c.setopt(c.POST, 1)
c.setopt(c.PROXY, 'companyproxy.net:8080')
c.setopt(c.CAINFO, certifi.where())
c.setopt(c.SSL_VERIFYPEER, 0)
c.setopt(c.SSL_VERIFYHOST, 0)
c.setopt(c.CONNECTTIMEOUT, 3)
c.setopt(c.TIMEOUT, 3)
refsbuffer = BytesIO()
c.setopt(c.WRITEDATA, refsbuffer)
c.perform()
body = refsbuffer.getvalue()
refsraw.write(body)
c.close()
print("Got em!")

I fixed the issue myself by launching a separate process using multiprocessing to launch the API call in a separate process, killing and restarting if it goes on for longer than 5 seconds. It's not very pretty but is cross-platform. For those looking for a solution that is more elegant but only works on *nix look into the signal library, specifically SIGALRM.
Code below:
# As this request for scan references sometimes hangs it will be run in a separate thread here
# This will be terminated and relaunched if no response is received within 5 seconds
def performRequest(usr, pwd):
today = DT.date.today()
week_ago = today - DT.timedelta(days=7)
strtoday = str(today)
strweek_ago = str(week_ago)
c = pycurl.Curl()
c.setopt(c.URL, 'https://qualysapi.qualys.eu/api/2.0/fo/scan/?action=list&launched_after_datetime=' + strweek_ago + '&launched_before_datetime=' + strtoday)
c.setopt(c.HTTPHEADER, ['X-Requested-With: pycurl', 'Content-Type: text/xml'])
c.setopt(c.USERPWD, usr + ':' + pwd)
c.setopt(c.POST, 1)
c.setopt(c.PROXY, 'companyproxy.net:8080')
c.setopt(c.CAINFO, certifi.where())
c.setopt(c.SSL_VERIFYPEER, 0)
c.setopt(c.SSL_VERIFYHOST, 0)
refsBuffer = BytesIO()
c.setopt(c.WRITEDATA, refsBuffer)
c.perform()
c.close()
body = refsBuffer.getvalue()
refsraw = open('refs_raw.txt', 'wb')
refsraw.write(body)
refsraw.close()
# Retrieve a list of all scans conducted in the past week
# Save this to refs_raw.txt
def getScanRefs(usr, pwd):
print("Getting scan references...")
# Occasionally the request will hang infinitely. Launch in separate method and retry if no response in 5 seconds
success = False
while success != True:
sendRequest = multiprocessing.Process(target=performRequest, args=(usr, pwd))
sendRequest.start()
for seconds in range(5):
print("...")
time.sleep(1)
if sendRequest.is_alive():
print("Maximum allocated time reached... Resending request")
sendRequest.terminate()
del sendRequest
else:
success = True
print("Got em!")

The question is old but i will add this answer, it might help someone.
the only way to terminate a running curl after executing "perform()" is by using callbacks:
1- using CURLOPT_WRITEFUNCTION:
as stated from docs:
Your callback should return the number of bytes actually taken care of. If that amount differs from the amount passed to your callback function, it'll signal an error condition to the library. This will cause the transfer to get aborted and the libcurl function used will return CURLE_WRITE_ERROR.
the drawback with this method is curl calls the write function only when receives new data from the server, so in case of server stopped sending data curl will just keep waiting at the server side and will not receive your kill signal
2- the alternative and the best so far is using progress callback:
the beauty of progress callback is being curl will call it at least once per seconds even if no data coming from the server which will give you the opportunity to return non zero value as a kill switch to curl
use option CURLOPT_XFERINFOFUNCTION,
note it is better than using CURLOPT_PROGRESSFUNCTION as quoted in docs:
We encourage users to use the newer CURLOPT_XFERINFOFUNCTION instead, if you can.
also you need to set option CURLOPT_NOPROGRESS
CURLOPT_NOPROGRESS must be set to 0 to make this function actually get called.
This is an example to show you both write and progress functions implementations in python:
# example of using write and progress function to terminate curl
import pycurl
open('mynewfile', 'w') as f # used to save downloaded data
counter = 0
# define callback functions which will be used by curl
def my_write_func(data):
"""write to file"""
f.write(data)
counter += len(data)
# an example to terminate curl: tell curl to abort if the downloaded data exceeded 1024 byte by returning -1 or any number
# not equal to len(data)
if counter >= 1024:
return -1
def progress(*data):
"""it receives progress from curl and can be used as a kill switch
Returning a non-zero value from this callback will cause curl to abort the transfer
"""
d_size, downloaded, u_size, uploade = data
# an example to terminate curl: tell curl to abort if the downloaded data exceeded 1024 byte by returning non zero value
if downloaded >= 1024:
return -1
# initialize curl object and options
c = pycurl.Curl()
# callback options
c.setopt(pycurl.WRITEFUNCTION, my_write_func)
self.c.setopt(pycurl.NOPROGRESS, 0) # required to use a progress function
self.c.setopt(pycurl.XFERINFOFUNCTION, self.progress)
# self.c.setopt(pycurl.PROGRESSFUNCTION, self.progress) # you can use this option but pycurl.XFERINFOFUNCTION is recommended
# put other curl options as required
# executing curl
c.perform()