I created a web service in Flask over uwsgi. I thought I would follow good practice and create a SessionPool with 20 connections to be safe. Each call to a web service endpoint, I acquire a connection from the pool, and at the end I release it.
When using Locust to swarm test the API, I was getting hundreds of failures, nearly 100% on some of the longer responses (30Mb JSON response). Smaller payloads were much better, but with intermittent failures.
The minute I switched back to bad practice and created a brand new connection and cursor within the method itself, all my problems vanished. 100% success on 1000s of stress test calls.
My errors were varied. TNS Bad Packet, incorrect number of connections from pool, request cancelled by user....you name it, it was there.
So I can't use Oracle connection pooling with flask it seems, or have a single connection at the Flask application level (this generated errors, not sure why, which is why I switched to connection pooling).
Any advice on creating scalable apps using cx_Oracle in flask.
My original code was:
pool = cx_Oracle.SessionPool("user", "password", "myserver.company.net:1521/myservice", min=10, max=10, increment=0, getmode=cx_Oracle.SPOOL_ATTRVAL_WAIT, encoding="UTF-8")
def read_products_search(search=None):
"""
This function responds to a request for /api/products
with the complete lists of people
:return: json string of list of people
"""
conn_ariel = pool.acquire()
cursor_ariel = conn_ariel.cursor()
search=search.lower()
print("product search term is: ", search)
# Create the list of products from our data
sql = """
SELECT DRUG_PRODUCT_ID, PREFERRED_TRADE_NAME, PRODUCT_LINE, PRODUCT_TYPE, FLAG_PASSIVE, PRODUCT_NUMBER
FROM DIM_DRUG_PRODUCT
WHERE lower(PREFERRED_TRADE_NAME) LIKE '%' || :search1 || '%' or lower(PRODUCT_LINE) LIKE '%' || :search2 || '%' or lower(PRODUCT_NUMBER) LIKE '%' || :search3 || '%'
ORDER BY PREFERRED_TRADE_NAME ASC
"""
cursor_ariel.execute(sql, {"search1":search,"search2":search, "search3":search })
products = []
for row in cursor_ariel.fetchall():
r = reg(cursor_ariel, row, False)
product = {
"drug_product_id" : r.DRUG_PRODUCT_ID,
"preferred_trade_name" : r.PREFERRED_TRADE_NAME,
"product_line" : r.PRODUCT_LINE,
"product_type" : r.PRODUCT_TYPE,
"flag_passive" : r.FLAG_PASSIVE,
"product_number" : r.PRODUCT_NUMBER
}
# logging.info("Adding Product: %r", product)
products.append(product)
if len(products) == 0:
products = None
pool.release(conn_ariel)
return products
When you create the pool, use threaded=True.
See How to use Python Flask with Oracle Database.
Related
As a test I have been trying to load Ensembl gene-transcript-protein data through a python program into OrientDB (logged in as root). Nodes work fine,(transcript to protein) edges load tens of thousands of edges fine, but then give an error message like
Invalid authentication info for access to the database com.orientechnologies.orient.core.metadata.security.auth.OTokenAuthInfo#
hInfo#3b531bc6 etc
for all remaining edges
Terminal output:
.....
DB name="ensembl"
228190
com.orientechnologies.orient.core.exception.OSecurityAccessException - Invalid authentication info for access to the database com.orientechnologies.orient.core.metadata.security.auth.OTokenAuthInfo#48703c3d
DB name="ensembl"
228191
com.orientechnologies.orient.core.exception.OSecurityAccessException - Invalid authentication info for access to the database com.orientechnologies.orient.core.metadata.security.auth.OTokenAuthInfo#1c80d217
DB name="ensembl"
228192
Debug console (vscode) and inspectin of the input data show nothing unexpected
relevant section of the code:
enter code here
Create the tr2prot edge
tr2protTable=pandas.read_table(dataPath + "tr2prot.txt",sep=',')
(s1,s2)=tr2protTable.shape
for i1 in range(s1):
if i1%1000 == 0: print(i1)
row=tr2protTable.iloc[i1]
uIDfrom=(row['TranscriptStableID']).replace("'","\'")
uIDto = (row['ProteinStableID']).replace("'","\'")
commandStr=f"create edge Tr2prot from (select from Transcript where uID= '{uIDfrom}') to (select from Protein where uID = '{uIDto}')"
tr2protList.append(commandStr)
try:
tr2prot = client.command(commandStr)
except Exception as e:
print(e)
print(i1)
In case its relevant, I tried to set the session token as indicated in the manual
client.connect("root", root-password)
client.set_session_token(True)
import pyorient
client = pyorient.OrientDB("localhost", 2424) # host, port
# Reattach to Session
if sessionToken != '':
client.set_session_token(sessionToken)
else:
# Open Database
client.db_drop("ensembl")
client.db_open("ensembl", "root", root-password)
# Fetch Session Token
sessionToken = client.get_session_token()
o
If I generate a batch submission, I do not seem to have the problem, but errors, when, for some reason, the vertex is not there. Is there something like the neo4j apoc.load.csv switch failOnError:false in OrientDB?
Would also help.
Any suggestions would be great
Hans
I have a successfully compiled and run a django rest consuming cocktaildb api. On local server when I run http://127.0.0.1:8000/api/ I get
{
"ingredients": "http://127.0.0.1:8000/api/ingredients/",
"drinks": "http://127.0.0.1:8000/api/drinks/",
"feeling-lucky": "http://127.0.0.1:8000/api/feeling-lucky/"
}
But when I go to one of the links mentioned in the json result above, for example:
http://127.0.0.1:8000/api/ingredients/
I get an empty [] with a status 200OK!
I need an endpoint to GET drinks and ingredients before I can destructure to specific details using angular.
I implemented helper folder in the app with the the API function as below:
class TheCoctailDBAPI:
THECOCTAILDB_URL = 'https://www.thecocktaildb.com/api/json/v1/1/'
async def __load_coctails_for_drink(self, drink, session):
for i in range(1, 16):
ingredientKey = 'strIngredient' + str(i)
ingredientName = drink[ingredientKey]
if not ingredientName:
break
if ingredientName not in self.ingredients:
async with session.get(f'{TheCoctailDBAPI.THECOCTAILDB_URL}search.php?i={ingredientName}') \
as response:
result = json.loads(await response.text())
self.ingredients[ingredientName] = result['ingredients'][0]
What was your expected responce?
Add the function that is called by this API as well as the DB settings in the question, so that we can properly help you.
Are you sure that you are connecting and pulling data from a remote location? It looks to me like your local DB is empty, so the API has no data to return.
I am trying to listen to documents in collection from Python code using example from Firestore Documentation. I receive correct data when listening root collection, but got nothing when listening subcollection.
Here's my code:
db = firestore.client()
# Create a callback on_snapshot function to capture changes
def on_snapshot(col_snapshot, changes, read_time):
print(col_snapshot, type(col_snapshot))
print(changes, type(col_snapshot))
root_collection = u'shared-streams'
subcollection = u'shared-streams/eFC4T~lLyT/messages'
# Watch the root collection query (1)
col_query = db.collection(root_collection)
query_watch = col_query.on_snapshot(on_snapshot)
# Watch the subcollection query (2)
col_query = db.collection(subcollection)
query_watch = col_query.on_snapshot(on_snapshot)
Subcollection exists in Firestore and non-empty. But in first case (1) I got non-empty lists of elements and changes (and updates), and in other case (2) just two empty lists (and nothing when update subcollection). As I know there are no differences in root/sub- collections, so, please, explain where am I wrong.
UPD: Similar code in node.js works fine, so looks like it is error in python client library.
node.js snippet:
var db = admin.firestore();
var query = db.collection('shared-streams/eFC4T~lLyT/messages')
var observer = query.onSnapshot(querySnapshot => {
console.log(`Received query snapshot of size ${querySnapshot.size}`);
// ...
}, err => {
console.log(`Encountered error: ${err}`);
});
I recently have acquired an ACS Linear Actuator (Tolomatic Stepper) that I am attempting to send data to from a Python application. The device itself communicates using Ethernet/IP protocol.
I have installed the library cpppo via pip. When I issue a command
in an attempt to read status of the device, I get None back. Examining the
communication with Wireshark, I see that it appears like it is
proceeding correctly however I notice a response from the device indicating:
Service not supported.
Example of the code I am using to test reading an "Input Assembly":
from cpppo.server.enip import client
HOST = "192.168.1.100"
TAGS = ["#4/100/3"]
with client.connector(host=HOST) as conn:
for index, descr, op, reply, status, value in conn.synchronous(
operations=client.parse_operations(TAGS)):
print(": %20s: %s" % (descr, value))
I am expecting to get a "input assembly" read but it does not appear to be
working that way. I imagine that I am missing something as this is the first
time I have attempted Ethernet/IP communication.
I am not sure how to proceed or what I am missing about Ethernet/IP that may make this work correctly.
clutton -- I'm the author of the cpppo module.
Sorry for the delayed response. We only recently implemented the ability to communicate with simple (non-routing) CIP devices. The ControlLogix/CompactLogix controllers implement an expanded set of EtherNet/IP CIP capability, something that most simple CIP devices do not. Furthermore, they typically also do not implement the *Logix "Read Tag" request; you have to struggle by with the basic "Get Attribute Single/All" requests -- which just return raw, 8-bit data. It is up to you to turn that back into a CIP REAL, INT, DINT, etc.
In order to communicate with your linear actuator, you will need to disable these enhanced encapsulations, and use "Get Attribute Single" requests. This is done by specifying an empty route_path=[] and send_path='', when you parse your operations, and to use cpppo.server.enip.getattr's attribute_operations (instead of cpppo.server.enip.client's parse_operations):
from cpppo.server.enip import client
from cpppo.server.enip.getattr import attribute_operations
HOST = "192.168.1.100"
TAGS = ["#4/100/3"]
with client.connector(host=HOST) as conn:
for index, descr, op, reply, status, value in conn.synchronous(
operations=attribute_operations(
TAGS, route_path=[], send_path='' )):
print(": %20s: %s" % (descr, value))
That should do the trick!
We are in the process of rolling out a major update to the cpppo module, so clone the https://github.com/pjkundert/cpppo.git Git repo, and checkout the feature-list-identity branch, to get early access to much better APIs for accessing raw data from these simple devices, for testing. You'll be able to use cpppo to convert the raw data into CIP REALs, instead of having to do it yourself...
...
With Cpppo >= 3.9.0, you can now use much more powerful cpppo.server.enip.get_attribute 'proxy' and 'proxy_simple' interfaces to routing CIP devices (eg. ControlLogix, Compactlogix), and non-routing "simple" CIP devices (eg. MicroLogix, PowerFlex, etc.):
$ python
>>> from cpppo.server.enip.get_attribute import proxy_simple
>>> product_name, = proxy_simple( '10.0.1.2' ).read( [('#1/1/7','SSTRING')] )
>>> product_name
[u'1756-L61/C LOGIX5561']
If you want regular updates, use cpppo.server.enip.poll:
import logging
import sys
import time
import threading
from cpppo.server.enip import poll
from cpppo.server.enip.get_attribute import proxy_simple as device
params = [('#1/1/1','INT'),('#1/1/7','SSTRING')]
# If you have an A-B PowerFlex, try:
# from cpppo.server.enip.ab import powerflex_750_series as device
# parms = [ "Motor Velocity", "Output Current" ]
hostname = '10.0.1.2'
values = {} # { <parameter>: <value>, ... }
poller = threading.Thread(
target=poll.poll, args=(device,), kwargs={
'address': (hostname, 44818),
'cycle': 1.0,
'timeout': 0.5,
'process': lambda par,val: values.update( { par: val } ),
'params': params,
})
poller.daemon = True
poller.start()
# Monitor the values dict (updated in another Thread)
while True:
while values:
logging.warning( "%16s == %r", *values.popitem() )
time.sleep( .1 )
And, Voila! You now have regularly updating parameter names and values in your 'values' dict. See the examples in cpppo/server/enip/poll_example*.py for further details, such as how to report failures, control exponential back-off of connection retries, etc.
Version 3.9.5 has recently been released, which has support for writing to CIP Tags and Attributes, using the cpppo.server.enip.get_attribute proxy and proxy_simple APIs. See cpppo/server/enip/poll_example_many_with_write.py
hope this is obvious, but accessing HOST = "192.168.1.100" will only be possible from a system located on the subnet 192.168.1.*
I'm using the python-ldap library to connect to our LDAP server and run queries. The issue I'm running into is that despite setting a size limit on the search, I keep getting SIZELIMIT_EXCEEDED errors on any query that would return too many results. I know that the query itself is working because I will get a result if the query returns a small subset of users. Even if I set the size limit to something absurd, like 1, I'll still get a SIZELIMIT_EXCEEDED on those bigger queries. I've pasted a generic version of my query below. Any ideas as to what I'm doing wrong here?
result = self.ldap.search_ext_s(self.base, self.scope, '(personFirstMiddle=<value>*)', sizelimit=5)
When the LDAP client requests a size-limit, that is called a 'client-requested' size limit. A client-requested size limit cannot override the size-limit set by the server. The server may set a size-limit for the server as a whole, for a particular authorization identity, or for other reasons - whichever the case, the client may not override the server size limit. The search request may have to be issued in multiple parts using the simple paged results control or the virtual list view control.
Here's a Python3 implementation that I came up with after heavily editing what I found here and in the official documentation. At the time of writing this it works with the pip3 package python-ldap version 3.2.0.
def get_list_of_ldap_users():
hostname = "google.com"
username = "username_here"
password = "password_here"
base = "dc=google,dc=com"
print(f"Connecting to the LDAP server at '{hostname}'...")
connect = ldap.initialize(f"ldap://{hostname}")
connect.set_option(ldap.OPT_REFERRALS, 0)
connect.simple_bind_s(username, password)
connect=ldap_server
search_flt = "(personFirstMiddle=<value>*)" # get all users with a specific middle name
page_size = 500 # how many users to search for in each page, this depends on the server maximum setting (default is 1000)
searchreq_attrlist=["cn", "sn", "name", "userPrincipalName"] # change these to the attributes you care about
req_ctrl = SimplePagedResultsControl(criticality=True, size=page_size, cookie='')
msgid = connect.search_ext(base=base, scope=ldap.SCOPE_SUBTREE, filterstr=search_flt, attrlist=searchreq_attrlist, serverctrls=[req_ctrl])
total_results = []
pages = 0
while True: # loop over all of the pages using the same cookie, otherwise the search will fail
pages += 1
rtype, rdata, rmsgid, serverctrls = connect.result3(msgid)
for user in rdata:
total_results.append(user)
pctrls = [c for c in serverctrls if c.controlType == SimplePagedResultsControl.controlType]
if pctrls:
if pctrls[0].cookie: # Copy cookie from response control to request control
req_ctrl.cookie = pctrls[0].cookie
msgid = connect.search_ext(base=base, scope=ldap.SCOPE_SUBTREE, filterstr=search_flt, attrlist=searchreq_attrlist, serverctrls=[req_ctrl])
else:
break
else:
break
return total_results
This will return a list of all users but you can edit it as required to return what you want without hitting the SIZELIMIT_EXCEEDED issue :)