Develop Reference

Pass elements of a list in a function

I have a function that is able to create triples and relationships from text. However, when I create a list of a column that contains text and pass it through the function, it only processes the first row, or item of the list. Therefore, I am wondering how the whole list can be processed within this function. Maybe a for loop would work?
The following line contains the list
rez_dictionary = {'Decent Little Reader, Poor Tablet',
'Ok For What It Is',
'Too Heavy and Poor weld quality,',
'difficult mount',
'just got it installed'}
from transformers import pipeline
triplet_extractor = pipeline('text2text-generation', model='Babelscape/rebel-large', tokenizer='Babelscape/rebel-large')
# We need to use the tokenizer manually since we need special tokens.
extracted_text = triplet_extractor.tokenizer.batch_decode([triplet_extractor(rez_dictionary, return_tensors=True, return_text=False)[0]["generated_token_ids"]])
print(extracted_text[0])
If anyone has a suggestion, I am looking forward for it.
Would it also be possible to get the output adjusted to the following format:
# Function to parse the generated text and extract the triplets
def extract_triplets(text):
triplets = []
relation, subject, relation, object_ = '', '', '', ''
text = text.strip()
current = 'x'
for token in text.replace("<s>", "").replace("<pad>", "").replace("</s>", "").split():
if token == "<triplet>":
current = 't'
if relation != '':
triplets.append({'head': subject.strip(), 'type': relation.strip(),'tail': object_.strip()})
relation = ''
subject = ''
elif token == "<subj>":
current = 's'
if relation != '':
triplets.append({'head': subject.strip(), 'type': relation.strip(),'tail': object_.strip()})
object_ = ''
elif token == "<obj>":
current = 'o'
relation = ''
else:
if current == 't':
subject += ' ' + token
elif current == 's':
object_ += ' ' + token
elif current == 'o':
relation += ' ' + token
if subject != '' and relation != '' and object_ != '':
triplets.append({'head': subject.strip(), 'type': relation.strip(),'tail': object_.strip()})
return triplets
extracted_triplets = extract_triplets(extracted_text[0])
print(extracted_triplets)

You are removing the other entries of rez_dictionary inside the batch_decode:
triplet_extractor(rez_dictionary, return_tensors=True, return_text=False)[0]["generated_token_ids"]
Use a list comprehension instead:
from transformers import pipeline
rez = ['Decent Little Reader, Poor Tablet',
'Ok For What It Is',
'Too Heavy and Poor weld quality,',
'difficult mount',
'just got it installed']
triplet_extractor = pipeline('text2text-generation', model='Babelscape/rebel-large', tokenizer='Babelscape/rebel-large')
model_output = triplet_extractor(rez, return_tensors=True, return_text=False)
extracted_text = triplet_extractor.tokenizer.batch_decode([x["generated_token_ids"] for x in model_output])
print("\n".join(extracted_text))
Output:
<s><triplet> Decent Little Reader <subj> Poor Tablet <obj> different from <triplet> Poor Tablet <subj> Decent Little Reader <obj> different from</s>
<s><triplet> Ok For What It Is <subj> film <obj> instance of</s>
<s><triplet> Too Heavy and Poor <subj> weld quality <obj> subclass of</s>
<s><triplet> difficult mount <subj> mount <obj> subclass of</s>
<s><triplet> 2008 Summer Olympics <subj> 2008 <obj> point in time</s>
Regarding the extension of the OP's question, OP wanted to know how to run the function extract_triplets. OP can simply do that via a for-loop:
for text in extracted_text:
print(extract_triplets(text))
Output:
[{'head': 'Decent Little Reader', 'type': 'different from', 'tail': 'Poor Tablet'}, {'head': 'Poor Tablet', 'type': 'different from', 'tail': 'Decent Little Reader'}]
[{'head': 'Ok For What It Is', 'type': 'instance of', 'tail': 'film'}]
[{'head': 'Too Heavy and Poor', 'type': 'subclass of', 'tail': 'weld quality'}]
[{'head': 'difficult mount', 'type': 'subclass of', 'tail': 'mount'}]
[{'head': '2008 Summer Olympics', 'type': 'point in time', 'tail': '2008'}]

Python concurrent.future not running in Parallel

I am writing code for querying CMDB data from ServiceNow through REST API calls using python. To gain results faster I am using concurrent.future module of python to allow parallel query executions.
Below is the code and output:
import requests
import os
import base64
import threading
import concurrent.futures
import datetime
class ServiceNowAPI:
__ServiceNowUserName = os.environ.get('ServiceNow_USERNAME')
__ServiceNowPassword = None
__ServiceNowPasswordProd = None
__BASE_URL = None
__WSI_URL = 'https://esiocecr.contoso.com:9443/rsrv_servicenow-outbound/'
__ServiceNow_Cert_Path = 'C:\Certificates\OSMCert.pem'
# ServiceNow API URL paths
__GET_CI_DETAILS = "/api/now/table/cmdb_ci_circuit?sysparm_query=u_port_circuit_idLIKE__CINAME__^ORnameLIKE__CINAME__&sysparam_fields=*"
def __init__(self, ServiceNowEnvironment):
self.ServiceNowEnvironment = ServiceNowEnvironment
self.__ServiceNowPassword = os.environ.get('ServiceNow_PROD_PWD') if ServiceNowEnvironment.lower() == "prod" or ServiceNowEnvironment.lower() == "production" else os.environ.get('ServiceNow_EAGLE_PWD')
self.__BASE_URL = "https://contososervicenow.service-now.com" if ServiceNowEnvironment.lower() == "prod" or ServiceNowEnvironment.lower() == "production" else "https://contosoeagle.service-now.com"
CredsPair = "{0}:{1}".format(self.__ServiceNowUserName, self.__ServiceNowPassword)
CredBytes = CredsPair.encode('ascii')
Base64Creds = base64.b64encode(CredBytes).decode('utf-8')
self.__Authorization = "Basic {0}".format(Base64Creds)
def _GetRouterName(self, RouterLink):
RouterName = ''
with requests.Session() as ServiceNowCall:
ServiceNowCall.headers.update({ 'Authorization': self.__Authorization, 'content-type': 'application/json', 'DP_EXTERNAL_URL': RouterLink})
ServiceNowCall.cert = self.__ServiceNow_Cert_Path
ServiceNowCall.verify = self.__ServiceNow_Cert_Path
with ServiceNowCall.get(self.__WSI_URL) as ResponseObject:
ResponseJSON = ResponseObject.json()
Results = ResponseJSON['result']
RouterName = Results['name']
return RouterName
def _GetCircuitCIDetails(self, CircuitID):
print('Started for ' + CircuitID + ' at ' + datetime.datetime.now().strftime("%d-%b-%Y %H:%M:%S.%f"))
ResponseJSON = ''
URL = "{0}{1}".format(self.__BASE_URL, self.__GET_CI_DETAILS.replace('__CINAME__', CircuitID))
with requests.Session() as ServiceNowCall:
ServiceNowCall.headers.update({ 'Authorization': self.__Authorization, 'content-type': 'application/json', 'DP_EXTERNAL_URL': URL})
ServiceNowCall.cert = self.__ServiceNow_Cert_Path
ServiceNowCall.verify = self.__ServiceNow_Cert_Path
with ServiceNowCall.get(self.__WSI_URL) as ResponseObject:
ResponseJSON = ResponseObject.json()
AllRecords = list(ResponseJSON['result'])
ActiveRecord = [rec for rec in AllRecords if rec['u_lifecycle_status'] != 'end of life'][0]
Router_Name = self._GetRouterName(ActiveRecord['u_router_name']['link'])
Results = {
'Name': ActiveRecord['name'],
'CarrierName': ActiveRecord['u_carrier_name'],
'NetworkType': ActiveRecord['u_network_type'],
'NetworkSubType': ActiveRecord['u_network_sub_type'],
'RouterName': Router_Name,
'PortCircuitID': ActiveRecord['name'],
'AccessCircuitID': ActiveRecord['u_port_circuit_id']
}
print('Finished ' + CircuitID + ' at ' + datetime.datetime.now().strftime("%d-%b-%Y %H:%M:%S.%f"))
yield Results
def GetCIDetails(self, CICSV):
CircuitDetails = []
CircuitIDList = [Circuit.strip() for Circuit in CICSV.split(',')]
CircuitDetails = []
with concurrent.futures.ThreadPoolExecutor() as executor:
CIDetailsResult = { executor.submit(self._GetCircuitCIDetails, CircuitID): CircuitID for CircuitID in CircuitIDList }
for future in concurrent.futures.as_completed(CIDetailsResult):
CircuitCI = CIDetailsResult[future]
try:
CurrentResult = future.result()
except Exception as exc:
ErrorResult = dict({ 'Name': CircuitCI, 'CarrierName': 'NA', 'NetworkType': 'NA', 'NetworkSubType': 'NA', 'RouterName': 'NA', 'PortCircuitID': 'Error', 'AccessCircuitID': 'Error'})
CircuitDetails.extend(ErrorResult)
else:
CircuitDetails.extend(CurrentResult)
return CircuitDetails
if __name__ == "__main__":
ServiceNowAPIClass = ServiceNowAPI('NONPROD')
CIDetails = ServiceNowAPIClass.GetCIDetails('Circuit1,Circuit2')
print(CIDetails)
Output:
Started for Circuit1 at 30-Apr-2022 13:40:06.784841
Finished Circuit1 at 30-Apr-2022 13:40:09.749164
Started for Circuit2 at 30-Apr-2022 13:40:09.751166
Finished Circuit2 at 30-Apr-2022 13:40:12.479171
[{'Name': 'Circuit1', 'CarrierName': 'CenturyLink', 'NetworkType': 'EU-BTM', 'NetworkSubType': 'N/A', 'RouterName': 'RT1234ABCD03', 'PortCircuitID': 'Circuit1', 'AccessCircuitID': 'Circuit1'}, {'Name': 'Circuit2', 'CarrierName': 'Verizon', 'NetworkType': 'DPS-NA', 'NetworkSubType': 'N/A', 'RouterName': 'RT12345678ABC', 'PortCircuitID': 'Circuit2', 'AccessCircuitID': 'Circuit2'}]
However, as you can see the executions are not happening in Parallel. It is finishing the queries for each circuit one after another.
How can I fix this to run the _GetCircuitCIDetails(self, CircuitID) function against all CircuitIDs in parallel?

Thread returning same value in infinite loop

I am new to threading, I'm able to read the data from the database using (def value) function and passing that data to (def main). I've created an infinite loop so that whenever new value is added to database, the (def value) function can read it and pass the new data accordingly, which will be passed to another device. But new data is not acquired automatically, I've to start the program again in order to load the new program and pass it to different device. I'm confused why I'm not getting new value while the loop is still running.
Here is the data format:
data = {'data': 'xyz', 'code': '<:c:605445> **[Code](https://traindata/35.6547,56475', 'time': '2021-12-30T09:56:53.547', 'value': 'True', 'stats': '96/23', 'dupe_id': 'S<:c-74.18'}
Here is the code:
def value(id):
headers = { 'authorization': 'xyz'} #authorization token here
r = requests.get(f'https://traindata.com/api/v9/{id}', headers=headers) #website link along with variable passed here
jsonn = json.loads(r.text) #read the values
i = 0
d = {}
while i < len(jsonn):
data = jsonn[i]["content].split("")[1] #splitting information which is needed
d[str(i)] = {}
d[str(i)]["data"] = data
d[str(i)]["code"] = code
i+= 1
return d
X = value('987654') #passing the id which gives data from specific file
def main(cds):
data = X #passing value which was outside the loop
for s in data.values():
print(s)
while data != "stop": # unless I press stop the program should run
if data == "stop": #when I press stop the program stops
os.system("disconnect")
else:
cds = d['code'].split('/')[-1].split(',')
#splits value which is needed to be passed onto another device
return cds
m = main(cds)
if __name__ == "__main__":
t1 = Thread(target=value, args=(987654, )) #thread 1
t2 = Thread(target=main, args=(m, )) #thread 2
t1.setDaemon(True)
t2.setDaemon(True)
t1.start()
t2.start()
t1.join()
t2.join()
while True:
pass
The output I'm getting is this:
Number of active threads: 1
Number of active threads: 1
{'data': 'xyz', 'code': '<:c:605445> **[Code](https://traindata/35.6547,56475', 'time': '2021-12-30T09:56:53.547', 'value': 'True', 'stats': '95/23', 'dupe_id': 'S<:c-74.18'}
35.6547,56475
Number of active threads ae: 2
Number of active threads: 3
{'data': 'xyz', 'code': '<:c:605445> **[Code](https://traindata/35.6547,56475', 'time': '2021-12-30T09:56:53.547', 'value': 'True', 'stats': '95/23', 'dupe_id': 'S<:c-74.18'} #same data is printed as previous one
Same data is printed even after thread is running in infinite loop. I want something like this:
Number of active threads: 1
Number of active threads: 1
{'data': 'xyz', 'code': '<:c:605445> **[Code](https://traindata/35.6547,56475', 'time': '2021-12-30T09:56:53.547', 'value': 'True', 'stats': '95/23', 'dupe_id': 'S<:c-74.18'}
35.6547,56475
Number of active threads ae: 2
Number of active threads: 3
{'data': 'xyz', 'code': '<:c:605455> **[Code](https://traindata/42.6247,28.47023', 'time': '2021-12-30T09:59:57.758', 'value': 'True', 'stats': '90/110', 'dupe_id': 'S<:c-74.18'} #different data should be printed (having different time stamp, and code)
42.6247,28.47023

Run a function in background using thread in Flask

I am trying to implement limit order book using flask and I am working on the backend part right now. I am new to flask so I am still learning and I am not much aware about how backend of trading works but I am trying to learn via this small project.
I have created 3 endpoints in my application which add order, remove order and give a response of the order status and these three endpoints are working fine checked them with postman. Now I am trying to run a function in background which will continuously check the new orders (buy/sell) from a json file which save all new orders. It will pick them one by one and will find a match based on price if a user's buy order matches a different user's sell order it will process and store it in a dict which I want to return or store all those successful order to the user.
Here is my code for the class I have created:
import json
import bisect
import random
import os
class Process(object):
def __init__(self):
self.trade_book = []
self.bid_prices = []
self.ask_prices = []
self.ask_book = {}
self.bid_book = {}
self.confirm_traded = []
self.orders_history = {}
self.traded = False
self.counter = 0
def save_userdata(self,order, newId):
orderid = order['order']['trader'] +"_"+ str(newId)
user_list = order
newJson = {
"orders":[
{ orderid: order['order']}
]
}
with open('data_user.json', 'a+') as jsonFile:
with open('data_user.json', 'r') as readableJson:
try:
jsonObj = json.load(readableJson)
except Exception as e:
jsonObj = {}
if jsonObj == {}:
json.dump(newJson, jsonFile)
else:
with open('data_user.json', 'w+') as writeFile:
exists = False
for item in jsonObj['orders']:
if item.get(orderid, None) is not None:
item[orderid] = order['order']
exists = True
break
if not exists:
jsonObj['orders'].append(newJson['orders'][0])
json.dump(jsonObj, writeFile)
return orderid
def get_userdata(self):
with open('data_user.json', 'r') as readableJson:
return json.load(readableJson)
def removeOrder(self, orderid):
order_id = list(orderid.values())[0]
with open('data_user.json') as data_file:
data = json.load(data_file)
newData = []
for item in data['orders']:
if item.get(order_id, None) is not None:
del item[order_id]
else:
newData.append(item)
data['orders'] = newData
with open('data_user.json', 'w') as data_file:
data = json.dump(data, data_file)
return order_id
def add_order_to_book(self, order):
index = list(order.keys())[0]
book_order = order[index]
print(index)
if order[index]['side'] == 'buy':
book_prices = self.bid_prices
book = self.bid_book
else: #order[index]['side'] == 'sell'
book_prices = self.ask_prices
book = self.ask_book
if order[index]['price'] in book_prices:
book[order[index]['price']]['num_orders'] += 1
book[order[index]['price']]['size'] += order[index]['quantity']
book[order[index]['price']]['order_ids'].append(index)
book[order[index]['price']]['orders'][index] = book_order
else:
bisect.insort(book_prices, order[index]['price'])
book[order[index]['price']] = {'num_orders': 1, 'size': order[index]['quantity'],'order_ids':
[index],
'orders': {index: book_order}}
def confirm_trade(self,order_id, timestamp, order_quantity, order_price, order_side):
trader = order_id.partition('_')[0]
self.confirm_traded.append({ 'trader': trader,'quantity': order_quantity, 'side': order_side,
'price': order_price,
'status': 'Successful'})
return self.confirm_traded
def process_trade_orders(self, order):
self.traded = False
index = list(order.keys())[0]
if order[index]['side'] == 'buy':
book = self.ask_book
if order[index]['price'] in self.ask_prices:
remainder = order[index]['quantity']
while remainder > 0:
book_order_id = book[order[index]['price']]['order_ids'][0]
book_order = book[order[index]['price']]['orders'][book_order_id]
if remainder >= book_order['quantity']:
self.trade_book.append({'order_id': book_order_id, 'timestamp': order[index]['timestamp'],
'price': order[index]['price'],
'quantity': order[index]['quantity'], 'side': book_order['side']})
self.confirm_trade(index, order[index]['timestamp'], order[index]['quantity'], order[index]['price'], order[index]['side'])
self.traded = True
remainder = remainder - book_order['quantity']
self.save_historty_orders(index, order[index])
break
else:
self.traded = True
self.trade_book.append({'order_id': index, 'timestamp': order[index]['timestamp'],
'price': order[index]['price'],
'quantity': order[index]['quantity'], 'side': order[index]['side']})
self.confirm_trade(index, order[index]['timestamp'], order[index]['quantity'], order[index]['price'], order[index]['side'])
self.save_historty_orders(index, order[index])
break
else:
self.add_order_to_book(order)
self.save_historty_orders(index, order[index])
else: #order['side'] == 'sell'
book = self.bid_book
if order[index]['price'] in self.bid_prices:
remainder = order[index]['quantity']
while remainder > 0:
book_order_id = book[order[index]['price']]['order_ids'][0]
book_order = book[order[index]['price']]['orders'][book_order_id]
if remainder >= book_order['quantity']:
self.trade_book.append({'order_id': book_order_id, 'timestamp': order[index]['timestamp'],
'price': order[index]['price'],
'quantity': order[index]['quantity'], 'side': order[index]['side']})
self.traded = True
self.confirm_trade(index, order[index]['timestamp'], order[index]['quantity'], order[index]['price'], order[index]['side'])
remainder = remainder - book_order['quantity']
self.save_historty_orders(index, order[index])
break
else:
self.traded = True
self.trade_book.append({'order_id': book_order_id, 'timestamp': order[index]['timestamp'],
'price': order[index]['price'],
'quantity': order[index]['quantity'], 'side': order[index]['side']})
self.confirm_trade(index, order[index]['timestamp'], order[index]['quantity'], order[index]['price'], order[index]['side'])
self.save_historty_orders(index, order[index])
break
else:
self.add_order_to_book(order)
self.save_historty_orders(index, order[index])
This class process I create object in my app.py and call the function process_trade_orders in that inside a function processing():
app = Flask(__name__)
app.config['DEBUG'] = True
newUser = Process()
succorder = Success()
#sched = BackgroundScheduler()
def generate_orderid():
num = 0
while num < 1000:
yield num
num = num + 1
genid = generate_orderid()
proc = Process()
sucorder = Success()
#Processing orders to find if they have a match
def processing():
get_orders_data = proc.get_userdata()
print(get_orders_data)
print("\n")
for data in get_orders_data['orders']:
index = list(data.keys())[0]
if data[index]['status'] == 'Successful':
sucorder.add_trader_orders(data[index],index)
else:
proc.process_trade_orders(data)
# sched = BackgroundScheduler()
# sched.add_job(func = processing, trigger="interval", seconds = 2)
# sched.start()
I did use APSbackground-scheduler for the same but I want to use thread for it. I was thinking of running a main thread in infinite loop as a daemon and use worker thread to run this function processing() in app.py which will be called after every few seconds to check if there are any successful order it will return the value to the main thread and those list of dict every new one I can return a response or some other way to the user about this successful order getting matched.
Note that this will be running is short intervals like 5 seconds and multiple add orders will be added and will be continuously running the checks asynchronously so I am not sure how will I return those values. I am just confused so if anyone can help me will be grateful.

If you want to make a threaded function that runs in background, just use the threading module, like this:
from threading import Thread
def bg_func():
doSomething
t = Thread(target=bg_func)
t.start() # will start the function and continue, even if the function still runs
doSomethingelseAtSameTime # runs with bg_func
You can also have multiple background threads.
Check the documentation for more info.

Why python dictonary gives me Keyerror?

I am reading excel file and creating dictionary using following code in file a.py.
productNo = int(worksheet.cell_value(curr_row, 1))
print "Product No :" + str(productNo)
products[productNo] = {}
while curr_cell < num_cells:
curr_cell += 1
header = str(worksheet.cell_value(0, curr_cell))
# Cell Types: 0=Empty, 1=Text, 2=Number, 3=Date, 4=Boolean, 5=Error, 6=Blank
cell_type = worksheet.cell_type(curr_row, curr_cell)
#print ' ',curr_cell,' <-> ', cell_type
cell_value = worksheet.cell_value(curr_row, curr_cell)
if cell_type == 0 or cell_type == 6:
products[productNo][header] = ""
if cell_type == 1:
#products[productNo] = {header :cell_value}
products[productNo][header] = cell_value
elif cell_type == 2:
cell_value = int(cell_value)
print "Header " + header
products[productNo][header] = cell_value
elif cell_type == 3:
products[productNo] = {header :cell_value}
elif cell_type == 4:
products[productNo] = {header :cell_value}
In file main.py I am calling above written code (in function) and catalouge.readExcel('D:/Personal/CatalogNo_1134.xls', products) by passing variable products as dictionary.
When dump it I am getting all keys and values.
print "Dump " + str(products)
print products.keys()
print products['28852']['ProductNo']
print products['28852']['Style']
Dump {28852: {'Category': u'Party Wear', 'Style': u'Designer', 'ProductNo': 28852, 'ProductGroup': u'Salwar Kameez', 'CatalogNo': 1134, 'ProductType': u'Salwar Kameez', 'SubCategoryDefinition': '', 'Fabric': u'Faux Georgette', 'MinWebPrice': 3395, 'Description': u'Capture The Exuberance Of Womanhood In Its Full Glory That Will Bring Out Your Fragility And Femininity. Genuine Magnificence Will Come Out Through The Dressing Style With This Off White Faux Georgette Salwar Kameez. The Appealing Lace|Resham Work Through The Attire Is Awe-Inspiring.', 'OptionOfSleeves': '', 'Rate/FreeSize': 2095, 'XLRate': 0, 'GenericName': u'Dress Material', 'SizeAvailable': u'Not Applicable', 'MaxLengthCholi': '', 'ItemTag': u'Enigmatic Off White Salwar Kameez', 'SubCategory': u'Other - Party Wear', 'Description of Blouse/Choli/Bottom': u'Paired With A Matching Bottom', 'Colour': u' Off White', 'Work': u'Lace|Resham', 'ExpressQty': 0, 'Occasion': u'Christmas,Diwali,Eid,Festival,Kitty Party,Mehendi,Navratri,Party Wear,Sangeet,Wedding', 'Weight': 1, 'Description of Dupatta': u'Comes With A Matching Dupatta', 'DeliveryDays': 8}, 28853: {...}}
keys are
[28852, 28853, 28854, 28855, 28856, 28857, 28858, 28859, 28860, 28861, 28862, 28863]
When I access it using products['28852']
Traceback (most recent call last):
print products['28852']['ProductNo']
KeyError: '28852'

You are accessing str 28852 while the key is integer. Please access with 28852.