Develop Reference

(Pymongo) Commit and rollback issue

(Pymongo) several functions cover with Commit and rollback, and one of them stll proceed while whole function should stop
I read the manual, that all the example commit and rollback only cover two operation, is that the limit?, usually should contain 3 or more operations and either operate in the same time or not operate if error https://pymongo.readthedocs.io/en/stable/api/pymongo/client_session.html
I tried to contain 3 operation inside commit and rollback but
mycol_two.insert_one() didn't stop proceed like other function when error occur
brief description:
I have three collections in same DB
collection "10_20_cash_all"
collection "10_20_cash_log"
collection "10_20_cash_info"
commit and rollback on line 39 to 44
line 42 print( 3/0 ) , I intent to make an error, expect all function would stop proceed
import pymongo
import datetime
import json
from bson.objectid import ObjectId
from bson import json_util
import re
myclient = pymongo.MongoClient("mongodb://localhost:27017/")
mydb = myclient["(practice_10_14)-0004444"]
mycol_one = mydb["10_20_cash_all"]
mycol_two = mydb["10_20_cash_log"]
mycol_3rd = mydb["10_20_cash_info"]
# already store 100$ in bank
# doc_two = {"ID" : 100998 , "Cash_log$" : 5 } # withdraw 70$ from bank
doc_two = input("Enter ID and log amount$: ")
doc_3rd = input("Enter extra info: ")
doc_two_dic = json.loads(doc_two)
doc_3rd_dic = json.loads(doc_3rd)
# doc_3rd = {"note" : "today is good" }
ID_input = doc_two_dic['ID']
print("ur id is :" + str(ID_input))
doc_one = {"ID" : ID_input}
with myclient.start_session() as s:
cash_all_result = mycol_one.find_one(doc_one, session=s)
def cb(s):
try:
while True:
cash_all_result = mycol_one.find_one(doc_one, session=s)
mycol_two.insert_one(doc_two_dic, session=s)
print( 3/0 )
mycol_3rd.insert_one(doc_3rd_dic, session=s)
print( "now total is :" + str(cash_all_result['Cash_$']) )
Cash_total_int = int(cash_all_result['Cash_$'])
log_int = int(doc_two_dic['Cash_log$'])
if Cash_total_int < log_int:
print("error: withdraw is over ur balance")
break
new_Cash_total = Cash_total_int - log_int
print("now total is :" + str(new_Cash_total))
newvalues_json = { "$set" : {"Cash_$" : new_Cash_total } }
mycol_one.update_one(doc_one , newvalues_json, session=s)
fail_condition_json = {"ok" : 1 , "fail reason" : "no error "}
print(fail_condition_json)
return fail_condition_json
except Exception as e:
fail_condition_json = {"ok" : 0 , "fail reason" : "error raise on start_session()"}
print(fail_condition_json)
return fail_condition_json
s.with_transaction(cb)
command prompt:
Enter ID and log amount$: {"ID" : 100998 , "Cash_log$" : 5 }
Enter extra info: {"note" : "today is good" }
ur id is :100998
{'ok': 0, 'fail reason': 'error raise on start_session()'}
the "10_20_cash_log" still store new value which shoud empty/not run like '"10_20_cash_info"' is empty
{
"_id" : ObjectId("635262e502725626c39cbe9e"),
"ID" : 100998,
"Cash_log$" : 5
}

Why Kinesis stream calls my Lambda function more than one time?

I am consuming Amazon Connect CTRs through Amazon Kinesis and inserting my data into Postgres. I am facing very unexpected behavior from Kinesis and Lambda function. Whenever a CTR record comes through kinesis, my lambda gets invoked and after inserting that record into Postgres, it again gets invoked and is very unexpected behavior. Although, I have received only one record. Here is my code, if anything is wrong with the code please correct me:
def lambda_handler(event, context):
print(event['Records'])
print(event)
for record in event['Records']:
conn = psycopg2.connect(
host = hostt,
user = username,
password = passwordd,
database = databasee
)
cur = conn.cursor(cursor_factory = RealDictCursor)
payload = base64.b64decode(record['kinesis']['data'])
de_serialize_payload = json.loads(payload)
print(len(de_serialize_payload))
print(de_serialize_payload)
try:
for dsp in de_serialize_payload:
if de_serialize_payload['Agent'] != None and de_serialize_payload['CustomerEndpoint'] != None and de_serialize_payload['Recording'] != None and de_serialize_payload['TransferredToEndpoint'] != None:
required_data = {
'arn' : de_serialize_payload['Agent']['ARN'],
'aftercontactworkduration' : de_serialize_payload['Agent']['AfterContactWorkDuration'],
'aftercontactworkendtimestamp' : de_serialize_payload['Agent']['AfterContactWorkEndTimestamp'],
'aftercontactworkstarttimestamp' : de_serialize_payload['Agent']['AfterContactWorkStartTimestamp'],
'agentconnectionattempts' : de_serialize_payload['AgentConnectionAttempts'],
'agentinteractionduration' : de_serialize_payload['Agent']['AgentInteractionDuration'],
'answeringmachinedetectionstatus' : de_serialize_payload['AnsweringMachineDetectionStatus'],
'channel' : de_serialize_payload['Channel'],
'connectedtoagenttimestamp' : de_serialize_payload['Agent']['ConnectedToAgentTimestamp'],
'connectedtosystemtimestamp' : de_serialize_payload['ConnectedToSystemTimestamp'],
'customerendpointaddress' : de_serialize_payload['CustomerEndpoint']['Address'],
'customerendpointtype' : de_serialize_payload['CustomerEndpoint']['Type'],
'customerholdduration' : de_serialize_payload['Agent']['CustomerHoldDuration'],
'dequeuetimestamp' : de_serialize_payload['Queue']['DequeueTimestamp'],
'disconnectreason' : de_serialize_payload['DisconnectReason'],
'disconnecttimestamp' : de_serialize_payload['DisconnectTimestamp'],
'queueduration' : de_serialize_payload['Queue']['Duration'],
'enqueuetimestamp' : de_serialize_payload['Queue']['EnqueueTimestamp'],
'hierarchygroups' : de_serialize_payload['Agent']['HierarchyGroups'],
'initialcontactid' : de_serialize_payload['InitialContactId'],
'initiationmethod' : de_serialize_payload['InitiationMethod'],
'initiationtimestamp' : de_serialize_payload['InitiationTimestamp'],
'instancearn' : de_serialize_payload['InstanceARN'],
'lastupdatetimestamp' : de_serialize_payload['LastUpdateTimestamp'],
'longestholdduration' : de_serialize_payload['Agent']['LongestHoldDuration'],
'nextcontactid' : de_serialize_payload['NextContactId'],
'numberofholds' : de_serialize_payload['Agent']['NumberOfHolds'],
'previouscontactid': de_serialize_payload['PreviousContactId'],
'queuearn' : de_serialize_payload['Queue']['ARN'],
'queuename' : de_serialize_payload['Queue']['Name'],
'recordingdeletionreason' : de_serialize_payload['Recording']['DeletionReason'],
'recordinglocation' : de_serialize_payload['Recording']['Location'],
'recordingstatus' : de_serialize_payload['Recording']['Status'],
'recordingtype' : de_serialize_payload['Recording']['Type'],
'routingprofilearn' : de_serialize_payload['Agent']['RoutingProfile']['ARN'],
'routingprofilename' : de_serialize_payload['Agent']['RoutingProfile']['Name'],
'scheduledtimestamp' : de_serialize_payload['ScheduledTimestamp'],
'systemendpointaddress' : de_serialize_payload['SystemEndpoint']['Address'],
'systemendpointtype' : de_serialize_payload['SystemEndpoint']['Type'],
'transfercompletedtimestamp' : de_serialize_payload['TransferCompletedTimestamp'],
'transferredtoendpoint' : de_serialize_payload['TransferredToEndpoint']['Address'],
'username' : de_serialize_payload['Agent']['Username'],
'voiceidresult' : de_serialize_payload['VoiceIdResult'],
'id' : de_serialize_payload['ContactId']
}
columns = required_data.keys()
print(columns)
values = [required_data[column] for column in columns]
print(values)
insert_statement = "insert into public.ctr (%s) values %s;"
cur.execute(insert_statement, (AsIs(','.join(columns)), tuple(values)))
print(cur.mogrify(insert_statement, (AsIs(','.join(columns)), tuple(values))))
conn.commit()
count = cur.rowcount
print(count, "Record inserted successfully into mobile table")
print("Agent, customer endpoint, transfer endpoint and recording data is available")
After one successful iteration it again starts iterating. I have spent more than two days on it and didn't figure out what's the problem.
I would really appreciate if someone guides me and sort out this query.

The issue was in my code. I was not ending my function successfully. It is Kinesis behavior if you are not ending your function successfully (200 OK) then kinesis reinvokes your function several times. So it it necessary to end your function properly.

MongoDB $geoIntersect query runs in Compass and Studio 3T, but not in Python - any ideas?

Here is the script which i have working inside MongoDBs Compass and working inside Studio 3T
db.getCollection("collection").find({
geometry: {
$geoIntersects: {
$geometry: {
type: "Polygon" ,
coordinates: [ [
[-2.4478329206542915, 52.679303638992494], [-2.4423397565917915, 52.677534343091544], [-2.445601322753901, 52.67430779548455], [-2.4509228254394477, 52.676129262942176], [-2.4478329206542915, 52.679303638992494]
] ]
}
}
}
})
and it returns the following results:
As you can see the code seems to work fine, but when I try to run this inside a Python script it keeps failing and I need a little help pointing out the seemingly not so obvious error.
I have changed the fin statement to a simple field find and it works, just not when its a geoIntersects find statement.
However, if I try to do this inside python I keep getting errors.
Please see below for a copy of my python script, sorry for the redacted comments but need to keep data secure.
import pymongo
#mongodb service vars
dbStartc = 'mongodb+srv://'
dbDomain = '#cluster0.o7cud.azure.mongodb.net'
dbUser = 'redacted'
dbPass = 'redacted'
dbName = 'redacted'
dbColl = 'redacted'
dbSettings = '/?retryWrites=true&w=majority'
#test vars
dbName_connected = False
dbColl_connected = False
try:
#conect to the mongodb instance
mongoURL = dbStartc + dbUser + ':' + dbPass + dbDomain + dbSettings
mongocon = pymongo.MongoClient(mongoURL)
#connect to the database
dblist = mongocon.list_database_names()
if dbName in dblist:
mongocdb = mongocon[dbName]
#print("MongoDB Service - Database connected")
dbName_connected = True
#connect to the collection we need
collist = mongocdb.list_collection_names()
if dbColl in collist:
mongocol = mongocdb[dbColl]
#print("MongoDB Service - Collection connected")
dbColl_connected = True
#pre checks test
if dbName_connected and dbColl_connected :
#print("MongoDB Service - Connection Checks Complete")
find_result = []
found_count = 0
found_count = mongocol.count_documents( )
if found_count > 0 :
print("Collection Document Count: " + found_count)
mydoc = mongocol.find({ geometry: { $geoIntersects: { $geometry: { type: "Polygon" , coordinates: [ [ [-2.44783, 52.67930], [-2.44233, 52.67753], [-2.44560, 52.67430], [-2.45092, 52.67612], [-2.44783, 52.67930] ] ] } } } })
#for x in
for x in mydoc:
find_result += [x]
print(find_result)
else :
print("MongoDB Service - Connection Checks Failed")
except Exception as ex:
print ("Something you were not expecting went wrong! (" + ex + ")")
Here is the error:
Any help in getting python to work would be greatly appreciated

How to properly set up a json config file so that it can be used to replace hardcoded values in python program

I am trying to modify my python method so that its reading values from a separate json config file.
I have a separate valid json file that looks like this testtok.json:
{
"email" : "user#domain.com",
"audience" : "http://someaudience.com",
"jti" : "MYJTI1234",
"name" : "John Smith",
"humid" : "ABC1234"
}
I want to pass in those values to my function here:
def tokengen(self, privatekey):
with open('config/testtok.json', 'r') as config:
data = json.load(config)
try:
"""Simulate Token Manager creating a token"""
email = config["email"]
audience = config["audience"]
jti = config["jti"]
name = config["name"]
humid = config["humid"]
#email = "user#domain.com"
#audience = "http://someaudience.com"
#jti = "MYJTI1234"
#name = "John Smith"
#humid = "ABC1234"
"""Time Component"""
timestamp = testdate()
timestamp.now()
issued = int(time.time())
expires_in=2400
expires = issued + expires_in
additional_headers = {
"alg": "RS256",
"typ": "JWT"
}
payload = {
"iss": email,
"sub": email,
"aud": audience,
"iat": issued,
"nbf": issued,
"exp": expires,
"jti": jti,
"name": name,
"humid": humid,
"email": email
}
self.testtoken = jwt.encode(payload, privatekey, algorithm="RS256", headers=additional_headers).decode("utf-8")
valid_response = {'validation' : 'VALID', 'ts' : timestamp.utc_datetime_str, 'test_token' : self.testtoken}
return(valid_response)
except Exception as ex:
invalid_response = {'validation' : 'INVALID', 'ts' : timestamp.utc_datetime_str, 'test_token' : self.testtoken}
return(invalid_response)
I'm seeing this error and unclear how to troubleshoot this.
Traceback (most recent call last):
File "testTokClass.py", line 25, in tokengen
config["email"]
TypeError: '_io.TextIOWrapper' object is not subscriptable
Is there a better way to do this? Ideally, I would like to have the config file as json. Thanks.

Th problem is that the config is a file handle; in the line data = json.load(config) the JSON is read from the file pointed to by config and into the data variable.
So, just change:
email = config["email"]
to:
email = data["email"]
And similarly for the next four lines.

Unable to set category to 'Education' in Upload

I am using Dailymotion python sdk to upload videos to dailymotion. I am unable to set category to 'Education'
Unable to set category to 'Education', works fine if I set it to 'news'.
upload_folder = r"D:\My Folder\Dailymotion\Download_DM\automated_upload"
for file in os.listdir(upload_folder):
try:
d.set_grant_type(
"password",
api_key=_API_KEY,
api_secret=_API_SECRET,
scope=["manage_videos"],
info={"username": _USERNAME, "password": _PASSWORD},
)
# Uploading the file on dailymotion servers
file_path = upload_folder +'\\' + os.path.splitext(file)[0] + ".mp4"
url = d.upload(file_path)
# Filling the information about the video
parameters = {
"url": url,
"title": os.path.splitext(file)[0],
"tags": "life,love,reality,god,spirituality,education,truth,saints,scriptures",
#"description": "my first automatic uplaod",
"published": 1,
"channel": "education"
}
# Sending the information to create the video on dailymotion
result = d.post("/me/videos?fields=id,url", parameters)
print("Uploaded video: ", os.path.splitext(file)[0], "\nDailymotion url: ", result['url'], "\n\n")
except Exception as e:
print("An error occured: %s" % str(e))
Error Message: DailymotionApiError: not_found: Can't find object channel for `channel' parameter

Education is the label of the channel, you have to pass its id which is 'school'.
You can retrieve all the channel ids at this endpoint: https://api.dailymotion.com/channels?fields=id,name