Develop Reference

When I attempt to create a JSON Object in Python, It errors, despite having validated the JSON Online

I believe that the issue is due to python formatting all ' to ", which would result in the error message which I recieved upon running the program.
My Code is as follows:
import requests
import json
import pandas as pd
username = input('enter username here: ')
print('')
passw = input('enter password here: ')
mcpayload = {"agent": {"name": "Minecraft", "version": 1}, "username": "{}".format(username), "password": "{}".format(passw), "requestUser": "true"}
header = {"Content-Type": "application/json"}
logintoken = requests.post('https://authserver.mojang.com/authenticate', data = mcpayload, headers = header)
print(logintoken.text)
print('')
print(logintoken.json)
print('')
print(logintoken.content)
It returns this err message on run:
{"error":"JsonParseException","errorMessage":"Unrecognized token 'agent': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')\n at [Source: (org.eclipse.jetty.server.HttpInputOverHTTP); line: 1, column: 7]"}
<bound method Response.json of <Response [400]>>
b'{"error":"JsonParseException","errorMessage":"Unrecognized token \'agent\': was expecting (JSON String, Number, Array, Object or token \'null\', \'true\' or \'false\')\\n at [Source: (org.eclipse.jetty.server.HttpInputOverHTTP); line: 1, column: 7]"}'

The problem is that you are not sending json string but you are sending a python dictionary. you would need to convert it to json first then send it.
You will need to use json.dumps to convert dictionary object to json string.
header=json.dumps(header) #converting dict to json
mcpayload=json.dumps(mcpayload) #converting dict to json
The code should look like this:
import requests
import json
import pandas as pd
username = input('enter username here: ')
print('')
passw = input('enter password here: ')
mcpayload = {"agent": {"name": "Minecraft", "version": 1}, "username": "{}".format(username), "password": "{}".format(passw), "requestUser": "true"}
header = {"Content-Type": "application/json"}
header=json.dumps(header) #converting dict to json
mcpayload=json.dumps(mcpayload) #converting dict to json
logintoken = requests.post('https://authserver.mojang.com/authenticate', data = mcpayload, headers = header)
print(logintoken.text)
print('')
print(logintoken.json)
print('')
print(logintoken.content)

If you want to send JSON, pass the JSON object via the kwarg json, not data:
logintoken = requests.post(
'https://authserver.mojang.com/authenticate',
json=mcpayload
)
Also, you can omit sending the custom header then, since requests.post() will automagically serialize the object for you and set appropriate headers.

Why am I getting this error "TypeError: string indices must be integers" when trying to fetch data from an api?

json file =
{
"success": true,
"terms": "https://curr
"privacy": "https://cu
"timestamp": 162764598
"source": "USD",
"quotes": {
"USDIMP": 0.722761,
"USDINR": 74.398905,
"USDIQD": 1458.90221
}
}
The json file is above. i deleted lot of values from the json as it took too many spaces. My python code is in below.
import urllib.request, urllib.parse, urllib.error
import json
response = "http://api.currencylayer.com/live?access_key="
api_key = "42141e*********************"
parms = dict()
parms['key'] = api_key
url = response + urllib.parse.urlencode(parms)
mh = urllib.request.urlopen(url)
source = mh.read().decode()
data = json.loads(source)
pydata = json.dumps(data, indent=2)
print("which curreny do you want to convert USD to?")
xm = input('>')
print(f"Hoe many USD do you want to convert{xm}to")
value = input('>')
fetch = pydata["quotes"][0]["USD{xm}"]
answer = fetch*value
print(fetch)
--------------------------------
Here is the
output
"fetch = pydata["quotes"][0]["USD{xm}"]
TypeError: string indices must be integers"

First of all the JSON data you posted here is not valid. There are missing quotes and commas. For example here "terms": "https://curr. It has to be "terms": "https://curr",. The same at "privacy" and the "timestamp" is missing a comma. After i fixed the JSON data I found a solution. You have to use data not pydata. This mean you have to change fetch = pydata["quotes"][0]["USD{xm}"] to fetch = data["quotes"][0]["USD{xm}"]. But this would result in the next error, which would be a KeyError, because in the JSON data you provided us there is no array after the "qoutes" key. So you have to get rid of this [0] or the json data has to like this:
"quotes":[{
"USDIMP": 0.722761,
"USDINR": 74.398905,
"USDIQD": 1458.90221
}]
At the end you only have to change data["quotes"]["USD{xm}"] to data["quotes"]["USD"+xm] because python tries to find a key called USD{xm} and not for example USDIMP, when you type "IMP" in the input.I hope this fixed your problem.

Generate HMAC Sha256 in python 3

I write code to verify an HMAC Auth incoming POST request with JSON to our API. The HMAC I received is OD5ZxL4tdGgWr78e9vO3cYrjuOFT8WOrTbTIuuIH1PQ=
When I try to generate it by my self using Python, it is always different.
Here is the JSON request I received:
{
"shipper_id": 4841,
"status": "Cancelled",
"shipper_ref_no": "",
"tracking_ref_no": "",
"shipper_order_ref_no": "",
"timestamp": "2018-05-23T15:13:28+0800",
"id": "61185ecf-3484-4985-b625-ffe30ba36e28",
"previous_status": "Pending Pickup",
"tracking_id": "NVSGBHINK000000001"
}
And the client secret is 817a3723917f4c7fac24b1f1b324bbab.
The HMAC secret I received is OD5ZxL4tdGgWr78e9vO3cYrjuOFT8WOrTbTIuuIH1PQ=.
Here is the code when I write it in PHP:
<?php
define('CLIENT_SECRET', 'my_shared_secret');
function verify_webhook($data, $hmac_header){
$calculated_hmac = base64_encode(hash_hmac('sha256', $data, CLIENT_SECRET, true));
return ($hmac_header == $calculated_hmac);
}
$hmac_header = $_SERVER['X-NINJAVAN-HMAC-SHA256'];
$data = file_get_contents('php://input');
$verified = verify_webhook($data, $hmac_header);
error_log('Webhook verified: '.var_export($verified, true)); //check error.log to see result
?>
But I have no idea how to do it in Python 3.

In Python 3 you basically want something like the following, taken from how you handle GitHub webhook requests.
import hashlib
import hmac
secret = 'CLIENT_SECRET'
data = rsp.content # assumes you're using requests for data/sig
signature = rsp.headers['X-Something-Signature']
signature_computed = 'sha1=' + hmac.new(
key=secret.encode('utf-8'),
msg=data.encode('utf-8'),
digestmod=hashlib.sha1
).hexdigest()
if not hmac.compare_digest(signature, signature_computed):
log("Invalid payload")

If you want to recreate the hashing code from PHP to Python do it thusly:
def create_signature(key, data):
sig_hash = hmac.new(key.encode('utf8'), data.encode('utf8'), hashlib.sha256).digest()
base64_message = base64.b64encode(sig_hash).decode()
return base64_message
This will create the signature that should match what your PHP code is creating. Just compare the signature to what is sent in the header.

from collections import OrderedDict
params = orderedDict()
params["shipper_id"] = 4841
params["status"] = "Cancelled"
params["shipper_ref_no"] = ""
params["tracking_ref_no"] = ""
params["shipper_order_ref_no"] = ""
params["timestamp"] = "2018-05-23T15:13:28+0800"
params["id"] = "61185ecf-3484-4985-b625-ffe30ba36e28"
params["previous_status"] = "Pending Pickup"
params["tracking_id"] = "NVSGBHINK000000001"
mes = json(params, separator = (";",",")).highdigest()
sighnature = hmac.new(mes, sha256)
# separators = (";",",") - i'm not shure
params['sighnature'] = sighnature
r = response.post(url,params,sighnature)
print(r.text())

How to update dictionary values side by side?

I have a dictionary, I'm creating string with sorting values and adding the length of each field value at the beginning of dictionaries field value.
I have to update this values side by side and i have to hash it:
But this string format is like that.
25http://a1ff569e.ngrok.io/
2TR
17foootest#mail.com
3Adı
6Soyadı
1105316806562
3000
164355084355084358
212
42018
8OPU_TEST
192017-08-28 10:24:57
8Ürünkodu
15Ürün Açıklaması
9Ürün İsmi
3100
11
7Test123
15
218
8CCVISAMC
3TRY
How can i update this string values side by side like below ?
**I know i can print hashstring with end='' but this is not updating hashstring varabile's value like that just shown like that and **
25http://a1ff569e.ngrok.io/
2TR
17payutest#mail.com
3Adı
6Soyadı
1105316806562
3000
164355084355084358
212
42018
8PYKDMNSZ
192017-08-28 10:24:57
8Ürünkodu
15Ürün Açıklaması
9Ürün İsmi
3100
11
7Test123
15
218
8CCVISAMC
3TRY
# Importing required libraries for sample.
from datetime import datetime
import hmac
import hashlib
from urllib.parse import urlencode
from urllib.request import Request, urlopen
# Endpoint
url = "https://secure.payu.com.tr/order/alu/v3"
# PayU Merchant's Secret Key
secret = 'SECRET_KEY'
# Array Begin
array = {
# PayU Merchant's Merchant ID
'MERCHANT': "OPU_TEST",
'ORDER_REF': "Test123",
'ORDER_DATE': datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S'),
'BACK_REF': "http://a1ff569e.ngrok.io/",
'ORDER_PNAME[0]': "Ürün İsmi",
'ORDER_PCODE[0]': "Ürünkodu",
'ORDER_PINFO[0]': "Ürün Açıklaması",
'ORDER_PRICE[0]': "100",
'ORDER_VAT[0]': "18",
'ORDER_QTY[0]': "1",
'ORDER_SHIPPING': "5",
'PRICES_CURRENCY': "TRY",
'PAY_METHOD': "CCVISAMC",
'SELECTED_INSTALLMENTS_NUMBER': "2",
'CC_NUMBER': "4355084355084358",
'EXP_MONTH': "12",
'EXP_YEAR': "2018",
'CC_CVV': "000",
'BILL_FNAME': "Adı",
'BILL_LNAME': "Soyadı",
'BILL_PHONE': "05316806562",
'BILL_EMAIL': "TEST#mail.com",
'BILL_COUNTRYCODE': "TR",
}
# Sorting Array params
for k, v in sorted(array.items()):
# Adding the length of each field value at the beginning of field value
hashstring = str(len(v)) + str(v)
print(hashstring)
# Calculating ORDER_HASH
signature = hmac.new(secret.encode('utf-8'), hashstring.encode('utf-8'), hashlib.md5).hexdigest()
# Adding ORDER_HASH param to dictionary
array['ORDER_HASH'] = signature
print(signature)
print()
# Sending Request to Endpoint
request = Request(url, urlencode(array).encode())
json = urlopen(request).read().decode()
# Printing result
print(json)

I'm not sure but maybe it's works.
Please try and share results with us.
hashstring=print(hashstring+" ", end="")

Example of update_item in dynamodb boto3

Following the documentation, I'm trying to create an update statement that will update or add if not exists only one attribute in a dynamodb table.
I'm trying this
response = table.update_item(
Key={'ReleaseNumber': '1.0.179'},
UpdateExpression='SET',
ConditionExpression='Attr(\'ReleaseNumber\').eq(\'1.0.179\')',
ExpressionAttributeNames={'attr1': 'val1'},
ExpressionAttributeValues={'val1': 'false'}
)
The error I'm getting is:
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the UpdateItem operation: ExpressionAttributeNames contains invalid key: Syntax error; key: "attr1"
If anyone has done anything similar to what I'm trying to achieve please share example.

Found working example here, very important to list as Keys all the indexes of the table, this will require additional query before update, but it works.
response = table.update_item(
Key={
'ReleaseNumber': releaseNumber,
'Timestamp': result[0]['Timestamp']
},
UpdateExpression="set Sanity = :r",
ExpressionAttributeValues={
':r': 'false',
},
ReturnValues="UPDATED_NEW"
)

Details on dynamodb updates using boto3 seem incredibly sparse online, so I'm hoping these alternative solutions are useful.
get / put
import boto3
table = boto3.resource('dynamodb').Table('my_table')
# get item
response = table.get_item(Key={'pkey': 'asdf12345'})
item = response['Item']
# update
item['status'] = 'complete'
# put (idempotent)
table.put_item(Item=item)
actual update
import boto3
table = boto3.resource('dynamodb').Table('my_table')
table.update_item(
Key={'pkey': 'asdf12345'},
AttributeUpdates={
'status': 'complete',
},
)

If you don't want to check parameter by parameter for the update I wrote a cool function that would return the needed parameters to perform a update_item method using boto3.
def get_update_params(body):
"""Given a dictionary we generate an update expression and a dict of values
to update a dynamodb table.
Params:
body (dict): Parameters to use for formatting.
Returns:
update expression, dict of values.
"""
update_expression = ["set "]
update_values = dict()
for key, val in body.items():
update_expression.append(f" {key} = :{key},")
update_values[f":{key}"] = val
return "".join(update_expression)[:-1], update_values
Here is a quick example:
def update(body):
a, v = get_update_params(body)
response = table.update_item(
Key={'uuid':str(uuid)},
UpdateExpression=a,
ExpressionAttributeValues=dict(v)
)
return response

The original code example:
response = table.update_item(
Key={'ReleaseNumber': '1.0.179'},
UpdateExpression='SET',
ConditionExpression='Attr(\'ReleaseNumber\').eq(\'1.0.179\')',
ExpressionAttributeNames={'attr1': 'val1'},
ExpressionAttributeValues={'val1': 'false'}
)
Fixed:
response = table.update_item(
Key={'ReleaseNumber': '1.0.179'},
UpdateExpression='SET #attr1 = :val1',
ConditionExpression=Attr('ReleaseNumber').eq('1.0.179'),
ExpressionAttributeNames={'#attr1': 'val1'},
ExpressionAttributeValues={':val1': 'false'}
)
In the marked answer it was also revealed that there is a Range Key so that should also be included in the Key. The update_item method must seek to the exact record to be updated, there's no batch updates, and you can't update a range of values filtered to a condition to get to a single record. The ConditionExpression is there to be useful to make updates idempotent; i.e. don't update the value if it is already that value. It's not like a sql where clause.
Regarding the specific error seen.
ExpressionAttributeNames is a list of key placeholders for use in the UpdateExpression, useful if the key is a reserved word.
From the docs, "An expression attribute name must begin with a #, and be followed by one or more alphanumeric characters". The error is because the code hasn't used an ExpressionAttributeName that starts with a # and also not used it in the UpdateExpression.
ExpressionAttributeValues are placeholders for the values you want to update to, and they must start with :

Based on the official example, here's a simple and complete solution which could be used to manually update (not something I would recommend) a table used by a terraform S3 backend.
Let's say this is the table data as shown by the AWS CLI:
$ aws dynamodb scan --table-name terraform_lock --region us-east-1
{
"Items": [
{
"Digest": {
"S": "2f58b12ae16dfb5b037560a217ebd752"
},
"LockID": {
"S": "tf-aws.tfstate-md5"
}
}
],
"Count": 1,
"ScannedCount": 1,
"ConsumedCapacity": null
}
You could update it to a new digest (say you rolled back the state) as follows:
import boto3
dynamodb = boto3.resource('dynamodb', 'us-east-1')
try:
table = dynamodb.Table('terraform_lock')
response = table.update_item(
Key={
"LockID": "tf-aws.tfstate-md5"
},
UpdateExpression="set Digest=:newDigest",
ExpressionAttributeValues={
":newDigest": "50a488ee9bac09a50340c02b33beb24b"
},
ReturnValues="UPDATED_NEW"
)
except Exception as msg:
print(f"Oops, could not update: {msg}")
Note the : at the start of ":newDigest": "50a488ee9bac09a50340c02b33beb24b" they're easy to miss or forget.

Small update of Jam M. Hernandez Quiceno's answer, which includes ExpressionAttributeNames to prevent encoutering errors such as:
"errorMessage": "An error occurred (ValidationException) when calling the UpdateItem operation:
Invalid UpdateExpression: Attribute name is a reserved keyword; reserved keyword: timestamp",
def get_update_params(body):
"""
Given a dictionary of key-value pairs to update an item with in DynamoDB,
generate three objects to be passed to UpdateExpression, ExpressionAttributeValues,
and ExpressionAttributeNames respectively.
"""
update_expression = []
attribute_values = dict()
attribute_names = dict()
for key, val in body.items():
update_expression.append(f" #{key.lower()} = :{key.lower()}")
attribute_values[f":{key.lower()}"] = val
attribute_names[f"#{key.lower()}"] = key
return "set " + ", ".join(update_expression), attribute_values, attribute_names
Example use:
update_expression, attribute_values, attribute_names = get_update_params(
{"Status": "declined", "DeclinedBy": "username"}
)
response = table.update_item(
Key={"uuid": "12345"},
UpdateExpression=update_expression,
ExpressionAttributeValues=attribute_values,
ExpressionAttributeNames=attribute_names,
ReturnValues="UPDATED_NEW"
)
print(response)

An example to update any number of attributes given as a dict, and keep track of the number of updates. Works with reserved words (i.e name).
The following attribute names shouldn't be used as we will overwrite the value: _inc, _start.
from typing import Dict
from boto3 import Session
def getDynamoDBSession(region: str = "eu-west-1"):
"""Connect to DynamoDB resource from boto3."""
return Session().resource("dynamodb", region_name=region)
DYNAMODB = getDynamoDBSession()
def updateItemAndCounter(db_table: str, item_key: Dict, attributes: Dict) -> Dict:
"""
Update item or create new. If the item already exists, return the previous value and
increase the counter: update_counter.
"""
table = DYNAMODB.Table(db_table)
# Init update-expression
update_expression = "SET"
# Build expression-attribute-names, expression-attribute-values, and the update-expression
expression_attribute_names = {}
expression_attribute_values = {}
for key, value in attributes.items():
update_expression += f' #{key} = :{key},' # Notice the "#" to solve issue with reserved keywords
expression_attribute_names[f'#{key}'] = key
expression_attribute_values[f':{key}'] = value
# Add counter start and increment attributes
expression_attribute_values[':_start'] = 0
expression_attribute_values[':_inc'] = 1
# Finish update-expression with our counter
update_expression += " update_counter = if_not_exists(update_counter, :_start) + :_inc"
return table.update_item(
Key=item_key,
UpdateExpression=update_expression,
ExpressionAttributeNames=expression_attribute_names,
ExpressionAttributeValues=expression_attribute_values,
ReturnValues="ALL_OLD"
)
Hope it might be useful to someone!

In a simple way you can use below code to update item value with new one:
response = table.update_item(
Key={"my_id_name": "my_id_value"}, # to get record
UpdateExpression="set item_key_name=:item_key_value", # Operation action (set)
ExpressionAttributeValues={":value": "new_value"}, # item that you need to update
ReturnValues="UPDATED_NEW" # optional for declarative message
)

Simple example with multiple fields:
import boto3
dynamodb_client = boto3.client('dynamodb')
dynamodb_client.update_item(
TableName=table_name,
Key={
'PK1': {'S': 'PRIMARY_KEY_VALUE'},
'SK1': {'S': 'SECONDARY_KEY_VALUE'}
}
UpdateExpression='SET #field1 = :field1, #field2 = :field2',
ExpressionAttributeNames={
'#field1': 'FIELD_1_NAME',
'#field2': 'FIELD_2_NAME',
},
ExpressionAttributeValues={
':field1': {'S': 'FIELD_1_VALUE'},
':field2': {'S': 'FIELD_2_VALUE'},
}
)

using previous answer from eltbus , it worked for me , except for minor bug,
You have to delete the extra comma using update_expression[:-1]