I am building a web application that includes a file upload feature. My goal is to initiate upload from users directly to an S3 bucket. The strategy is to pre-sign a POST request that will get submitted as a form.
The roadblock is a SignatureDoesNotMatch error - as far as I can tell I've conformed to the documentation, and have explored a lot of options, but still unable to resolve. I am able to generate presigned download links.
Referencing:
AWS POST documentation
Example
boto3 generate_presigned_post reference
Generate signed request:
def s3_upload_creds(name, user):
s3 = boto3.client('s3')
key = '${filename}'
region = 'us-east-1'
date_short = datetime.datetime.utcnow().strftime('%Y%m%d')
date_long = datetime.datetime.utcnow().strftime('%Y%m%dT000000Z')
fields = {
'acl': 'private',
'date': date_short,
'region': region,
'x-amz-algorithm': 'AWS4-HMAC-SHA256',
'x-amz-date': date_long
}
return s3.generate_presigned_post(
Bucket = 'leasy',
Fields = fields,
Key = key,
Conditions = [
{'acl': 'private'},
{'x-amz-algorithm': 'AWS4-HMAC-SHA256'},
{'x-amz-credential': '/'.join(['AKI--snip--', date_short, region, 's3', 'aws4_request'])},
{'x-amz-date': date_long}
]
)
Upload form (populated with fields above):
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
{{ creds }}
<form action="{{ creds.url }}" method="post" enctype="multipart/form-data">
Key to upload:
<input type="input" name="key" value="${filename}" /><br />
<input type="input" name="acl" value="{{ creds.fields.acl }}" />
<input type="hidden" name="Policy" value="{{ creds.fields.policy }}" />
<input type="text" name="X-Amz-Algorithm" value="{{ creds.fields['x-amz-algorithm'] }}" />
<input type="input" name="X-Amz-Credential" value="{{ creds.fields.AWSAccessKeyId }}/{{ creds.fields.date }}/us-east-1/s3/aws4_request" />
<input type="input" name="X-Amz-Date" value="{{ creds.fields['x-amz-date'] }}" />
<input type="input" name="X-Amz-Signature" value="{{ creds.fields.signature }}" />
File:
<input type="file" name="file" /> <br />
<!-- The elements after this will be ignored -->
<input type="submit" name="submit" value="Upload to Amazon S3" />
</form>
</html>
Relevant portion of response:
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>
The request signature we calculated does not match the signature you provided. Check your key and signing method.
</Message>
<AWSAccessKeyId>AKI--snip--</AWSAccessKeyId>
<StringToSign>
eyJjb25kaXRpb25zIjogW3siYWNsIjogInByaXZhdGUifSwgeyJ4LWFtei1hbGdvcml0aG0iOiAiQVdTNC1ITUFDLVNIQTI1NiJ9LCB7IngtYW16LWNyZWRlbnRpYWwiOiAiQUtJQUlDVjRNVlBUUlFHU1lLV1EvMjAxNTEyMTgvdXMtZWFzdC0xL3MzL2F3czRfcmVxdWVzdCJ9LCB7IngtYW16LWRhdGUiOiAiMjAxNTEyMThUMDAwMDAwWiJ9LCB7ImJ1Y2tldCI6ICJsZWFzeSJ9LCBbInN0YXJ0cy13aXRoIiwgIiRrZXkiLCAiIl1dLCAiZXhwaXJhdGlvbiI6ICIyMDE1LTEyLTE4VDA1OjEwOjU2WiJ9
</StringToSign>
<SignatureProvided>wDOjsBRc0iIW7JNtz/4GHgfvKaU=</SignatureProvided>
Base64 decode of StringToSign in above error:
{u'conditions': [{u'acl': u'private'},
{u'x-amz-algorithm': u'AWS4-HMAC-SHA256'},
{u'x-amz-credential': u'AKI--snip--/20151218/us-east-1/s3/aws4_request'},
{u'x-amz-date': u'20151218T000000Z'},
{u'bucket': u'leasy'},
[u'starts-with', u'$key', u'']],
u'expiration': u'2015-12-18T04:59:32Z'}
Found a solution: had to explicitly configure the s3 client to use Amazon's new signature v4. The error occurs since it defaults to an older version, causing the mismatch. Bit of a facepalm - at the time this wasn't written in boto3 docs, although folks at Amazon say it should be soon.
The method is simplified since it now returns exactly the fields required:
def s3_upload_creds(name):
BUCKET = 'mybucket'
REGION = 'us-west-1'
s3 = boto3.client('s3', region_name=REGION, config=Config(signature_version='s3v4'))
key = '${filename}'
return s3.generate_presigned_post(
Bucket = BUCKET,
Key = key
)
Which means the form can be easily generated:
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
{{ creds }}
<form action="https://mybucket.s3.amazonaws.com" method="post" enctype="multipart/form-data">
{% for key, value in creds.fields.items() %}
<input type="hidden" name="{{ key }}" value="{{ value }}" />
{% endfor %}
File:
<input type="file" name="file" /> <br />
<input type="submit" name="submit" value="Upload to Amazon S3" />
</form>
</html>
Cheers
Been a few years since the last response, but I've been stuck on this for the last day or two so I'll share my experience for anyone it may help.
I had been getting the error: "403: The AWS Access Key Id you provided does not exist in our records" when trying to upload to an s3 bucket via my presigned url.
I was able to successfully generate a presigned url similarly to above, using the server-side code:
signed_url_dict = self.s3_client.generate_presigned_post(
self.bucket_name,
object_name,
ExpiresIn=300
This returned a dictionary with the structure:
{
url: "https://___",
fields: {
key: "___",
AWSAccesKeyId: "___",
x-amz-security-token: "___",
policy: "___",
signature: "___"
}
}
This lead to the part where things were a little different now in 2019 with the browser-side javascript, where the required form inputs seem to have changed. Instead of setting up the form as OP did, I had to create my form as seen below:
<form action="https://pipeline-poc-ed.s3.amazonaws.com/" method="post" enctype="multipart/form-data" name="upload_form">
<!-- Copy ALL of the 'fields' key:values returned by S3Client.generate_presigned_post() -->
<input type="hidden" name="key" value="___" />
<input type="hidden" name="AWSAccessKeyId" value="___" />
<input type="hidden" name="policy" value="___"/>
<input type="hidden" name="signature" value="___" />
<input type="hidden" name="x-amz-security-token" value="___" />
File:
<input type="file" name="file" /> <br />
<input type="submit" name="submit" value="Upload to Amazon S3" />
</form>
My error was that I followed an example in the boto3 1.9.138 docs and left out "x-amz-security-token" on the form, which turned out to be quite necessary. A thoughtless oversight on may part, but hopefully this will help someone else.
EDIT: My results above were based on a N. Virginia Lambda Function. When I ran generate_presigned_post(...) in Ohio (the region containing my bucket), I got results similar to OP:
{
"url": "https://__",
"fields": {
"key": "___",
"x-amz-algorithm": "___",
"x-amz-credential": "___",
"x-amz-date": "___",
"x-amz-security-token": "___",
"policy": "___",
"x-amz-signature": "___"
}
}
Perhaps the results of the function are region specific?
In my case, I was generating a form with Base64-encoded.
The problem was due to Firefox inherently encoding the Policy and Security token values into Base64-encoded on top of it.
Thus there was double encoding and therefore the signature did not match as required.
Related
I am trying to do a simple create_presigned_post using python and boto3.
import boto3
from botocore.config import Config
def s3_upload_creds():
REGION = 'eu-west-2'
s3 = boto3.client('s3',
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key, region_name=REGION, config=Config(signature_version='s3v4'))
return s3.generate_presigned_post(
Bucket = bucket,
Key = key,
ExpiresIn=3600
)
upload_fields = s3_upload_creds()
url = upload_fields['url']
upload_fields = upload_fields['fields']
"""<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
<form action="{url}" method="post" enctype="multipart/form-data">
<input type="text" id="x-amz-algorithm" name="x-amz-algorithm" value="AWS4-HMAC-SHA256" /><br />
<input type="text" id="x-amz-credential" name="x-amz-credential" value="{creds}" /><br />
<input type="text" id="x-amz-date" name="x-amz-date" value="{date}" /><br />
<input type="text" id="x-amz-policy" name="policy" value="{policy}" /><br />
<input type="text" id="signature" name="signature" value="{signature}" />
<input type="text" id="key" name="key" value="{key}" />
File:
<input type="file" name="file" /> <br />
<input type="submit" name="submit" value="Upload to Amazon S3" />
</form>
</html>""".format(
url=url,
creds=upload_fields['x-amz-credential'],
date=upload_fields['x-amz-date'],
policy=upload_fields['policy'],
signature=upload_fields['x-amz-signature'],
key=upload_fields['key']
)
However I receive this error.
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>InvalidArgument</Code>
<Message>Bucket POST must contain a field named 'AWSAccessKeyId'. If it is specified, please check the order of the fields.</Message>
<ArgumentName>AWSAccessKeyId</ArgumentName>
<ArgumentValue></ArgumentValue>
<RequestId>DFEDBZFEES4PFQV3</RequestId>
<HostId>k7wj2Ehd/DjtpVk+OtG0qGFRECtTYkQv64hEwLRFkqKR4Qfhj0nbOHKS5DNqWo/TTGR3BbC6k=</HostId>
</Error>
Browsing solutions online I found that if the file is specified before the other fields, it will not work. I have verified that it is not the case on my browser and Postman.
I tried to add the field but it gives me this error.
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>InvalidRequest</Code>
<Message>The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.</Message>
<RequestId>AYGPAQPQFDWFE3Q6N</RequestId>
<HostId>aYTEn/SyElUsyFPRFEDWtZ8SYKuQWB7bvaxvmq0tvXkPFDWlYAixsO6ue+tEbVstaWGRF4KY=</HostId>
</Error>
I do not understand why aws needs AWSAccessKeyId since I can see it at the start of x-amz-credential:
AK-----------------R/date/region/s3/aws4_requests
I Have looked
at theses:
Link
Link
Thanks Anon Coward for your help.
The issue come from the html form I used. One of the fields where rename with the name html attribute wrongly, causing and error which message is out of context. The sample above should change name='signature' to name='x-amz-signature'.
Note that the field names returned from create_presigned_post are:
key
x-amz-algorithm
x-amz-credential
x-amz-date
policy
x-amz-signature
Getting a 400, when trying to upload a file ad send other form elements to flask from html. Tried to use ajax, but that throws me an error as well.
Python:
#app.route('/prod_diff_result', methods=['POST', 'GET'])
def prod_diff_result():
try:
host = request.form['host-prod-iterator']
print(host)
if request.files['file']:
f = request.files['file']
f.save(secure_filename(f.filename))
HTML:
<div class="main-div">
<form action="/prod_diff_result" method="POST" enctype="multipart/form-data">
<div class="grid-container">
<div class="grid-item">
<span class="label label-default ">PROD</span><br>
<p>Iterator Host : <input type="text" class="form-control" id="host-prod-iterator" value="10.47.7.57"
required></p>
<input type="radio" name="data_fetch_type" value="file" onclick="showfile()">Upload File
<input type="file" name="file" />
<input type="radio" name="data_fetch_type" value="db"> Get from DB
<input type="submit" />
</div>
</form>
</div>
I want to be able send hostname and file back to flask error in one request and using one form.
It gives an error because you try to access a form field that it cannot find, and assumes that somehow the request was bad, because it didn't include a required form field. You are trying to access:
host = request.form['host-prod-iterator']
However you have simply not given it a name in your HTML. If you give it a name, it should work:
<p>Iterator Host :
<input type="text" class="form-control" name="host-prod-iterator" id="host-prod-iterator" value="10.47.7.57" required>
</p>
I have the following form in my forms.py :
class PaymentMethodForm(forms.Form):
def __init__(self, *args, **kwargs):
super(PaymentMethodForm, self).__init__(*args, **kwargs)
payment_choices = ['online payment', 'payment at delivery']
self.payment_method_choice = forms.ChoiceField(choices=payment_choices, widget=forms.RadioSelect)
now this is the page in an online shop where you select your payment method. as you see, we have two methods, one is payment at delivery time which means there should not be any payments in the website, and the other one is online payment. In the case of the user choosing online payment, I should submit a form via post that looks something like this and after that redirect the user to the action url :
<form id="Form2" method="post" Action="https://somepaymentsite.com/gateway.aspx"
>
<input type="hidden" name="invoiceNumber" value="<%= invoiceNumber %>"
/>
<input type="hidden" name="invoiceDate" value="<%= invoiceDate %>" />
<input type="hidden" name="amount" value="<%= amount %>" />
<input type="hidden" name="terminalCode" value="<%= terminalCode %>" />
<input type="hidden" name="merchantCode" value="<%= merchantCode %>" />
<input type="hidden" name="redirectAddress" value="<%= redirectAddress
%>" />
<input type="hidden" name="timeStamp" value="<%= timeStamp %>" />
<input type="hidden" name="action" value="<%= action %>" />
<input type="hidden" name="sign" value="<%= sign %>" />
<input type="submit" name="submit" value=""continue />
</form>
now what I have in mind for doing this is that instead of putting this html stuff in my template (html file) I will get the radio button choice the user has chosen and if they chose online payment, I will submit a form like the above in my views.py . the problem is, I do not know how to go about this. I googled things but I didn't find anything good on how to post a form in a view in django. Can anybody help me? thanks.
If I understood correctly, "submit form from the view" is the same as do POST request to https://somepaymentsite.com/gateway.aspx with data {'invoiceNumber': 12345, 'invoiceDate': '10.10.2015', ...}. You can use requests library in this case.
# in view
# if online payment then
r = request.post('https://somepaymentsite.com/gateway.aspx', data=payload)
# processing r (error handling or something else)
where payload is dictionary {'invoiceNumber': 12345, 'invoiceDate': '10.10.2015', ...} that you construct somehow.
So, I've hacked this together from a few sources, so if I'm totally going about it the wrong way I welcome feed back. It also occurs to me that this is not possible, as it's probably a security check designed to prevent this behavior being used maliciously.
But anyway:
I have a form on our Django site where people can request to change the name of one of our items, which should automatically create a jira ticket. Here's the form:
<form target="_blank" action='http://issues.dowjones.net/secure/CreateIssueDetails!init.jspa' method='get' id='create_jira_ticket_form'>
<a id='close_name_change_form' class="close">×</a>
<label for="new_name">New name: </label>
<input id="new_name" type="text" name="new_name" value="{{item.name}}">
<input type="hidden" value="10517" name="pid">
<input type="hidden" value="3" name="issuetype">
<input type="hidden" value="5" name="priority">
<input type="hidden" value="Change name of {{item.name}} to " name="summary" id='summary'>
<input type="hidden" value="{{request.user}}" name="reporter">
<input type="hidden" value="user123" name="assignee">
<input type="hidden" value="" name="description" id="description">
<input id='name_change_submit' class="btn btn-primary btn-sm" type="submit" value="Create JIRA ticket">
</form>
Then I have a little JS to amend the fields with the new values:
$(document).ready(function(){
$('#create_jira_ticket_form').submit(function(){
var watchers = ' \[\~watcher1\] \[\watcher2\]';
var new_name = $('#new_name').val();
var summary = $('#summary').val();
$('#summary').val(summary + new_name);
$('#description').val(summary + new_name + watchers);
})
})
It comes very close to working, but the description field is escaped, leaving it looking like:
Change name of OLDNAME to NEWNAME %5B%7Ewatcher1t%5D %5B%7Ewatcher2%5D
Which is less than helpful. How can I keep it as is so I can add watchers?
This happens when your form encodes the fields and values in your form.
You can try this out by this simple snippet:
console.log($('form').serialize());
you should see something like
description=ejdd+%5B~watcher1%5D+%5Bwatcher2%5D
in order to prevent this you should change your method='get' to method='post'.
The encoding happens because it's apart of HTTP, read here why
You can also read the spec paragraph
17.13.3 Processing form data
scriptInfo.py
import os, sys, platform, webbrowser
def main()
template = open('scriptHmtl.phtml').read()
scriptHtml.phtml
<html>
<head>
</head>
<body>
<h2><center> welcome </center></h2>
<br/><br/><br/>
...
variables
..
<form name="sendData" method="get" action="http://localhost:8000/cgi/scriptGet.py">
Name: <input type="text" name="n"><br/><br/>
First Name: <input type="text" name="fn"/><br/><br/>
Mail: <input type="text" name="ma"/><br/><br/>
Address: <input type="text" name="add"/> <br/><br/>
<input type="submit" value="OK"/>
</form>
Instead of action="http://localhost:8000/cgi/scriptGet.py", there must be a variable which contain the code to recover the server address, but I don't want how to do it.
With HTML forms you can just ignore the server and go straight to the script.
For example like
<form name="sendData" method="get" action="cgi/scriptGet.py">