Until just a couple of weeks ago i was able to get the json data from a website after performing some post requests
payload = {'NumDossier': numCase_com, 'idJuridiction': numJuri, 'typeDossier': "DP"}
r = requests.post(url, json=payload)
Now i am getting this strange HTML on every post request
<html xmlns="http://www.w3.org/1999/xhtml">
<body>
<form method="post" action="./getJuridiction1instance" id="form1">
<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="/wEPDwULLTE2MTY2ODcyMjlkZDDGgir+XqoIRtkvd//GurKfYTbq8hIisZRyOefPLUDj" />
<input type="hidden" name="__VIEWSTATEGENERATOR" id="__VIEWSTATEGENERATOR" value="79390B9D" />
<div>
</div>
</form>
</body>
</html>
When inspecting the newtwork on chrome, in no time it sends __VIEWSTATE or __VIEWSTATEGENERATOR, i spent two days trying to figure out what i am doing wrong but nothing. (I am no expert in ASP.Net)
PS: I get the same result with postman
I am building a web application that includes a file upload feature. My goal is to initiate upload from users directly to an S3 bucket. The strategy is to pre-sign a POST request that will get submitted as a form.
The roadblock is a SignatureDoesNotMatch error - as far as I can tell I've conformed to the documentation, and have explored a lot of options, but still unable to resolve. I am able to generate presigned download links.
Referencing:
AWS POST documentation
Example
boto3 generate_presigned_post reference
Generate signed request:
def s3_upload_creds(name, user):
s3 = boto3.client('s3')
key = '${filename}'
region = 'us-east-1'
date_short = datetime.datetime.utcnow().strftime('%Y%m%d')
date_long = datetime.datetime.utcnow().strftime('%Y%m%dT000000Z')
fields = {
'acl': 'private',
'date': date_short,
'region': region,
'x-amz-algorithm': 'AWS4-HMAC-SHA256',
'x-amz-date': date_long
}
return s3.generate_presigned_post(
Bucket = 'leasy',
Fields = fields,
Key = key,
Conditions = [
{'acl': 'private'},
{'x-amz-algorithm': 'AWS4-HMAC-SHA256'},
{'x-amz-credential': '/'.join(['AKI--snip--', date_short, region, 's3', 'aws4_request'])},
{'x-amz-date': date_long}
]
)
Upload form (populated with fields above):
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
{{ creds }}
<form action="{{ creds.url }}" method="post" enctype="multipart/form-data">
Key to upload:
<input type="input" name="key" value="${filename}" /><br />
<input type="input" name="acl" value="{{ creds.fields.acl }}" />
<input type="hidden" name="Policy" value="{{ creds.fields.policy }}" />
<input type="text" name="X-Amz-Algorithm" value="{{ creds.fields['x-amz-algorithm'] }}" />
<input type="input" name="X-Amz-Credential" value="{{ creds.fields.AWSAccessKeyId }}/{{ creds.fields.date }}/us-east-1/s3/aws4_request" />
<input type="input" name="X-Amz-Date" value="{{ creds.fields['x-amz-date'] }}" />
<input type="input" name="X-Amz-Signature" value="{{ creds.fields.signature }}" />
File:
<input type="file" name="file" /> <br />
<!-- The elements after this will be ignored -->
<input type="submit" name="submit" value="Upload to Amazon S3" />
</form>
</html>
Relevant portion of response:
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>
The request signature we calculated does not match the signature you provided. Check your key and signing method.
</Message>
<AWSAccessKeyId>AKI--snip--</AWSAccessKeyId>
<StringToSign>
eyJjb25kaXRpb25zIjogW3siYWNsIjogInByaXZhdGUifSwgeyJ4LWFtei1hbGdvcml0aG0iOiAiQVdTNC1ITUFDLVNIQTI1NiJ9LCB7IngtYW16LWNyZWRlbnRpYWwiOiAiQUtJQUlDVjRNVlBUUlFHU1lLV1EvMjAxNTEyMTgvdXMtZWFzdC0xL3MzL2F3czRfcmVxdWVzdCJ9LCB7IngtYW16LWRhdGUiOiAiMjAxNTEyMThUMDAwMDAwWiJ9LCB7ImJ1Y2tldCI6ICJsZWFzeSJ9LCBbInN0YXJ0cy13aXRoIiwgIiRrZXkiLCAiIl1dLCAiZXhwaXJhdGlvbiI6ICIyMDE1LTEyLTE4VDA1OjEwOjU2WiJ9
</StringToSign>
<SignatureProvided>wDOjsBRc0iIW7JNtz/4GHgfvKaU=</SignatureProvided>
Base64 decode of StringToSign in above error:
{u'conditions': [{u'acl': u'private'},
{u'x-amz-algorithm': u'AWS4-HMAC-SHA256'},
{u'x-amz-credential': u'AKI--snip--/20151218/us-east-1/s3/aws4_request'},
{u'x-amz-date': u'20151218T000000Z'},
{u'bucket': u'leasy'},
[u'starts-with', u'$key', u'']],
u'expiration': u'2015-12-18T04:59:32Z'}
Found a solution: had to explicitly configure the s3 client to use Amazon's new signature v4. The error occurs since it defaults to an older version, causing the mismatch. Bit of a facepalm - at the time this wasn't written in boto3 docs, although folks at Amazon say it should be soon.
The method is simplified since it now returns exactly the fields required:
def s3_upload_creds(name):
BUCKET = 'mybucket'
REGION = 'us-west-1'
s3 = boto3.client('s3', region_name=REGION, config=Config(signature_version='s3v4'))
key = '${filename}'
return s3.generate_presigned_post(
Bucket = BUCKET,
Key = key
)
Which means the form can be easily generated:
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
{{ creds }}
<form action="https://mybucket.s3.amazonaws.com" method="post" enctype="multipart/form-data">
{% for key, value in creds.fields.items() %}
<input type="hidden" name="{{ key }}" value="{{ value }}" />
{% endfor %}
File:
<input type="file" name="file" /> <br />
<input type="submit" name="submit" value="Upload to Amazon S3" />
</form>
</html>
Cheers
Been a few years since the last response, but I've been stuck on this for the last day or two so I'll share my experience for anyone it may help.
I had been getting the error: "403: The AWS Access Key Id you provided does not exist in our records" when trying to upload to an s3 bucket via my presigned url.
I was able to successfully generate a presigned url similarly to above, using the server-side code:
signed_url_dict = self.s3_client.generate_presigned_post(
self.bucket_name,
object_name,
ExpiresIn=300
This returned a dictionary with the structure:
{
url: "https://___",
fields: {
key: "___",
AWSAccesKeyId: "___",
x-amz-security-token: "___",
policy: "___",
signature: "___"
}
}
This lead to the part where things were a little different now in 2019 with the browser-side javascript, where the required form inputs seem to have changed. Instead of setting up the form as OP did, I had to create my form as seen below:
<form action="https://pipeline-poc-ed.s3.amazonaws.com/" method="post" enctype="multipart/form-data" name="upload_form">
<!-- Copy ALL of the 'fields' key:values returned by S3Client.generate_presigned_post() -->
<input type="hidden" name="key" value="___" />
<input type="hidden" name="AWSAccessKeyId" value="___" />
<input type="hidden" name="policy" value="___"/>
<input type="hidden" name="signature" value="___" />
<input type="hidden" name="x-amz-security-token" value="___" />
File:
<input type="file" name="file" /> <br />
<input type="submit" name="submit" value="Upload to Amazon S3" />
</form>
My error was that I followed an example in the boto3 1.9.138 docs and left out "x-amz-security-token" on the form, which turned out to be quite necessary. A thoughtless oversight on may part, but hopefully this will help someone else.
EDIT: My results above were based on a N. Virginia Lambda Function. When I ran generate_presigned_post(...) in Ohio (the region containing my bucket), I got results similar to OP:
{
"url": "https://__",
"fields": {
"key": "___",
"x-amz-algorithm": "___",
"x-amz-credential": "___",
"x-amz-date": "___",
"x-amz-security-token": "___",
"policy": "___",
"x-amz-signature": "___"
}
}
Perhaps the results of the function are region specific?
In my case, I was generating a form with Base64-encoded.
The problem was due to Firefox inherently encoding the Policy and Security token values into Base64-encoded on top of it.
Thus there was double encoding and therefore the signature did not match as required.
I got the following code:
import urllib
import re
html = urllib.urlopen("http://jshawl.com/python-playground/").read()
lines = [html]
for line in lines:
if re.findall("jesseshawl", line):
print line
My output when I run this code, is that it wil return the full website. How can I only display the row where it did found "jesseshawl". It should return something like:
jesseshawl#gmail.com
And is there a way to not return all html tags when I run this?
My output:
<html>
<head></head>
<body>
<h1>Some images to download:</h1>
<img src='python.gif'/><br />
<img src='terminal.png' />
<hr />
<h1>Email addresses to extract:</h1>
jesseshawl#gmail.com<br />
sudojesse#gmail.com<br />
<hr />
<h1>Login Form:</h1>
Login here:<br />
User: user<br />
Pass: pass
<form method="POST" action="login.php">
User: <input type="text" name="username" /><br />
Pass: <input type="password" name="password" /><br />
<input type="submit" />
</form>
<h1>Memorable Quotes</h1>
<ul>
<li></li>
</ul>
</body>
</html>
You are reading the whole page .S0 it prints all the thing .You have to read it line by line.There is no need for findall you can use in operator
Code:
import urllib
import re
html = urllib.urlopen("http://jshawl.com/python-playground/").readlines()
for line in html :
if "jesseshawl" in line:
print line
Output:
jesseshawl#gmail.com<br />
And if you don't want tags you could remove them using sub
Code2:
import urllib
import re
html = urllib.urlopen("http://jshawl.com/python-playground/").readlines()
for line in html :
if "jesseshawl" in line:
print re.sub("<[^>]*?>","",line)
Output2:
jesseshawl#gmail.com
scriptInfo.py
import os, sys, platform, webbrowser
def main()
template = open('scriptHmtl.phtml').read()
scriptHtml.phtml
<html>
<head>
</head>
<body>
<h2><center> welcome </center></h2>
<br/><br/><br/>
...
variables
..
<form name="sendData" method="get" action="http://localhost:8000/cgi/scriptGet.py">
Name: <input type="text" name="n"><br/><br/>
First Name: <input type="text" name="fn"/><br/><br/>
Mail: <input type="text" name="ma"/><br/><br/>
Address: <input type="text" name="add"/> <br/><br/>
<input type="submit" value="OK"/>
</form>
Instead of action="http://localhost:8000/cgi/scriptGet.py", there must be a variable which contain the code to recover the server address, but I don't want how to do it.
With HTML forms you can just ignore the server and go straight to the script.
For example like
<form name="sendData" method="get" action="cgi/scriptGet.py">
I am using django framework 1.6.2, eclipse 3.8.1, python 2.7.5
i have followed This link as my sample test program, unfortunately i am getting issue but its working perfect with out using eclipse. please help out, your help is really appreciable
Client side problem:
The code is
<html>
<head>
</head>
<body>
<h1>Braintree Credit Card Transaction Form</h1>
<div>
<form action="/create_transaction" method="POST" id="braintree-payment-form">
<p>
<label>Card Number</label>
<input type="text" size="20" autocomplete="off" data-encrypted-name="number" />
</p>
<p>
<label>CVV</label>
<input type="text" size="4" autocomplete="off" data-encrypted-name="cvv" />
</p>
<p>
<label>Expiration (MM/YYYY)</label>
<input type="text" size="2" data-encrypted-name="month" /> / <input type="text" size="4" data-encrypted-name="year" />
</p>
<input type="submit" id="submit" />
</form>
</div>
<script src="https://js.braintreegateway.com/v1/braintree.js"></script>
<script>
var braintree = Braintree.create("YourClientSideEncryptionKey");
braintree.onSubmitEncryptForm('braintree-payment-form');
</script>
</body>
</html>
The Bug is
Uncaught ReferenceError: Braintree is not defined
Server side problem:
The code is
import braintree
braintree.Configuration.configure(braintree.Environment.Sandbox,
merchant_id="use_your_merchant_id",
public_key="use_your_public_key",
private_key="use_your_private_key")
Bug is
Undefined variable from import:Sandbox
I used v2 script, but have the same error. The problem was in async loading of scripts.
Your script braintree.js is not loaded yet, but next script is already executed and you see error that braintree is not defined.
For solving this problem I used jQuery getScript, and executed second part in callback.
<script>
$.getScript( "https://js.braintreegateway.com/v2/braintree.js", function() {
var clientToken = "myTokenClient";
braintree.setup(clientToken, "dropin", {
container: "payment-form"
});
});
</script>
I fixed it by changing to
src="https://js.braintreegateway.com/v2/braintree.js"
otherwise only Braintree is available in v1 namespace it seems.
Moving the code that needed the Braintree script to be properly loaded inside a
$( window ).load(function() {}
worked for me.