Develop Reference

Failing to login to website using python's requests library

edit I believe the issue is that when opening the website for the first time, you must click a "Our privacy policy has been updated..." Accept button. I'm now looking into how I might go about "clicking" this button using python requests, however the button calls a javascript function so I'm not sure if it's possible to do with just the requests library.
I am attempting to write a script for this website: https://rocket-league.com/
I need to be logged in to perform my tasks, but I'm having trouble logging in.
I feel as though I've accounted for all the correct params. I fetch the csrf_token dynamically using regex. Maybe I need to do something with the cookies, but I'm not sure what?
This is my first attempt at writing a script that interacts with websites so I'm sorry if I'm naive.
I'd be grateful for any insights that aren't just a disguised way of telling me I'm dumb and/or lazy.
Here is my code so far:
import requests;
import re;
payload = {
'csrf_token': '',
'email': 'myemail',
'password': 'mypassword',
'submit': 'Login'
}
url = 'https://rocket-league.com/login'
with requests.Session() as s:
r = s.get(url)
m = re.search("<input type='hidden' name='csrf_token' value='(.+)'", r.text)
if m: payload['csrf_token'] = m.group(1)
else: print("couldnt find csrf_token")
p = s.post('https://rocket-league.com/functions/login.php', data=payload)
print(p.text)
Running this code prints out HTML that still has the login form in it which means I am not being logged in.
When I login from my browser with the developer tools > network open, I get this information for my post request:
Headers:
Request URL: https://rocket-league.com/functions/login.php
Request method: POST
Status code: 302 Found
Version HTTP/2.0
Cookies:
__cfduid: d2c83b2c9ad728195366656a56592f6d71549577451
acceptedPrivacyPolicy: 2.0
euconsent: BObnsHAObnsHAABABAENCF-AAAAkF7_______9______9uz_Ov_v_f__33e8__9v_l_7_-___u_-33d4-_1vf99yfm1-7ftr3tp_87ues2_Xur__59__3z3_tphPhA
fantasy_rlcs_id: hA8fga9ghIgaFGA9
PHPSESSID: lrse3tgov95eg574sqga6la9jc
Params:
csrf_token: 3f2588113e8921f52dc3eb78e51246a1
email: myemail
password: mypassword
submit: Login
Here is the actual HTML for the login form:
<form class="rlg-form" method="post" action="/functions/login.php">
<input type="hidden" name="csrf_token" value="3accad82ad0957cab634f805a7e28beb">
<input class="rlg-input" type="email" name="email" placeholder="Email" required="">
<input class="rlg-input" type="password" name="password" placeholder="Password" autocomplete="off" required="">
<fieldset class="rlg-checkbox">
<input type="checkbox" name="rememberme" id="rememberme-login">
<label for="rememberme-login">Remember me?</label>
</fieldset>
<input class="rlg-btn-primary" type="submit" name="submit" value="Login">
</form>

Python - Login to Website

I know that there are several posts on this subject and I believe I have read a significant amount of them, however I still can't login to this website.
Below is my inspection of the login page:
<form id="login" name="login" method="POST" action="/signin">
<div id="login_username">
<label>Email</label>
<input class="textfield" id="email" name="email" type="text" autocomplete="off" value="">
</div>
<div id="login_password">
<label>Password</label>
<input class="textfield" id="password" name="password"
type="password" autocomplete="off" value="">
</div>
<input type="hidden" id="hash" name="hash" value="">
<div id="login_submit">
<a id="forgot_password_link">Forgot Password?</a>
<input class="submitbutton" type="submit" value="Sign In">
</div>
</form>
Below is my code:
username = 'XXXXX#gmail.com'
password = 'XXXX'
hash = ''
data = {'password':password, 'email':username,'hash':hash}
login_url = "https://carmel.orangetheoryfitness.com/login"
s = requests.session()
result = s.post(login_url, data=data, headers = dict(referer=login_url))
scrape_url = 'https://carmel.orangetheoryfitness.com/apps/otf/classes/view?id=16297&loc=0'
result = s.get(url=scrape_url)
From here I go on to search the html document but I'm not finding what I want as I am sent back to the login page when getting the scrape_url. I have verified this by inspecting the resulting html document.
Things I have considered:
-Almost all blog posts or SO responses indicate that usually a CSRF token is required. I have searched the login page and can't find a CSRF token.

The form has an action="/signin" attribute so you need to post to https://carmel.orangetheoryfitness.com/signin instead.
result = s.post('https://carmel.orangetheoryfitness.com/signin', data=data, headers = dict(referer=login_url))

Amazon AWS S3 browser-based upload using POST -

I am building a web application that includes a file upload feature. My goal is to initiate upload from users directly to an S3 bucket. The strategy is to pre-sign a POST request that will get submitted as a form.
The roadblock is a SignatureDoesNotMatch error - as far as I can tell I've conformed to the documentation, and have explored a lot of options, but still unable to resolve. I am able to generate presigned download links.
Referencing:
AWS POST documentation
Example
boto3 generate_presigned_post reference
Generate signed request:
def s3_upload_creds(name, user):
s3 = boto3.client('s3')
key = '${filename}'
region = 'us-east-1'
date_short = datetime.datetime.utcnow().strftime('%Y%m%d')
date_long = datetime.datetime.utcnow().strftime('%Y%m%dT000000Z')
fields = {
'acl': 'private',
'date': date_short,
'region': region,
'x-amz-algorithm': 'AWS4-HMAC-SHA256',
'x-amz-date': date_long
}
return s3.generate_presigned_post(
Bucket = 'leasy',
Fields = fields,
Key = key,
Conditions = [
{'acl': 'private'},
{'x-amz-algorithm': 'AWS4-HMAC-SHA256'},
{'x-amz-credential': '/'.join(['AKI--snip--', date_short, region, 's3', 'aws4_request'])},
{'x-amz-date': date_long}
]
)
Upload form (populated with fields above):
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
{{ creds }}
<form action="{{ creds.url }}" method="post" enctype="multipart/form-data">
Key to upload:
<input type="input" name="key" value="${filename}" /><br />
<input type="input" name="acl" value="{{ creds.fields.acl }}" />
<input type="hidden" name="Policy" value="{{ creds.fields.policy }}" />
<input type="text" name="X-Amz-Algorithm" value="{{ creds.fields['x-amz-algorithm'] }}" />
<input type="input" name="X-Amz-Credential" value="{{ creds.fields.AWSAccessKeyId }}/{{ creds.fields.date }}/us-east-1/s3/aws4_request" />
<input type="input" name="X-Amz-Date" value="{{ creds.fields['x-amz-date'] }}" />
<input type="input" name="X-Amz-Signature" value="{{ creds.fields.signature }}" />
File:
<input type="file" name="file" /> <br />
<!-- The elements after this will be ignored -->
<input type="submit" name="submit" value="Upload to Amazon S3" />
</form>
</html>
Relevant portion of response:
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>
The request signature we calculated does not match the signature you provided. Check your key and signing method.
</Message>
<AWSAccessKeyId>AKI--snip--</AWSAccessKeyId>
<StringToSign>
eyJjb25kaXRpb25zIjogW3siYWNsIjogInByaXZhdGUifSwgeyJ4LWFtei1hbGdvcml0aG0iOiAiQVdTNC1ITUFDLVNIQTI1NiJ9LCB7IngtYW16LWNyZWRlbnRpYWwiOiAiQUtJQUlDVjRNVlBUUlFHU1lLV1EvMjAxNTEyMTgvdXMtZWFzdC0xL3MzL2F3czRfcmVxdWVzdCJ9LCB7IngtYW16LWRhdGUiOiAiMjAxNTEyMThUMDAwMDAwWiJ9LCB7ImJ1Y2tldCI6ICJsZWFzeSJ9LCBbInN0YXJ0cy13aXRoIiwgIiRrZXkiLCAiIl1dLCAiZXhwaXJhdGlvbiI6ICIyMDE1LTEyLTE4VDA1OjEwOjU2WiJ9
</StringToSign>
<SignatureProvided>wDOjsBRc0iIW7JNtz/4GHgfvKaU=</SignatureProvided>
Base64 decode of StringToSign in above error:
{u'conditions': [{u'acl': u'private'},
{u'x-amz-algorithm': u'AWS4-HMAC-SHA256'},
{u'x-amz-credential': u'AKI--snip--/20151218/us-east-1/s3/aws4_request'},
{u'x-amz-date': u'20151218T000000Z'},
{u'bucket': u'leasy'},
[u'starts-with', u'$key', u'']],
u'expiration': u'2015-12-18T04:59:32Z'}

Found a solution: had to explicitly configure the s3 client to use Amazon's new signature v4. The error occurs since it defaults to an older version, causing the mismatch. Bit of a facepalm - at the time this wasn't written in boto3 docs, although folks at Amazon say it should be soon.
The method is simplified since it now returns exactly the fields required:
def s3_upload_creds(name):
BUCKET = 'mybucket'
REGION = 'us-west-1'
s3 = boto3.client('s3', region_name=REGION, config=Config(signature_version='s3v4'))
key = '${filename}'
return s3.generate_presigned_post(
Bucket = BUCKET,
Key = key
)
Which means the form can be easily generated:
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
{{ creds }}
<form action="https://mybucket.s3.amazonaws.com" method="post" enctype="multipart/form-data">
{% for key, value in creds.fields.items() %}
<input type="hidden" name="{{ key }}" value="{{ value }}" />
{% endfor %}
File:
<input type="file" name="file" /> <br />
<input type="submit" name="submit" value="Upload to Amazon S3" />
</form>
</html>
Cheers

Been a few years since the last response, but I've been stuck on this for the last day or two so I'll share my experience for anyone it may help.
I had been getting the error: "403: The AWS Access Key Id you provided does not exist in our records" when trying to upload to an s3 bucket via my presigned url.
I was able to successfully generate a presigned url similarly to above, using the server-side code:
signed_url_dict = self.s3_client.generate_presigned_post(
self.bucket_name,
object_name,
ExpiresIn=300
This returned a dictionary with the structure:
{
url: "https://___",
fields: {
key: "___",
AWSAccesKeyId: "___",
x-amz-security-token: "___",
policy: "___",
signature: "___"
}
}
This lead to the part where things were a little different now in 2019 with the browser-side javascript, where the required form inputs seem to have changed. Instead of setting up the form as OP did, I had to create my form as seen below:
<form action="https://pipeline-poc-ed.s3.amazonaws.com/" method="post" enctype="multipart/form-data" name="upload_form">
<!-- Copy ALL of the 'fields' key:values returned by S3Client.generate_presigned_post() -->
<input type="hidden" name="key" value="___" />
<input type="hidden" name="AWSAccessKeyId" value="___" />
<input type="hidden" name="policy" value="___"/>
<input type="hidden" name="signature" value="___" />
<input type="hidden" name="x-amz-security-token" value="___" />
File:
<input type="file" name="file" /> <br />
<input type="submit" name="submit" value="Upload to Amazon S3" />
</form>
My error was that I followed an example in the boto3 1.9.138 docs and left out "x-amz-security-token" on the form, which turned out to be quite necessary. A thoughtless oversight on may part, but hopefully this will help someone else.
EDIT: My results above were based on a N. Virginia Lambda Function. When I ran generate_presigned_post(...) in Ohio (the region containing my bucket), I got results similar to OP:
{
"url": "https://__",
"fields": {
"key": "___",
"x-amz-algorithm": "___",
"x-amz-credential": "___",
"x-amz-date": "___",
"x-amz-security-token": "___",
"policy": "___",
"x-amz-signature": "___"
}
}
Perhaps the results of the function are region specific?

In my case, I was generating a form with Base64-encoded.
The problem was due to Firefox inherently encoding the Policy and Security token values into Base64-encoded on top of it.
Thus there was double encoding and therefore the signature did not match as required.

Passing File Objects with Django

I am attempting to retrieve a file and then upload it to Parse.com by submitting via POST. My HTML:
<script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js"></script>
<div class="container">
<form action="" id="fileupload" name="fileupload" enctype="multipart/form-data" method="post">
{% csrf_token %}
<fieldset>
<input type="file" name="fileselect" id="fileselect" /></input>
<input type="hidden" name="myFile" id="myFile" />
<input id="uploadbutton" type="submit" value="Upload to Parse" />
</fieldset>
</form>
</div>
And using the following function to retrieve the file:
<script type="text/javascript">
$(function() {
var file;
// Set an event listener on the Choose File field.
$('#fileselect').bind("change", function(e) {
var files = e.target.files || e.dataTransfer.files;
// Our file var now holds the selected file
file = files[0];
document.getElementById('myFile').value = file;
});
});
</script>
However, this "myFile" field is not posting the file as an object nor does it jive with the Parse API documentation, which appears to be looking for a file path name, which I don't think I can pull from an unknown machine.
import json,httplib
connection = httplib.HTTPSConnection('api.parse.com', 443)
connection.connect()
connection.request('POST', '/1/files/pic.jpg', open('myPicture.jpg','rb').read(), {
"X-Parse-Application-Id": "xxxxxxxxxxxxxxxxxxxxx",
"X-Parse-REST-API-Key": "xxxxxxxxxxxxxxxxxxxxxxx",
"Content-Type": "image/jpeg"
})
result = json.loads(connection.getresponse().read())
print result
This seems like it would be a common use case but the only documentation I found was for ajax, which I would prefer not to use because it exposes my API credentials https://www.parse.com/questions/uploading-files-to-parse-using-javascript-and-the-rest-api.
I am not sure what is the best way to handle the file... if there's a way to handle within the Django framework or if I need to convert to JSON. And, even when the file object is captured, I'm not clear on how to use the Parse.com API with the file object.

After some research, it turns out that the answer is pretty straightforward. Using the "Basic File Upload" documentation from the Django website: https://docs.djangoproject.com/en/1.8/topics/http/file-uploads/ and replacing
open('myPicture.jpg', 'rb').read()
in the Parse documentation (referenced in the question above) with
request.FILES['file']
I was able to successfully upload the file to Parse. No javascript necessary.

Uploading file with AngularJS fails

Below are the snippets of my code regarding file upload.
Here is my HTML code where I will choose and upload the file:
<form ng-click="addImportFile()" enctype="multipart/form-data">
<label for="importfile">Import Time Events File:</label><br><br>
<label for="select_import_file">SELECT FILE:</label><br>
<input id="import_file" type="file" class="file btn btn-default" ng-disabled="CutOffListTemp.id== Null" data-show-preview="false">
<input class="btn btn-primary" type="submit" name="submit" value="Upload" ng-disabled="CutOffListTemp.id== Null"/><br/><br/>
</form>
This is my controller that will link both html and my python file:
angular.module('hrisWebappApp').controller('ImportPayrollCtrl', function ($scope, $state, $stateParams, $http, ngTableParams, $modal, $filter) {
$scope.addImportFile = function() {
$http.post('http://127.0.0.1:5000/api/v1.0/upload_file/' + $scope.CutOffListTemp.id, {})
.success(function(data, status, headers, config) {
console.log(data);
if (data.success) {
console.log('import success!');
} else {
console.log('importing of file failed' );
}
})
.error(function(data, status, headers, config) {});
};
This is my python file:
#api.route('/upload_file/<int:id>', methods=['GET','POST'])
#cross_origin(headers=['Content-Type'])
def upload_file(id):
print "hello"
try:
os.stat('UPLOAD_FOLDER')
except:
os.mkdir('UPLOAD_FOLDER')
file = request.files['file']
print 'filename: ' + file.filename
if file and allowed_file(file.filename):
print 'allowing file'
filename = secure_filename(file.filename)
path=(os.path.join(current_app.config['UPLOAD_FOLDER'], filename))
file.save(path) #The end of the line which save the file you uploaded.
return redirect(url_for('uploaded_file',
filename=filename))
return '''
<!doctype html>
<title>Upload new File</title>
<h1>Upload new File</h1>
<p>opsss it seems you uploaded an invalid filename please use .csv only</p>
<form action="" method=post enctype=multipart/form-data>
<p><input type=file name=file>
<input type=submit value=Upload>
</form>
'''
And the result in the console gave me this even if I select the correct format of file:
<!doctype html>
<title>Upload new File</title>
<h1>Upload new File</h1>
<p>opsss it seems you uploaded an invalid filename please use .csv only</p>
<form action="" method=post enctype=multipart/form-data>
<p><input type=file name=file>
<input type=submit value=Upload>
</form>
This is not returning to my HTML and I cannot upload the file.

Hi I can finally upload the file, I change the angular part, I change it by this:
$scope.addImportFile = function() {
var f = document.getElementById('file').files[0]; console.log(f);
var formData = new FormData();
formData.append('file', f);
$http({method: 'POST', url: 'http://127.0.0.1:5000/api/v1.0/upload_file/' +$scope.CutOffListTemp.id,
data: formData,
headers: {'Content-Type': undefined},
transformRequest: angular.identity})
.success(function(data, status, headers, config) {console.log(data);
if (data.success) {
console.log('import success!');
}
})
.error(function(data, status, headers, config) {
});
// }
};

The first thing is about the post request. Without ng-click="addImportFile()", the browser will usually take care of serializing form data and sending it to the server. So if you try:
<form method="put" enctype="multipart/form-data" action="http://127.0.0.1:5000/api/v1.0/upload_file">
<label for="importfile">Import Time Events File:</label><br><br>
<label for="select_import_file">SELECT FILE:</label><br>
<input id="import_file" type="file" name="file" class="file btn btn-default" ng-disabled="CutOffListTemp.id== Null" data-show-preview="false">
<input class="btn btn-primary" type="submit" name="submit" value="Upload" ng-disabled="CutOffListTemp.id== Null"/><br/><br/>
</form>
and then in python, make your request url independent of scope.CutOffListTemp.id:
#api.route('/upload_file', methods=['GET','POST'])
It probably will work.
Alternatively, if you want to use your custom function to send post request, the browser will not take care of the serialization stuff any more, you will need to do it yourself.
In angular, the API for $http.post is:
$http.post('/someUrl', data).success(successCallback);
If we use "{}" for the data parameter, which means empty, the server will not find the data named "file" (file = request.files['file']). Thus you will see Bad Request
To fix it, we need to use formData to make file upload which requires your browser supports HTML5:
$scope.addImportFile = function() {
var f = document.getElementById('file').files[0]
var fd = new FormData();
fd.append("file", f);
$http.post('http://127.0.0.1:5000/api/v1.0/upload_file/'+$scope.CutOffListTemp.id,
fd,
headers: {'Content-Type': undefined})
.success......
Other than using the native javascript code above, there are plenty great angular file upload libraries which can make file upload much easier for angular, you may probably want to have a look at one of them (reference: File Upload using AngularJS):
https://github.com/nervgh/angular-file-upload
https://github.com/leon/angular-upload
......