Develop Reference

Python to GAS translation not working: get bad request error

I want the following code to be translated into GAS from python. I wrote the GAS version pasted below but it is not working. It must be something simple but I don't know the reason why I get this error. Any advice will be appreciated. Thanks.
import requests
requestId = "*******************"
url = "http://myapi/internal/ocr/"+requestid+"/ng"
payload={}
headers = {
'X-Authorization': 'abcdefghijklmn'
}
response = requests.request("POST", url, headers=headers, data=payload)
print(response.text)
I wrote this at the moment but I get bad request error.
function sending(yesorno, requestId) {
var requestId = "*******************"
　var STAGING_KEY = "abcdefghijklmn"
var url = url = "http://myapi/internal/ocr/"+requestId+"/ng"
var data = {}
var options = {
'muteHttpExceptions': true,
'method': 'post',
'payload': JSON.stringify(data),
'headers': {
'X-Authorization': STAGING_KEY
}
};
//Error processing
try {
var response = JSON.parse(UrlFetchApp.fetch(url, options));
if (response && response["id"]) {
return 'sent';
} else {
//reportError("Invalid response: " + JSON.stringify(response));
//return 'error';
Logger.log('error')
}
} catch (e) {
//reportError(e.toString());
//return 'error';
Logger.log('error')
}
}
Modified Code
function sending() {
var requestId = "*************************"
var STAGING_KEY = "abcdefghijklmn"
var url = "http://myapi/internal/ocr/"+requestId+"/ng";
var data = {}
var options = {
'muteHttpExceptions': true,
'method': 'post',
'payload': data,
'headers': {
'X-Authorization': STAGING_KEY
}
};
try {
var response = JSON.parse(UrlFetchApp.fetch(url, options).getContentText());
Logger.log(response)
if (response && response["id"]) {
return 'sent';
} else {
//reportError("Invalid response: " + JSON.stringify(response));
//return 'error';
Logger.log('error1')
}
} catch (e) {
//reportError(e.toString());
//return 'error';
Logger.log('error2: '+ e.toString())
}
}
Error
error2: Exception: Bad request:

I understood your situation as follows.
Your python script works fine.
You want to convert the python script to Google Apps Script.
When your Google Apps Script is run, an error Exception: Bad request: occurs.
In this case, how about the following modification? When response = requests.request("POST", url, headers=headers, data=payload) is used with payload={}, I think that at Google Apps Script, it's 'payload': {}.
Modified script:
function sending() {
var requestId = "*******************"
var STAGING_KEY = "abcdefghijklmn"
var url = "http://myapi/internal/ocr/" + requestId + "/ng"
var data = {}
var options = {
'muteHttpExceptions': true,
'method': 'post',
'payload': data,
'headers': {
'X-Authorization': STAGING_KEY
}
};
try {
var response = JSON.parse(UrlFetchApp.fetch(url, options).getContentText());
console.log(response)
if (response && response["id"]) {
return 'sent';
} else {
//reportError("Invalid response: " + JSON.stringify(response));
//return 'error';
Logger.log('error')
}
} catch (e) {
//reportError(e.toString());
//return 'error';
Logger.log('error')
}
}
Note:
By the above modification, the request of Google Apps Script is the same as that of the python script. But if an error occurs, please check the URL and your STAGING_KEY, again. And, please check whether the API you want to use can access from the Google side.
Reference:
fetch(url, params)

Anything similar to Python Request.session() but on Node.js

I have a web scraping application written completely in python. The information that I am web scraping is behind a login and I am using Request.session to save login sessions. I am trying to port the code to Node.js and wasn't able to find anything similar to request.session on Node.js. Please let me know if such a thing exists. Thank you.

Your best bet is using axios, which allows for extensive customization of request headers and types of authentication.
You can add axios-cookiejar for cookie management.
Simple example:
const axios = require('axios').default;
const axiosCookieJarSupport = require('axios-cookiejar-support').default;
axiosCookieJarSupport(axios);
// you can add an Authorization header to all requests:
// axios.defaults.headers.common['Authorization'] = 'Bearer token';
const cookie = 'lang=en; token=hello';
const body = {message: 'I\'m a message body'};
axios.post(url, body, {
auth: {
username: 'username',
password: 'mypassword',
},
headers: {
'Cookie': cookie,
'Content-Type': 'application/json',
},
})
.then(res => console.log(res))
.catch(err => console.error(err));
Old Answer:
I believe the Request object of the default HTTP server does not have what you are looking for.
But if you opted to use the Express framework, you could find similar functionalities in the express-session package.
To ilustrate it's capabilities and simplicity, here's a slightly tweaked version of the code example provided in their documentation:
const express = require('express')
const parseurl = require('parseurl')
const session = require('express-session')
const app = express()
app.use(session({
secret: 'keyboard cat',
resave: false,
saveUninitialized: true,
}));
app.use((req, res, next) => {
if (!req.session.views) {
req.session.views = {}
}
// get the url pathname
const pathname = parseurl(req).pathname;
// count the views
req.session.views[pathname] = (req.session.views[pathname] || 0) + 1;
next();
});
app.get('/foo', (req, res, next) => {
res.send('you viewed this page ' + req.session.views['/foo'] + ' times');
});
app.get('/bar', (req, res, next) => {
res.send('you viewed this page ' + req.session.views['/bar'] + ' times');
});
app.use((req, res, next) => {
if (res.headersSent) {
return next();
} else {
console.error('Not found');
const err = new Error('Not Found');
err.status = 404;
return next(err);
}
});
app.use((err, req, res, next) => {
console.error(req.app.get('env') === 'production' ? err.message : err.stack);
if (res.headersSent) {
return next(err);
}
res.status(err.status || 500);
res.send(`${err.status} - ${err.message}`);
});
const http = require('http');
const server = http.createServer(app).listen(3000);
server.on('error', (e) => {
console.log('>>> Server error');
console.error(e);
});
console.log(`>>> Server running on localhost:3000`);

python - Directly Send Files to S3 From Django on Heroku

So I'm trying to uploaded larger files to my site, and due to Herokus limitations with the sizes, I'm trying to upload them directly to S3. Heroku provides great documentation on how to do so, seen here. I'm following along with the guide, and adjusting my views.py based on this Git. The problem is my signing request doesn't work when trying to post to S3. I get the error
The request signature we calculated does not match the signature you provided. Check your key and signing method.
So i am unsure of how my signature is wrong or if there is something wrong in my javascript. Any help would be appreciated.
My views.py
def sign_s3(request):
"""
https://devcenter.heroku.com/articles/s3-upload-python
"""
if request.user.is_authenticated():
user = request.user.id
AWS_ACCESS_KEY = AWS_ACCESS_KEY_ID
AWS_SECRET_KEY = AWS_SECRET_ACCESS_KEY
S3_BUCKET = AWS_STORAGE_BUCKET_NAME
object_name = urllib.parse.quote_plus(request.GET['file_name'])
mime_type = request.GET['file_type']
secondsPerDay = 24*60*60
expires = int(time.time()+secondsPerDay)
amz_headers = "x-amz-acl:public-read"
string_to_sign = "PUT\n\n%s\n%d\n%s\n/%s/%s" % (mime_type, expires, amz_headers, S3_BUCKET, object_name)
encodedSecretKey = AWS_SECRET_KEY.encode()
encodedString = string_to_sign.encode()
h = hmac.new(encodedSecretKey, encodedString, sha1)
hDigest = h.digest()
signature = base64.encodebytes(hDigest).strip()
print(signature)
signature = urllib.parse.quote_plus(signature)
print(signature)
url = 'https://%s.s3.amazonaws.com/media/user_%s/%s/' % (S3_BUCKET, user, object_name)
return JsonResponse({
'data': '%s?AWSAccessKeyId=%s&Expires=%s&Signature=%s' % (url, AWS_ACCESS_KEY, expires, signature),
'url': url,
})
My javascript:
function getSignedRequest(file){
var token = $('input[name=csrfmiddlewaretoken]').val();
var xhr = new XMLHttpRequest();
xhr.open("GET", "/dashboard/sign_s3/?file_name="+file.name+"&file_type="+file.type);
// xhr.setRequestHeader('X-CSRF-Token', token);
xhr.onreadystatechange = function(){
if(xhr.readyState === 4){
if(xhr.status === 200){
var response = JSON.parse(xhr.responseText);
uploadFile(file, response.data, response.url);
}
else{
alert("Could not get signed URL.");
}
}
};
xhr.send();
}
function uploadFile(file, s3Data, url){
var awskey = getParameterByName('AWSAccessKeyId', s3Data);
var expires = getParameterByName('Expires', s3Data);
var signature = getParameterByName('Signature', s3Data);
var xhr = new XMLHttpRequest();
var token = $('input[name=csrfmiddlewaretoken]').val();
xhr.open("POST", s3Data);
xhr.setRequestHeader('X-CSRF-Token', token);
var postData = new FormData();
postData.append('AWSAccessKeyId', awskey);
postData.append('Expires', expires);
postData.append('Signature', signature);
postData.append('file', file);
xhr.onreadystatechange = function() {
if(xhr.readyState === 4){
if(xhr.status === 200 || xhr.status === 204){
//do something
}
else{
alert("Could not upload file.");
}
}
};
xhr.send(postData);
}

Uploading two files with POST in node.JS

I've got the following code in Python:
import requests
fileslist = [('file[]',('user_query.txt', open('user_query.txt', 'rb'), 'text/plain')),
('file[]',('wheatData.csv', open('wheatData.csv', 'rb'), 'text/csv')),]
r = requests.post('url',
files=fileslist)
And I'm trying to convert it to a node.JS version. So far I've got this:
var request = require('request');
var fs = require('fs');
var req = request.post(url, function (err, resp, body) {
if (err) {
console.log('Error!');
} else {
console.log(body);
}
});
var form = req.form();
form.append('wheatData.csv', fs.createReadStream('wheatData.csv'));
form.append('user_query.txt', fs.createReadStream('user_query.txt'));
What am I doing wrong?

This is how you do it using express and body-parser module to parse the post request and fetch the files you need .This is what goes in your node.js server.
Import all the modules :
var express = require("express");
var bodyParser = require('body-parser')
var app = express(); //init express app()
var util = require('util');
//APP CONFIGURATION >> Skip this if you dont want CORS
app.use(express.static('app')); // use this as resource directory
app.use(function (req, res, next) {
res.header("Access-Control-Allow-Origin", "*");
res.header("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept");
next();
});
Configure the post Url :
//url => url/for/mypostrequest
app.post(url, done, function (req, res) {
//Handle the post request body here...
var filesUploaded = 0;
//check if files present
if (Object.keys(req.files).length === 0) {
console.log('no files uploaded');
} else {
console.log(req.files);
var files = req.files.file1;
//If multiple files store in array..
if (!util.isArray(req.files.file1)) {
files = [req.files.file1];
}
filesUploaded = files.length;
}
res.json({message: 'Finished! Uploaded ' + filesUploaded + ' files. Route is /files1'});
});
Make sure all the modules are installed and present as dependencies in package.json
CODE for making an api post call from node..
Include the http module first in your server .
var http = require('http');
var querystring = require('querystring');
var fs = require('fs');
Theninclude following code to make a post request from node server
var file1, file2;
//Read first File ...
fs.readFileSync('wheatData.csv', function (err, data) {
if (err) {
console.log('Error in file reading...');
}
file1 = data;
//Read second file....
fs.readFileSync('wheatData.csv', function (err, data) {
if (err) {
console.log('Error in file reading...');
}
file2 = data;
//Construct the post request data..
var postData = querystring.stringify({
'msg': 'Hello World!',
'file1': file1,
'file2': file2
});
var options = { //setup option for you request
hostname: 'www.path/to/api/',
port: 80,
path: '/upload',
method: 'POST',
headers: {
'Content-Type': 'application/x-www-form-urlencoded',
'Content-Length': postData.length
}
};
var req = http.request(options, function (res) {
console.log('STATUS:' + res.statusCode);
console.log('HEADERS:' + JSON.stringify(res.headers));
res.setEncoding('utf8');
res.on('data', function (chunk) {
console.log('BODY: ' + chunk);
});
res.on('end', function () {
console.log('No more data in response.');
});
});
req.on('error', function (e) {
console.log('problem with request: ' + e.message);
});
// write data to request body
req.write(postData);
req.end();
});
});
Please note that code has not been tested on live server , you may need to make alteration as per your configuration.
Also you can use other libraries like request or needler..etc to make post calls from node server as suggested in this post.

unable to redirect to a url in flask

I am not able to redirect to the main.demo. Everything is working fine until data upload after that redirect is not happening. Why?
EDIT: app.py
from flask import Blueprint
main = Blueprint('main', __name__)
import json
import os
from flask import Flask, request,render_template,url_for,redirect
from werkzeug import secure_filename
import glob2
from uuid import uuid4
#main.route('/')
def index():
"""Main index page """
return render_template('index.html')
#main.route('/upload', methods=["GET","POST"])
def upload():
if request.method == 'POST':
form = request.form
# Create a unique "session ID" for this particular batch of uploads.
upload_key = str(uuid4())
# Is the upload using Ajax, or a direct POST by the form?
is_ajax = False
if form.get("__ajax", None) == "true":
is_ajax = True
# Target folder for these uploads.
target = "upload/{}".format(upload_key)
try:
os.mkdir(target)
except:
if is_ajax:
return ajax_response(False, "Couldn't create upload directory: {}".format(target))
else:
return "Couldn't create upload directory: {}".format(target)
print "=== Form Data ==="
for key, value in form.items():
print key, "=>", value
for upload in request.files.getlist("file"):
filename = upload.filename.rsplit("/")[0]
destination = "/".join([target, filename])
print "Accept incoming file:", filename
print "Save it to:", destination
upload.save(destination)
return redirect(url_for("main.demo"))
return render_template("upload.html")
#main.route('/demo',methods=["GET","POST"])
def demo():
if request.method == "GET":
return render_template("demo.html")
def ajax_response(status, msg):
status_code = "ok" if status else "error"
return json.dumps(dict(
status=status_code,
msg=msg,
))
def create_app():
global app
app = Flask(__name__,template_folder=os.path.join(os.path.dirname(os.path.abspath(__file__)),'templates'))
app.debug = True
app.config['UPLOAD_FOLDER'] = UPLOAD_FOLDER
app.register_blueprint(main)
#app.config['MAX_CONTENT_LENGTH'] = 16 * 1024 * 1024
return app
Following are some the logs which I am getting:
=== Form Data ===
__ajax => true
Accept incoming file: demo.txt
Save it to: upload/XXXXXXX-XXXXXX-XXXXX-XXXXXXX/demo.txt
"POST /upload HTTP/1.1" 302 231 "http://localhost:5000/upload"
"GET /demo HTTP/1.1" 200 3624 "http://localhost:5000/upload"
It's showing that it's going for demo, but the final url is wrong. Why is it like that?
EDIT 1:
Is it like I am unable to submit the form as the page doesn't gets refreshed ? But its getting redirected to demo() function execute it but doesn't but doesn't render_template. Or it does but somehow gets back to the same function ?
EDIT 2:
More to add this code is using following JavaScript in the background
// Constants
var MAX_UPLOAD_FILE_SIZE = 1024*1024; // 1 MB
var UPLOAD_URL = "/upload";
var NEXT_URL = "/demo";
// List of pending files to handle when the Upload button is finally clicked.
var PENDING_FILES = [];
$(document).ready(function() {
// Set up the drag/drop zone.
initDropbox();
// Set up the handler for the file input box.
$("#file-picker").on("change", function() {
handleFiles(this.files);
});
// Handle the submit button.
$("#upload-button").on("click", function(e) {
// If the user has JS disabled, none of this code is running but the
// file multi-upload input box should still work. In this case they'll
// just POST to the upload endpoint directly. However, with JS we'll do
// the POST using ajax and then redirect them ourself when done.
e.preventDefault();
doUpload();
})
});
function doUpload() {
$("#progress").show();
var $progressBar = $("#progress-bar");
// Gray out the form.
$("#upload-form :input").attr("disabled", "disabled");
// Initialize the progress bar.
$progressBar.css({"width": "0%"});
// Collect the form data.
fd = collectFormData();
// Attach the files.
for (var i = 0, ie = PENDING_FILES.length; i < ie; i++) {
// Collect the other form data.
fd.append("file", PENDING_FILES[i]);
}
// Inform the back-end that we're doing this over ajax.
fd.append("__ajax", "true");
var xhr = $.ajax({
xhr: function() {
var xhrobj = $.ajaxSettings.xhr();
if (xhrobj.upload) {
xhrobj.upload.addEventListener("progress", function(event) {
var percent = 0;
var position = event.loaded || event.position;
var total = event.total;
if (event.lengthComputable) {
percent = Math.ceil(position / total * 100);
}
// Set the progress bar.
$progressBar.css({"width": percent + "%"});
$progressBar.text(percent + "%");
}, false)
}
return xhrobj;
},
url: UPLOAD_URL,
method: "POST",
contentType: false,
processData: false,
cache: false,
data: fd,
success: function(data) {
$progressBar.css({"width": "100%"});
data = JSON.parse(data);
// How'd it go?
if (data.status === "error") {
// Uh-oh.
window.alert(data.msg);
$("#upload-form :input").removeAttr("disabled");
return;
}
else {
// Ok! Get the UUID.
var uuid = data.msg;
//window.location = NEXT_URL + uuid;
window.location = NEXT_URL;
}
},
});
}
function collectFormData() {
// Go through all the form fields and collect their names/values.
var fd = new FormData();
$("#upload-form :input").each(function() {
var $this = $(this);
var name = $this.attr("name");
var type = $this.attr("type") || "";
var value = $this.val();
// No name = no care.
if (name === undefined) {
return;
}
// Skip the file upload box for now.
if (type === "file") {
return;
}
// Checkboxes? Only add their value if they're checked.
if (type === "checkbox" || type === "radio") {
if (!$this.is(":checked")) {
return;
}
}
fd.append(name, value);
});
return fd;
}
function handleFiles(files) {
// Add them to the pending files list.
for (var i = 0, ie = files.length; i < ie; i++) {
PENDING_FILES.push(files[i]);
}
}
function initDropbox() {
var $dropbox = $("#dropbox");
// On drag enter...
$dropbox.on("dragenter", function(e) {
e.stopPropagation();
e.preventDefault();
$(this).addClass("active");
});
// On drag over...
$dropbox.on("dragover", function(e) {
e.stopPropagation();
e.preventDefault();
});
// On drop...
$dropbox.on("drop", function(e) {
e.preventDefault();
$(this).removeClass("active");
// Get the files.
var files = e.originalEvent.dataTransfer.files;
handleFiles(files);
// Update the display to acknowledge the number of pending files.
$dropbox.text(PENDING_FILES.length + " files ready for upload!");
});
// If the files are dropped outside of the drop zone, the browser will
// redirect to show the files in the window. To avoid that we can prevent
// the 'drop' event on the document.
function stopDefault(e) {
e.stopPropagation();
e.preventDefault();
}
$(document).on("dragenter", stopDefault);
$(document).on("dragover", stopDefault);
$(document).on("drop", stopDefault);
}
I have tried to integrate the functionality from the following link: Flask Multiple Upload
Unable to understand that why its still going to /upload even after hitting the /demo
Can somebody help me with this that what is happening in the background ?

In short, my guess is that you are falsely expecting a response to XHR request to change browser's address.
Here's a more detailed explanation of what I think is happening:
A user clicks the upload button
doUpload() Javascript function is called
doUpload() makes an XHR (ajax) request to UPLOAD_URL (POST /upload)
upload() Python function is called and after successful upload generates a "302 Found" redirect response to /demo
browser receives this response and follows the provided URL, in this case it goes to /demo (note that this does not change the address, it happens in the background as part of the XHR request)
demo() Python function renders some HTML, which is returned as a "200 OK" response
the XHR call from step 3 receives this HTML from step 5 and checks whether the response indicates success
success function is triggered and tries to parse HTML as JSON JSON.parse(data), which should fail
I have made some assumptions about your browser, Javascript libraries, HTML templates, and user interaction, but I see this as the most likely explanation from the code you have provided.
My suggestion is to first try replacing return redirect(url_for("main.demo")) with something like this:
if is_ajax:
return ajax_response(True, upload_key)
else:
return redirect(url_for("main.demo"))
In this case, after a successful upload window.location = NEXT_URL will get executed inside your doUpload() and that will change the address in your browser.