I am running into an issue that I can't seem to get past. Any insight would be great.
The script is supposed to get memory allocation information from a database, and return that information as a formatted JSON object. The script works fine when I give it a static JSON object will stack_ids (the information I would be passing) but it won't work when I try to pass the information via POST.
Although the current state of my code uses request.json("") to access the passed data, I have also tried request.POST.get("").
My HTML includes this post request, using D3's xhr post:
var stacks = [230323, 201100, 201108, 229390, 201106, 201114];
var stack_ids = {'stack_ids': stacks};
var my_request = d3.xhr('/pie_graph');
my_request.header("Content-Type", "application/json")
my_request.post(stack_ids, function(stuff){
stuff = JSON.parse(stuff);
var data1 = stuff['allocations'];
var data2 = stuff['allocated bytes'];
var data3 = stuff['frees'];
var data4 = stuff['freed bytes'];
...
...
}, "json");
while my server script has this route:
#views.webapp.route('/pie_graph', method='POST')
def server_pie_graph_json():
db = views.db
config = views.config
ret = {
'allocations' : [],
'allocated bytes' : [],
'frees' : [],
'freed bytes' : [],
'leaks' : [],
'leaked bytes' : []
}
stack_ids = request.json['stack_ids']
#for each unique stack trace
for pos, stack_id in stack_ids:
stack = db.stacks[stack_id]
nallocs = format(stack.nallocs(db, config))
nalloc_bytes = format(stack.nalloc_bytes(db, config))
nfrees = format(stack.nfrees(db, config))
nfree_bytes = format(stack.nfree_bytes(db, config))
nleaks = format(stack.nallocs(db, config) - stack.nfrees(db, config))
nleaked_bytes = format(stack.nalloc_bytes(db, config) - stack.nfree_bytes(db, config))
# create a dictionary representing the stack
ret['allocations'].append({'label' : stack_id, 'value' : nallocs})
ret['allocated bytes'].append({'label' : stack_id, 'value' : nalloc_bytes})
ret['frees'].append({'label' : stack_id, 'value' : nfrees})
ret['freed bytes'].append({'label' : stack_id, 'value' : nfree_bytes})
ret['leaks'].append({'label' : stack_id, 'value' : nleaks})
ret['leaked bytes'].append({'label' : stack_id, 'value' : nfree_bytes})
# return dictionary of allocation information
return ret
Most of that can be ignored, the script works when I give it a static JSON object full of data.
The request currently returns a 500 Internal Server Error: JSONDecodeError('Expecting value: line 1 column 2 (char 1)',).
Can anyone explain to me what I am doing wrong?
Also, if you need me to explain anything further, or include any other information, I am happy to do that. My brain is slightly fried after working on this for so long, so I may have missed something.
Here is what I do with POST and it works:
from bottle import *
#post('/')
def do_something():
comment = request.forms.get('comment')
sourcecode = request.forms.get('sourceCode')
Source
function saveTheSourceCodeToServer(comment) {
var path = saveLocation();
var params = { 'sourceCode' : getTheSourceCode() , 'comment' : comment};
post_to_url(path, params, 'post');
}
Source with credits to JavaScript post request like a form submit
Related
I need to send an API request to SMS API.
My Python code is working but my X++/C# code is not working.
I tried with Postman, it is working as well.
Here's my Python code:
import requests
headers = {"Accept":"application/json"}
data = {"AppSid":"#############YxLtXtaN###",
"SenderID":"####23423#####",
"Body":"This is a test message.",
"Recipient":"###45645######",
"responseType":"JSON",
"CorrelationID":"",
"baseEncode":"true",
"statusCallback":"sent",
"async":"false"}
r = requests.post('http://myapi/rest/SMS/messages', auth=('user#domain.com', 'password'), headers=headers,
data=data)
Here's my X++/C# code:
class Class1
{
public static void main(Args _args)
{
str destinationUrl = 'myapi', requestXml, responseXml;
System.Net.HttpWebRequest request;
System.Net.HttpWebResponse response;
CLRObject clrObj;
System.Byte[] bytes;
System.Text.Encoding utf8;
System.IO.Stream requestStream, responseStream;
System.IO.StreamReader streamReader;
System.Exception ex;
System.Net.WebHeaderCollection httpHeader;
str byteStr;
System.Byte[] byteArray;
System.IO.Stream stream;
System.IO.Stream dataStream;
byteStr = strfmt('%1:%2', "user#domain.com", "password");
requestXml = " {\"AppSid\":\"###########\", \"SenderID\":\"########-AD\", \"Body\":\"This is a test message from ## from X++ Coding Language..\", \"Recipient\":\"######\", \"responseType\":\"JSON\", \"CorrelationID\":\"\", \"baseEncode\":\"true\", \"statusCallback\":\"sent\", \"async\":\"false\"}";
try
{
new InteropPermission(InteropKind::ClrInterop).assert();
httpHeader = new System.Net.WebHeaderCollection();
clrObj = System.Net.WebRequest::Create(destinationUrl);
request = clrObj;
utf8 = System.Text.Encoding::get_UTF8();
bytes = utf8.GetBytes(requestXml);
request.set_KeepAlive(true);
request.set_ContentType("application/xml");
request.AllowAutoRedirect=true;
utf8 = System.Text.Encoding::get_UTF8();
byteArray = utf8.GetBytes(byteStr);
byteStr = System.Convert::ToBase64String(byteArray);
httpHeader.Add("Authorization", 'Basic ' + byteStr);
request.set_ContentType("text/xml; encoding='utf-8'");
request.set_ContentLength(bytes.get_Length());
request.set_Method("POST");
request.set_Headers(httpHeader);
requestStream = request.GetRequestStream();
requestStream.Write(bytes, 0, bytes.get_Length());
response = request.GetResponse();
responseStream = response.GetResponseStream();
streamReader = new System.IO.StreamReader(responseStream);
responseXml = streamReader.ReadToEnd();
info(responseXml);
}
catch (Exception::CLRError)
{
//bp deviation documented
ex = CLRInterop::getLastException().GetBaseException();
error(ex.get_Message());
}
requestStream.Close();
streamReader.Close();
responseStream.Close();
response.Close();
}
}
I'm getting this error:
error code : The remote server returned an error: (308) Permanent
Redirect.
I made two changes.
removed the Authorization by commenting it out
//httpHeader.Add("Authorization", 'Basic ' + byteStr);
Made it HTTPS instead of HTTP. I could see the Location: HTTPS in the exception Response object in Visual Studio.
These 2 things resolved the issue for me.
I don't know about x++, but you could try explicitly allowing redirect on your webRequest object.
webRequest = System.Net.WebRequest::Create('http://myapi/rest/SMS/messages') as System.Net.HttpWebRequest;
webRequest.AllowAutoRedirect = true;
webRequest.MaximumAutomaticRedirections = 1; //Set value according to your requirements
I am trying to create in python the following payload in order to make a POST request. The following line comes from the REST client in my browser and creates the result I want, and I know it works.
(URLencoded form data)
features=[{"attributes":{"POINT_NAME":"tpoint","EASTING":"338250","NORTHING":"342300","CREATION_DATE":"16/12/2020",},"geometry":{"x":338250,"y":342300}}]&f=json
So far, I did a number of attempts with the closest being the following:
import json
import requests
#Create inner dictionary
data_dictionary = {}
data_dictionary['attributes'] = {"POINT_NAME" : "tpoint", "EASTING" : "338250", "NORTHING" : "342300", "CREATION_DATE" : "16/12/2020"}
data_dictionary['geometry'] ={"x" : 338250, "y" : 342300}
data_list = [data_dictionary]
payload = {"features": data_list , "f" : "json"}
r = request.post(API_ENDPOINT_POINT, data = payload)
r.text
payload output is:
{'features': [{'attributes': {'POINT_NAME': 'tpoint', 'EASTING': '338250',
'NORTHING': '342300','CREATION_DATE': '16/12/2020'},'geometry': {'x': 338250, 'y': 342300}}],
'f': 'json'},
but when I make the request I get:
'{"error":{"code":500,"message":"Unable to complete operation.","details":["Parser error: Some parameters could not be recognized."]}}'
Any ideas would be helpful, I am stuck.
Thanks
Copying the answer from comments: try to use payload as
payload = {"features": json.dumps(data_list) , "f" : "json"}
I have a small mongo DB and I am trying to write code that will reverence the value of parameters (keys) imbedded in a document.
I have a mongo DB server on localhost and am able to successfully input the json format data structure into my DB. However, when I try to reference the values of this data within, I get the errors as shown.
import pymongo
import json
import time
from datetime import datetime
server = pymongo.MongoClient('localhost')
database = server['testData']
collection = database['testCollection']
test_entry1 = '{"Level1" : {"level2_1" : {"param1" : "1.6","param2" : "32.3","param3" : "11.0"}, "level2_2" : {"param1" : "2.6","param2" : "9.3","param3" : "112.0"}}}'
mongo_import = json.loads(test_entry1)
collection.insert_one(mongo_import)
The above works fine and when I query the database I get the following response (as expected):
{ "_id" : ObjectId("5d0081e931775cbc28cf7704"), "Level1" : { "level2_1" : { "param1" : "1.6", "param2" : "32.3", "param3" : "11.0" }, "level2_2" : { "param1" : "2.6", "param2" : "9.3", "param3" : "112.0" } } }
Now, I would like to reference the data in these parameters. I would like to get a response for all of what is in "level 2_1". The attempt #I made is below...and the error received is below that.
level_1_param_1 = collection.find("level1" : "level2_1")
error:
File "Desktop/Scripts/StackOverflowSnippit.py", line 29, in <module>
level_1_param_1 = collection.find('"level1" : "level2_1"')
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pymongo/collection.py", line 1456, in find
return Cursor(self, *args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pymongo/cursor.py", line 146, in __init__
validate_is_mapping("filter", spec)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pymongo/common.py", line 452, in validate_is_mapping
"collections.Mapping" % (option,))
TypeError: filter must be an instance of dict, bson.son.SON, or other type that inherits from collections.Mapping
I would also like to be able to get the value of a parameter that is one layer lower like "level1 --> level2_1 --> param2" for instance but so far have been unable to do so.
My hope is to be able to reference the data in this structure as needed.
Your filter should have brackets:
level_1_param_1 = collection.find({"level1": "level2_1"})
Trying to collect data on book price fluctuations for a school project. I'm using Python to scrape from a book buyback aggregator (in this case, bookscouter), but I find that since the site has to load in the data, grabbing the source code through the urllib2 package gives me the source code from before the data is loaded. How do I pull from after the data is loaded?
Example: http://bookscouter.com/prices.php?isbn=9788498383621&searchbutton=Sell
You cannot this with Python only. You need a JavaScript engine API like PhantomJS
With Phantom, will be very easy to setup the web scraping of all the page contents, static and dynamic JavaScript contents (like Ajax calls results in your case). Infact you can register page event handlers to your page parser like (this is a node.js + phantom.js example)
/*
* Register Page Handlers as functions
{
onLoadStarted : onLoadStarted,
onLoadFinished: onLoadFinished,
onError : onError,
onResourceRequested : onResourceRequested,
onResourceReceived : onResourceReceived,
onNavigationRequested : onNavigationRequested,
onResourceError : onResourceError
}
*/
registerHandlers : function(page, handlers) {
if(handlers.onLoadStarted) page.set('onLoadStarted',handlers.onLoadStarted)
if(handlers.onLoadFinished) page.set('onLoadFinished',handlers.onLoadFinished)
if(handlers.resourceError) page.set('onResourceError', handlers.resourceError)
if(handlers.onResourceRequested) page.set('onResourceRequested',handlers.onResourceRequested)
if(handlers.onResourceReceived) page.set('onResourceReceived',handlers.onResourceReceived)
if(handlers.onNavigationRequested) page.set('onNavigationRequested',handlers.onNavigationRequested)
if(handlers.onError) page.set('onError',handlers.onError)
}
At this point you have full control of what is going on and when in the page you have to download like:
var onResourceError = function(resourceError) {
var errorReason = resourceError.errorString;
var errorPageUrl = resourceError.url;
}
var onResourceRequested = function (request) {
var msg = ' request: ' + JSON.stringify(request, undefined, 4);
};
var onResourceReceived = function(response) {
var msg = ' id: ' + response.id + ', stage: "' + response.stage + '", response: ' + JSON.stringify(response);
};
var onNavigationRequested = function(url, type, willNavigate, main) {
var msg = ' destination_url: ' + url;
msg += ' type (cause): ' + type;
msg += ' will navigate: ' + willNavigate;
msg += ' from page\'s main frame: ' + main;
};
page.onResourceRequested(
function(requestData, request) {
//request.abort()
//request.changeUrl(url)
//request.setHeader(key,value)
var msg = ' request: ' + JSON.stringify(request, undefined, 4);
//console.log( msg )
},
function(requestData) {
//console.log(requestData.url)
})
PageHelper.registerHandlers(page,
{
onLoadStarted : onLoadStarted,
onLoadFinished: onLoadFinished,
onError : null, // onError THIS HANDLER CRASHES PHANTOM-NODE
onResourceRequested : null, // MUST BE ON PAGE OBJECT
onResourceReceived : onResourceReceived,
onNavigationRequested : onNavigationRequested,
onResourceError : onResourceError
});
As you can see you can define you page handlers and take control of the flow and so of the resources loaded on that page. So you can be sure that all data are ready and set, before you take the whole page source like:
var Parser = {
parse : function(page) {
var onSuccess = function (page) { // page loaded
var pageContents=page.evaluate(function() {
return document.body.innerText;
});
}
var onError = function (page,elapsed) { // error
}
page.evaluate(function(func) {
return func(document);
}, function(dom) {
return true;
});
}
} // Parser
Here you can see the whole page contents loaded in the onSuccess callback:
var pageContents=page.evaluate(function() {
return document.body.innerText;
});
The page comes from Phantomjs directly like in the following snippet:
phantom.create(function (ph) {
ph.createPage(function (page) {
Parser.parse(page)
})
},options)
Of course this to give you and idea of what you can do with node.js + Phantomjs, that are super powerful when combined together.
You can run phantomjs in a Python env, calling it like
try:
output = ''
for result in runProcess([self.runProcess,
self.runScript,
self.jobId,
self.protocol,
self.hostname,
self.queryString]):
output += '' + result
print output
except Exception as e:
print e
print(traceback.format_exc())
where you use subprocess Popen to execute the binary:
def runProcess(exe):
p = subprocess.Popen(exe, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while(True):
retcode = p.poll() #returns None while subprocess is running
line = p.stdout.readline()
yield line
if(retcode is not None):
break
Of course the process to run is node.js in this case
self.runProcess='node'
with the args you need as params.
The challenge is reading the data once its been rendered by a web browser, which will require some extra tricks to do. If you can see if the site has a pre-rendered version* or an API.
This article (linked from the Web archive) has a pretty good breakdown of what you'll need to do. It can be summed up however as:
Pick a good python-webkit renderer (in the case of the article PyQT)
Use a windowing widget to fetch and render the page
Fetch the rendered HTML from the widget
Parse this HTML as normal using a library like lXML or BeautifulSoup.
* Minor rant - the idea of having to hope for a pre-rendered version ofwhat should be a static webpage angers me.
I am new to python, and I have looked at some examples of scripts already made.
I started my script and it works fine but now I want to add some extras and I would appreciate your help.
def listar_videos(url):
codigo_fonte = abrir_url(url)
soup = BeautifulSoup(abrir_url(url))
content = BeautifulSoup(soup.find("div", { "id" : "dle-content" }).prettify())
filmes = content("div", { "class" : "short-film" })
for filme in filmes:
nome_filme = filme.img["alt"]
url = filme.a["href"].replace('Assistir ','')
img = filme.img["src"]
addDir(nome_filme.encode('utf8'),url,4,img,False,len(filmes))
pagina = BeautifulSoup(soup.find('div', { "class" : "pages" }).prettify())("div", { "class" : "pnext" })[0]["href"]
addDir('Página Seguinte >>',pagina,2,artfolder + 'prox.png')
IndexError: list index out of range on this line:
pagina = BeautifulSoup(soup.find('div', { "class" : "pages" }).prettify())("div", { "class" : "pnext" })[0]["href"]
I alredy try another different code but no sucess.
It's weird because I already used this code in a website Worpress and it works fine, but this new one gave me this error.
I need your help guys.