I faced a trouble to properly encode a query string to use Bing image search API.
I got my account key for using Bing API, which contains "/" and "+". So when I try Bing example query like
http://api.bing.net/json.aspx?AppId=MY_APP_ID&Query=xbox site:microsoft.com&Sources=Image
I got reply that my AppId value is invalid:
{"SearchResponse":{"Version":"2.2","Query":{"SearchTerms":"xbox site\u003amicrosoft.com"},"Errors":[{"Code":1002,"Message":"Parameter has invalid value.","Parameter":"SearchRequest.AppId","Value":"*******\u002*****\u002b***","HelpUrl":"http\u003a\u002f\u002fmsdn.microsoft.com\u002fen-us\u002flibrary\u002fdd251042.aspx"}]}}
Where *** are valid characters of my account key
I tried all possible ways that came to my mind and found in web, but still failed to solve it. So what I tried:
import requests
url = "http://api.bing.net/json.aspx?AppId=****/***+***&Query=xbox site:microsoft.com&Sources=Image"
r = requests.get(url)
I got an error that the value is invalid "****\u002*** ***"
I tried doing the same thing using urllib2, trying to encode and quote both the whole query and the account key only. The code for separately quoting each part of request is like this:
import urllib2
urlStart = u"http://api.bing.net/json.aspx?AppId=%s&Query=xbox"
quotedUrlStart = urllib2.quote(urlStart.encode("utf8"), safe="%/:=&?~#+!$,;'#()*[]")
urlEnd = u" site:microsoft.com&Sources=Image"
quotedUrlEnd = urllib2.quote(urlEnd.encode("utf8"), safe="")
key = u"**/**+**"
quotedKey = urllib2.quote(key.encode("utf8") , safe="%:=&?~#!$,;'#()*[]")
fullUrl = (quotedUrlStart % quotedKey) + quotedUrlEnd
reply = urllib2.urlopen(fullUrl).read()
print reply
I also tried to replace "/" with %2F and "+" with %2B, but the error is the same.
What is a mess for me here is what I have to quote and what not. I actually don't have a clear understanding so far how these things work. I guess that I have to encode everything and quote it different ways - qoute slashes in one place and do not qoute in another.
This question addresses the same issue: XCODE Swift Replacing HTTP-Get App-ID with a space
Also there are numerous questions on SO on escaping symbols, but they were unhelpful for me
I appreciate your time guys
Related
EDIT:
In a similar vein, when I now try to log into their account with a post request, what is returned is none of the errors they suggest on their site, but is in fact a "JSON exception". Is there any way to debug this, or is an error code 500 completely impossible to deal with?
I'm well aware this question has been asked before. Sadly, when trying the proposed answers, none worked. I have an extremely simple Python project with urllib, and I've never done web programming in Python before, nor am I even a regular Python user. My friend needs to get access to content from this site, but their user-friendly front-end is down and I learned that they have a public API to access their content. Not knowing what I'm doing, but glad to try to help and interested in the challenge, I have very slowly set out.
Note that it is necessary for me to only use standard Python libraries, so that any finished project could easily be emailed to their computer and just work.
The following works completely fine minus the "originalLanguage" query, but when using it, which the API has documented as an array value, no matter whether I comma-separate things, or write "originalLanguage[0]" or "originalLanguage0" or anything that I've seen online, this creates the error message from the server: "Array value expected but string detected" or something along those lines.
Is there any way for me to get this working? Because it clearly can work, otherwise the API wouldn't document it. Many thanks.
In case it helps, when using "[]" or "<>" or "{}" or any delimeter I could think of, my IDE didn't recognise it as part of the URL.
import urllib.request as request
import urllib.parse as parse
def make_query(url, params):
url += "?"
for i in range(len(params)):
url += list(params)[i]
url += '='
url += list(params.values())[i]
if i < len(params) - 1:
url += '&'
return url
base = "https://api.mangadex.org/manga"
params = {
"limit": "50",
"originalLanguage": "en"
}
url = make_query(base, params)
req = request.Request(url)
response = request.urlopen(req)
I'm getting a string url request, I want to handle it and send the parameters, because it is very long.
I get plus in req.url instead of a space like query.
query = "https://www.superjob.ru/resume/search_resume.html?sbmit=1&detail_search=1&keywords%5B0%5D%5Bkeys%5D=%D0%9A%D0%BB%D0%B0%D0%B4%D0%BE%D0%B2%D1%89%D0%B8%D0%BA+%D0%9A%D0%BE%D0%BC%D0%BF%D0%BB%D0%B5%D0%BA%D1%82%D0%BE%D0%B2%D1%89%D0%B8%D0%BA+%D0%93%D1%80%D1%83%D0%B7%D1%87%D0%B8%D0%BA+%D0%A0%D0%B0%D0%B1%D0%BE%D1%82%D0%BD%D0%B8%D0%BA+%D1%81%D0%BA%D0%BB%D0%B0%D0%B4%D0%B0&keywords%5B0%5D%5Bskwc%5D=or&keywords%5B0%5D%5Bsrws%5D=60&keywords%5B1%5D%5Bkeys%5D=%D0%9A%D0%BB%D0%B0%D0%B4%D0%BE%D0%B2%D1%89%D0%B8%D0%BA+%D0%9A%D0%BE%D0%BC%D0%BF%D0%BB%D0%B5%D0%BA%D1%82%D0%BE%D0%B2%D1%89%D0%B8%D0%BA+%D0%93%D1%80%D1%83%D0%B7%D1%87%D0%B8%D0%BA+%D0%A0%D0%B0%D0%B1%D0%BE%D1%82%D0%BD%D0%B8%D0%BA+%D1%81%D0%BA%D0%BB%D0%B0%D0%B4%D0%B0&keywords%5B1%5D%5Bskwc%5D=or&keywords%5B1%5D%5Bsrws%5D=7&keywords%5B2%5D%5Bkeys%5D=%D0%AE%D0%BB%D0%BC%D0%B0%D1%80%D1%82&keywords%5B2%5D%5Bskwc%5D=nein&keywords%5B2%5D%5Bsrws%5D=50&keywords%5B3%5D%5Bkeys%5D=%D0%A0%D1%83%D0%BA%D0%BE%D0%B2%D0%BE%D0%B4%D0%B8%D1%82%D0%B5%D0%BB%D1%8C+%D0%B4%D0%B8%D1%80%D0%B5%D0%BA%D1%82%D0%BE%D1%80+%D0%BD%D0%B0%D1%87%D0%B0%D0%BB%D1%8C%D0%BD%D0%B8%D0%BA+%D1%8D%D0%BA%D0%BE%D0%BD%D0%BE%D0%BC%D0%B8%D1%81%D1%82+%D0%B0%D0%BD%D0%B0%D0%BB%D0%B8%D1%82%D0%B8%D0%BA+%D0%BB%D0%BE%D0%B3%D0%B8%D1%81%D1%82+%D0%9E%D1%84%D0%B8%D1%81-%D0%BC%D0%B5%D0%BD%D0%B5%D0%B4%D0%B6%D0%B5%D1%80+%D0%93%D0%BE%D1%81%D1%83%D0%B4%D0%B0%D1%80%D1%81%D0%B2*&keywords%5B3%5D%5Bskwc%5D=nein&keywords%5B3%5D%5Bsrws%5D=60&exclude_words=%D0%A0%D1%83%D0%BA%D0%BE%D0%B2%D0%BE%D0%B4%D0%B8%D1%82%D0%B5%D0%BB%D1%8C+%D0%B4%D0%B8%D1%80%D0%B5%D0%BA%D1%82%D0%BE%D1%80+%D0%BD%D0%B0%D1%87%D0%B0%D0%BB%D1%8C%D0%BD%D0%B8%D0%BA+%D1%8D%D0%BA%D0%BE%D0%BD%D0%BE%D0%BC%D0%B8%D1%81%D1%82+%D0%B0%D0%BD%D0%B0%D0%BB%D0%B8%D1%82%D0%B8%D0%BA+%D0%BB%D0%BE%D0%B3%D0%B8%D1%81%D1%82+%D0%9E%D1%84%D0%B8%D1%81-%D0%BC%D0%B5%D0%BD%D0%B5%D0%B4%D0%B6%D0%B5%D1%80+%D0%93%D0%BE%D1%81%D1%83%D0%B4%D0%B0%D1%80%D1%81%D0%B2*&period=7&place_of_work=0&t%5B%5D=4&m%5B%5D=194&m%5B%5D=195&m%5B%5D=193&m%5B%5D=192&m%5B%5D=191&m%5B%5D=164&m%5B%5D=162&m%5B%5D=163&m%5B%5D=150&m%5B%5D=534&m%5B%5D=540&m%5B%5D=149&m%5B%5D=148&m%5B%5D=147&m%5B%5D=146&m%5B%5D=535&m%5B%5D=536&m%5B%5D=159&m%5B%5D=157&m%5B%5D=158&m%5B%5D=156&m%5B%5D=155&m%5B%5D=154&m%5B%5D=161&m%5B%5D=20&m%5B%5D=21&m%5B%5D=22&m%5B%5D=24&m%5B%5D=23&m%5B%5D=25&m%5B%5D=28&m%5B%5D=573&m%5B%5D=29&m%5B%5D=27&m%5B%5D=26&m%5B%5D=145&m%5B%5D=144&m%5B%5D=143&m%5B%5D=142&m%5B%5D=141&m%5B%5D=30&m%5B%5D=31&m%5B%5D=115&m%5B%5D=153&m%5B%5D=72&m%5B%5D=84&m%5B%5D=152&m%5B%5D=83&m%5B%5D=82&m%5B%5D=81&m%5B%5D=79&m%5B%5D=80&m%5B%5D=542&m%5B%5D=151&m%5B%5D=46&m%5B%5D=85&paymentfrom=20000&paymentto=35000&type_of_work=0&citizenship%5B0%5D=1&old1=18&old2=40&maritalstatus=0&pol=0&children=0&education=0&eduform=0&id_institute=0&institution=&languages%5B0%5D%5Blanguage_id%5D=0&languages%5B0%5D%5Blanguage_level%5D=0&business_trip=0"
def getparams(url):
params = {}
urlarr = url.split("&")
i = 0
while i < len(urlarr):
params[urllib2.unquote(urlarr[i].split("=")[0].encode("utf-8"))] = urllib2.unquote(urlarr[i].split("=")[1].encode('utf-8'))
i += 1
return params
s = Session()
req = s.get(query.split("?")[0], params=getparams(query.split("?")[1]))
print(req.url)
I tried decode() instead encode() and without encode() and without urllib2.unquote()
Have you looked at the url you are posting?
you set
query = "https://www.superjob.ru/resume/search_resume.html?sbmit=1&detail_search=1&keywords%5B0%5D%5Bkeys%5D=%D0%9A%D0%BB%D0%B0%D0%B4%D0%BE%D0%B2%D1%89%D0%B8%D0%BA+%D0%9A%D0%BE%D0%BC%D0%BF%D0%BB%D0%B5%D0%BA%D1%82%D0%BE%D0%B2%D1%89%D0%B8%D0%BA+%D0%93%D1%80%D1%83%D0%B7%D1%87%D0%B8%D0%BA+%D0%A0%D0%B0%D0%B1%D0%BE%D1%82%D0%BD%D0%B8%D0%BA+%D1%81%D0%BA%D0%BB%D0%B0%D0%B4%D0%B0&keywords%5B0%5D%5Bskwc%5D=or&keywords%5B0%5D%5Bsrws%5D=60&keywords%5B1%5D%5Bkeys%5D=%D0%9A%D0%BB%D0%B0%D0%B4%D0%BE%D0%B2%D1%89%D0%B8%D0%BA+%D0%9A%D0%BE%D0%BC%D0%BF%D0%BB%D0%B5%D0%BA%D1%82%D0%BE%D0%B2%D1%89%D0%B8%D0%BA+%D0%93%D1%80%D1%83%D0%B7%D1%87%D0%B8%D0%BA+%D0%A0%D0%B0%D0%B1%D0%BE%D1%82%D0%BD%D0%B8%D0%BA+%D1%81%D0%BA%D0%BB%D0%B0%D0%B4%D0%B0&keywords%5B1%5D%5Bskwc%5D=or&keywords%5B1%5D%5Bsrws%5D=7&keywords%5B2%5D%5Bkeys%5D=%D0%AE%D0%BB%D0%BC%D0%B0%D1%80%D1%82&keywords%5B2%5D%5Bskwc%5D=nein&keywords%5B2%5D%5Bsrws%5D=50&keywords%5B3%5D%5Bkeys%5D=%D0%A0%D1%83%D0%BA%D0%BE%D0%B2%D0%BE%D0%B4%D0%B8%D1%82%D0%B5%D0%BB%D1%8C+%D0%B4%D0%B8%D1%80%D0%B5%D0%BA%D1%82%D0%BE%D1%80+%D0%BD%D0%B0%D1%87%D0%B0%D0%BB%D1%8C%D0%BD%D0%B8%D0%BA+%D1%8D%D0%BA%D0%BE%D0%BD%D0%BE%D0%BC%D0%B8%D1%81%D1%82+%D0%B0%D0%BD%D0%B0%D0%BB%D0%B8%D1%82%D0%B8%D0%BA+%D0%BB%D0%BE%D0%B3%D0%B8%D1%81%D1%82+%D0%9E%D1%84%D0%B8%D1%81-%D0%BC%D0%B5%D0%BD%D0%B5%D0%B4%D0%B6%D0%B5%D1%80+%D0%93%D0%BE%D1%81%D1%83%D0%B4%D0%B0%D1%80%D1%81%D0%B2*&keywords%5B3%5D%5Bskwc%5D=nein&keywords%5B3%5D%5Bsrws%5D=60&exclude_words=%D0%A0%D1%83%D0%BA%D0%BE%D0%B2%D0%BE%D0%B4%D0%B8%D1%82%D0%B5%D0%BB%D1%8C+%D0%B4%D0%B8%D1%80%D0%B5%D0%BA%D1%82%D0%BE%D1%80+%D0%BD%D0%B0%D1%87%D0%B0%D0%BB%D1%8C%D0%BD%D0%B8%D0%BA+%D1%8D%D0%BA%D0%BE%D0%BD%D0%BE%D0%BC%D0%B8%D1%81%D1%82+%D0%B0%D0%BD%D0%B0%D0%BB%D0%B8%D1%82%D0%B8%D0%BA+%D0%BB%D0%BE%D0%B3%D0%B8%D1%81%D1%82+%D0%9E%D1%84%D0%B8%D1%81-%D0%BC%D0%B5%D0%BD%D0%B5%D0%B4%D0%B6%D0%B5%D1%80+%D0%93%D0%BE%D1%81%D1%83%D0%B4%D0%B0%D1%80%D1%81%D0%B2*&period=7&place_of_work=0&t%5B%5D=4&m%5B%5D=194&m%5B%5D=195&m%5B%5D=193&m%5B%5D=192&m%5B%5D=191&m%5B%5D=164&m%5B%5D=162&m%5B%5D=163&m%5B%5D=150&m%5B%5D=534&m%5B%5D=540&m%5B%5D=149&m%5B%5D=148&m%5B%5D=147&m%5B%5D=146&m%5B%5D=535&m%5B%5D=536&m%5B%5D=159&m%5B%5D=157&m%5B%5D=158&m%5B%5D=156&m%5B%5D=155&m%5B%5D=154&m%5B%5D=161&m%5B%5D=20&m%5B%5D=21&m%5B%5D=22&m%5B%5D=24&m%5B%5D=23&m%5B%5D=25&m%5B%5D=28&m%5B%5D=573&m%5B%5D=29&m%5B%5D=27&m%5B%5D=26&m%5B%5D=145&m%5B%5D=144&m%5B%5D=143&m%5B%5D=142&m%5B%5D=141&m%5B%5D=30&m%5B%5D=31&m%5B%5D=115&m%5B%5D=153&m%5B%5D=72&m%5B%5D=84&m%5B%5D=152&m%5B%5D=83&m%5B%5D=82&m%5B%5D=81&m%5B%5D=79&m%5B%5D=80&m%5B%5D=542&m%5B%5D=151&m%5B%5D=46&m%5B%5D=85&paymentfrom=20000&paymentto=35000&type_of_work=0&citizenship%5B0%5D=1&old1=18&old2=40&maritalstatus=0&pol=0&children=0&education=0&eduform=0&id_institute=0&institution=&languages%5B0%5D%5Blanguage_id%5D=0&languages%5B0%5D%5Blanguage_level%5D=0&business_trip=0"
which contains a lot of escape quotes. None of them are a space (%20)
The reason you get a plus instead of a space, is because your url contains a plus in place of every space. This is expected behaviour, as proper URLs should not contain any spaces, and thus, are often converted into "+" beforehand.
If I put your escaped URL into my browser hotbar and look at it, all I see is "+", no spaces there.
This is from somewhere at the start of your URL:
detail_search=1&keywords%5B0%5D%5Bkeys%5D=%D0%9A%D0%BB%D0%B0%D0%B4%D0%BE%D0%B2%D1%89%D0%B8%D0%BA+%D0%9A%D0%BE%D0%BC%D0%BF%D0%BB%D0%B5%D0%BA%D1%82%D0%BE%D0%B2%D1%89%D0%B8%D0%BA
which is essentially detail_search=1&keywords[0][keys]=Кладовщик+Комплектовщик
if unescaped. As you can clearly see, both versions, quoted and unquoted, contain the plus. This is done by the website, and can't (shouldn't) be changed, especially if you want to reconstruct the URL somewhere else.
If you need the parameters without the plus, just do "myUnquotedString".replace("+"," ") (after unquoting, because i don't know what urllib2 does if it gets a malformed string as parameter for unquote().
I solved this problem like this:
query = "https://www.superjob.ru/resume/search_resume.html?sbmit=1&detail_search=1&keywords%5B0%5D%5Bkeys%5D=%D0%9A%D0%BB%D0%B0%D0%B4%D0%BE%D0%B2%D1%89%D0%B8%D0%BA+%D0%9A%D0%BE%D0%BC%D0%BF%D0%BB%D0%B5%D0%BA%D1%82%D0%BE%D0%B2%D1%89%D0%B8%D0%BA+%D0%93%D1%80%D1%83%D0%B7%D1%87%D0%B8%D0%BA+%D0%A0%D0%B0%D0%B1%D0%BE%D1%82%D0%BD%D0%B8%D0%BA+%D1%81%D0%BA%D0%BB%D0%B0%D0%B4%D0%B0&keywords%5B0%5D%5Bskwc%5D=or&keywords%5B0%5D%5Bsrws%5D=60&keywords%5B1%5D%5Bkeys%5D=%D0%9A%D0%BB%D0%B0%D0%B4%D0%BE%D0%B2%D1%89%D0%B8%D0%BA+%D0%9A%D0%BE%D0%BC%D0%BF%D0%BB%D0%B5%D0%BA%D1%82%D0%BE%D0%B2%D1%89%D0%B8%D0%BA+%D0%93%D1%80%D1%83%D0%B7%D1%87%D0%B8%D0%BA+%D0%A0%D0%B0%D0%B1%D0%BE%D1%82%D0%BD%D0%B8%D0%BA+%D1%81%D0%BA%D0%BB%D0%B0%D0%B4%D0%B0&keywords%5B1%5D%5Bskwc%5D=or&keywords%5B1%5D%5Bsrws%5D=7&keywords%5B2%5D%5Bkeys%5D=%D0%AE%D0%BB%D0%BC%D0%B0%D1%80%D1%82&keywords%5B2%5D%5Bskwc%5D=nein&keywords%5B2%5D%5Bsrws%5D=50&keywords%5B3%5D%5Bkeys%5D=%D0%A0%D1%83%D0%BA%D0%BE%D0%B2%D0%BE%D0%B4%D0%B8%D1%82%D0%B5%D0%BB%D1%8C+%D0%B4%D0%B8%D1%80%D0%B5%D0%BA%D1%82%D0%BE%D1%80+%D0%BD%D0%B0%D1%87%D0%B0%D0%BB%D1%8C%D0%BD%D0%B8%D0%BA+%D1%8D%D0%BA%D0%BE%D0%BD%D0%BE%D0%BC%D0%B8%D1%81%D1%82+%D0%B0%D0%BD%D0%B0%D0%BB%D0%B8%D1%82%D0%B8%D0%BA+%D0%BB%D0%BE%D0%B3%D0%B8%D1%81%D1%82+%D0%9E%D1%84%D0%B8%D1%81-%D0%BC%D0%B5%D0%BD%D0%B5%D0%B4%D0%B6%D0%B5%D1%80+%D0%93%D0%BE%D1%81%D1%83%D0%B4%D0%B0%D1%80%D1%81%D0%B2*&keywords%5B3%5D%5Bskwc%5D=nein&keywords%5B3%5D%5Bsrws%5D=60&exclude_words=%D0%A0%D1%83%D0%BA%D0%BE%D0%B2%D0%BE%D0%B4%D0%B8%D1%82%D0%B5%D0%BB%D1%8C+%D0%B4%D0%B8%D1%80%D0%B5%D0%BA%D1%82%D0%BE%D1%80+%D0%BD%D0%B0%D1%87%D0%B0%D0%BB%D1%8C%D0%BD%D0%B8%D0%BA+%D1%8D%D0%BA%D0%BE%D0%BD%D0%BE%D0%BC%D0%B8%D1%81%D1%82+%D0%B0%D0%BD%D0%B0%D0%BB%D0%B8%D1%82%D0%B8%D0%BA+%D0%BB%D0%BE%D0%B3%D0%B8%D1%81%D1%82+%D0%9E%D1%84%D0%B8%D1%81-%D0%BC%D0%B5%D0%BD%D0%B5%D0%B4%D0%B6%D0%B5%D1%80+%D0%93%D0%BE%D1%81%D1%83%D0%B4%D0%B0%D1%80%D1%81%D0%B2*&period=7&place_of_work=0&t%5B%5D=4&m%5B%5D=194&m%5B%5D=195&m%5B%5D=193&m%5B%5D=192&m%5B%5D=191&m%5B%5D=164&m%5B%5D=162&m%5B%5D=163&m%5B%5D=150&m%5B%5D=534&m%5B%5D=540&m%5B%5D=149&m%5B%5D=148&m%5B%5D=147&m%5B%5D=146&m%5B%5D=535&m%5B%5D=536&m%5B%5D=159&m%5B%5D=157&m%5B%5D=158&m%5B%5D=156&m%5B%5D=155&m%5B%5D=154&m%5B%5D=161&m%5B%5D=20&m%5B%5D=21&m%5B%5D=22&m%5B%5D=24&m%5B%5D=23&m%5B%5D=25&m%5B%5D=28&m%5B%5D=573&m%5B%5D=29&m%5B%5D=27&m%5B%5D=26&m%5B%5D=145&m%5B%5D=144&m%5B%5D=143&m%5B%5D=142&m%5B%5D=141&m%5B%5D=30&m%5B%5D=31&m%5B%5D=115&m%5B%5D=153&m%5B%5D=72&m%5B%5D=84&m%5B%5D=152&m%5B%5D=83&m%5B%5D=82&m%5B%5D=81&m%5B%5D=79&m%5B%5D=80&m%5B%5D=542&m%5B%5D=151&m%5B%5D=46&m%5B%5D=85&paymentfrom=20000&paymentto=35000&type_of_work=0&citizenship%5B0%5D=1&old1=18&old2=40&maritalstatus=0&pol=0&children=0&education=0&eduform=0&id_institute=0&institution=&languages%5B0%5D%5Blanguage_id%5D=0&languages%5B0%5D%5Blanguage_level%5D=0&business_trip=0"
query = query.encode('utf-8')
s = Session()
req = s.get(query.split("?")[0], params=parse_qs(urlparse(query).query))
I have a url that I need to add an api key to as a parameter. The key has % and other characters in it and should not be url encoded, is there a way to do this with furl?
Heres my current code:
request_url = furl('http://www.domain.com/service/') \
.add({
'format' : 'json',
'location' : address_line_1 + ',' + city + ',' + state,
'key' : APP_KEY
}).url
I have a url that I need to add an api key to as a parameter. The key has % and other characters in it and should not be url encoded
That can't be true. It must be URL-encoded. Otherwise, you end up with either an invalid URL, or a URL that doesn't mean what you wanted.
For example, let's say the key is 123%456%789. If you send http://www.domain.com/service/?format=json&key=123%25456%25789, the web service on the other end will get 123%456%789, which will succeed. If you just send http://www.domain.com/service/?format=json&key=123%456%789, the web service on the other end will get 123E6x9, which will fail.
So, your example is already correct, and you shouldn't need to do anything.
If, for some reason, you're already URL-encoding APP_KEY before you get here… well, don't. With a name like that, I'm guessing it's a constant literal just copied and pasted into your module, which means you had to manually URL-encode it and copy and paste the result. Just don't do that, and you're fine.
If you got the APP_KEY in URL-encoded form, you can always decode it with, e.g., urlparse.parse_qs, or even with furl.
But if none of that seems reasonable…
is there a way to do this with furl?
No. See the Encoding section of the docs. furl only works on decoded query string names and values.
But there's an easy way around this. A URL isn't that complicated of a thing, and furl is specifically designed to allow you to mix and match manually- and programmatically- created bits. Most of the examples in the README show exactly that, such as furl('http://www.google.com/?one=1').add({'two':2}).url. So, if you've already got a pre-encoded "key=123%25456%25789" from somewhere, just stick it on the string manually before giving it to furl:
request_url = furl('http://www.domain.com/service/?key={}'.format(APP_KEY)) \
.add({
'format' : 'json',
'location' : address_line_1 + ',' + city + ',' + state,
}).url
Hacky? Well, yes, in that you're side-stepping furl's encoding, but that's exactly what you're asking how to do.
I am trying to use the requests library in Python to push data (a raw value) to a firebase location.
Say, I have urladd (the url of the location with authentication token). At the location, I want to push a string, say International. Based on the answer here, I tried
data = {'.value': 'International'}
p = requests.post(urladd, data = sjson.dumps(data))
I get <Response [400]>. p.text gives me:
u'{\n "error" : "Invalid data; couldn\'t parse JSON object, array, or value. Perhaps you\'re using invalid characters in your key names."\n}\n'
It appears that they key .value is invalid. But that is what the answer linked above suggests. Any idea why this may not be working, or how I can do this through Python? There are no problems with connection or authentication because the following works. However, that pushes an object instead of a raw value.
data = {'name': 'International'}
p = requests.post(urladd, data = sjson.dumps(data))
Thanks for your help.
The answer you've linked is a special case for when you want to assign a priority to a value. In general, '.value' is an invalid name and will throw an error.
If you want to write just "International", you should write the stringified-JSON version of that data. I don't have a python example in front of me, but the curl command would be:
curl -X POST -d "\"International\"" https://...
Andrew's answer above works. In case someone else wants to know how to do this using the requests library in Python, I thought this would be helpful.
import simplejson as sjson
data = sjson.dumps("International")
p = requests.post(urladd, data = data)
For some reason I had thought that the data had to be in a dictionary format before it is converted to stringified JSON version. That is not the case, and a simple string can be used as an input to sjson.dumps().
I am experimenting with mechanize and re to find the websites which correspond to a list of retail stores.
I have been parsing Bing search results to grab the top result's url. Unfortunately, seemingly independent of the query, at random times I've been getting an httplib.IncompleteRead error. Even though I've got a workaround which follows, I'd like to understand what's happening.
def bingSearch(query): #query is the store's name, i.e. "Bob's Pet Shop"
while True:
try:
bingBrowser.open('http://www.bing.com/search?q="' + query.replace(' ','+') + '"' )
htmlCode = bingBrowser.response().read()
break
except httplib.IncompleteRead:
#Sleep for a little while and try again.
Other relevant info:
Sometimes, for a single bing url, the program will to attempt to open and read that url multiple times, before a successful read without an IncompleteRead error.
bingBrowser's headers attribute is set up to look nice.
bingBrowser's robots attribute is set to false.
httplib: incomplete read ... I don't know anything about Apache so I wasn't able to understand the answer to the question, but it may be helpful to you. That said, I doubt that I'm having a similar problem (Why would bing.com be suffering from an Apache error?!)
Edit:
Replaced query.replace(' ','+') + '"' ) with urllib.urlencode(dict(q=query)) per JF Sebastian's suggestion - no change (I know this wasn't proposed as a solution).
Suffered from an inexplicable urllib2.URLError on bingBrowser.open('http://www.bing.com/search?q="' + query.replace(' ','+') + '"' )
Got an xlwt related "String longer than 65535 characters" error - probably unrelated.
Thanks in advance.