I'm getting a string url request, I want to handle it and send the parameters, because it is very long.
I get plus in req.url instead of a space like query.
query = "https://www.superjob.ru/resume/search_resume.html?sbmit=1&detail_search=1&keywords%5B0%5D%5Bkeys%5D=%D0%9A%D0%BB%D0%B0%D0%B4%D0%BE%D0%B2%D1%89%D0%B8%D0%BA+%D0%9A%D0%BE%D0%BC%D0%BF%D0%BB%D0%B5%D0%BA%D1%82%D0%BE%D0%B2%D1%89%D0%B8%D0%BA+%D0%93%D1%80%D1%83%D0%B7%D1%87%D0%B8%D0%BA+%D0%A0%D0%B0%D0%B1%D0%BE%D1%82%D0%BD%D0%B8%D0%BA+%D1%81%D0%BA%D0%BB%D0%B0%D0%B4%D0%B0&keywords%5B0%5D%5Bskwc%5D=or&keywords%5B0%5D%5Bsrws%5D=60&keywords%5B1%5D%5Bkeys%5D=%D0%9A%D0%BB%D0%B0%D0%B4%D0%BE%D0%B2%D1%89%D0%B8%D0%BA+%D0%9A%D0%BE%D0%BC%D0%BF%D0%BB%D0%B5%D0%BA%D1%82%D0%BE%D0%B2%D1%89%D0%B8%D0%BA+%D0%93%D1%80%D1%83%D0%B7%D1%87%D0%B8%D0%BA+%D0%A0%D0%B0%D0%B1%D0%BE%D1%82%D0%BD%D0%B8%D0%BA+%D1%81%D0%BA%D0%BB%D0%B0%D0%B4%D0%B0&keywords%5B1%5D%5Bskwc%5D=or&keywords%5B1%5D%5Bsrws%5D=7&keywords%5B2%5D%5Bkeys%5D=%D0%AE%D0%BB%D0%BC%D0%B0%D1%80%D1%82&keywords%5B2%5D%5Bskwc%5D=nein&keywords%5B2%5D%5Bsrws%5D=50&keywords%5B3%5D%5Bkeys%5D=%D0%A0%D1%83%D0%BA%D0%BE%D0%B2%D0%BE%D0%B4%D0%B8%D1%82%D0%B5%D0%BB%D1%8C+%D0%B4%D0%B8%D1%80%D0%B5%D0%BA%D1%82%D0%BE%D1%80+%D0%BD%D0%B0%D1%87%D0%B0%D0%BB%D1%8C%D0%BD%D0%B8%D0%BA+%D1%8D%D0%BA%D0%BE%D0%BD%D0%BE%D0%BC%D0%B8%D1%81%D1%82+%D0%B0%D0%BD%D0%B0%D0%BB%D0%B8%D1%82%D0%B8%D0%BA+%D0%BB%D0%BE%D0%B3%D0%B8%D1%81%D1%82+%D0%9E%D1%84%D0%B8%D1%81-%D0%BC%D0%B5%D0%BD%D0%B5%D0%B4%D0%B6%D0%B5%D1%80+%D0%93%D0%BE%D1%81%D1%83%D0%B4%D0%B0%D1%80%D1%81%D0%B2*&keywords%5B3%5D%5Bskwc%5D=nein&keywords%5B3%5D%5Bsrws%5D=60&exclude_words=%D0%A0%D1%83%D0%BA%D0%BE%D0%B2%D0%BE%D0%B4%D0%B8%D1%82%D0%B5%D0%BB%D1%8C+%D0%B4%D0%B8%D1%80%D0%B5%D0%BA%D1%82%D0%BE%D1%80+%D0%BD%D0%B0%D1%87%D0%B0%D0%BB%D1%8C%D0%BD%D0%B8%D0%BA+%D1%8D%D0%BA%D0%BE%D0%BD%D0%BE%D0%BC%D0%B8%D1%81%D1%82+%D0%B0%D0%BD%D0%B0%D0%BB%D0%B8%D1%82%D0%B8%D0%BA+%D0%BB%D0%BE%D0%B3%D0%B8%D1%81%D1%82+%D0%9E%D1%84%D0%B8%D1%81-%D0%BC%D0%B5%D0%BD%D0%B5%D0%B4%D0%B6%D0%B5%D1%80+%D0%93%D0%BE%D1%81%D1%83%D0%B4%D0%B0%D1%80%D1%81%D0%B2*&period=7&place_of_work=0&t%5B%5D=4&m%5B%5D=194&m%5B%5D=195&m%5B%5D=193&m%5B%5D=192&m%5B%5D=191&m%5B%5D=164&m%5B%5D=162&m%5B%5D=163&m%5B%5D=150&m%5B%5D=534&m%5B%5D=540&m%5B%5D=149&m%5B%5D=148&m%5B%5D=147&m%5B%5D=146&m%5B%5D=535&m%5B%5D=536&m%5B%5D=159&m%5B%5D=157&m%5B%5D=158&m%5B%5D=156&m%5B%5D=155&m%5B%5D=154&m%5B%5D=161&m%5B%5D=20&m%5B%5D=21&m%5B%5D=22&m%5B%5D=24&m%5B%5D=23&m%5B%5D=25&m%5B%5D=28&m%5B%5D=573&m%5B%5D=29&m%5B%5D=27&m%5B%5D=26&m%5B%5D=145&m%5B%5D=144&m%5B%5D=143&m%5B%5D=142&m%5B%5D=141&m%5B%5D=30&m%5B%5D=31&m%5B%5D=115&m%5B%5D=153&m%5B%5D=72&m%5B%5D=84&m%5B%5D=152&m%5B%5D=83&m%5B%5D=82&m%5B%5D=81&m%5B%5D=79&m%5B%5D=80&m%5B%5D=542&m%5B%5D=151&m%5B%5D=46&m%5B%5D=85&paymentfrom=20000&paymentto=35000&type_of_work=0&citizenship%5B0%5D=1&old1=18&old2=40&maritalstatus=0&pol=0&children=0&education=0&eduform=0&id_institute=0&institution=&languages%5B0%5D%5Blanguage_id%5D=0&languages%5B0%5D%5Blanguage_level%5D=0&business_trip=0"
def getparams(url):
params = {}
urlarr = url.split("&")
i = 0
while i < len(urlarr):
params[urllib2.unquote(urlarr[i].split("=")[0].encode("utf-8"))] = urllib2.unquote(urlarr[i].split("=")[1].encode('utf-8'))
i += 1
return params
s = Session()
req = s.get(query.split("?")[0], params=getparams(query.split("?")[1]))
print(req.url)
I tried decode() instead encode() and without encode() and without urllib2.unquote()
Have you looked at the url you are posting?
you set
query = "https://www.superjob.ru/resume/search_resume.html?sbmit=1&detail_search=1&keywords%5B0%5D%5Bkeys%5D=%D0%9A%D0%BB%D0%B0%D0%B4%D0%BE%D0%B2%D1%89%D0%B8%D0%BA+%D0%9A%D0%BE%D0%BC%D0%BF%D0%BB%D0%B5%D0%BA%D1%82%D0%BE%D0%B2%D1%89%D0%B8%D0%BA+%D0%93%D1%80%D1%83%D0%B7%D1%87%D0%B8%D0%BA+%D0%A0%D0%B0%D0%B1%D0%BE%D1%82%D0%BD%D0%B8%D0%BA+%D1%81%D0%BA%D0%BB%D0%B0%D0%B4%D0%B0&keywords%5B0%5D%5Bskwc%5D=or&keywords%5B0%5D%5Bsrws%5D=60&keywords%5B1%5D%5Bkeys%5D=%D0%9A%D0%BB%D0%B0%D0%B4%D0%BE%D0%B2%D1%89%D0%B8%D0%BA+%D0%9A%D0%BE%D0%BC%D0%BF%D0%BB%D0%B5%D0%BA%D1%82%D0%BE%D0%B2%D1%89%D0%B8%D0%BA+%D0%93%D1%80%D1%83%D0%B7%D1%87%D0%B8%D0%BA+%D0%A0%D0%B0%D0%B1%D0%BE%D1%82%D0%BD%D0%B8%D0%BA+%D1%81%D0%BA%D0%BB%D0%B0%D0%B4%D0%B0&keywords%5B1%5D%5Bskwc%5D=or&keywords%5B1%5D%5Bsrws%5D=7&keywords%5B2%5D%5Bkeys%5D=%D0%AE%D0%BB%D0%BC%D0%B0%D1%80%D1%82&keywords%5B2%5D%5Bskwc%5D=nein&keywords%5B2%5D%5Bsrws%5D=50&keywords%5B3%5D%5Bkeys%5D=%D0%A0%D1%83%D0%BA%D0%BE%D0%B2%D0%BE%D0%B4%D0%B8%D1%82%D0%B5%D0%BB%D1%8C+%D0%B4%D0%B8%D1%80%D0%B5%D0%BA%D1%82%D0%BE%D1%80+%D0%BD%D0%B0%D1%87%D0%B0%D0%BB%D1%8C%D0%BD%D0%B8%D0%BA+%D1%8D%D0%BA%D0%BE%D0%BD%D0%BE%D0%BC%D0%B8%D1%81%D1%82+%D0%B0%D0%BD%D0%B0%D0%BB%D0%B8%D1%82%D0%B8%D0%BA+%D0%BB%D0%BE%D0%B3%D0%B8%D1%81%D1%82+%D0%9E%D1%84%D0%B8%D1%81-%D0%BC%D0%B5%D0%BD%D0%B5%D0%B4%D0%B6%D0%B5%D1%80+%D0%93%D0%BE%D1%81%D1%83%D0%B4%D0%B0%D1%80%D1%81%D0%B2*&keywords%5B3%5D%5Bskwc%5D=nein&keywords%5B3%5D%5Bsrws%5D=60&exclude_words=%D0%A0%D1%83%D0%BA%D0%BE%D0%B2%D0%BE%D0%B4%D0%B8%D1%82%D0%B5%D0%BB%D1%8C+%D0%B4%D0%B8%D1%80%D0%B5%D0%BA%D1%82%D0%BE%D1%80+%D0%BD%D0%B0%D1%87%D0%B0%D0%BB%D1%8C%D0%BD%D0%B8%D0%BA+%D1%8D%D0%BA%D0%BE%D0%BD%D0%BE%D0%BC%D0%B8%D1%81%D1%82+%D0%B0%D0%BD%D0%B0%D0%BB%D0%B8%D1%82%D0%B8%D0%BA+%D0%BB%D0%BE%D0%B3%D0%B8%D1%81%D1%82+%D0%9E%D1%84%D0%B8%D1%81-%D0%BC%D0%B5%D0%BD%D0%B5%D0%B4%D0%B6%D0%B5%D1%80+%D0%93%D0%BE%D1%81%D1%83%D0%B4%D0%B0%D1%80%D1%81%D0%B2*&period=7&place_of_work=0&t%5B%5D=4&m%5B%5D=194&m%5B%5D=195&m%5B%5D=193&m%5B%5D=192&m%5B%5D=191&m%5B%5D=164&m%5B%5D=162&m%5B%5D=163&m%5B%5D=150&m%5B%5D=534&m%5B%5D=540&m%5B%5D=149&m%5B%5D=148&m%5B%5D=147&m%5B%5D=146&m%5B%5D=535&m%5B%5D=536&m%5B%5D=159&m%5B%5D=157&m%5B%5D=158&m%5B%5D=156&m%5B%5D=155&m%5B%5D=154&m%5B%5D=161&m%5B%5D=20&m%5B%5D=21&m%5B%5D=22&m%5B%5D=24&m%5B%5D=23&m%5B%5D=25&m%5B%5D=28&m%5B%5D=573&m%5B%5D=29&m%5B%5D=27&m%5B%5D=26&m%5B%5D=145&m%5B%5D=144&m%5B%5D=143&m%5B%5D=142&m%5B%5D=141&m%5B%5D=30&m%5B%5D=31&m%5B%5D=115&m%5B%5D=153&m%5B%5D=72&m%5B%5D=84&m%5B%5D=152&m%5B%5D=83&m%5B%5D=82&m%5B%5D=81&m%5B%5D=79&m%5B%5D=80&m%5B%5D=542&m%5B%5D=151&m%5B%5D=46&m%5B%5D=85&paymentfrom=20000&paymentto=35000&type_of_work=0&citizenship%5B0%5D=1&old1=18&old2=40&maritalstatus=0&pol=0&children=0&education=0&eduform=0&id_institute=0&institution=&languages%5B0%5D%5Blanguage_id%5D=0&languages%5B0%5D%5Blanguage_level%5D=0&business_trip=0"
which contains a lot of escape quotes. None of them are a space (%20)
The reason you get a plus instead of a space, is because your url contains a plus in place of every space. This is expected behaviour, as proper URLs should not contain any spaces, and thus, are often converted into "+" beforehand.
If I put your escaped URL into my browser hotbar and look at it, all I see is "+", no spaces there.
This is from somewhere at the start of your URL:
detail_search=1&keywords%5B0%5D%5Bkeys%5D=%D0%9A%D0%BB%D0%B0%D0%B4%D0%BE%D0%B2%D1%89%D0%B8%D0%BA+%D0%9A%D0%BE%D0%BC%D0%BF%D0%BB%D0%B5%D0%BA%D1%82%D0%BE%D0%B2%D1%89%D0%B8%D0%BA
which is essentially detail_search=1&keywords[0][keys]=Кладовщик+Комплектовщик
if unescaped. As you can clearly see, both versions, quoted and unquoted, contain the plus. This is done by the website, and can't (shouldn't) be changed, especially if you want to reconstruct the URL somewhere else.
If you need the parameters without the plus, just do "myUnquotedString".replace("+"," ") (after unquoting, because i don't know what urllib2 does if it gets a malformed string as parameter for unquote().
I solved this problem like this:
query = "https://www.superjob.ru/resume/search_resume.html?sbmit=1&detail_search=1&keywords%5B0%5D%5Bkeys%5D=%D0%9A%D0%BB%D0%B0%D0%B4%D0%BE%D0%B2%D1%89%D0%B8%D0%BA+%D0%9A%D0%BE%D0%BC%D0%BF%D0%BB%D0%B5%D0%BA%D1%82%D0%BE%D0%B2%D1%89%D0%B8%D0%BA+%D0%93%D1%80%D1%83%D0%B7%D1%87%D0%B8%D0%BA+%D0%A0%D0%B0%D0%B1%D0%BE%D1%82%D0%BD%D0%B8%D0%BA+%D1%81%D0%BA%D0%BB%D0%B0%D0%B4%D0%B0&keywords%5B0%5D%5Bskwc%5D=or&keywords%5B0%5D%5Bsrws%5D=60&keywords%5B1%5D%5Bkeys%5D=%D0%9A%D0%BB%D0%B0%D0%B4%D0%BE%D0%B2%D1%89%D0%B8%D0%BA+%D0%9A%D0%BE%D0%BC%D0%BF%D0%BB%D0%B5%D0%BA%D1%82%D0%BE%D0%B2%D1%89%D0%B8%D0%BA+%D0%93%D1%80%D1%83%D0%B7%D1%87%D0%B8%D0%BA+%D0%A0%D0%B0%D0%B1%D0%BE%D1%82%D0%BD%D0%B8%D0%BA+%D1%81%D0%BA%D0%BB%D0%B0%D0%B4%D0%B0&keywords%5B1%5D%5Bskwc%5D=or&keywords%5B1%5D%5Bsrws%5D=7&keywords%5B2%5D%5Bkeys%5D=%D0%AE%D0%BB%D0%BC%D0%B0%D1%80%D1%82&keywords%5B2%5D%5Bskwc%5D=nein&keywords%5B2%5D%5Bsrws%5D=50&keywords%5B3%5D%5Bkeys%5D=%D0%A0%D1%83%D0%BA%D0%BE%D0%B2%D0%BE%D0%B4%D0%B8%D1%82%D0%B5%D0%BB%D1%8C+%D0%B4%D0%B8%D1%80%D0%B5%D0%BA%D1%82%D0%BE%D1%80+%D0%BD%D0%B0%D1%87%D0%B0%D0%BB%D1%8C%D0%BD%D0%B8%D0%BA+%D1%8D%D0%BA%D0%BE%D0%BD%D0%BE%D0%BC%D0%B8%D1%81%D1%82+%D0%B0%D0%BD%D0%B0%D0%BB%D0%B8%D1%82%D0%B8%D0%BA+%D0%BB%D0%BE%D0%B3%D0%B8%D1%81%D1%82+%D0%9E%D1%84%D0%B8%D1%81-%D0%BC%D0%B5%D0%BD%D0%B5%D0%B4%D0%B6%D0%B5%D1%80+%D0%93%D0%BE%D1%81%D1%83%D0%B4%D0%B0%D1%80%D1%81%D0%B2*&keywords%5B3%5D%5Bskwc%5D=nein&keywords%5B3%5D%5Bsrws%5D=60&exclude_words=%D0%A0%D1%83%D0%BA%D0%BE%D0%B2%D0%BE%D0%B4%D0%B8%D1%82%D0%B5%D0%BB%D1%8C+%D0%B4%D0%B8%D1%80%D0%B5%D0%BA%D1%82%D0%BE%D1%80+%D0%BD%D0%B0%D1%87%D0%B0%D0%BB%D1%8C%D0%BD%D0%B8%D0%BA+%D1%8D%D0%BA%D0%BE%D0%BD%D0%BE%D0%BC%D0%B8%D1%81%D1%82+%D0%B0%D0%BD%D0%B0%D0%BB%D0%B8%D1%82%D0%B8%D0%BA+%D0%BB%D0%BE%D0%B3%D0%B8%D1%81%D1%82+%D0%9E%D1%84%D0%B8%D1%81-%D0%BC%D0%B5%D0%BD%D0%B5%D0%B4%D0%B6%D0%B5%D1%80+%D0%93%D0%BE%D1%81%D1%83%D0%B4%D0%B0%D1%80%D1%81%D0%B2*&period=7&place_of_work=0&t%5B%5D=4&m%5B%5D=194&m%5B%5D=195&m%5B%5D=193&m%5B%5D=192&m%5B%5D=191&m%5B%5D=164&m%5B%5D=162&m%5B%5D=163&m%5B%5D=150&m%5B%5D=534&m%5B%5D=540&m%5B%5D=149&m%5B%5D=148&m%5B%5D=147&m%5B%5D=146&m%5B%5D=535&m%5B%5D=536&m%5B%5D=159&m%5B%5D=157&m%5B%5D=158&m%5B%5D=156&m%5B%5D=155&m%5B%5D=154&m%5B%5D=161&m%5B%5D=20&m%5B%5D=21&m%5B%5D=22&m%5B%5D=24&m%5B%5D=23&m%5B%5D=25&m%5B%5D=28&m%5B%5D=573&m%5B%5D=29&m%5B%5D=27&m%5B%5D=26&m%5B%5D=145&m%5B%5D=144&m%5B%5D=143&m%5B%5D=142&m%5B%5D=141&m%5B%5D=30&m%5B%5D=31&m%5B%5D=115&m%5B%5D=153&m%5B%5D=72&m%5B%5D=84&m%5B%5D=152&m%5B%5D=83&m%5B%5D=82&m%5B%5D=81&m%5B%5D=79&m%5B%5D=80&m%5B%5D=542&m%5B%5D=151&m%5B%5D=46&m%5B%5D=85&paymentfrom=20000&paymentto=35000&type_of_work=0&citizenship%5B0%5D=1&old1=18&old2=40&maritalstatus=0&pol=0&children=0&education=0&eduform=0&id_institute=0&institution=&languages%5B0%5D%5Blanguage_id%5D=0&languages%5B0%5D%5Blanguage_level%5D=0&business_trip=0"
query = query.encode('utf-8')
s = Session()
req = s.get(query.split("?")[0], params=parse_qs(urlparse(query).query))
Related
It is not supposed to be a hard problem, but I've worked on it for almost a day!
I want to create a query which has to be in this format: 'lat':'###','long':'###' where ###s represent latitude and longitude.
I am using the following code to generate the queries:
coordinateslist=[]
for i in range(len(lat)):
coordinateslist.append("'lat':'{}','long':'-{}'".format(lat[i],lon[i]))
coordinateslist
However the result would be some thing similar to this which has "" at the beginning and end of it: "'lat':'40.66','long':'-73.93'"
Ridiculously enough it's impossible to remove the " with either .replace or .strip! and wrapping the terms around repr doesn't solve the issue.
Do you know how I can get rid of those double quotation marks?
P.S. I know that when I print the command the "will not be shown but when i use each element of the array in my query, a " will appear at the end of the query which stops it from working.
directly writing the line like this:
query_params = {'lat':'43.57','long':'-116.56'}
works perfectly fine.
but using either of the codes below will lead to an error.
aa=print(coordinateslist[0])
bb=coordinateslist[0]
query_params = {aa}
query_params = {bb}
query_params = aa
query_params = bb
Try using a dictionary instead, if you don't want to see the " from string representation:
coordinateslist.append({
"lat": lat[i],
"long": "-{}".format(lon[i])
})
It is likely that the error you are getting is something else entirely (i.e. unrelated to the quotes you are seeing in printed outputs). From the format of the query, I would guess that it is expecting a properly formatted URL parameters: reverse_geocode.json?... which 'lat':'41.83','long':'-87.68' is not. Did you try to manually call it with a fixed string, (e.g. using with postman) ?
Assuming you're calling the twitter geo/ API, you might want to ry it out with properly separated URL parameters.
geo/reverse_geocode.json?lat=41.83&long=-87.68
I am kind of very new to python.
I tried to loop through an URL request via python and I want to change one variable each time it loops.
My code looks something like this:
codes = ["MCDNDF3","MCDNDF4"]
#count = 0
for x in codes:
response = requests.get(url_part1 + str(codes) + url_part3, headers=headers)
print(response.content)
print(response.status_code)
print(response.url)
I want to have the url change at every loop to like url_part1+code+url_part3 and then url_part1+NEXTcode+url_part3.
Sadly my request badly formats the string from the variable to "%5B'MCDNDF3'%5D".
It should get inserted as a raw string each loop. I don't know if I need url encoding as I don't have any special chars in the request. Just change code to MCDNDF3 and in the next request to MCDNDF4.
Any thoughts?
Thanks!
In your for loop, the first line should be:
response = requests.get(url_part1 + x + url_part3, headers=headers)
This will work assuming url_part1 and url_part3 are regular strings. x is already a string, as your codes list (at least in your example) contains only strings. %5B and %5D are [ and ] URL-encoded, respectively. You got that error because you called str() on a single-membered list:
>>> str(["This is a string"])
"['This is a string']"
If url_part1 and url_part3 are raw strings, as you seem to indicate, please update your question to show how they are defined. Feel free to use example.com if you don't want to reveal your actual target URL. You should probably be calling str() on them before constructing the full URL.
You’re putting the whole list in (codes) when you probably want x.
I have this link below ( where the freepeople+top has the + replacing a space)
https://poshmark.com/search?query=freepeople+top&type=listings&department=Women
I was doing this to query the link :
search='https://poshmark.com/search?'
brand="freepeople"
style="top"
# & seperates parameters
queryParameters={'query':[brand,style],'type':'listings','department':'Women'}
response=requests.get(search,params=queryParameters)
I'm confused why when I did print(response.text) it appears like it gives me all the html but when I do:
MacBook-Air-4:finalproject BCohen$ python3 poshmart.py >/tmp/poshmart.html
MacBook-Air-4:finalproject BCohen$ open /tmp/poshmart.html
It doesn't take me to a valid page
I was assuming that maybe I queried the Indexed search ( the + ) wrong but I'm not sure how to properly query it.
You can use '+'.join([brand,style]) to turn that array into a string where the values are joined with a +. The result of that would be what you're looking for: freepeople+top
import requests
search='https://poshmark.com/search?'
brand="freepeople"
style="top"
print('+'.join([brand,style]))
# & seperates parameters
queryParameters={'query':'+'.join([brand,style]),'type':'listings','department':'Women'}
response=requests.get(search,params=queryParameters)
print(response.request.url)
and the output is
freepeople+top
https://poshmark.com/search?query=freepeople%2Btop&type=listings&department=Women
the reason it shows up as %2B in the 2nd print is because that is the urlencoded value of +
I faced a trouble to properly encode a query string to use Bing image search API.
I got my account key for using Bing API, which contains "/" and "+". So when I try Bing example query like
http://api.bing.net/json.aspx?AppId=MY_APP_ID&Query=xbox site:microsoft.com&Sources=Image
I got reply that my AppId value is invalid:
{"SearchResponse":{"Version":"2.2","Query":{"SearchTerms":"xbox site\u003amicrosoft.com"},"Errors":[{"Code":1002,"Message":"Parameter has invalid value.","Parameter":"SearchRequest.AppId","Value":"*******\u002*****\u002b***","HelpUrl":"http\u003a\u002f\u002fmsdn.microsoft.com\u002fen-us\u002flibrary\u002fdd251042.aspx"}]}}
Where *** are valid characters of my account key
I tried all possible ways that came to my mind and found in web, but still failed to solve it. So what I tried:
import requests
url = "http://api.bing.net/json.aspx?AppId=****/***+***&Query=xbox site:microsoft.com&Sources=Image"
r = requests.get(url)
I got an error that the value is invalid "****\u002*** ***"
I tried doing the same thing using urllib2, trying to encode and quote both the whole query and the account key only. The code for separately quoting each part of request is like this:
import urllib2
urlStart = u"http://api.bing.net/json.aspx?AppId=%s&Query=xbox"
quotedUrlStart = urllib2.quote(urlStart.encode("utf8"), safe="%/:=&?~#+!$,;'#()*[]")
urlEnd = u" site:microsoft.com&Sources=Image"
quotedUrlEnd = urllib2.quote(urlEnd.encode("utf8"), safe="")
key = u"**/**+**"
quotedKey = urllib2.quote(key.encode("utf8") , safe="%:=&?~#!$,;'#()*[]")
fullUrl = (quotedUrlStart % quotedKey) + quotedUrlEnd
reply = urllib2.urlopen(fullUrl).read()
print reply
I also tried to replace "/" with %2F and "+" with %2B, but the error is the same.
What is a mess for me here is what I have to quote and what not. I actually don't have a clear understanding so far how these things work. I guess that I have to encode everything and quote it different ways - qoute slashes in one place and do not qoute in another.
This question addresses the same issue: XCODE Swift Replacing HTTP-Get App-ID with a space
Also there are numerous questions on SO on escaping symbols, but they were unhelpful for me
I appreciate your time guys
I have a url that I need to add an api key to as a parameter. The key has % and other characters in it and should not be url encoded, is there a way to do this with furl?
Heres my current code:
request_url = furl('http://www.domain.com/service/') \
.add({
'format' : 'json',
'location' : address_line_1 + ',' + city + ',' + state,
'key' : APP_KEY
}).url
I have a url that I need to add an api key to as a parameter. The key has % and other characters in it and should not be url encoded
That can't be true. It must be URL-encoded. Otherwise, you end up with either an invalid URL, or a URL that doesn't mean what you wanted.
For example, let's say the key is 123%456%789. If you send http://www.domain.com/service/?format=json&key=123%25456%25789, the web service on the other end will get 123%456%789, which will succeed. If you just send http://www.domain.com/service/?format=json&key=123%456%789, the web service on the other end will get 123E6x9, which will fail.
So, your example is already correct, and you shouldn't need to do anything.
If, for some reason, you're already URL-encoding APP_KEY before you get here… well, don't. With a name like that, I'm guessing it's a constant literal just copied and pasted into your module, which means you had to manually URL-encode it and copy and paste the result. Just don't do that, and you're fine.
If you got the APP_KEY in URL-encoded form, you can always decode it with, e.g., urlparse.parse_qs, or even with furl.
But if none of that seems reasonable…
is there a way to do this with furl?
No. See the Encoding section of the docs. furl only works on decoded query string names and values.
But there's an easy way around this. A URL isn't that complicated of a thing, and furl is specifically designed to allow you to mix and match manually- and programmatically- created bits. Most of the examples in the README show exactly that, such as furl('http://www.google.com/?one=1').add({'two':2}).url. So, if you've already got a pre-encoded "key=123%25456%25789" from somewhere, just stick it on the string manually before giving it to furl:
request_url = furl('http://www.domain.com/service/?key={}'.format(APP_KEY)) \
.add({
'format' : 'json',
'location' : address_line_1 + ',' + city + ',' + state,
}).url
Hacky? Well, yes, in that you're side-stepping furl's encoding, but that's exactly what you're asking how to do.