python urllib2.Request or urllib2.urlopen seems cutting response - python

I need to write a python script to do some validation on GSH number, the page is here: https://www.businessregistration-inscriptionentreprise.gc.ca/ebci/brom/registry/registryPrompt.do
As you can see there are three parameters to be provided. My script is as below (thanks to enter link description here):
import urllib
import urllib2
url = 'https://www.businessregistration-inscriptionentreprise.gc.ca/ebci/brom/registry/registryPromptSubmit.do'
values = {'businessNumber' : 'Michael Foord',
'businessName' : 'Northampton',
'requestDate' : '2016-10-23' }
data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
the_page = response.read()
print the_page
I inspect the result and it shows the response ends with something like these:
, here is a screenshot:
What happened to the response? where is the mainContentOfPage indicated in the right after the section?
Thank you very much.

Related

Accessing data from a json array in python JSONDecodeError at /get/

My response looks like:
[
{"_id":"5f6060d0d279373c0017447d","name":"Faizan","email":"faizan#test.com"}
]
I want to get the name in python. I am tryting:
response = requests.get(url)
data = response.json()
The error I am getting is:
JSONDecodeError at /get/
this might help you
import requests
import json
dburl = 'https://postman-9e13.restdb.io/rest/contact'
headers = {'x-apikey': '7267cbb3c251a01dd8563ca447194e78af67d', 'Content-Type': 'application/json'}
params = {"name":"Faizan","email":"faizan#test.com"}
r = requests.get(dburl, params=params, headers=headers)
print(r.json())
after seeing the result, you can pass to your templates like this
geodata = response.json()
return render(request, 'main/get.html', { 'name': geodata[0]['name'] })
or you can change accordingly.
The snippet you shared is correct, however the problem might be with the URL.
response = requests.get(url)
data = response.json()
Here, .json() only works for the response which is in the JSON format only. If it is showing that error that means your URL returning the HTML document.

POST request using urllib2 doesn't correctly send data (401 error)

I am trying to make a POST request in Python 2, using urllib2. My code is currently as follows;
url = 'http://' + server_url + '/playlists/upload?'
data = urllib.urlencode(OrderedDict([("sectionID", section_id), ("path", current_playlist), ("X-Plex-Token", plex_token)]))
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
d = response.read()
print(d)
'url' and 'data' return correctly formatted with the variables, I know this because I can copy their output into Postman for checking and the POST works fine (see example url below)
http://192.168.1.96:32400/playlists/upload?sectionID=11&path=D%3A%5CMedia%5CPPP%5Ctmp%5Cplex%5CAmbient.m3u&X-Plex-Token=XXXXXXXXX
When I run my Python code I get a 401 error returned, presumably meaning the X-Plex-Token parameter was not correctly sent, hence I am not allowed access.
Can anyone tell me where I'm going wrong? Help is greatly appreciated.
Have you tried removing the question mark and not using OrderedDict (no idea why you would need that) ?
url = 'http://' + server_url + '/playlists/upload'
data = urllib.urlencode({"sectionID":section_id), "path":current_playlist,"X-Plex-Token":plex_token})
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
d = response.read()
print(d)
Of course you should be using requests instead anyway:
import requests
r = requests.post('http://{}/playlists/upload'.format(server_url), data = {"sectionID":section_id), "path":current_playlist,"X-Plex-Token":plex_token})
print r.url
print r.text
print r.json
I've ended up switching to Python 3, as I didn't realise that the requests module was included by default. Still no idea why the above wasn't working, but maybe something to do with the lack of headers
headers = {'cache-control': "no-cache"}
edit:
This is what I'm using now, as mentioned above I probably don't need OrderedDict.
import requests
url = 'http://' + server_url + '/playlists/upload'
headers = {'cache-control': "no-cache"}
querystring = urllib.parse.urlencode(OrderedDict([("sectionID", section_id), ("path", current_playlist), ("X-Plex-Token", plex_token)]))
response = requests.request("POST", url, data = "", headers = headers, params = querystring)
print(response.text)

how to send cookies inside post request

trying to send Post request with the cookies on my pc from get request
#! /usr/bin/python
import re #regex
import urllib
import urllib2
#get request
x = urllib2.urlopen("http://www.example.com) #GET Request
cookies=x.headers['set-cookie'] #to get the cookies from get request
url = 'http://example' # to know the values type any password to know the cookies
values = {"username" : "admin",
"passwd" : password,
"lang" : "" ,
"option" : "com_login",
"task" : "login",
"return" : "aW5kZXgucGhw" }
data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
result = response.read()
cookies=response.headers['set-cookie'] #to get the last cookies from post req in this variable
then i searched in google
how to send cookies inside same post request and found
opener = urllib2.build_opener() # send the cookies
opener.addheaders.append(('Cookie', cookies)) # send the cookies
f = opener.open("http://example")
but i don't exactly where should i type it in my code
what i need to do exactly is to
send GET request, put the cookies from the request in variable,then make post request with the value that i got from the GET request
if anyone know answer i need edit on my code
Just create a HTTP opener and a cookiejar handler. So cookies will be retrieved and will be passed together to next request automatically. See:
import urllib2 as net
import cookielib
import urllib
cookiejar = cookielib.CookieJar()
cookiejar.clear_session_cookies()
opener = net.build_opener(net.HTTPCookieProcessor(cookiejar))
data = urllib.urlencode(values)
request = net.Request(url, urllib.urlencode(data))
response = opener.open(request)
As opener is a global handler, just make any request and the previous cookies sent from previous request will be in the next request (POST/GET), automatically.
You should really look into the requests library python has to offer. All you need to do is make a dictionary for you cookies key/value pair and pass it is as an arg.
Your entire code could be replaced by
#import requests
url = 'http://example' # to know the values type any password to know the cookies
values = {"username" : "admin",
"passwd" : password,
"lang" : "" ,
"option" : "com_login",
"task" : "login",
"return" : "aW5kZXgucGhw" }
session = requests.Session()
response = session.get(url, data=values)
cookies = session.cookies.get_dict()
response = reqeusts.post(url, data=values, cookies=cookies)
The second piece of code is probably what you want, but depends on the format of the response.

Having problems with Post via python

I'm having a bit of trouble. I'm trying to send a POST and trying to follow the documentation, but I can't seem to get it right.
on github: https://github.com/trtmn/Python
Pull requests welcomed!
# Getting documentation from :
#https://docs.python.org/2/howto/urllib2.html
import urllib
import urllib2
url = 'https://hooks.slack.com/services/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'
values = {"username": "webhookbot", "text": "This is posted to #general and comes from a bot named webhookbot.", "icon_emoji": ":ghost:"}
data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
the_page = response.read()
looks like I needed to stringify it as JSON (which I knew, but didn't know how). Thanks to Tim G. for the assist.
So here's the functional code:
import urllib2
import json
url = 'https://hooks.slack.com/services/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'
values = {"username": "webhookbot", "text": "This is posted to #general and comes from a bot named webhookbot.", "icon_emoji": ":ghost:"}
data = json.dumps(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
the_page = response.read()
Using httplib as an alternative for POST:
*import httplib
conn = httplib.HTTPSConnection(host)
conn.request('POST',urI,request_body, headers)
response = conn.getresponse()
resp_status=response.status
resp_reason=response.reason
resp_body=response.read()
conn.close()*
See if this helps.
Not sure if this solves the problem but if I add a slash to the end of your url I do receive a response when I execute your code.
url = 'https://hooks.slack.com/services/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/'

Converting urllib2 POST to Requests

I have an existing Http POST using urllib2:
data = 'client_id=%s&client_secret=%s&grant_type=authorization_code&code=%s&redirect_uri=%s' % (settings.GOOGLE_CLIENT_ID, settings.GOOGLE_CLIENT_SECRET, code, redirect_uri)
req = urllib2.Request(access_token_url, data=data)
response = urllib2.urlopen(req)
response_content = response.read()
json_response = json.loads(response_content)
I'm trying to convert this to the Requests library instead (http://docs.python-requests.org/) but I'm getting a 400 Invalid Request.
Here's my attempt:
params = {'redirect_uri' : redirect_uri}
params['client_id'] = settings.GOOGLE_CLIENT_ID
params['client_secret'] = settings.GOOGLE_CLIENT_SECRET
params['grant_type'] = 'authorization_code'
params['code'] = code
req = requests.post(access_token_url, data=params)
json_response = req.json()
I tried tweaking it to use params instead of data but I got the same error.
Anything I'm missing?
Make sure the values of the data dict are not already escaped as requests will do that for you. Please notice how your original example does not do any escaping.

Categories

Resources