I want to send JSON encoded data to a server using only native Python libraries. I love requests but I simply can't use it because I can't use it on the machine which runs the script. I need to do it without.
newConditions = {"con1":40, "con2":20, "con3":99, "con4":40, "password":"1234"}
params = urllib.parse.urlencode(newConditions)
params = params.encode('utf-8')
req = urllib.request.Request(conditionsSetURL, data=params)
urllib.request.urlopen(req)
My server is a local WAMP server. I always get an
urllib.error.HTTPError: HTTP Error 500: Internal Server Error
I am 100% sure that this is NOT a server issue, because the same data, with the same url, on the same machine, with the same server works with the requests library and Postman.
You are not posting JSON, you are posting a application/x-www-form-urlencoded request.
Encode to JSON and set the right headers:
import json
newConditions = {"con1":40, "con2":20, "con3":99, "con4":40, "password":"1234"}
params = json.dumps(newConditions).encode('utf8')
req = urllib.request.Request(conditionsSetURL, data=params,
headers={'content-type': 'application/json'})
response = urllib.request.urlopen(req)
Demo:
>>> import json
>>> import urllib.request
>>> conditionsSetURL = 'http://httpbin.org/post'
>>> newConditions = {"con1":40, "con2":20, "con3":99, "con4":40, "password":"1234"}
>>> params = json.dumps(newConditions).encode('utf8')
>>> req = urllib.request.Request(conditionsSetURL, data=params,
... headers={'content-type': 'application/json'})
>>> response = urllib.request.urlopen(req)
>>> print(response.read().decode('utf8'))
{
"args": {},
"data": "{\"con4\": 40, \"con2\": 20, \"con1\": 40, \"password\": \"1234\", \"con3\": 99}",
"files": {},
"form": {},
"headers": {
"Accept-Encoding": "identity",
"Connection": "close",
"Content-Length": "68",
"Content-Type": "application/json",
"Host": "httpbin.org",
"User-Agent": "Python-urllib/3.4",
"X-Request-Id": "411fbb7c-1aa0-457e-95f9-1af15b77c2d8"
},
"json": {
"con1": 40,
"con2": 20,
"con3": 99,
"con4": 40,
"password": "1234"
},
"origin": "84.92.98.170",
"url": "http://httpbin.org/post"
}
Related
I would get access to this website, I get it from another website, https://www.betexplorer.com/soccer/algeria/ligue-1/bordj-bou-arreridj-tlemcen/tfvAHu7U/, in the Network section of Developer tools.
Code
import requests
url = 'https://www.betexplorer.com/archive-odds/4urejxv464x0xd4645/18/'
response = requests.get(url, headers={'User-Agent': 'my user agent'})
print(response)
Output <Response [404]>
Set Referer HTTP header to obtain correct response:
import json
import requests
url = "https://www.betexplorer.com/archive-odds/4urejxv464x0xd4645/18/"
headers = {
"Referer": "https://www.betexplorer.com",
}
data = requests.get(url, headers=headers).json()
print(json.dumps(data, indent=4))
Prints:
[
{
"date": "23.07. 17:36",
"odd": "1.88",
"change": "+0.08"
},
{
"date": "23.07. 17:34",
"odd": "1.80",
"change": "-0.01"
},
...
I want to send a multipart/form-data as a post body using python request but I am not getting bad request issue.
import requests
headers = {"Content-Type": "multipart/form-data"}
data = {
"#context": "http://semantro.com/", "#type": "KiranaSearch", "actionName": "listCategoryProducts",
"pageLimit": {"#context": "http://semantro.com/", "#type": "PageProperty", "start": 0, "end": 24},
"data": {"#context": "http://semantro.com/", "#type": "KiranaCategory",
"identifier": "c5394d1d5c6c4cb8-adc77dd996876dba"}
}
response = requests.post('https://merokirana.com/semantro-web-interface/query',
data=data, headers=headers)
print(response.text)
Response
{
"statusTitle" : "ServiceUnsuccessful",
"statusMessage" : "Invalid type of data received. The request should have multipart query data.",
"#context" : "http://semantro.com",
"#type" : "RemoteServiceStatus"
}
But I can retrieve the required data using postman the same formdata.
import http.client
import mimetypes
from codecs import encode
conn = http.client.HTTPSConnection("merokirana.com")
dataList = []
boundary = 'wL36Yn8afVp8Ag7AmP8qZ0SA4n1v9T'
dataList.append(encode('--' + boundary))
dataList.append(encode('Content-Disposition: form-data; name=data;'))
dataList.append(encode('Content-Type: {}'.format('text/plain')))
dataList.append(encode(''))
dataList.append(encode("{ \"#context\": \"http://semantro.com/\", \"#type\": \"KiranaSearch\", \"actionName\": \"listCategoryProducts\",\"pageLimit\": {\"#context\": \"http://semantro.com/\", \"#type\": \"PageProperty\", \"start\": 0, \"end\": 24}, \"data\": {\"#context\": \"http://semantro.com/\", \"#type\": \"KiranaCategory\",\"identifier\": \"c5394d1d5c6c4cb8-adc77dd996876dba\"}}"))
dataList.append(encode('--'+boundary+'--'))
dataList.append(encode(''))
body = b'\r\n'.join(dataList)
payload = body
headers = {
'Content-type': 'multipart/form-data; boundary={}'.format(boundary)
}
conn.request("POST", "/semantro-web-interface/query", payload, headers)
res = conn.getresponse()
data = res.read()
print(data.decode("utf-8"))
output:
I'm trying to import contacts into a contact list in Qualtrics. I am using python to do this.
Token = 'MyToken' #when running the code I put in my actual token and id
ContactsID = 'MyContactsID'
data = open('contacts.json', 'rb')
headers = {'X-API-TOKEN': Token, 'Content-Type':'application/json',}
r = requests.post('https://az1.qualtrics.com/API/v3/mailinglists/' + ContactsID +'/contactimports', headers=headers, data=data)
r.text
This code gives me the following error: '{"meta":{"httpStatus":"400 - Bad Request","error":{"errorMessage":"Invalid Content-Type. expected=multipart/form-data found=application/json","errorCode":"RP_0.1"},"requestId":null}}'
I changed the content type to multipart/form-data that it says it is expecting and received the response "413", which qualtrics explains means "The request body was too large. This can also happen in cases where a multipart/form-data request is malformed."
I have tested my json and verified that it is valid. Also, I don't know why the request body would be too large because it's only 13 contacts that I'm trying to import. Any ideas?
With the help of Qualtrics Support, I was eventually able to get the following code to work:
Token = 'MyToken' #when running the code I put in my actual token and id
ContactsID = 'MyContactsID'
url = "https://az1.qualtrics.com/API/v3/mailinglists/" + ContactsID + "/contactimports/"
headers = {
'content-type': "multipart/form-data; boundary=---BOUNDRY",
'x-api-token': "Token"
}
files = {'contacts': ('contacts', open('contacts.json', 'rb'), 'application/json')}
request = requests.post(url, headers=headers, files=files)
print(request.text)
Please note that if you want to use this code, you will need to change "az1" in the URL to your own Qualtrics datacenter ID.
You need to use files = .. for a multipart request:
Token = 'MyToken' #when running the code I put in my actual token and id
ContactsID = 'MyContactsID'
data = open('contacts.json', 'rb')
headers = {'X-API-TOKEN': Token}
r = requests.post('https://az1.qualtrics.com/API/v3/mailinglists/' + ContactsID +'/contactimports',files={"file":data}, headers=headers)
r.text
Once you do requests will take care of the rest:
In [36]: url = 'http://httpbin.org/post'
In [37]: headers = {'X-API-TOKEN': "123456789"}
In [38]: files = {'file': open('a.csv', 'rb')}
In [39]: r = requests.post(url, files=files, headers=headers)
In [40]: print r.text
{
"args": {},
"data": "",
"files": {
"file": "a,b,c\n1,2,3"
},
"form": {},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Content-Length": "152",
"Content-Type": "multipart/form-data; boundary=3830dbe5fa6141f69d3d85dee4ba6e78",
"Host": "httpbin.org",
"User-Agent": "python-requests/2.10.0",
"X-Api-Token": "123456789"
},
"json": null,
"origin": "51.171.98.185",
"url": "http://httpbin.org/post"
}
In [41]: print(r.request.body)
--3830dbe5fa6141f69d3d85dee4ba6e78
Content-Disposition: form-data; name="file"; filename="a.csv"
a,b,c
1,2,3
--3830dbe5fa6141f69d3d85dee4ba6e78--
looking at the docs, you actually want something closer to:
Token = 'MyToken' #when running the code I put in my actual token and id
ContactsID = 'MyContactsID'
data = open('contacts.json', 'rb')
files = {'file': ('contact', data ,'application/json', {'X-API-TOKEN': Token})}
r = requests.post('https://az1.qualtrics.com/API/v3/mailinglists/' + ContactsID +'/contactimports',files=files)
I have the following cookie, stored as a dict:
cookies = session.cookies.get_dict()
{'nflx-rgn': '"ue1|1433799654706|uw2:2"', 'SecureNetflixId': 'v%3D2%26mac%3DAQEAEQABABTh6lNR0cbOgGRD7DT7CChBIhA83qsfV_k.%26dt%3D1433799654968', 'memclid': '9a55d2c6-4d85-44bb-8338-0b1a180a7c44', 'NetflixId': 'v%3D2%26ct%3DBQAOAAEBEFP07qwktIitvHYO-Xn6gU2BYKymzutSFtIyfD_zEWVo4yMjbjJ4LgPmtZhjgE5lRWg5OTP8C1N2zNbTc8VsdzUS3OaLnsGsV4AkXZzkewt0X0WcMwSYVl3-pX8Rs0jH2PCXixD8K2oARmtQIrWCvv35EUOpGCb3v-tExqpZaN_6FBfTrF68FXcR4kDGAjXYooBWKnCv1aS1VVscso_GRg6HH7--qOsjW7WoPkxyRAAqt0GmeD8tY5aQHMbD779s7oiAr7YArEmUC_3Hdej0huyGo0Iq-_IlY_jGZhyupIO0ZIpznvGB2teF8YeaiSh2LlQ4RIlUOo3hO5n9lmlmBnRXAYGYbJ_64HiwTfsw3jRWkgAJWnnxSHo9sGLJibspLCQ3RzdA-JwxSPYoBh-HkhnAxxPveWmPbH_1JtXR9rvhfnenIhDgzu_nCUH7i-sxg4bnKhGjpVzH1Y8x9wSNE2-xWdax912FBEzQWyT5I2HxnQQ.%26bt%3Dusr%26ch%3DAQEAEAABABRxRbrqqSNQFSxPcuqeaLCNvVMfYe40K6A.%26mac%3DAQEAEAABABTQs51fmHZefZn26dZQFm9ZDMjlNgmXis8.', 'tlr': 'US|1433799655947'}
How would I then add that to my requests header so it would be valid. For example:
headers = {
"Accept-Language" : "en-US,en;q=0.8,pt;q=0.6",
"Connection" : "keep-alive",
"Cookies": ???
}
requests.get(url, headers=headers)
I would prefer to do this over adding in the cookies argument to requests.
Use this:
cookie_string = "; ".join([str(x)+"="+str(y) for x,y in cookies.items()])
headers = {
"Accept-Language" : "en-US,en;q=0.8,pt;q=0.6",
"Connection" : "keep-alive",
"Cookies": cookie_string
}
requests.get(url, headers=headers)
import requests
r = requests.get('http://httpbin.org/get');
r.text
returns:
u'{\n "url": "http://httpbin.org/get",\n "headers": {\n "Host": "httpbin.org",\n "Accept-Encoding": "gzip, deflate, compress",\n "Connection": "close",\n "Accept": "*/*",\n "User-Agent": "python-requests/2.2.1 CPython/2.7.5 Windows/7",\n "X-Request-Id": "db302999-d07f-4dd6-8c1e-14db45d39fb0"\n },\n "origin": "61.228.172.190",\n "args": {}\n}'
How can get the origin and headers/Host values?
What's being returned is a JSON string; you need to parse it before you can conveniently use it. Requests can do this for you if you call r.json() instead of r.text.
resp = r.json()
print resp['origin']
print resp['headers']['Host']