I am trying to reproduce a x-http request captured with Charles (Web Debugging Proxy) with Python but I can't find any documentation (or don't know what or where to look for).
I'd use the requests library for this, as it makes tasks like these easier.
The request you captured seems to be posting JSON data, albeit with a text/javascript content type:
import requests
import json
headers = {'Content-Type': 'text/javascript;charset=utf-8')
data = json.dumps({'mod': 'calendar.field', 'action': 'mini', 'vars': {"current": 0}})
r = requests.post('http://www.kavka.be/xhttp.mod', data=data, headers=headers)
where data is a JSON string created from the same information as your proxy-captured POST.
Alternatively, if you only want to use the standard library, use urllib2:
import urllib2
import json
headers = {'Content-Type': 'text/javascript;charset=utf-8')
data = json.dumps({'mod': 'calendar.field', 'action': 'mini', 'vars': {"current": 0}})
req = urllib2.Request('http://www.kavka.be/xhttp.mod', data, headers)
r = urllib2.urlopen(req)
Related
I have call so many api with the help of urllib2 json type. But now i want to crate from-data api with the help of urllib2 and it is not working
I have post api and url and this data
Dummy url = https://www.example.com/xyz?id=32323232
dummy data {'data': "here"}
data should be sent by form-data not raw json type
how can we write code in python with urllib2
url = https://www.example.com/xyz?id=32323232
data = {'data': "here"}
header = {'ContentType' : 'multipart/form-data'}
request = urllib2.Request(url, data, header)
response = urllib2.urlopen(request)
Refer : https://www.pythonforbeginners.com/python-on-the-web/how-to-use-urllib2-in-python/
I am currently working with and API that requires "RFC2045-MIME variant of Base64, except not limited to 76 char/line" this seems to be different from the normal basic auth used in the requests library. Curious if anyone else has come across this and been able to solve it? I imagine I will have to write a function to do this encoding and build the header manually.
Yes, you can create a function as shown below where the headers for the request are created manually.
For this implementation, you will need these variables:
API_PUBLIC_KEY,
API_SECRET_KEY,
host_url,
endpoint (where the API is supposed to hit).
import base64
import requests
def create_request():
auth_header = base64.b64encode(bytes(f'{API_PUBLIC_KEY}:{API_SECRET_KEY}'.encode('ascii'))).decode('utf-8')
headers = {
'Host': host_url,
'Authorization': f'Basic {auth_header}',
'Content-Type': 'application/json'
}
api_endpoint = f'https://{host_url}/{endpoint}'
data = {}
response = requests.request("POST", api_endpoint, headers=headers, data=data)
print(response.text.encode('utf8'))
I am trying to send a simple POST to an api.
import requests
url ="http://someapi/v1/auth"
payload = {'username': '', 'password': ''}
s1 = requests.post(url, headers={"content-type":"application/x-www-form-urlencoded"}, data=json.dumps(payload))
print s1.status_code
I keep getting status code 401.
Same steps Works fine in POSTMAN.
Any Ideas/pointers ?
Post data in raw format.
payload = "username=;password=;"
s1 = requests.post(
url,
headers={"content-type":"application/x-www-form-urlencoded"},
data=payload)
FWIW, you can click on Code below the Save button on the top right corner of Postman to view code in a couple of languages for your request.
It will only works if the API accepts also JSON body.
Otherwise you can use the #Oluwafemi Sule's answer.
import requests
url ="http://someapi/v1/auth"
payload = {'username': '', 'password': ''}
s1 = requests.post(url, headers={"content-type":"application/json"}, data=json.dumps(payload))
print s1.status_code
This code worked for me.
import requests
from requests_ntlm import HttpNtlmAuth
payload = "username=;password=;"
s= requests.post(
"http://someapi/v1/auth",
headers={"content-type":"application/x-www-form-urlencoded"},
data = payload,
auth=HttpNtlmAuth('',''))
print s.status_code
I am trying to do a HTTP POST using cookies in Python.
I have the values of URL, POST data and cookie.
import urllib2
url="http://localhost/testing/posting.php"
data="subject=Alice-subject&addbbcode18=%23444444&addbbcode20=0&helpbox=Close+all+open+bbCode+tags&message=alice-body&poll_title=&add_poll_option_text=&poll_length=&mode=newtopic&sid=5b2e663a3d724cc873053e7ca0f59bd0&f=1&post=Submit"
cookie = "phpbb2mysql_data=a%3A2%3A%7Bs%3A11%3A%22autologinid%22%3Bs%3A0%3A%22%22%3Bs%3A6%3A%22userid%22%3Bs%3A1%3A%223%22%3B%7D; phpbb2mysql_t=a%3A9%3A%7Bi%3A3%3Bi%3A1330156986%3Bi%3A1%3Bi%3A1330160737%3Bi%3A5%3Bi%3A1330161702%3Bi%3A6%3Bi%3A1330179284%3Bi%3A2%3Bi%3A1330160743%3Bi%3A7%3Bi%3A1330163187%3Bi%3A8%3Bi%3A1330164442%3Bi%3A9%3Bi%3A1330164739%3Bi%3A10%3Bi%3A1330176335%3B%7D; phpbb2mysql_sid=5b2e663a3d724cc873053e7ca0f59bd0"
#creating HTTP Req
req = urllib2.Request(url,data,cookie)
f = urllib2.urlopen(req)
print f.read()
However, if I try to run the program, it is throwing an error:
Traceback (most recent call last):
File "task-4.py", line 7, in <module>
req = urllib2.Request(url,data,cookie)
File "/usr/lib/python2.6/urllib2.py", line 197, in __init__
for key, value in headers.items():
AttributeError: 'str' object has no attribute 'items'
I have two questions:
1. Is my HTTP POST request proper? (I have properly been able to execute the same thing in Java and got a HTTP 200 with a successful post to phpBB, however, I am new to Python)
2. Can someone show me a toy example of handling HTTP POST with POST data and cookies?
Thanks in advance,
Roy
You can try requests, which makes life easier when dealing with HTTP queries.
import requests
url="http://localhost/testing/posting.php"
data= {
'subject': 'Alice-subject',
'addbbcode18': '%23444444',
'addbbcode20': '0',
'helpbox': 'Close all open bbCode tags',
'message': 'alice-body',
'poll_title': '',
'add_poll_option_text': '',
'poll_length': '',
'mode': 'newtopic',
'sid': '5b2e663a3d724cc873053e7ca0f59bd0',
'f': '1',
'post': 'Submit',
}
cookies = {'phpbb2mysql_data': 'a%3A2%3A%7Bs%3A11%3A%22autologinid%22%3Bs%3A0%3A%22%22%3Bs%3A6%3A%22userid%22%3Bs%3A1%3A%223%22%3B%7D',
'phpbb2mysql_t': 'a%3A9%3A%7Bi%3A3%3Bi%3A1330156986%3Bi%3A1%3Bi%3A1330160737%3Bi%3A5%3Bi%3A1330161702%3Bi%3A6%3Bi%3A1330179284%3Bi%3A2%3Bi%3A1330160743%3Bi%3A7%3Bi%3A1330163187%3Bi%3A8%3Bi%3A1330164442%3Bi%3A9%3Bi%3A1330164739%3Bi%3A10%3Bi%3A1330176335%3B%7D',
'phpbb2mysql_sid': '5b2e663a3d724cc873053e7ca0f59bd0',
}
print requests.get(url, data=data, cookies=cookies).text
http://python-requests.org/
the 3rd argument you pass is a header and should be a dictionary. This should do it
cookie = {"Cookie" : "phpbb2mysql_data=a%3A2%3A%7Bs%3A11%3A%22autologinid%22%3Bs%3A0%3A%22%22%3Bs%3A6%3A%22userid%22%3Bs%3A1%3A%223%22%3B%7D; phpbb2mysql_t=a%3A9%3A%7Bi%3A3%3Bi%3A1330156986%3Bi%3A1%3Bi%3A1330160737%3Bi%3A5%3Bi%3A1330161702%3Bi%3A6%3Bi%3A1330179284%3Bi%3A2%3Bi%3A1330160743%3Bi%3A7%3Bi%3A1330163187%3Bi%3A8%3Bi%3A1330164442%3Bi%3A9%3Bi%3A1330164739%3Bi%3A10%3Bi%3A1330176335%3B%7D; phpbb2mysql_sid=5b2e663a3d724cc873053e7ca0f59bd0"}
I like httplib:
from urlparse import urlparse
from httplib import HTTPConnection
url = "http://localhost/testing/posting.php"
data = "subject=Alice-subject&addbbcode18=%23444444&addbbcode20=0&helpbox=Close+all+open+bbCode+tags&message=alice-body&poll_title=&add_poll_option_text=&poll_length=&mode=newtopic&sid=5b2e663a3d724cc873053e7ca0f59bd0&f=1&post=Submit"
cookie = "phpbb2mysql_data=a%3A2%3A%7Bs%3A11%3A%22autologinid%22%3Bs%3A0%3A%22%22%3Bs%3A6%3A%22userid%22%3Bs%3A1%3A%223%22%3B%7D; phpbb2mysql_t=a%3A9%3A%7Bi%3A3%3Bi%3A1330156986%3Bi%3A1%3Bi%3A1330160737%3Bi%3A5%3Bi%3A1330161702%3Bi%3A6%3Bi%3A1330179284%3Bi%3A2%3Bi%3A1330160743%3Bi%3A7%3Bi%3A1330163187%3Bi%3A8%3Bi%3A1330164442%3Bi%3A9%3Bi%3A1330164739%3Bi%3A10%3Bi%3A1330176335%3B%7D; phpbb2mysql_sid=5b2e663a3d724cc873053e7ca0f59bd0"
urlparts = urlparse(url)
conn = HTTPConnection(urlparts.netloc, urlparts.port or 80)
conn.request("POST", urlparts.path, data, {'Cookie': cookie})
resp = conn.getresponse()
body = resp.read()
Not really. That error is because urllib2 library is trying to iterate over the items of the cookie string you gave it. Try using:
cookies = urllib.urlencode({'phpbb2mysql_data':'foo', 'autologinid':'blahblah'})
# Can do the same for data, allowing you to store it as a map.
headers = {'Cookie': cookies}
req = urllib2.Request(url, data, headers)
See python: urllib2 how to send cookie with urlopen request but your best reference is still really the urllib2 Request docs, but yes it's a tricky (but powerful) library compared to some newer ones.
I need to upload some data to a server using HTTP PUT in python. From my brief reading of the urllib2 docs, it only does HTTP POST. Is there any way to do an HTTP PUT in python?
I've used a variety of python HTTP libs in the past, and I've settled on requests as my favourite. Existing libs had pretty useable interfaces, but code can end up being a few lines too long for simple operations. A basic PUT in requests looks like:
payload = {'username': 'bob', 'email': 'bob#bob.com'}
>>> r = requests.put("http://somedomain.org/endpoint", data=payload)
You can then check the response status code with:
r.status_code
or the response with:
r.content
Requests has a lot synactic sugar and shortcuts that'll make your life easier.
import urllib2
opener = urllib2.build_opener(urllib2.HTTPHandler)
request = urllib2.Request('http://example.org', data='your_put_data')
request.add_header('Content-Type', 'your/contenttype')
request.get_method = lambda: 'PUT'
url = opener.open(request)
Httplib seems like a cleaner choice.
import httplib
connection = httplib.HTTPConnection('1.2.3.4:1234')
body_content = 'BODY CONTENT GOES HERE'
connection.request('PUT', '/url/path/to/put/to', body_content)
result = connection.getresponse()
# Now result.status and result.reason contains interesting stuff
You can use the requests library, it simplifies things a lot in comparison to taking the urllib2 approach. First install it from pip:
pip install requests
More on installing requests.
Then setup the put request:
import requests
import json
url = 'https://api.github.com/some/endpoint'
payload = {'some': 'data'}
# Create your header as required
headers = {"content-type": "application/json", "Authorization": "<auth-key>" }
r = requests.put(url, data=json.dumps(payload), headers=headers)
See the quickstart for requests library. I think this is a lot simpler than urllib2 but does require this additional package to be installed and imported.
This was made better in python3 and documented in the stdlib documentation
The urllib.request.Request class gained a method=... parameter in python3.
Some sample usage:
req = urllib.request.Request('https://example.com/', data=b'DATA!', method='PUT')
urllib.request.urlopen(req)
You should have a look at the httplib module. It should let you make whatever sort of HTTP request you want.
I needed to solve this problem too a while back so that I could act as a client for a RESTful API. I settled on httplib2 because it allowed me to send PUT and DELETE in addition to GET and POST. Httplib2 is not part of the standard library but you can easily get it from the cheese shop.
I also recommend httplib2 by Joe Gregario. I use this regularly instead of httplib in the standard lib.
Have you taken a look at put.py? I've used it in the past. You can also just hack up your own request with urllib.
You can of course roll your own with the existing standard libraries at any level from sockets up to tweaking urllib.
http://pycurl.sourceforge.net/
"PyCurl is a Python interface to libcurl."
"libcurl is a free and easy-to-use client-side URL transfer library, ... supports ... HTTP PUT"
"The main drawback with PycURL is that it is a relative thin layer over libcurl without any of those nice Pythonic class hierarchies. This means it has a somewhat steep learning curve unless you are already familiar with libcurl's C API. "
If you want to stay within the standard library, you can subclass urllib2.Request:
import urllib2
class RequestWithMethod(urllib2.Request):
def __init__(self, *args, **kwargs):
self._method = kwargs.pop('method', None)
urllib2.Request.__init__(self, *args, **kwargs)
def get_method(self):
return self._method if self._method else super(RequestWithMethod, self).get_method()
def put_request(url, data):
opener = urllib2.build_opener(urllib2.HTTPHandler)
request = RequestWithMethod(url, method='PUT', data=data)
return opener.open(request)
You can use requests.request
import requests
url = "https://www.example/com/some/url/"
payload="{\"param1\": 1, \"param1\": 2}"
headers = {
'Authorization': '....',
'Content-Type': 'application/json'
}
response = requests.request("PUT", url, headers=headers, data=payload)
print(response.text)
A more proper way of doing this with requests would be:
import requests
payload = {'username': 'bob', 'email': 'bob#bob.com'}
try:
response = requests.put(url="http://somedomain.org/endpoint", data=payload)
response.raise_for_status()
except requests.exceptions.RequestException as e:
print(e)
raise
This raises an exception if there is an error in the HTTP PUT request.
Using urllib3
To do that, you will need to manually encode query parameters in the URL.
>>> import urllib3
>>> http = urllib3.PoolManager()
>>> from urllib.parse import urlencode
>>> encoded_args = urlencode({"name":"Zion","salary":"1123","age":"23"})
>>> url = 'http://dummy.restapiexample.com/api/v1/update/15410' + encoded_args
>>> r = http.request('PUT', url)
>>> import json
>>> json.loads(r.data.decode('utf-8'))
{'status': 'success', 'data': [], 'message': 'Successfully! Record has been updated.'}
Using requests
>>> import requests
>>> r = requests.put('https://httpbin.org/put', data = {'key':'value'})
>>> r.status_code
200