I am trying to send a request with the header: ":hello". However, the leading colon causes the script to not function properly, and emit this traceback:
Traceback (most recent call last):
(first few lines removed for my privacy)
File "C:\Python27\lib\site-packages\requests\api.py", line 109, in post
return request('post', url, data=data, json=json, **kwargs)
File "C:\Python27\lib\site-packages\requests\api.py", line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "C:\Python27\lib\site-packages\requests\adapters.py", line 370, in send
timeout=timeout
File "C:\Python27\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 559, in urlopen
body=body, headers=headers)
File "C:\Python27\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 353, in _make_request
conn.request(method, url, **httplib_request_kw)
File "C:\Python27\lib\httplib.py", line 1057, in request
self._send_request(method, url, body, headers)
File "C:\Python27\lib\httplib.py", line 1096, in _send_request
self.putheader(hdr, value)
File "C:\Python27\lib\httplib.py", line 1030, in putheader
raise ValueError('Invalid header name %r' % (header,))
ValueError: Invalid header name ':hello'
Is there a workaround for this? My script is:
import requests
headers = {'user-agent': 'alsotesting', ':hello': 'test'}
requests.post("my server", headers=headers)
As your error says, :header is not a valid HTTP header name (you cannot start headers with ":" - see documentation). You should change
headers = {'user-agent': 'alsotesting', ':hello': 'test'}
to
headers = {'user-agent': 'alsotesting', 'hello': 'test'}
Edit: HTTP/2 uses pseudo-headers fields, which start with a colon (see documentation). Also, as explained here, you may see some headers starting with a colon in Chrome's Developer Tools, which can happen when Chrome is talking to a web server using SPDY - and also HTTP/2 (which is based on SPDY/2), which correspond to pseudo-headers. As stated in documentation, pseudo-header fields are not HTTP header fields.
In conclusion, header fields starting with a colon are not allowed with standard HTTP protocol, so that is why you are getting the Invalid header name error
Related
I've been having trouble storing JSON to database with non ASCII characters.
Here's what I have so far. I'm using python requests library.
getJSON = requests.get('https://jsonapi.com/jsonfile')
storeDB = requests.post(
'http://databaseAPI/Database/item',
data='{"payload": %s}' % (getJSON.text),
headers={'Content-Type': 'application/json'}
)
But it's failing with this error because my JSON contains non ASCII character. How can I fix that?
Error -
Traceback (most recent call last):
File "addKonvo.py", line 134, in <module>
headers={'Content-Type': 'application/json'}
File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 119, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 387, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1297, in _send_request
body = _encode(body, 'body')
File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 170, in _encode
(name.title(), data[err.start:err.end], name)) from None
UnicodeEncodeError: 'latin-1' codec can't encode character '\u20ac' in position 505: Body ('€') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.```
you need to encode the data in UTF-8 format using getJSON.text.encode('utf8')
headers = {'Accept': 'application/json',}
response = requests.get(self.url, headers=headers, verify=self.ssl)
Getting error when passing False as input from command line. It works fine when verify=False is set in the code without passing it as input.
Error:
response = requests.get(self.api, headers=headers, verify=self.ssl)
File "/usr/lib/python3/dist-packages/requests/api.py", line 72, in get
return request('get', url, params=params, **kwargs)
File "/usr/lib/python3/dist-packages/requests/api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 520, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 630, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 407, in send
self.cert_verify(conn, request.url, verify, cert)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 226, in cert_verify
"invalid path: {0}".format(cert_loc))
OSError: Could not find a suitable TLS CA certificate bundle, invalid path: False
parser.add_argument('--ssl', default=True, action="store_false")
How do you pass None as the value of a parameter in a GET request header with urllib3 without raising a TypeError?
I have tried using requests and it works there, but the package lacks some features of urllib3 regarding proxy management, pooling, and simplicity, which sadly are necessities for this project.
My code currently looks like this:
headers = {}
headers.setdefault("Attribute", None)
r = urllib3.PoolManager().request("GET", url="someurl.xyz", headers=headers)
This raises a TypeError: expected string or bytes-like object, because None seems to be an illegal header value.
Here's the relevant part of the log:
File "/home/user/PycharmProjects/Project/src/main/classes/oof.py", line 54, in set_fingerprint
r = self.session.request('GET', url=experiment_url, headers=headers)
File "/home/user/anaconda3/lib/python3.7/site-packages/urllib3/request.py", line 68, in request
**urlopen_kw)
File "/home/user/anaconda3/lib/python3.7/site-packages/urllib3/request.py", line 89, in request_encode_url
return self.urlopen(method, url, **extra_kw)
File "/home/user/anaconda3/lib/python3.7/site-packages/urllib3/poolmanager.py", line 324, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/home/user/anaconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/home/user/anaconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/home/user/anaconda3/lib/python3.7/http/client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/home/user/anaconda3/lib/python3.7/http/client.py", line 1270, in _send_request
self.putheader(hdr, value)
File "/home/user/anaconda3/lib/python3.7/http/client.py", line 1206, in putheader
if _is_illegal_header_value(values[i]):
TypeError: expected string or bytes-like object
Is there a way to circumvent this error with urllib3?
You should pass empty string as an empty header, not None
I want to set proxy in requests and get content of url source:
privoxy = 'localhost:8118'
proxy_dict = {'http':privoxy, 'https':privoxy, 'ftp':privoxy, 'ssl':privoxy}
content = session.get(url=url, proxies=proxy_dict).content
The following error:
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 467, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 455, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 558, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 305, in send
conn = self.get_connection(request.url, proxies)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 206, in get_connection
except_on_missing_scheme(proxy)
File "/usr/lib/python2.7/dist-packages/requests/utils.py", line 636, in except_on_missing_scheme
raise MissingSchema('Proxy URLs must have explicit schemes.')
requests.exceptions.MissingSchema: Proxy URLs must have explicit schemes.
Set the scheme to http, in your case:
privoxy = 'http://localhost:8118'
(It can be others of course, like 'https', 'ftp', etc.)
More from Requests docs for Proxies:
Note that proxy URLs must include the scheme.
I'm doing pretty simple thing - get file from client and send it to process to another server:
import requests
import logging
logger = logging.getLogger(__name__)
data = request.environ['wsgi.input'].read()
url = request.cfg.DOC_PARSE_URL
params = {'AddedPath': str(request.cfg.FORM_TEMP_URL+uid+'/'), 'Button': 'Generate'}
logger.warning('url:'+repr(url))
for k in params:
logger.warning('params:'+repr(k)+' '+repr(params[k]))
logger.warning('data:'+repr(type(data))+str(len(data)))
#logger.warning('data:'+data)
files = {'WordFile': ('process.doc', data)}
r = requests.post(url, files=files, data=params, stream=False)
and everything works fine on my own PC and on dev server, but on production server this code crashes with UnicodeDecodeError:
Traceback (most recent call last):
File "/sites/mo/admin/utils/dispatcher.py", line 492, in __call__
response = self.view_map[endpoint](request, **params)
File "/sites/mo/admin/utils/admin_views.py", line 231, in parse_doc
r = requests.post(url, files=files, data=params, stream=False)
File "/sites/mo/admin/../third_party/requests/api.py", line 88, in post
return request('post', url, data=data, **kwargs)
File "/sites/mo/admin/../third_party/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/sites/mo/admin/../third_party/requests/sessions.py", line 360, in request
resp = self.send(prep, **send_kwargs)
File "/sites/mo/admin/../third_party/requests/sessions.py", line 463, in send
r = adapter.send(request, **kwargs)
File "/sites/mo/admin/../third_party/requests/adapters.py", line 292, in send
timeout=timeout
File "/sites/mo/admin/../third_party/requests/packages/urllib3/connectionpool.py", line 428, in urlopen
body=body, headers=headers)
File "/sites/mo/admin/../third_party/requests/packages/urllib3/connectionpool.py", line 283, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib64/python2.7/httplib.py", line 946, in request
self._send_request(method, url, body, headers)
File "/usr/lib64/python2.7/httplib.py", line 987, in _send_request
self.endheaders(body)
File "/usr/lib64/python2.7/httplib.py", line 940, in endheaders
self._send_output(message_body)
File "/usr/lib64/python2.7/httplib.py", line 801, in _send_output
msg += message_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 739: ordinal not in range(128)
All values are bytes string (I inserted logging to ensure) so there shouldn't be any unicode issues. Seems there is some environment influence, but i can't figure it out. Does anybody know what could cause such problem?