Develop Reference

python SOAP with zeep, requests.exception: 403 client error

I'm trying to make a SOAP client in python using zeepSo far I've had zero luck in trying to get it to run. I keep getting the following error.
Traceback (most recent call last):
File "C:/Users/z905/PycharmProjects/Soap_Test/soap_test.py", line 14, in <module>
client = Client(wsdl=client_location, transport=transport_with_basic_auth)
File "C:\Users\z905\PycharmProjects\Soap_Test\venv\lib\site-packages\zeep\client.py", line 134, in __init__
self.wsdl = Document(wsdl, self.transport, strict=strict)
File "C:\Users\z905\PycharmProjects\Soap_Test\venv\lib\site-packages\zeep\wsdl\wsdl.py", line 81, in __init__
root_definitions = Definition(self, document, self.location)
File "C:\Users\z905\PycharmProjects\Soap_Test\venv\lib\site-packages\zeep\wsdl\wsdl.py", line 178, in __init__
self.parse_imports(doc)
File "C:\Users\z905\PycharmProjects\Soap_Test\venv\lib\site-packages\zeep\wsdl\wsdl.py", line 270, in parse_imports
document = self.wsdl._get_xml_document(location)
File "C:\Users\z905\PycharmProjects\Soap_Test\venv\lib\site-packages\zeep\wsdl\wsdl.py", line 140, in _get_xml_document
location, self.transport, self.location, strict=self.strict)
File "C:\Users\z905\PycharmProjects\Soap_Test\venv\lib\site-packages\zeep\loader.py", line 72, in load_external
content = transport.load(url)
File "C:\Users\z905\PycharmProjects\Soap_Test\venv\lib\site-packages\zeep\transports.py", line 110, in load
content = self._load_remote_data(url)
File "C:\Users\z905\PycharmProjects\Soap_Test\venv\lib\site-packages\zeep\transports.py", line 126, in _load_remote_data
response.raise_for_status()
File "C:\Users\z905\PycharmProjects\Soap_Test\venv\lib\site-packages\requests\models.py", line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden (Forbidden port) for url: http://url:7104
I've tried the program with and without basic authentication.
from zeep import Client
from requests import Session
from requests.auth import HTTPBasicAuth
from zeep.transports import Transport
client_location = "file://C:/<path-to-local-file>"
loc = "<url-to-wsdl>"
session = Session()
session.auth = HTTPBasicAuth('usrname', 'pass')
transport_with_basic_auth = Transport(session=session)
client = Client(wsdl=client_location, transport=transport_with_basic_auth)
And version without Basic Auth.
from zeep import Client
from zeep.wsse.username import UsernameToken
client_location = "file://C:/<path-to-local-file>"
loc = "<url-to-wsdl>"
client = Client(wsdl=loc, wsse=UsernameToken(usr, password, use_digest=True))

I was able to solve the issue by doing the following. The issue was caused by some firewall issues and the proxy not working.
import requests
session = requests.session()
session.trust_env = False
transport = Transport(session=session)
client = Client(wsdl=client_location, wsse=UsernameToken(usr, password, use_digest=True), transport=transport)

for range to send post request by using requests.Session(), it alert 'module' object has no attribute 'kqueue'

macOS 10.12.3 python 2.7.13 requests 2.13.0
I use requests package to send post request.This request need to login before post data.So I use request.Session() and load a logined cookie.
Then I use this session to send post data in cycle mode.
It is no error that I used to run this code in Windows and Linux.
Simple Code:
s = request.Session()
s.cookies = cookieslib.LWPCookieJar('cookise')
s.cookies.load(ignore_discard=True)
for user_id in range(100,200):
url = 'http://xxxx'
data = { 'user': user_id, 'content': '123'}
r = s.post(url, data)
...
But the program frequently (about every interval) crash, the error isAttributeError: 'module' object has no attribute 'kqueue'
Traceback (most recent call last):
File "/Users/gasxia/Dev/Projects/TgbookSpider/kfz_send_msg.py", line 90, in send_msg
r = requests.post(url, data) # catch error if user isn't exist
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 535, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 423, in send
timeout=timeout
File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 588, in urlopen
conn = self._get_conn(timeout=pool_timeout)
File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 241, in _get_conn
if conn and is_connection_dropped(conn):
File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/util/connection.py", line 27, in is_connection_dropped
return bool(wait_for_read(sock, timeout=0.0))
File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/util/wait.py", line 33, in wait_for_read
return _wait_for_io_events(socks, EVENT_READ, timeout)
File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/util/wait.py", line 22, in _wait_for_io_events
with DefaultSelector() as selector:
File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/util/selectors.py", line 431, in __init__
self._kqueue = select.kqueue()
AttributeError: 'module' object has no attribute 'kqueue'

This looks like a problem that commonly arises if you're using something like eventlet or gevent, both of which monkeypatch the select module. If you're using those to achieve asynchrony, you will need to ensure that those monkeypatches are applied before importing requests. This is a known bug, being tracked in this issue.

Python: Requests patch method doesn't work

I have the code below which works fine and brings back what I need
import requests
from requests.auth import HTTPBasicAuth
response = requests.get('https://example/answers/331', auth=HTTPBasicAuth('username', 'password'),json={"solution": "12345"})
print response.content
However when I change it to a patch method, which is accepted by the server, I get the following errors. Any idea on why?
Traceback (most recent call last):
File "auth.py", line 8, in <module>
response = requests.patch('https://example/answers/331', auth=HTTPBasicAuth('username', 'password'),json={"solution": "12345"})
File "C:\Python27\lib\site-packages\requests-2.12.0-py2.7.egg\requests\api.py", line 138, in patch
return request('patch', url, data=data, **kwargs)
File "C:\Python27\lib\site-packages\requests-2.12.0-py2.7.egg\requests\api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Python27\lib\site-packages\requests-2.12.0-py2.7.egg\requests\sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python27\lib\site-packages\requests-2.12.0-py2.7.egg\requests\sessions.py", line 609, in send
r = adapter.send(request, **kwargs)
File "C:\Python27\lib\site-packages\requests-2.12.0-py2.7.egg\requests\adapters.py", line 473, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",))
Thanks

Try using a POST request with the following header: X-HTTP-Method-Override: PATCH
This is unique to the Oracle Service Cloud REST API implementation and is documented.

In cases where the browser or client application does not support PATCH requests, or network intermediaries block PATCH requests, HTTP tunneling can be used with a POST request by supplying an X-HTTP-Method-Override header.
Example:
import requests
restURL = <Your REST URL>
params = {'field': 'val'}
headers = {'X-HTTP-Method-Override':'PATCH'}
try:
resp = requests.post(restURL, json=params, auth=('<uname>', '<pwd>'), headers=headers)
print resp
except requests.exceptions.RequestException as err:
errMsg = "Error: %s" % err
print errMsg

Ignore certificate validation with urllib3

I'm using urllib3 against private services that have self signed certificates. Is there any way to have urllib3 ignore the certificate errors and make the request anyways?
import urllib3
c = urllib3.HTTPSConnectionPool('10.0.3.168', port=9001)
c.request('GET', '/')
When using the following:
import urllib3
c = urllib3.HTTPSConnectionPool('10.0.3.168', port=9001, cert_reqs='CERT_NONE')
c.request('GET', '/')
The following error is raised:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3/dist-packages/urllib3/request.py", line 67, in request
**urlopen_kw)
File "/usr/lib/python3/dist-packages/urllib3/request.py", line 80, in request_encode_url
return self.urlopen(method, url, **urlopen_kw)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 415, in urlopen
body=body, headers=headers)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 267, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.3/http/client.py", line 1061, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python3.3/http/client.py", line 1099, in _send_request
self.endheaders(body)
File "/usr/lib/python3.3/http/client.py", line 1057, in endheaders
self._send_output(message_body)
File "/usr/lib/python3.3/http/client.py", line 902, in _send_output
self.send(msg)
File "/usr/lib/python3.3/http/client.py", line 840, in send
self.connect()
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 103, in connect
match_hostname(self.sock.getpeercert(), self.host)
File "/usr/lib/python3/dist-packages/urllib3/packages/ssl_match_hostname/__init__.py", line 32, in match_hostname
raise ValueError("empty or no certificate")
ValueError: empty or no certificate
Using cURL I'm able to get the expected response from the service
$ curl -k https://10.0.3.168:9001/
Please read the documentation for API endpoints

Try following code:
import urllib3
c = urllib3.HTTPSConnectionPool('10.0.3.168', port=9001, cert_reqs='CERT_NONE',
assert_hostname=False)
c.request('GET', '/')
See Setting assert_hostname to False will disable SSL hostname verification

In this question I see many answers but, IMHO, too much unnecessary information that can lead to confusion.
Just add the cert_reqs='CERT_NONE' parameter
import urllib3
http = urllib3.PoolManager(cert_reqs='CERT_NONE')

I found the answer to my problem. The urllib3 documentation does not, in fact, completely explain how to suppress SSL certificate validation. What is missing is a reference to ssl.CERT_NONE.
My code has a boolean, ssl_verify, to indicate whether or not I want SSL validation. The code now looks like this:
import ssl
import urllib3
#
#
#
if (ssl_verify):
cert_reqs = ssl.CERT_REQUIRED
else:
cert_reqs = ssl.CERT_NONE
urllib3.disable_warnings()
http = urllib3.PoolManager(cert_reqs = cert_reqs)
auth_url = f'https://{fmc_ip}/api/fmc_platform/v1/auth/generatetoken'
type = {'Content-Type': 'application/json'}
auth = urllib3.make_headers(basic_auth=f'{username}:{password}')
headers = { **type, **auth }
resp = http.request('POST',
auth_url,
headers=headers,
timeout=10.0)

Try to instanciate your connection pool this way:
HTTPSConnectionPool(self.host, self.port, cert_reqs=ssl.CERT_NONE)
or this way:
HTTPSConnectionPool(self.host, self.port, cert_reqs='CERT_NONE')
Source: https://github.com/shazow/urllib3/blob/master/test/with_dummyserver/test_https.py
EDIT (after seeing your edit):
It looks like the remote host didn't send a certificate (is it possible?).
This is the code (from urllib3) which raised an exception:
def match_hostname(cert, hostname):
"""Verify that *cert* (in decoded format as returned by
SSLSocket.getpeercert()) matches the *hostname*. RFC 2818 rules
are mostly followed, but IP addresses are not accepted for *hostname*.
CertificateError is raised on failure. On success, the function
returns nothing.
"""
if not cert:
raise ValueError("empty or no certificate")
So it looks like cert is empty, which means that self.sock.getpeercert() returned an empty string.

Intermittent DownloadError Application Error 2 on Google App Engine

We have two applications that are both running on Google App Engine. App1 makes requests to app2 as an authenticated user. The authentication works by requesting an authentication token from Google ClientLogin that is exchanged for a cookie. The cookie is then used for subsequent requests (as described here). App1 runs the following code:
class AuthConnection:
def __init__(self):
self.cookie_jar = cookielib.CookieJar()
self.opener = urllib2.OpenerDirector()
self.opener.add_handler(urllib2.ProxyHandler())
self.opener.add_handler(urllib2.UnknownHandler())
self.opener.add_handler(urllib2.HTTPHandler())
self.opener.add_handler(urllib2.HTTPRedirectHandler())
self.opener.add_handler(urllib2.HTTPDefaultErrorHandler())
self.opener.add_handler(urllib2.HTTPSHandler())
self.opener.add_handler(urllib2.HTTPErrorProcessor())
self.opener.add_handler(urllib2.HTTPCookieProcessor(self.cookie_jar))
self.headers = {'User-Agent': 'Mozilla/5.0 (Windows; U; ' +\
'Windows NT 6.1; en-US; rv:1.9.1.2) ' +\
'Gecko/20090729 Firefox/3.5.2 ' +\
'(.NET CLR 3.5.30729)'
}
def fetch(self, url, method, payload=None):
self.__updateJar(url)
request = urllib2.Request(url)
request.get_method = lambda: method
for key, value in self.headers.iteritems():
request.add_header(key, value)
response = self.opener.open(request)
return response.read()
def __updateJar(self, url):
cache = memcache.Client()
cookie = cache.get('auth_cookie')
if cookie:
self.cookie_jar.set_cookie(cookie)
else:
cookie = self.__retrieveCookie(url=url)
cache.set('auth_cookie', cookie, 5000)
def __getCookie(self, url):
auth_url = 'https://www.google.com/accounts/ClientLogin'
auth_data = urllib.urlencode({'Email': USER_NAME,
'Passwd': PASSPHRASE,
'service': 'ah',
'source': 'app1',
'accountType': 'HOSTED_OR_GOOGLE' })
auth_request = urllib2.Request(auth_url, data=auth_data)
auth_response_body = self.opener.open(auth_request).read()
auth_response_dict = dict(x.split('=')
for x in auth_response_body.split('\n') if x)
cookie_args = {}
cookie_args['continue'] = url
cookie_args['auth'] = auth_response_dict['Auth']
cookie_url = 'https://%s/_ah/login?%s' %\
('app2.appspot.com', (urllib.urlencode(cookie_args)))
cookie_request = urllib2.Request(cookie_url)
for key, value in self.headers.iteritems():
cookie_request.add_header(key, value)
try:
self.opener.open(cookie_request)
except:
pass
for cookie in self.cookie_jar:
if cookie.domain == 'app2domain':
return cookie
For 10-30% of the requests a DownloadError is raised:
Error fetching https://app2/Resource
Traceback (most recent call last):
File "/base/data/home/apps/app1/5.344034030246386521/source/main/connection/authenticate.py", line 112, in fetch
response = self.opener.open(request)
File "/base/python_runtime/python_dist/lib/python2.5/urllib2.py", line 381, in open
response = self._open(req, data)
File "/base/python_runtime/python_dist/lib/python2.5/urllib2.py", line 399, in _open
'_open', req)
File "/base/python_runtime/python_dist/lib/python2.5/urllib2.py", line 360, in _call_chain
result = func(*args)
File "/base/python_runtime/python_dist/lib/python2.5/urllib2.py", line 1115, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "/base/python_runtime/python_dist/lib/python2.5/urllib2.py", line 1080, in do_open
r = h.getresponse()
File "/base/python_runtime/python_dist/lib/python2.5/httplib.py", line 197, in getresponse
self._allow_truncated, self._follow_redirects)
File "/base/data/home/apps/app1/5.344034030246386521/source/main/connection/monkeypatch_urlfetch_deadline.py", line 18, in new_fetch
follow_redirects, deadline, *args, **kwargs)
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/urlfetch.py", line 241, in fetch
return rpc.get_result()
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 501, in get_result
return self.__get_result_hook(self)
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/urlfetch.py", line 325, in _get_fetch_result
raise DownloadError(str(err))
DownloadError: ApplicationError: 2
The request logs for app2 (the "server") seem fine, as expected (according to the docs DownloadError is only raised if there was no valid HTTP response).
Why is the exception raised?

see this:
http://bitbucket.org/guilin/gae-rproxy/src/tip/gae_rproxy/niceurllib.py
because of urllib and urllib2 default to handle http 302 code, and automatically redirect to what the server told it. But when redirect it does not contains the cookie which the server told it.
for example:
urllib2 request //server/login
server response 302,
//server/profile , set-cookie :
session-id:xxxx
urllib2 request
//server/profile
server response not login error or
500 error cause there is no
session-id found.
urllib2 throw error
so, there is no chance for you to set cookie.
self.opener.add_handler(urllib2.HTTPRedirectHandler())
I think you should remove this line and add your own HTTPRedirectHandler which neighter throw error nor automatically redirect , just return the http code and headers, so you have the chance to set cookie.