Develop Reference

How can I use SSPI to negotiate requests handled by external libraries?

I'll set expectations with the fact that I've been pushed well outside my area of expertise here. I'm behind a corporate firewall, and it's interfering with a lot of external code I use.
For example, I'm trying to use HuggingFace's from_pretrained method. Behind the scenes, this eventually makes a request similar to this:
import requests
requests.get('https://huggingface.co/distilbert-base-uncased-distilled-squad/resolve/main/tokenizer_config.json')
This requests fails with an error from my proxy telling me my credentials are missing, but it can be fixed with the following using this excellent library:
import requests
from requests_negotiate_sspi import HttpNegotiateAuth
s = requests.Session()
s.auth = HttpNegotiateAuth()
s.get('https://huggingface.co/distilbert-base-uncased-distilled-squad/resolve/main/tokenizer_config.json', verify='/path/to/cert.pem')
Unfortunately though, that request is made behind the scenes in the HuggingFace library. I can set env variables to save the path to the cert, but I can't use the SSPI negotiation unless I control that code directly (so far as I can tell). Is there any way around this problem?

Google Appengine: Requests Alternative

I have a non-GAE application/request-handler that uses the Python requests module in a to post an uploaded imaged via a POST request, as binary:
headers = {"MyAuth" : "xyz"}
r = requests.post(base_uri, data=open('0.jpg')), headers=headers)
The user uploads an image, the uploaded image is saved locally, opened for reading, then sent to a remote classifier pipeline via post request - this returns some JSON regarding the image features, which can then be returned to the user.
I need to implement this behaviour in a GAE app, but know that GAE has no traditional file system, so I will have to use StringIO:
data = ... #some jpg => str
headers = {"MyAuth" : "xyz"}
r = requests.post(base_uri, data=StringIO.StringIO(data), headers=headers)
How could I completely replace the requests module in this example in a GAE friendly way?
Many thanks.

Commonly used module for making HTTP requests on app engine is urlfetch, it is available in the default runtime via google.appengine.api.urlfetch. Supposedly urllib2 and/or urllib3 are also options, but I have not used those myself so I can't say for sure.
You can also install requests in your app engine directory and upload it with the project, but I find that a bit of a hassle, since requests has its own dependencies that you will need to include as well.
Also see Using the Requests python library in Google App Engine

Although probably not the best solution to this problem, I managed to get requests 2.3.0 to work in the GAE project with:
pip install --target myproject/externals/ requests==2.3.0
I can now use requests as I would normally.

Pyramid subrequests

I need to call GET, POST, PUT, etc. requests to another URI because of search, but I cannot find a way to do that internally with pyramid. Is there any way to do it at the moment?

Simply use the existing python libraries for calling other webservers.
On python 2.x, use urllib2, for python 3.x, use urllib.request instead. Alternatively, you could install requests.
Do note that calling external sites from your server while serving a request yourself could mean your visitors end up waiting for a 3rd-party web server that stopped responding. Make sure you set decent time outs.

pyramid uses webob which has a client api as of version 1.2
from webob import Request
r = Request.blank("http://google.com")
response = r.send()
generally anything you want to override for the request you would just pass in as a parameter.
from webob import Request
r = Request.blank("http://facebook.com",method="DELETE")
another handy feature is that you can see the request as the http that is passed over the wire
print r
DELETE HTTP/1.0
Host: facebook.com:80
docs

Also check the response status code: response.status_int
I use it for example, to introspect my internal URIs and see whether or not a given relative URI is really served by the framework (example to generate breadcrumbs and make intermediate paths as links only if there are pages behind)

Python and curl question

I will be transmitting purchase info (like CC) to a bank gateway and retrieve the result by using Django thus via Python.
What would be the efficient and secure way of doing this?
I have read a documentation of this gateway for php, they seem to use this method:
$xml= Some xml holding data of a purchase.
$curl = `/usr/bin/curl -s -d 'DATA=$xml' "https://url of the virtual bank POS"`;
$data=explode("\n",$curl); //return value is also an xml, seems like they are splitting by each `\n`
and using the $data, they process if the payment is accepted, rejected etc..
I want to achieve this under python language, for this I have done some searching and seems like there is a python curl application named pycurl yet I have no experience using curl and do not know if this is library is suitable for this task. Please keep in mind that as this transfer requires security, I will be using SSL.
Any suggestion will be appreciated.

Use of the standard library urllib2 module should be enough:
import urllib
import urllib2
request_data = urllib.urlencode({"DATA": xml})
response = urllib2.urlopen("https://url of the virtual bank POS", request_data)
response_data = response.read()
data = response_data.split('\n')
I assume that xml variable holds data to be sent.

Citing pycurl.sourceforge.net:
To sum up, PycURL is very fast (esp. for multiple concurrent operations) and very feature complete, but has a somewhat complex interface. If you need something simpler or prefer a pure Python module you might want to check out urllib2 and urlgrabber. There is also a good comparison of the various libraries.
Both curl and urllib2 can work with https so it's up to you.

What are the differences between the urllib, urllib2, urllib3 and requests module?

In Python, what are the differences between the urllib, urllib2, urllib3 and requests modules? Why are there three? They seem to do the same thing...

I know it's been said already, but I'd highly recommend the requests Python package.
If you've used languages other than python, you're probably thinking urllib and urllib2 are easy to use, not much code, and highly capable, that's how I used to think. But the requests package is so unbelievably useful and short that everyone should be using it.
First, it supports a fully restful API, and is as easy as:
import requests
resp = requests.get('http://www.mywebsite.com/user')
resp = requests.post('http://www.mywebsite.com/user')
resp = requests.put('http://www.mywebsite.com/user/put')
resp = requests.delete('http://www.mywebsite.com/user/delete')
Regardless of whether GET / POST, you never have to encode parameters again, it simply takes a dictionary as an argument and is good to go:
userdata = {"firstname": "John", "lastname": "Doe", "password": "jdoe123"}
resp = requests.post('http://www.mywebsite.com/user', data=userdata)
Plus it even has a built in JSON decoder (again, I know json.loads() isn't a lot more to write, but this sure is convenient):
resp.json()
Or if your response data is just text, use:
resp.text
This is just the tip of the iceberg. This is the list of features from the requests site:
International Domains and URLs
Keep-Alive & Connection Pooling
Sessions with Cookie Persistence
Browser-style SSL Verification
Basic/Digest Authentication
Elegant Key/Value Cookies
Automatic Decompression
Unicode Response Bodies
Multipart File Uploads
Connection Timeouts
.netrc support
List item
Python 2.7, 3.6—3.9
Thread-safe.

urllib2 provides some extra functionality, namely the urlopen() function can allow you to specify headers (normally you'd have had to use httplib in the past, which is far more verbose.) More importantly though, urllib2 provides the Request class, which allows for a more declarative approach to doing a request:
r = Request(url='http://www.mysite.com')
r.add_header('User-Agent', 'awesome fetcher')
r.add_data(urllib.urlencode({'foo': 'bar'})
response = urlopen(r)
Note that urlencode() is only in urllib, not urllib2.
There are also handlers for implementing more advanced URL support in urllib2. The short answer is, unless you're working with legacy code, you probably want to use the URL opener from urllib2, but you still need to import into urllib for some of the utility functions.
Bonus answer
With Google App Engine, you can use any of httplib, urllib or urllib2, but all of them are just wrappers for Google's URL Fetch API. That is, you are still subject to the same limitations such as ports, protocols, and the length of the response allowed. You can use the core of the libraries as you would expect for retrieving HTTP URLs, though.

In the Python 2 standard library there were two HTTP libraries that existed side-by-side. Despite the similar name, they were unrelated: they had a different design and a different implementation.
urllib was the original Python HTTP client, added to the standard library in Python 1.2. Earlier documentation for urllib can be found in Python 1.4.
urllib2 was a more capable HTTP client, added in Python 1.6, intended as a replacement for urllib:
urllib2 - new and improved but incompatible version of urllib (still experimental).
Earlier documentation for urllib2 can be found in Python 2.1.
The Python 3 standard library has a new urllib which is a merged/refactored/rewritten version of the older modules.
urllib3 is a third-party package (i.e., not in CPython's standard library). Despite the name, it is unrelated to the standard library packages, and there is no intention to include it in the standard library in the future.
Finally, requests internally uses urllib3, but it aims for an easier-to-use API.

urllib and urllib2 are both Python modules that do URL request related stuff but offer different functionalities.
1) urllib2 can accept a Request object to set the headers for a URL request, urllib accepts only a URL.
2) urllib provides the urlencode method which is used for the generation of GET query strings, urllib2 doesn't have such a function. This is one of the reasons why urllib is often used along with urllib2.
Requests - Requests’ is a simple, easy-to-use HTTP library written in Python.
1) Python Requests encodes the parameters automatically so you just pass them as simple arguments, unlike in the case of urllib, where you need to use the method urllib.encode() to encode the parameters before passing them.
2) It automatically decoded the response into Unicode.
3) Requests also has far more convenient error handling.If your authentication failed, urllib2 would raise a urllib2.URLError, while Requests would return a normal response object, as expected. All you have to see if the request was successful by boolean response.ok

Just to add to the existing answers, I don't see anyone mentioning that python requests is not a native library. If you are ok with adding dependencies, then requests is fine. However, if you are trying to avoid adding dependencies, urllib is a native python library that is already available to you.

One considerable difference is about porting Python2 to Python3. urllib2 does not exist for python3 and its methods ported to urllib.
So you are using that heavily and want to migrate to Python3 in future, consider using urllib.
However 2to3 tool will automatically do most of the work for you.

I think all answers are pretty good. But fewer details about urllib3.urllib3 is a very powerful HTTP client for python.
For installing both of the following commands will work,
urllib3
using pip,
pip install urllib3
or you can get the latest code from Github and install them using,
$ git clone git://github.com/urllib3/urllib3.git
$ cd urllib3
$ python setup.py install
Then you are ready to go,
Just import urllib3 using,
import urllib3
In here, Instead of creating a connection directly, You’ll need a PoolManager instance to make requests. This handles connection pooling and thread-safety for you. There is also a ProxyManager object for routing requests through an HTTP/HTTPS proxy
Here you can refer to the documentation.
example usage :
>>> from urllib3 import PoolManager
>>> manager = PoolManager(10)
>>> r = manager.request('GET', 'http://google.com/')
>>> r.headers['server']
'gws'
>>> r = manager.request('GET', 'http://yahoo.com/')
>>> r.headers['server']
'YTS/1.20.0'
>>> r = manager.request('POST', 'http://google.com/mail')
>>> r = manager.request('HEAD', 'http://google.com/calendar')
>>> len(manager.pools)
2
>>> conn = manager.connection_from_host('google.com')
>>> conn.num_requests
3
As mentioned in urrlib3 documentations,urllib3 brings many critical features that are missing from the Python standard libraries.
Thread safety.
Connection pooling.
Client-side SSL/TLS verification.
File uploads with multipart encoding.
Helpers for retrying requests and dealing with HTTP redirects.
Support for gzip and deflate encoding.
Proxy support for HTTP and SOCKS.
100% test coverage.
Follow the user guide for more details.
Response content (The HTTPResponse object provides status, data,
and header attributes)
Using io Wrappers with Response content
Creating a query parameter
Advanced usage of urllib3
requests
requests uses urllib3 under the hood and make it even simpler to make requests and retrieve data.
For one thing, keep-alive is 100% automatic, compared to urllib3 where it's not. It also has event hooks which call a callback function when an event is triggered, like receiving a response
In requests, each request type has its own function. So instead of creating a connection or a pool, you directly GET a URL.
For install requests using pip just run
pip install requests
or you can just install from source code,
$ git clone git://github.com/psf/requests.git
$ cd requests
$ python setup.py install
Then, import requests
Here you can refer the official documentation,
For some advanced usage like session object, SSL verification, and Event Hooks please refer to this url.

I like the urllib.urlencode function, and it doesn't appear to exist in urllib2.
>>> urllib.urlencode({'abc':'d f', 'def': '-!2'})
'abc=d+f&def=-%212'

To get the content of a url:
try: # Try importing requests first.
import requests
except ImportError:
try: # Try importing Python3 urllib
import urllib.request
except AttributeError: # Now importing Python2 urllib
import urllib
def get_content(url):
try: # Using requests.
return requests.get(url).content # Returns requests.models.Response.
except NameError:
try: # Using Python3 urllib.
with urllib.request.urlopen(index_url) as response:
return response.read() # Returns http.client.HTTPResponse.
except AttributeError: # Using Python3 urllib.
return urllib.urlopen(url).read() # Returns an instance.
It's hard to write Python2 and Python3 and request dependencies code for the responses because they urlopen() functions and requests.get() function return different types:
Python2 urllib.request.urlopen() returns a http.client.HTTPResponse
Python3 urllib.urlopen(url) returns an instance
Request request.get(url) returns a requests.models.Response

You should generally use urllib2, since this makes things a bit easier at times by accepting Request objects and will also raise a URLException on protocol errors. With Google App Engine though, you can't use either. You have to use the URL Fetch API that Google provides in its sandboxed Python environment.

A key point that I find missing in the above answers is that urllib returns an object of type <class http.client.HTTPResponse> whereas requests returns <class 'requests.models.Response'>.
Due to this, read() method can be used with urllib but not with requests.
P.S. : requests is already rich with so many methods that it hardly needs one more as read() ;>