Develop Reference

can't download a file with python but i can from the browser

I am trying to make a function that will download file from the internet. If I go to the direct web address I do get the images or download the files. But, when I am running my code, it just hangs and then I get the timeout error. Is there any particular reason why that might be happening?
#fpp = "http://www.blog.pythonlibrary.org/wp-content/uploads/2012/06/wxDbViewer.zip"
fpp = "http://www.gunnerkrigg.com//comics/00000001.jpg"
download_file(fpp)
This is my function:
import urllib2
def download_file(url_path):
response = urllib2.urlopen(url_path)
data = response.read()
Is there any particular reason why might work from the browser but not in the code?
this is the error i get:
Traceback (most recent call last):
File "/Users/dk/testing/myfile.py", line 42, in <module>
download_file(fpp)
File "/Users/dk/Documents/testing/code_project.py", line 154, in download_file
response = urllib2.urlopen(url_path)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 431, in open
response = self._open(req, data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 449, in _open
'_open', req)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1227, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1197, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno 60] Operation timed out>

Python broken urllib2 request

I get this error when trying to run my program:
Traceback (most recent call last):
File "C:/Users/Adasli199/Desktop/Minetek-Testy-Stuff/tools/soldering-iron.py", line 400, in <module>
MCVersionRegEx = cacheMCVersions()
File "C:/Users/Adasli199/Desktop/Minetek-Testy-Stuff/tools/soldering-iron.py", line 141, in cacheMCVersions
feeddata = opener.open(request).read()
File "C:\Python27\lib\urllib2.py", line 404, in open
response = self._open(req, data)
File "C:\Python27\lib\urllib2.py", line 422, in _open
'_open', req)
File "C:\Python27\lib\urllib2.py", line 382, in _call_chain
result = func(*args)
File "C:\Python27\lib\urllib2.py", line 1222, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "C:\Python27\lib\urllib2.py", line 1184, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno 1] _ssl.c:510: error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error>
I am trying to get some data data to help with processing my RegEx command in another part of the code, here is the function that it is referencing to which throws the error.
import httplib
import urllib2
def cacheMCVersions():
request = urllib2.Request('https://mcversions.net/')
request.add_header('User-Agent', 'SolderingIron/1.0 +http://tetrarch.co/') # identify ourselves so we don't get blocked
opener = urllib2.build_opener()
feeddata = opener.open(request).read()
There is some more regex code after this to interpret the data but I feel this is all that is needed to resolve the issues. At the time of running, https://mcversions.net/ was up and still is as far as I am aware, which is what makes this error even more strange.

urllib is broken on python 2.7.9 and lower for https requests.
Install python 2.7.10 inside a virtual enviornment
and it should work. You may need to install libssl-dev or openssl depending on your linux platform.

Different behaviour between double quoted and single quoted string when using urllib in python

I am new to python and I learned there are no different between single and double quoted string.
But I have found some different behavior.
from bs4 import BeautifulSoup
import urllib.request
url1 = "http://www.backpackers.com.tw/forum/forumdisplay.php?f=310"
url2 = 'http://www.backpackers.com.tw/forum/forumdisplay.php?f=310'
If I run:
response = urllib.request.urlopen(url1)
Result: Script finished without error
And If I run:
response = urllib.request.urlopen(url2)
Result: error
C:\Users\user1\Desktop\scrape>python backpacker_tw.py
Traceback (most recent call last):
File "C:\Python34\lib\urllib\request.py", line 1189, in do_open
h.request(req.get_method(), req.selector, req.data, headers)
File "C:\Python34\lib\http\client.py", line 1090, in request
self._send_request(method, url, body, headers)
File "C:\Python34\lib\http\client.py", line 1128, in _send_request
self.endheaders(body)
File "C:\Python34\lib\http\client.py", line 1086, in endheaders
self._send_output(message_body)
File "C:\Python34\lib\http\client.py", line 924, in _send_output
self.send(msg)
File "C:\Python34\lib\http\client.py", line 859, in send
self.connect()
File "C:\Python34\lib\http\client.py", line 836, in connect
self.timeout, self.source_address)
File "C:\Python34\lib\socket.py", line 509, in create_connection
raise err
File "C:\Python34\lib\socket.py", line 500, in create_connection
sock.connect(sa)
ConnectionRefusedError: [WinError 10061] No connection could be made because the
target machine actively refused it
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "backpacker_tw.py", line 7, in <module>
response = urllib.request.urlopen(url2)
File "C:\Python34\lib\urllib\request.py", line 153, in urlopen
return opener.open(url, data, timeout)
File "C:\Python34\lib\urllib\request.py", line 455, in open
response = self._open(req, data)
File "C:\Python34\lib\urllib\request.py", line 473, in _open
'_open', req)
File "C:\Python34\lib\urllib\request.py", line 433, in _call_chain
result = func(*args)
File "C:\Python34\lib\urllib\request.py", line 1215, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "C:\Python34\lib\urllib\request.py", line 1192, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [WinError 10061] No connection could be ma
de because the target machine actively refused it>
Is it a bug or anything i missed?
C:\Users\user1\Desktop\scrape>python -V
Python 3.4.1

To the documentation! PEP 8, pretty much the be all end all on python code formatting, states "In Python, single-quoted strings and double-quoted strings are the same." This is written by the creator of Python, and I'm going to take his word for it.
Looking at your stack trace I see the error No connection could be ma
de because the target machine actively refused it, so maybe that means something wonky was going on with the server at that point?

How to crawl Twitter pages using Python?

When I try to crawl Twitter using this code:
import urllib2
s = "https://mobile.twitter.com/bing/"
html = urllib2.urlopen(s).read()
print html
... I get the following error:
Traceback (most recent call last):
File "C:\Users\arpit\Downloads\Desktop\Wiki Code\final Crawler_wiki.py", line 14, in <module>
html = urllib2.urlopen(s).read()
File "C:\Python27\lib\urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "C:\Python27\lib\urllib2.py", line 400, in open
response = self._open(req, data)
File "C:\Python27\lib\urllib2.py", line 418, in _open
'_open', req)
File "C:\Python27\lib\urllib2.py", line 378, in _call_chain
result = func(*args)
File "C:\Python27\lib\urllib2.py", line 1215, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "C:\Python27\lib\urllib2.py", line 1177, in do_open
raise URLError(err)
URLError: <urlopen error [Errno 10061] No connection could be made because the target machine actively refused it>
If I replace mobile.twitter.com with twitter.com then it works, but I want it to work with mobile.twitter.com.

The twitter site is probably looking for a user-agent which you dont have set when you make the request through the urllib api.
You will likely need to use something like mechanize to fake your user-agent.
But I highly suggest your use the twitter api which provide a lot of easy and awesome way to play with data.

Possible message size limitation when using SUDS to consume an RPClib service?

I have been using SUDS and RPCLib to develop a SOAP interface to a software solution that takes a PDF document and returns a PNG, and have found a very interesting problem.
I have written the testing client (using SUDS) and server (using RPCLib), and they work successfully when when the documents to be uploaded and returned are less than about 3.5Mb. However, when uploading larger documents I get the SUDS Error:
Traceback (most recent call last):
File "MyFunc.py", line 90, in <module>
callMyFuncSOAPService(fName, test_id, fNameOut)
File "MyFuncClient.py", line 77, in callMyFuncSOAPService
temp_list = client.service.createInstance(encoded_data, 19, test_id, 20)
File "/usr/local/lib/python2.7/dist-packages/suds/client.py", line 542, in __call__
return client.invoke(args, kwargs)
File "/usr/local/lib/python2.7/dist-packages/suds/client.py", line 602, in invoke
result = self.send(soapenv)
File "/usr/local/lib/python2.7/dist-packages/suds/client.py", line 637, in send
reply = transport.send(request)
File "/usr/local/lib/python2.7/dist-packages/suds/transport/https.py", line 64, in send
return HttpTransport.send(self, request)
File "/usr/local/lib/python2.7/dist-packages/suds/transport/http.py", line 77, in send
fp = self.u2open(u2request)
File "/usr/local/lib/python2.7/dist-packages/suds/transport/http.py", line 118, in u2open
return url.open(u2request, timeout=tm)
File "/usr/lib/python2.7/urllib2.py", line 400, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 418, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1207, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1177, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno 32] Broken pipe>
And when returning a document, the Server finishes processing and returns the document, but the Client hangs.
I have a feeling that this is due to a limit in the HTTP transport layer, but have no idea how to address this. Thanks!

You could just increase your allowed request length.
app = WsgiApplication(...)
app.max_content_length = 10 * 0x100000 # 10 MB
Spyne 2.10 and 2.9.4 will have these parameters in the constructor, so you'll be able to just do this:
WsgiApplication(..., max_content_length=10 * 0x100000)

I found a possible solution. It involved updating to and editing the spyne library [the successor to rpclib].
in WSGI.py, function __wsgi_input_to_iterable(), I commented out the 2 lines that threw the error:
raise RequestTooLongError()
The problem is that we are pulling a limit from length = str(http_env.get('CONTENT_LENGTH', self.max_content_length)) that seems to be incorrect.
I will check with the SPYNE developers to see what this bug could be coming from