Develop Reference

_cffi_ssl._stdssl.error.SSLEOFError: EOF occurred in violation of protocol while running in PyPy

The question
I created a package for maintaining HTTP requests - just for fun.
It runs perfectly in normal Python, however, when I try to run it in PyPy, the following error occurs while requesting over HTTPS:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/adam/python/httpy/httpy/__init__.py", line 2019, in request
http_version=http_version
File "/home/adam/python/httpy/httpy/__init__.py", line 1863, in _raw_request
proto.send_request(sock,method,defhdr,data,path,debug)
File "/home/adam/python/httpy/httpy/__init__.py", line 1339, in send_request
return self.sender(*args).send(sock)
File "/home/adam/python/httpy/httpy/__init__.py", line 1677, in send
sock.send(self.body)
File "/opt/pypy3/lib/pypy3.8/ssl.py", line 1173, in send
return self._sslobj.write(data)
File "/opt/pypy3/lib/pypy3.8/_cffi_ssl/_stdssl/__init__.py", line 546, in write
return self._write_with_length(_str_to_ffi_buffer(bytestring), len(bytestring))
File "/opt/pypy3/lib/pypy3.8/_cffi_ssl/_stdssl/__init__.py", line 603, in _write_with_length
raise pyssl_error(self, length)
_cffi_ssl._stdssl.error.SSLEOFError: EOF occurred in violation of protocol
Requests, for example, run well even in PyPy.
Could the issue be something with using ssl.wrap_socket() or it's an error on a deeper level?
System info
Python version: 3.10.5
PyPy version: 3.8.13
OS: x86_64 Linux 5.18.10-arch1-1

OK, I found it out. It was because of sending b'' after request if the request body was empty. I thought that sending empty string would be an empty operation, but obviously it is not - it is treated as an EOF. I am still not sure why this fails only using PyPy. It's maybe because it's older version than Python where the code was tested.
Anyway, if you get this error, make sure you are not accidentially sending b''.

Can't send packets (Errno 9: Bad file descriptor)

Hi I'm a beginner with Python and Scapy. When I try to use sendp() on a basic Layer 2 packet, I get a traceback error.
Using Python 3.8 and the latest development version of Scapy from https://scapy.readthedocs.io/en/latest/installation.html#platform-specific-instructions
This is what I'm inputting into the Python shell:
import scapy
from scapy.all import *
a=Ether()/IP(dst="www.google.ca")/ICMP()/"Hello world"
sendp(a)
This is the error message:
Traceback (most recent call last):
File "<pyshell#3>", line 1, in <module>
sendp(a)
File "C:\Users\hoang\AppData\Local\Programs\Python\Python38-32\lib\site-packages\scapy-git_archive.dev304758016-py3.8.egg\scapy\sendrecv.py", line 336, in sendp
results = __gen_send(socket, x, inter=inter, loop=loop,
File "C:\Users\hoang\AppData\Local\Programs\Python\Python38-32\lib\site-packages\scapy-git_archive.dev304758016-py3.8.egg\scapy\sendrecv.py", line 296, in __gen_send
os.write(1, b".")
OSError: [Errno 9] Bad file descriptor

As pointed by #furas, this is indeed an issue with your console (IDLE?).
Scapy tries to display that the packet was sent, which fails.
You can always use
sendp(p, verbose=False)
To disable the logs, therefore working around the issue.
However I must say that if os.write(1, ..) was the only option back in the days, it's a bit outdated nowadays. This could probably be fixed on upstream.

Twisted HTTPS Client

I am currently having some trouble accessing content hosted via https using the twisted python library. I am new to this library, and am assuming there is some concept I am missing that's causing the issue, but perhaps not based upon the example.
Here is a link to the page in which I gathered the example:
https://twistedmatrix.com/documents/current/web/howto/client.html
Under the heading HTTP over SSL
from twisted.python.log import err
from twisted.web.client import Agent
from twisted.internet import reactor
from twisted.internet.ssl import optionsForClientTLS
def display(response):
print("Received response")
print(response)
def main():
contextFactory = optionsForClientTLS(u"https://example.com/")
agent = Agent(reactor, contextFactory)
d = agent.request("GET", "https://example.com/")
d.addCallbacks(display, err)
d.addCallback(lambda ignored: reactor.stop())
reactor.run()
if __name__ == "__main__":
main()
When running this code, it straight up fails. I get an error that looks like this:
Traceback (most recent call last):
File "https.py", line 19, in <module>
main()
File "https.py", line 11, in main
contextFactory = optionsForClientTLS(u"https://example.com/")
File "/home/amaricich/.local/lib/python2.7/site-packages/twisted/internet/_sslverify.py", line 1336, in optionsForClientTLS
return ClientTLSOptions(hostname, certificateOptions.getContext())
File "/home/amaricich/.local/lib/python2.7/site-packages/twisted/internet/_sslverify.py", line 1198, in __init__
self._hostnameBytes = _idnaBytes(hostname)
File "/home/amaricich/.local/lib/python2.7/site-packages/twisted/internet/_sslverify.py", line 86, in _idnaBytes
return idna.encode(text)
File "/usr/local/lib/python2.7/dist-packages/idna/core.py", line 355, in encode
result.append(alabel(label))
File "/usr/local/lib/python2.7/dist-packages/idna/core.py", line 276, in alabel
check_label(label)
File "/usr/local/lib/python2.7/dist-packages/idna/core.py", line 253, in check_label
raise InvalidCodepoint('Codepoint {0} at position {1} of {2} not allowed'.format(_unot(cp_value), pos+1, repr(label)))
idna.core.InvalidCodepoint: Codepoint U+003A at position 6 of u'https://example' not allowed
This error lead me to believe the parameter being passed into optionsForClientTLS were incorrect. It calls for a hostname and not a full url, so I shortened the parameter to simply example.com. Once that change was made, the function completed successfully.
Unfortunately though, after making the change, the script now failed at the line invoking agent.request. The error it supplied was this:
Traceback (most recent call last):
File "https.py", line 19, in <module>
main()
File "https.py", line 13, in main
d = agent.request("GET", "https://example.com/")
File "/home/amaricich/.local/lib/python2.7/site-packages/twisted/web/client.py", line 1596, in request
endpoint = self._getEndpoint(parsedURI)
File "/home/amaricich/.local/lib/python2.7/site-packages/twisted/web/client.py", line 1580, in _getEndpoint
return self._endpointFactory.endpointForURI(uri)
File "/home/amaricich/.local/lib/python2.7/site-packages/twisted/web/client.py", line 1456, in endpointForURI
uri.port)
File "/home/amaricich/.local/lib/python2.7/site-packages/twisted/web/client.py", line 982, in creatorForNetloc
context = self._webContextFactory.getContext(hostname, port)
AttributeError: 'ClientTLSOptions' object has no attribute 'getContext'
This error leads me to believe that the object being produced by optionsForClientTLS is not the object type that is expected to be passed into the Agent upon creation. A function is trying to be invoked that does not exist. With all that said, I have two questions.
Is this example deprecated? The previous examples that make http requests all work like a charm. Am I doing something wrong, or is the example no longer valid?
I am only looking for a simple way to retrieve data from a server using HTTPS. If doing things this way is not the solution, is anyone familiar with how HTTPS requests can be made using twisted?

Yes you're absolutely correct that the example on the docs is wrong. I noticed the bug while working w/ treq. Try following this example from v14. With that being said, you should use treq as opposed to trying to use Twisted directly. Most of the heavy lifting has been taken care of for you. Here's an simple conversion of your example:
from __future__ import print_function
import treq
from twisted.internet import defer, task
from twisted.python.log import err
#defer.inlineCallbacks
def display(response):
content = yield treq.content(response)
print('Content: {0}'.format(content))
def main(reactor):
d = treq.get('https://twistedmatrix.com')
d.addCallback(display)
d.addErrback(err)
return d
task.react(main)
As you can see treq takes care of the SSL stuff for you. The display() callback function can be used to extract various components of the HTTP response, such as headers, status codes, body, etc. If you only need a single component, such as the response body, then you can simplify further like so:
def main(reactor):
d = treq.get('https://twistedmatrix.com')
d.addCallback(treq.content) # get response content when available
d.addErrback(err)
d.addCallback(print)
return d
task.react(main)

Python, gevent, urllib2.urlopen.read(), download accelerator

I am attempting to build a download accelerator for Linux. My program utilizes gevent, os, and urllib2. My program receives a URL and attempts to download the file concurrently. All of my code is valid. My only problem is that urllib2.urlopen.read() is blocking me from running the .read() function concurrently.
This is the exception thats thrown at me.
Traceback (most recent call last):
File "/usr/lib/pymodules/python2.7/gevent/greenlet.py", line 405, in run
result = self._run(*self.args, **self.kwargs)
File "gevent_concurrent_downloader.py", line 94, in childTasklet
_tempRead = handle.read(divisor) # Read/Download part
File "/usr/lib/python2.7/socket.py", line 380, in read
data = self._sock.recv(left)
File "/usr/lib/python2.7/httplib.py", line 561, in read
s = self.fp.read(amt)
File "/usr/lib/python2.7/socket.py", line 380, in read
data = self._sock.recv(left)
File "/usr/lib/pymodules/python2.7/gevent/socket.py", line 407, in recv
wait_read(sock.fileno(), timeout=self.timeout, event=self._read_event)
File "/usr/lib/pymodules/python2.7/gevent/socket.py", line 153, in wait_read
assert event.arg is None, 'This event is already used by another greenlet: %r' % (event.arg, )
AssertionError: This event is already used by another greenlet: (<Greenlet at 0x2304958: childTasklet(<__main__.NewFile object at 0x22c4390>, 4595517, <addinfourl at 37154616 whose fp = <socket._fileob, 459551, 1)>, timeout('timed out',))
<Greenlet at 0x2304ea8: childTasklet(<__main__.NewFile object at 0x22c4390>,4595517, <addinfourl at 37154616 whose fp = <socket._fileob, 7, -1)failed with AssertionError
My program works by getting the file byte size from the URL by invoking:
urllib2.urlopen(URL).info().get("Content-Length")
and dividing the file size by a divisor and thus breaking the download process into parts. In this example i am breaking the download into 10 parts.
Each greenlet runs a command in this fassion:
urllib2.urlopen(URL).read(offset)
Here's a link to my code hosted on pastie: http://pastie.org/3253705
Thank you for the help!
FYI: I am running on Ubuntu 11.10.

You're trying to read a response to a single request from different greenlets.
If you'd like to download the same file using several concurrent connections then you could use Range http header if the server supports it (you get 206 status instead of 200 for the request with Range header). See HTTPRangeHandler.

the argument to read is a number of bytes, not an offset.
It seems gevent will let you call urllib asynchronously, but not let you access the same resource from multiple greenlets.
Furthermore, since it is using wait_read, the effect will still be a synchronous, sequential read from the file (The complete opposite of what you wanted to achieve).
I'd suggest you might need to go lower than, or use a different library from, urllib2.

Python 3.x and TestLink xmlprc

Appreciate your helping first, I am new for the python 3.x.
When I try to use Python 3.x to parse the testlink xmlprc server. I got below error, but I can run the code under Python 2.x, any idea?
import xmlrpc.client
server = xmlrpc.client.Server("http://172.16.29.132/SITM/lib/api/xmlrpc.php") //here is my testlink server
print (server.system.listMethods()) //I can print the methods list here
print (server.tl.ping()) // Got error.
Here is the error:
['system.multicall', 'system.listMethods', 'system.getCapabilities', 'tl.repeat', 'tl.sayHello', 'tl.ping', 'tl.setTestMode', 'tl.about', 'tl.checkDevKey', 'tl.doesUserExist', 'tl.deleteExecution', 'tl.getTestSuiteByID', 'tl.getFullPath', 'tl.getTestCase', 'tl.getTestCaseAttachments', 'tl.getFirstLevelTestSuitesForTestProject', 'tl.getTestCaseCustomFieldDesignValue', 'tl.getTestCaseIDByName', 'tl.getTestCasesForTestPlan', 'tl.getTestCasesForTestSuite', 'tl.getTestSuitesForTestSuite', 'tl.getTestSuitesForTestPlan', 'tl.getLastExecutionResult', 'tl.getLatestBuildForTestPlan', 'tl.getBuildsForTestPlan', 'tl.getTotalsForTestPlan', 'tl.getTestPlanPlatforms', 'tl.getProjectTestPlans', 'tl.getTestPlanByName', 'tl.getTestProjectByName', 'tl.getProjects', 'tl.addTestCaseToTestPlan', 'tl.assignRequirements', 'tl.uploadAttachment', 'tl.uploadTestCaseAttachment', 'tl.uploadTestSuiteAttachment', 'tl.uploadTestProjectAttachment', 'tl.uploadRequirementAttachment', 'tl.uploadRequirementSpecificationAttachment', 'tl.uploadExecutionAttachment', 'tl.createTestSuite', 'tl.createTestProject', 'tl.createTestPlan', 'tl.createTestCase', 'tl.createBuild', 'tl.setTestCaseExecutionResult', 'tl.reportTCResult']
Traceback (most recent call last):
File "F:\SQA\Python\Testlink\Test.py", line 5, in <module>
print (server.tl.ping())
File "C:\Python31\lib\xmlrpc\client.py", line 1029, in __call__
return self.__send(self.__name, args)
File "C:\Python31\lib\xmlrpc\client.py", line 1271, in __request
verbose=self.__verbose
File "C:\Python31\lib\xmlrpc\client.py", line 1070, in request
return self.parse_response(resp)
File "C:\Python31\lib\xmlrpc\client.py", line 1164, in parse_response
p.feed(response)
File "C:\Python31\lib\xmlrpc\client.py", line 454, in feed
self._parser.Parse(data, 0)
xml.parsers.expat.ExpatError: junk after document element: line 2, column 0

When I've seen this message before, it happened because the contents of the transported data wasn't escaped for XML transport. The solution was to wrap the data in an XMLRPC Binary object.
In your case, you don't control the server side, so the above isn't a solution for you but it may suggest what the actual problem is.
Also, the Python 2 versus Python 3 difference suggests that there is a text/bytes issue at work.
To help diagnose the issue, set verbose=True so you can see the actual HTTP request/response headers and the XML request/response. That may show you what is at line 2: column 0. You may find that the issue may be with the PHP script not wrapping up binary data in base64 encoding as required by the XMLRPC spec.

Thank you , I find out all the methods list, only 'tl.sayHello', 'tl.ping','tl.about' has this problem, and all of them are pass a string with a PHP automatic loader empty file *.class.php to the parser, other methods are pass a xml file. So I give up to use those methods and the script works fine.