When using Python to make a connection to ShareFile via implicit FTPS I get the following:
Traceback (most recent call last):
ftps.storbinary("STOR /file, open(file, "rb"), 1024)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ftplib.py", line 769, in storbinary
conn.unwrap()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 791, in unwrap
s = self._sslobj.shutdown()
SSLError: ('The read operation timed out',)
My tyFTP (required because implicit FTPS is not directly supported in ftplib) class comes from here: Python FTP implicit TLS connection issue. Here's the code:
ftps = tyFTP()
try:
ftps.connect(‘ftps.host.domain’, 990)
except:
traceback.print_exc()
traceback.print_stack()
ftps.login(‘uid', ‘pwd')
ftps.prot_p()
try:
ftps.storbinary("STOR /file", open(file, "rb"), 1024)
# i also tried non-binary, but that didn't work either
# ftps.storlines("STOR /file", open(file, "r"))
except:
traceback.print_exc()
traceback.print_stack()
This question has been asked previously, but the only solution provided is to hack the python code. Is that the best/only option?
ShareFile upload with Python 2.7.5 code timesout on FTPS STOR
ftplib - file creation very slow: SSLError: The read operation timed out
ftps.storlines socket.timeout despite file upload completing
There is also an old discussion about this issue on python.org: http://bugs.python.org/issue8108. The suggestion there is that this is an ambiguous situation that's difficult to fix (and maybe never was?)
Please note: I would have added comments to the existing questions, but my reputation was not high enough to comment (new stack exchange user).
sometimes the help you need is your own.
In order to fix this without directly modifying ftplib code (which requires jumping through hoops on a Mac because you cannot easily write/modify files in your /System/Library) I overrode the storbinary method in ftplib.FTP_TLS. That's essentially using this fix for supporting implicit FTPS:
Python FTP implicit TLS connection issue
and then adding these lines to the class tyFTP, and commenting out the conn.unwrap() call, and replacing it with 'pass':
def storbinary(self, cmd, fp, blocksize=8192, callback=None, rest=None):
self.voidcmd('TYPE I')
conn = self.transfercmd(cmd, rest)
try:
while 1:
buf = fp.read(blocksize)
if not buf: break
conn.sendall(buf)
if callback: callback(buf)
if isinstance(conn, ssl.SSLSocket):
pass
# conn.unwrap()
finally:
conn.close()
return self.voidresp()
My issue with implicit ftp over TLS has bothered me for six months. This week, I decided it was time to fix it. Finally, I combined the code from George Leslie-Waksman and gaFF at here, and Manager_of_it here, and it work like champ! Thank you, those three persons.
Related
I am trying to check if a domain name has MX records resolved using dnspython module. I am getting the following error while connecting to the mx record server. Can anyone explain why I am facing this issue?
Traceback (most recent call last):
File "c:\Users\iamfa\OneDrive\Desktop\test\email_mx.py", line 26, in <module>
dns.resolver.resolve("cmrit.ac.in", 'MX')
File "c:\Users\iamfa\OneDrive\Desktop\test\env1\lib\site-packages\dns\resolver.py", line 1193, in resolve
return get_default_resolver().resolve(qname, rdtype, rdclass, tcp, source,
File "c:\Users\iamfa\OneDrive\Desktop\test\env1\lib\site-packages\dns\resolver.py", line 1066, in resolve
timeout = self._compute_timeout(start, lifetime,
File "c:\Users\iamfa\OneDrive\Desktop\test\env1\lib\site-packages\dns\resolver.py", line 879, in _compute_timeout
raise LifetimeTimeout(timeout=duration, errors=errors)
dns.resolver.LifetimeTimeout: The resolution lifetime expired after 5.001 seconds: Server 10.24.0.1 UDP port 53 answered The DNS operation timed out.; Server 198.51.100.1 UDP port 53 answered The DNS operation timed out.; Server 10.95.11.110 UDP port 53 answered The DNS operation timed out.
This is my code:
import dns.resolver
if dns.resolver.resolve("cmrit.ac.in", 'MX'):
print(True)
else:
print(False)
However it was working fine till yesterday but when I try to run the same code today I am facing this issue.
If the remote DNS server takes a long time to respond, or accepts the connection but does not respond at all, the only thing you can really do is move along. Perhaps try again later. You can catch the error with try/except:
import dns.resolver
try:
if dns.resolver.resolve("cmrit.ac.in", 'MX'):
print(True)
else:
print(False)
except dns.resolver.LifetimeError:
print("timed out, try again later maybe?")
If you want to apply a longer timeout, the resolve method accepts a lifetime keyword argument which is documented in the Resolver.resolve documentation.
The Resolver class (documented at the top of the same page) also has a timeout parameter you can tweak if you build your own resolver.
For production code, you should probably add the other possible errors to the except clause; the exemplary documentation shows you precisely which exceptions resolve can raise.
...
except (dns.resolver.LifetimeTimeout, dns.resolver.NXDOMAIN,
dns.resolver.YXDOMAIN, dns.resolver.NoAnswer,
dns.resolver.NoNameservers) as err:
print("halp, something went wrong:", err)
Probably there is a base exception class which all of these inherit from; I was too lazy to go back and check. Then you only have to list the base class in the except statement.
It's probably more useful to extract the actual MX record and display it, rather than just print True, but that's a separate topic.
Your error message indicates that you were able to connect to your own resolver at 10.24.0.1 but in the general case, this error could also happen if your network (firewall etc) prevents you from accessing DNS for some reason.
I am trying to upload a file to an FTP site using FTPS, but when I attempt to store the file, it just hangs after the file is fully transferred.
global f_blocksize
global total_size
global size_written
f_blocksize = 1024
total_size = os.path.getsize(file_path)
size_written = 0
file = open(file_path, "rb")
try:
ftps = FTP_TLS("ftp.example.com")
ftps.auth()
ftps.sendcmd("USER username")
ftps.sendcmd("PASS password")
ftps.prot_p()
print(ftps.getwelcome())
try:
print("File transfer started...")
ftps.storbinary("STOR myfile.txt", file, callback=handle, blocksize=f_blocksize)
print("File transfer complete!")
except OSError as ex:
print(ftps.getresp())
except Exception as ex:
print("FTP transfer failed.")
print("%s: %s" %(type(ex), str(ex)))
def handle(block):
global size_written
global total_size
global f_blocksize
size_written = size_written + f_blocksize if size_written + f_blocksize < total_size else total_size
percent_complete = size_written / total_size * 100
print("%s percent complete" %str(percent_complete))
I get the following output:
220 Microsoft FTP Service
File transfer started...
3.5648389904264577 percent complete
7.129677980852915 percent complete
10.694516971279374 percent complete
14.25935596170583 percent complete
17.824194952132288 percent complete
21.389033942558747 percent complete
24.953872932985206 percent complete
28.51871192341166 percent complete
32.083550913838124 percent complete
35.648389904264576 percent complete
39.213228894691035 percent complete
42.778067885117494 percent complete
46.342906875543946 percent complete
49.90774586597041 percent complete
53.472584856396864 percent complete
57.03742384682332 percent complete
60.60226283724979 percent complete
64.16710182767625 percent complete
67.7319408181027 percent complete
71.29677980852915 percent complete
74.8616187989556 percent complete
78.42645778938207 percent complete
81.99129677980854 percent complete
85.55613577023499 percent complete
89.12097476066144 percent complete
92.68581375108789 percent complete
96.25065274151436 percent complete
99.81549173194082 percent complete
100.0 percent complete
After which there is no further progress until the connection times out...
FTP transfer failed.
<class 'ftplib.error_temp'>: 425 Data channel timed out due to not meeting the minimum bandwidth requirement.
While the program is running I can see an empty myfile.txt in the FTP site if I connect and look manually, but when I either cancel it or the connection times out, this empty file disappears.
Is there something I'm missing that I need to invoke to close the file after it has been completely transferred?
This appears to be an issue with Python's SSLSocket class, which is waiting for data from the server when running unwrap. Since it never receives this data from the server, it is unable to unwrap SSL from the socket and therefore times out.
This server in particular I have identified by the welcome message as some Microsoft FTP server, which fits in well with the issue written about in this blog
The "fix" (if you can call it that) was to stop the SSLSocket from attempting to unwrap the connection altogether by editing ftplib.py and amending the FTP_TLS.storbinary() method.
def storbinary(self, cmd, fp, blocksize=8192, callback=None, rest=None):
self.voidcmd('TYPE I')
with self.transfercmd(cmd, rest) as conn:
while 1:
buf = fp.read(blocksize)
if not buf: break
conn.sendall(buf)
if callback: callback(buf)
# shutdown ssl layer
if isinstance(conn, ssl.SSLSocket):
# HACK: Instead of attempting unwrap the connection, pass here
pass
return self.voidresp()
I faced this issue on STORBINARY function when using python's ftplib.FTP_TLS, prot_p and Microsoft FTP server.
Example:
ftps = FTP_TLS(host,username,password)
ftps.prot_p
STORBINARY...
The error indicated timeout on the unwrap function.
It is related to the following issues:
https://www.sami-lehtinen.net/blog/python-32-ms-ftps-ssl-tls-lockup-fix
https://bugs.python.org/issue10808
https://bugs.python.org/issue34557
Resolution:
Open the python page for ftplib: https://docs.python.org/3/library/ftplib.html
Click on the source code which will take you to something like this: https://github.com/python/cpython/blob/3.10/Lib/ftplib.py
Create a copy of this code into your project (example: my_lib\my_ftplib.py)
For the method that is failing, in your case STORBINARY, the error looks to be on the line where it says conn.unwrap() in that method. Comment this line. Enter keyword pass otherwise the empty if block will give syntax error.
Import the above library in your file where you are instantiating the FTP_TLS. Now you will no longer face this error.
Reasoning:
The code in the function def ntransfercmd (under FTP_LTS class) encloses the conn object into a SSL session. The above line which you have commented is responsible for tearing down the SSL session post transfer. For some reason, when using Microsoft's FTP server, the code gets blocked on that line and results in timeout. This can be either because post transfer the server drops the connection or maybe the server unwraps the SSL from its side. I am not sure. Commenting that line is harmless because eventually the connection will be closed anyways - see below for details:
In ftplib's python code, you will notice that the conn object in STORBINARY function is enclosed in a with block, and that it is created using socket.create_connection. This means that .close() is automatically called when the code exits the with block (you can confirm this by looking at the __exit__ method on source code of python's socket class).
I'm writing application in python using Asyncio for networking. I have code similar too:
try:
data = await self._reader.readexactly(10000)
# Code that uses data
except IncompleteReadError as e:
data = e.parial
# More code
When I try running this code, it never seems to actually run. If I set a breakpoint on the second line, the breakpoint will trip, but the rest of the function is ignored.
The closest thing I get to an error is this from the asyncio logger:
Traceback (most recent call last):
File "c:\python36\Lib\asyncio\selector_events.py", line 724, in _read_ready
data = self._sock.recv(self.max_size)
ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine
Replacing the second line with data = await self._reader.read(10000) appears to solve this issue, but read() doesn't solve my issue, I need to use readexactly(). So why does readexactly() cause a socket error but read() not?
the only difference between the two is that "read" read up to n bytes while readexactly reads exactly n bytes and if reaches the end before n bytes is raises an IncompleteReadError , which may cause your socket to get the error you have pointed out.
I am trying to download a file from FTP server. I am able to connect to the server. But not able to change the directory.
#! /user/bin/python33
import os
import ftplib
ftp = ftplib.FTP("ftp.sec.gov")
ftp.login("anonymous", "abcd#yahoo.com")
data = []
ftp.dir(data.append)
ftp.quit()
for line in data:
print( "-", line)
print(os.getcwd())
path= "/edgar/full-index/2013/"
print(path)
ftp.cwd(path)
it fails in the last line. can some one suggest what needs to be done"
thanks a lot in advance
Your cwd call fails because you previously called ftp.quit().
The docs for that method say:
Send a QUIT command to the server and close the connection. This is the “polite” way to close a connection, but it may raise an exception if the server responds with an error to the QUIT command. This implies a call to the close() method which renders the FTP instance useless for subsequent calls (see below).
(The "below" reference is to the next part of the documentation which says you can't do any operations on a closed FTP object.)
Here is the relevant code that's causing the Error.
ftp = ftplib.FTP('server')
ftp.login(r'user', r'pass')
#change directories to the "incoming" folder
ftp.cwd('incoming')
fileObj = open(fromDirectory + os.sep + f, 'rb')
#push the file
try:
msg = ftp.storbinary('STOR %s' % f, fileObj)
except Exception as inst:
msg = inst
finally:
fileObj.close()
if '226' not in msg:
#handle error case
I've never seen this error before and any information about why I might be getting it would be useful and appreciated.
complete error message:
[Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
It should be noted that when I manually (i.e. open a dos-prompt and push the files using ftp commands) push the file from the same machine that the script is on, I have no problems.
Maybe you should increase the "timeout" option, and let the server more time to response.
In my case, changing to ACTV mode, as #Anders Lindahl suggested, got everything back into working order.