I'm appending values to a log file every 6th second. Every 30 sec I'm transferring this log to an FTP server as a file. But instead of transfering the whole file, I just want to append the collected data to the file on my server. I haven't been able to figure out how to open the server file and then append the values.
My code so far:
session = ftplib.FTP(authData[0],authData[1],authData[2])
session.cwd("//"+serverCatalog()+"//") # open server catalog
file = open(fileName(),'rb')
with open(fileName(), 'rb') as f:
f = f.readlines()
for line in f:
collected = line
# In some way open server file, write lines to it
session.storbinary('STOR ' + fileName(), open(fileName(), 'a'), 1024)
file.close()
session.quit()
Instead, do I have to download the server file open and append, then send it back?
Above was my question, the full solution is below:
session.cwd("//"+serverCatalog()+"//") # open server catalog
localfile = open("logfile.txt",'rb')
session.storbinary('APPE serverfile.txt', localfile)
localfile.close()
Use APPE instead of STOR.
Source: http://www.gossamer-threads.com/lists/python/python/22960 (link to web.archive.org)
Related
I am trying to read a CSV file from a folder in FTP. The file has 3072 rows. However, when I am running the code, it is not reading all the rows. Certain rows from the bottom are getting missed out.
## FTP host name and credentials
ftp = ftplib.FTP('IP', 'username','password')
## Go to the required directory
ftp.cwd("Folder_Name")
names = ftp.nlst()
final_names= [line for line in names if '.csv' in line]
latest_time = None
latest_name = None
#os.chdir(filepath)
for name in final_names:
time1 = ftp.sendcmd("MDTM " + name)
if (latest_time is None) or (time1 > latest_time):
latest_name = name
latest_time = time1
file = open(latest_name, 'wb')
ftp.retrbinary('RETR '+ latest_name, file.write)
dat = pd.read_csv(latest_name)
The CSV file to be read from FTP is as given below-
The output from the code is as-
Make sure you close the file, before you try to read it, using file.close(), or even better using with:
with open(latest_name, 'wb') as file:
ftp.retrbinary('RETR '+ latest_name, file.write)
dat = pd.read_csv(latest_name)
If you do not need to actually store the file to local file system, and the file is not too large, you can download it to memory only:
Reading files from FTP server to DataFrame in Python
I'm trying to download a file from an FTPS server, using Python ftplib.
But the downloaded file has always 0 bytes (is empty).
If I see the file in the server with WinSCP, the file has data (about 1Kb).
In WinSCP I'm using the options "Encryption: Explicit TSL" and "PassiveMode=False".
What is wrong with the code?
Thanks!!
This is the code I am using:
import ftplib
server='10.XX.XX.XX'
username='username'
password='password'
session = ftplib.FTP_TLS(server)
session.login(user=username,passwd=password)
session.prot_p()
session.set_pasv(False)
session.nlst()
session.cwd("home")
print(session.pwd())
filename = "test.txt"
# Open a local file to store the downloaded file
my_file = open(r'c:\temp\ftpTest.txt', 'wb')
session.retrbinary('RETR ' + filename, my_file.write, 1024)
session.quit()
You are not closing the local file after the download. You should use context manager for that. Similarly also for the FTP session:
with ftplib.FTP_TLS(server) as session:
session.login(user=username, passwd=password)
session.prot_p()
session.set_pasv(False)
session.nlst()
session.cwd("home")
print(session.pwd())
filename = "test.txt"
# Open a local file to store the downloaded file
with open(r'c:\temp\ftpTest.txt', 'wb') as my_file:
session.retrbinary('RETR ' + filename, my_file.write, 1024)
I am sending a file from one Linux VM to another Linux VM using python which is doing great. Files are sent successfully BUT before I send the file, I want to change the file encoding to "UTF-8" and line ending to "Unix/Linux". How to do that?
Below is the piece of code that is sending the file through sftp:
with pysftp.Connection(host=host, username=userName, password=passWord) as sftpVal:
#print(sftpVal.listdir()) #list directories in sftp home
sftpVal.put(source_file_path,'incoming/'+fileName) #(localPath, destinationPath)
One option would be to read the file contents, change them as you need, save them to a temp file, and send the temp file instead. For example:
with open('file_name.txt') as original_file, open('file_to_send.txt', 'w') as send_file: # Open the original and prepared files
content = original_file.read()
content = content.replace('\r\n', '\n') # Change the line endings
content = content.encode('utf-8') # Encode in UTF-8
send_file.write(content)
# Do your file send here but with the prepared file instead.
os.remove('file_to_send.txt') # Optional removal of temp file.
SOLVED
It looks like I needed to close the sys.stdout file (which was open for writing) with a
sys.stdout.close()
and then re-open it, but in read mode, with
sys.stdout = open ('/home/myuser/verify_yslog_conf/remote_hostname/hi', 'r')
and then it uploaded correctly. I used a
time.sleep(60)
to cat the file mid-run for troubleshooting. It was empty until the def had finished, which by default closed the "hi" file. So I realized that I needed to close it to fully write the file. thanks.
Code below is meant to check if syslog is running, and if so, comment out any lines with "#" (but NOT #192.168.x.xx or 192.168.x.xxx) from the pulled remote syslog.conf file. After commenting those lines locally, it is supposed to upload the new file to the remote server in the old location. However, all that happens is that my /etc/syslog.conf file gets erased. The file isn't removed, it is just empty.
I know the issue is from the put statement
fabric.operations.put('/home/myuser/verify_yslog_conf/remote_hostname/hi', '/etc/syslog.conf', use_sudo=True)
but nothing looks wrong with it.
def verify():
#lines below are good; keep them
ip1 = '192.168.x.xx'
ip2 = '92.168.x.xxx'
#check if syslog is running
output = sudo("/sbin/service syslog status")
if 'is running...' in output:
#pull the syslog file from the remote server
fabric.operations.get('/etc/syslog.conf')
#read thru the file
fh = open('/home/myuser/verify_yslog_conf/remote_hostname/syslog.conf', 'r')
f = fh.read()
fh.close()
#if the file needs commenting ...
if re.search(r'(#(?!({0}|{1})))'.format(ip1, ip2), f):
#get the file again -- maybe the problem? was advised to do this
fabric.operations.get('/etc/syslog.conf')
#save the file locally for reading and then for writing (r+ failed)
conf = open ('/home/myuser/verify_yslog_conf/remote_hostname/syslog.conf', 'r')
sys.stdout = open ('/home/myuser/verify_yslog_conf/remote_hostname/hi', 'w')
#for every line in the old /etc/syslog.conf
for line in conf:
#if it has an # but not followed by the permitted IPs
if re.search(r'(#(?!({0}|{1})))'.format(ip1, ip2), line):
#comment out the line then print it (does it twice, dont care)
line = "#" + line
sys.stdout.write(line)
sys.stdout.write(line)
conf.close()
#upload the newly commented file to the remote host
fabric.operations.put('/home/myuser/verify_yslog_conf/remote_hostname/hi', '/etc/syslog.conf', use_sudo=True)
else:
print GOOD
else:
print RSYSLOG
Try to sys.stdout.close() immediately prior to the fabric.operations.put command.
It's likely that before the put operation, the data isn't flushed to the file, so it's sending an empty file, but when the script finishes, it automatically flushes (writes) the data which is why the file appears normal on the local system.
this below code is working fine, but the issue is that the transferred file has no spaces , everything is in single line
ftp =FTP('x.x.x.x')
ftp.login('user','password')
filename = "test.txt"
local = open(filename, 'wb')
ftp.cwd('/root/Desktop')
ftp.retrbinary('RETR ' +filename, local.write,1024)
say test.txt had
1)one
2)two
3)three
the test.txt thats copied from the reemote server has
1)one2)two3)three
everything in one line,
any help pls.
To quote the internet:
http://www.cs.toronto.edu/~krueger/csc209h/tut/line-endings.html
Text files created on DOS/Windows machines have different line endings
than files created on Unix/Linux. DOS uses carriage return and line
feed ("\r\n") as a line ending, which Unix uses just line feed ("\n").
It's the writing binary that's causing your problem. It prevents the translation which should be automatic when writing as text. Can you transfer and save as a text file? You can use retrlines instead of retrbinary.
ftp =FTP('x.x.x.x')
ftp.login('user','password')
filename = "test.txt"
local = open(filename, 'w')
ftp.cwd('/root/Desktop')
ftp.retrlines('RETR ' +filename, local.write)