I am trying to perform a task to transfer files between two different FTP locations. And the simple goal is that I would want to specific file type from FTP Location A to FTP Location B for only last few hours using Python script.
I am using ftplib to perform the task and have put together below code.
So far the file transfer is working fine for single file defined in the from_sock variable, but I am hitting road block when I am wanting to loop through all files which were created within last 2 hours and copy them. So the script I have written is basically copying individual file but I want to I wan't to move all files with particular extension example *.jpg which were created within last 2 hours. I tired to use MDTM to find the file modification time but I am not able to implement in right way.
Any help on this is much appreciated. Below is the current code:
import ftplib
srcFTP = ftplib.FTP("test.com", "username", "pass")
srcFTP.cwd("/somefolder")
desFTP = ftplib.FTP("test2.com", "username", "pass")
desFTP.cwd("/")
from_Sock = srcFTP.transfercmd("RETR Test1.text")
to_Sock = desFTP.transfercmd("STOR test1.text")
state = 0
while 1:
block = from_Sock.recv(1024)
if len(block) == 0:
break
state += len(block)
while len(block) > 0:
sentlen = to_Sock.send(block)
block = block[sentlen:]
print state, "Total Bytes Transferred"
from_Sock.close()
to_Sock.close()
srcFTP.quit()
desFTP.quit()
Thanks,
DD
Here a short code that takes the path and uploads every file with an extension of .jpg via ftp. Its not exactly what you want but I stumbled on your answer and this might help you on your way.
import os
from ftplib import FTP
def ftpPush(filepathSource, filename, filepathDestination):
ftp = FTP(IP, username, password)
ftp.cwd(filepathDestination)
ftp.storlines("STOR "+filename, open(filepathSource+filename, 'r'))
ftp.quit()
path = '/some/path/'
for fileName in os.listdir(path):
if fileName.endswith(".jpg"):
ftpPush(filepathSource=path, filename=fileName, filepathDestination='/some/destination/')
The creation time of a file can be checked on an ftp server using this example.
fileName = "nameOfFile.txt"
modifiedTime = ftp.sendcmd('MDTM ' + fileName)
# successful response: '213 20120222090254'
ftp.quit()
Now you just need to check when the file that have been modified, download it if it is below you wished for threshold and then upload them to the other computer.
Related
I am trying to send each file from a disk image to a remote server using paramiko.
class Server:
def __init__(self):
self.ssh = paramiko.SSHClient()
self.ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
self.ssh.connect('xxx', username='xxx', password='xxx')
def send_file(self, i_node, name):
sftp = self.ssh.open_sftp()
serverpath = '/home/paul/Testing/'
try:
sftp.chdir(serverpath)
except IOError:
sftp.mkdir(serverpath)
sftp.chdir(serverpath)
serverpath = '/home/Testing/' + name
sftp.putfo(fs.open_meta(inode = i_node), serverpath)
However when I run this I get an error saying that "pytsk.File has no attribute read".
Is there any other way of sending this file to the server?
After a quick investigation I think I found what your problem is. Paramiko's sftp.putfo expects a Python file object as the first parameter. The file object of Pytsk3 is a completely different thing. Your sftp object tries to perform "read" on this, but Pytsk3 file object does not have a method "read", hence the error.
You could in theory try expanding Pytsk3.File class and adding this method but I would not hold my breath that it actually works.
I would just read the file to a temporary one and send that. Something like this (you would need to make temp file name handling more clever and delete the file afterwards but you will get the idea):
serverpath = '/home/Testing/' + name
tmp_path = "/tmp/xyzzy"
file_obj = fs.open_meta(inode = i_node)
# Add here tests to confirm this is actually a file, not a directory
tha = open(tmp_path, "wb")
tha.write(file_obj.read_random(0, file_obj.info.meta.size))
tha.close()
rha = open(tmp_path, "rb")
sftp.putfo(rha, serverpath)
rha.close()
# Delete temp file here
Hope this helps. This will read the whole file in memory from fs image to be written to temp file, so if the file is massive you would run out of memory.
To work around that, you should read the file in chunks looping through it with read_random in suitable chunks (the parameters are start offset and amount of data to read), allowing you to construct the temp file in a chunks of for example a couple of megabytes.
This is just a simple example to illustrate your problem.
Hannu
Backstory is im trying to pull some data from an ftp login I was given. This data constantly gets updated, about daily, and I believe they wipe the ftp at the end of each week or month. I was thinking about inputting a date and having the script run daily to see if there are any files that match the date, but if the servers time isn't accurate it could cause data loss. For now I just want to download ALL the files, and then ill work on fine-tuning it.
I haven't worked much with coding ftp before, but seems simple enough. However, the problem I'm having is small files get downloaded without a problem and their file sizes check out and match. When it tries to download a big file that would normally take a few minutes, it gets to a certain point (almost completing the file) and then it just stops and the script hangs.
For Example:
It tries to download a file that is 373485927 bytes in size. The script runs and downloads that file up until 373485568 bytes. It ALWAYS stops at this amount after trying different methods and changing some code.
Don't understand why it always stops at this byte and why it would work fine with smaller files (1000 bytes and under).
import os
import sys
import base64
import ftplib
def get_files(ftp, filelist):
for f in filelist:
try:
print "Downloading file " + f + "\n"
local_file = os.path.join('.', f)
file = open(local_file, "wb")
ftp.retrbinary('RETR ' + f, file.write)
except ftplib.all_errors, e:
print str(e)
file.close()
ftp.quit()
def list_files(ftp):
print "Getting directory listing...\n"
ftp.dir()
filelist = ftp.nlst()
#determine new files to DL, pass to get_files()
#for now we will download all each execute
get_files(ftp, filelist)
def get_conn(host,user,passwd):
ftp = ftplib.FTP()
try:
print "\nConnecting to " + host + "...\n"
ftp.connect(host, 21)
except ftplib.all_errors, e:
print str(e)
try:
print "Logging in...\n"
ftp.login(user, base64.b64decode(passwd))
except ftplib.all_errors, e:
print str(e)
ftp.set_pasv(True)
list_files(ftp)
def main():
host = "host.domain.com"
user = "admin"
passwd = "base64passwd"
get_conn(host,user,passwd)
if __name__ == '__main__':
main()
Output looks like this with file dddd.tar.gz being the big one and never finishes it.
Downloading file aaaa.del.gz
Downloading file bbbb.del.gz
Downloading file cccc.del.gz
Downloading file dddd.tar.gz
This could be caused by a timeout issue, perhaps try in:
def get_conn(host,user,passwd):
ftp = ftplib.FTP()
add in larger timeouts until you have more of an idea whats going on, like:
def get_conn(host,user,passwd):
ftp = ftplib.FTP(timeout=100)
I'm not sure if ftplib defaults to a timeout or not, it would be worth checking and worth checking if you are being timed-out from the server. Hope this helps.
If you are running your scrpit in windows cmd console, try to disable the "QuickEdit Mode" option of cmd.
I had encontered a problem that my ftp script hangs running in windows, but works normally in linux. At last i found that solution is working for me.
Ref:enter link description here
I'm not sure if this can be done or not but I thought I'd ask!
I'm running a windows 10 PC using Python 2.7. I'm wanting to download a file form sharepoint to a folder on my C: drive.
OpenPath = "https://office.test.com/sites/Rollers/New improved files/"
OpenFile = "ABC UK.xlsb"
The downside is the file is uploaded externally & due to human error it can be saved as a .xlsx or ABC_UK. Therefor I only want to use the first 3 characters with a wildcard (ABC*) to open that file. Thankfully the first 3 Characters are unique & there only be one file in the path that should match.
to find the file in your dir:
import os, requests, fnmatch
#loop over dir
for filename in os.listdir(OpenPath):
#find the file
if fnmatch.fnmatch(filename, 'ABC_UK*'):
#download the file
# open file handler
with open('C:\dwnlfile.xls', 'wb') as fh:
#try to get it
result = requests.get(OpenPath+filename)
#check u got it
if not result.ok:
print result.reason # or text
exit(1)
#save it
fh.write(result.content)
print 'got it and saved'
I'm trying to retrieve a zip folder(s) from an ftp site and save them to my local machine, using python (ideally I'd like to specify where they are saved on my C:).
The code below connects to the FTP site and then *something happens in the PyScripter window that looks like random characters for about 1000 lines... but nothing actually gets downloaded to my hard drive.
Any tips?
import ftplib
import sys
def gettext(ftp, filename, outfile=None):
# fetch a text file
if outfile is None:
outfile = sys.stdout
# use a lambda to add newlines to the lines read from the server
ftp.retrlines("RETR " + filename, lambda s, w=outfile.write: w(s+"\n"))
def getbinary(ftp, filename, outfile=None):
# fetch a binary file
if outfile is None:
outfile = sys.stdout
ftp.retrbinary("RETR " + filename, outfile.write)
ftp = ftplib.FTP("FTP IP Address")
ftp.login("username", "password")
ftp.cwd("/MCPA")
#gettext(ftp, "subbdy.zip")
getbinary(ftp, "subbdy.zip")
Well, it seems that you simply forgot to open the file you want to write into.
Something like:
getbinary(ftp, "subbdy.zip", open(r'C:\Path\to\subbdy.zip', 'wb'))
This question already has answers here:
Python FTP get the most recent file by date
(5 answers)
Closed 4 years ago.
At my company we have a scale that checks the weights of boxes before loading them into a truck. In the case the box contains more or less product than acceptable, the box is rejected and delivered to another conveyor belt. The electronic scale keeps a record of the performance of the operation. Files are stored in the scale's disk and accessed using ftp from a nearby desktop computer. My boss wants the reports to be automatically emailed to his account so he doesn't need to go to that facility just to check the rejections of the day before. I started writing a program in Python to do that, but got stucked in the part about retrieving the file from the folder. Here is my code:
#This program tries to retrieve the latest report and send it by email.
import urllib
import shutil
import ftplib
import os
import sys
import glob
import time
import datetime
import smtplib
import email
#Define the server and folder where the reports are stored.
carpetaftp = "/reports/"
#This function looks for the last file in the folder.
def obtenerultimoarchivo(camino):
for cur_path, dirnames, filenames in os.walk(camino):
for filename in filenames:
datos_archivo = os.stat(filename)
tiempo_archivo = datos_archivo.st_mtime
#Connects to an ftp folder and downloads the last report.
def descargareporteftp(carpetaftp):
ftp = ftplib.FTP("server.scale.com")
ftp.login()
ftp.cwd(carpetaftp)
#Uses 'ultimoreporte.pdf' as a copy of the last report.
archivo = open('C:\\Balanza\\Reportes\\ultimoreporte.pdf',"wb")
ftp.retrbinary("RETR " + obtenerultimoarchivo(),archivo.write)
archivo.close()
return archivo
#The function enviaemail() sends an email with an attachment.
def enviaemail(destinatario, adjunto):
remitente = "electronic_scale#email.com.uy"
msg = email.MIMEMultipart()
msg['From'] = remitente
msg['To'] = destinatario
msg['Subject'] = "Ultimo informe de la balanza."
adjunto = open('C:\\Balanza\\Reportes\\ultimoreporte.pdf', 'rb')
attach = email.MIMENonMultipart('application', 'pdf')
payload = base64.b64encode(adjunto.read()).decode('ascii')
attach.set_payload(payload)
attach['Content-Transfer-Encoding'] = 'base64'
adjunto.close()
attach.add_header('Content-Disposition', 'attachment', filename = 'ultimoreporte.pdf')
msg.attach(attach)
server = smtplib.SMTP('smtp.email.com.uy')
server.login('electronic_scale#email.com.uy', 'thermofischer')
server.sendmail('electronic_scale#email.com.uy',destinatario, msg.as_string())
server.quit()
#The main routine, to retrieve the last report and send it by email.
adjunto = descargareporteftp(carpetaftp)
print("Reporte descargado")
enviaemail('myemail#email.com.uy',reporte)
print("Reporte enviado")
Here is a dirty way I found by making a mistake:
If no file is specified in a urllib retrieve command it will return and list of files in the directory (as in an ls command).
So using this, the code does the following:
Retrieve a list of files in the FTP folder and save file locally
Read the file and find the last entry on the list
Retrieve the last file from the FTP folder
import urllib
def retrieve(address,output):
# Run the ftp retrieve command here
urllib.urlretrieve(address, output)
# Flush the urllib so that we can download a second file
urllib.urlcleanup()
def main():
#let's build the ftp address here:
usr = "usrname"
psswrd = "password"
host = "host.com"
path = "/foo/bar/"
list_file = "list"
address = 'ftp://%s:%s#%s%s' % (usr,psswrd,host,path)
print "Getting file listing from: " + address
# Retrieve the list
retrieve(address,list_file)
# read a text file as a list of lines
# find the last line, change to a file you have
list_file_reader = open ( 'list',"r" )
lineList = list_file_reader.readlines()
last_file = lineList[-1].split(' ')[-1:][0].rstrip()
output = "wanted_file.dat"
address = 'ftp://%s:%s#%s%s%s' % (usr,psswrd,host,path,last_file)
# Retrieve the file you are actually looking for
retrieve(address, output)
print address
main()
This is certainly no the most efficient way, but it works in my scenario.
References:
Python: download a file over an FTP server
https://www.daniweb.com/programming/software-development/threads/24544/how-do-i-read-the-last-line-of-a-text-file
Downloading second file from ftp fails