I am trying to read a CSV file from a folder in FTP. The file has 3072 rows. However, when I am running the code, it is not reading all the rows. Certain rows from the bottom are getting missed out.
## FTP host name and credentials
ftp = ftplib.FTP('IP', 'username','password')
## Go to the required directory
ftp.cwd("Folder_Name")
names = ftp.nlst()
final_names= [line for line in names if '.csv' in line]
latest_time = None
latest_name = None
#os.chdir(filepath)
for name in final_names:
time1 = ftp.sendcmd("MDTM " + name)
if (latest_time is None) or (time1 > latest_time):
latest_name = name
latest_time = time1
file = open(latest_name, 'wb')
ftp.retrbinary('RETR '+ latest_name, file.write)
dat = pd.read_csv(latest_name)
The CSV file to be read from FTP is as given below-
The output from the code is as-
Make sure you close the file, before you try to read it, using file.close(), or even better using with:
with open(latest_name, 'wb') as file:
ftp.retrbinary('RETR '+ latest_name, file.write)
dat = pd.read_csv(latest_name)
If you do not need to actually store the file to local file system, and the file is not too large, you can download it to memory only:
Reading files from FTP server to DataFrame in Python
Related
I want to print the content of the last saved text file in a folder using Python. I wrote the below code. It is printing out only the path of the file but not the content.
folder_path = r'C:\Users\Siciid\Desktop\restaurant\bill'
file_type = r'\*txt'
files = glob.glob(folder_path + file_type)
max_file = max(files, key=os.path.getctime)
filename=tempfile.mktemp('.txt')
open(filename,'w').write(max_file)
os.startfile(filename,"print")
Is it possible to do this in Python. Any suggestion. I would appreciate your help. Thank you.
You can do that using the following code. Just replace the line where you open and write a file with these two lines:
with open(max_file, "r") as f, open(filename, 'w') as f2:
f2.write(f.read())
The max_file variable contains a file name, not the contents of the file, so writing it to the temp file and printing that will simply print the file name instead of its contents. To put its contents into the temporary file, you need to open the file and then read it. That is what the above two lines of code do.
Using python, I'm uploading a csv file to Sharepoint online. I'm able to get the file to the Sharepoint site document list, but when I open the CSV file it is incorrectly formatted.
Example:
Source file:
"FirstName", "LastName", "Date"
"John", "Smith", "08-09-1990"
"Jane", "doe", "01-02-2010"
Destination Sharepoint file: (after upload to Sharepoint online)
"FirstName"\ "LastName"\ "Date"\ "John"\ "Smith"\ "08-09-1990"\ "Jane"\ "doe"\ "01-02-2010"
The destination Sharepoint file ends up having a modified delimiter and all of the data is parsed to a single row. I believe the single row parsing is due to the change from a "," delimiter to a "" delimiter.
How can I keep the source file formatting with rows/columns? Below is the code that I'm using.
#Connect to Sharepoint
ctx_auth = AuthenticationContext(url_shrpt)
if ctx_auth.acquire_token_for_user(username_shrpt, password_shrpt):
ctx = ClientContext(url_shrpt, ctx_auth)
web = ctx.web
ctx.load(web)
ctx.execute_query()
print('Authenticated into sharepoint as: ',web.properties['Title'])
else:
print(ctx_auth.get_last_error())
##### ---- ####
#Upload the files
try:
#list all of the files in the folder
for file in fileNames:
#Open individual files and read content
readFile = open(file_name + file, 'r')
data = readFile.read()
#Set the destination path for the files to be sent
remotepath = r'Shared Documents/Folder/SubFolder/'+file
#List the director, then split the files to get the file name
dir, name = os.path.split(file)
file = ctx.web.get_folder_by_server_relative_url(dir).upload_file(remotepath, data).execute_query()
print("File has been uploaded to Sharepoint")
#close the file so it can be archived
readFile.close()
except Exception as err:
print("Something went wrong with writing files: " + str(err))
I'm developing a Flask application where I want the user to download a set of files from the server. To achieve this objective, I use the zipfile module in order to zip all the files and then send this compressed file to the user.
Here is my code:
#app.route("/get_my_files/<file_name>", methods = ["GET"])
def get_my_files(file_name):
file_exts = [".csv", ".xlsx", ".json"]
file_path = "./user_files/" + file_name
# Create compress file.
memory_file = io.BytesIO()
with zipfile.ZipFile(memory_file, 'w') as zf:
for ext in file_exts:
#data = zipfile.ZipInfo(individualFile)
data = zipfile.ZipInfo("resultado" + ext)
data.date_time = time.localtime(time.time())[:6]
data.compress_type = zipfile.ZIP_DEFLATED
zf.writestr(data, open(file_path + ext, "r").read())
memory_file.seek(0)
# , encoding="ISO-8859-1"
# Delete files.
for ext in file_exts:
os.remove(file_path + ext)
# Return zip file to client.
return send_file(
memory_file,
mimetype = "zip",
attachment_filename='resultados.zip',
as_attachment=True,
cache_timeout=0
)
Unfortunately, once I decompress the zip file, the Excel file is getting corrupted (CSV and JSON file can be read and opened without problems). I have tried several different types of encoding when writing the zip file, however I haven't been able to find a solution.
What is the problem and how can I do this correctly?
You opened the files in text mode, which worked for JSON and CSV, but not for a binary file like Excel.
open(file_path + ext, "r")
You need to open them in binary mode, that is rb = read binary.
I have tried the following code by slightly modifying the example in documentation
class Upload():
def POST(self):
web.header('enctype','multipart/form-data')
print strftime("%Y-%m-%d %H:%M:%S", gmtime())
x = web.input(file={})
filedir = '/DiginUploads' # change this to the directory you want to store the file in.
if 'file' in x: # to check if the file-object is created
filepath=x.file.filename.replace('\\','/') # replaces the windows-style slashes with linux ones.
filename=filepath.split('/')[-1] # splits the and chooses the last part (the filename with extension)
fout = open(filedir +'/'+ filename,'w') # creates the file where the uploaded file should be stored
fout.write(x.file.file.read()) # writes the uploaded file to the newly created file.
fout.close() # closes the file, upload complete.
But this works only for csv and txt documents. For Excel/pdf etc file gets created but it can't be opened (corrupted). What should I do to handle this scenario?
I saw this but it is about printing the content which does not address my matter.
You need to use wb (binary) mode when opening the file:
fout = open(filedir +'/'+ filename, 'wb')
I'm appending values to a log file every 6th second. Every 30 sec I'm transferring this log to an FTP server as a file. But instead of transfering the whole file, I just want to append the collected data to the file on my server. I haven't been able to figure out how to open the server file and then append the values.
My code so far:
session = ftplib.FTP(authData[0],authData[1],authData[2])
session.cwd("//"+serverCatalog()+"//") # open server catalog
file = open(fileName(),'rb')
with open(fileName(), 'rb') as f:
f = f.readlines()
for line in f:
collected = line
# In some way open server file, write lines to it
session.storbinary('STOR ' + fileName(), open(fileName(), 'a'), 1024)
file.close()
session.quit()
Instead, do I have to download the server file open and append, then send it back?
Above was my question, the full solution is below:
session.cwd("//"+serverCatalog()+"//") # open server catalog
localfile = open("logfile.txt",'rb')
session.storbinary('APPE serverfile.txt', localfile)
localfile.close()
Use APPE instead of STOR.
Source: http://www.gossamer-threads.com/lists/python/python/22960 (link to web.archive.org)