Using Python 3.6
successfully retrieving data from mssqlserver and querying oracle for historical data and exporting it via Pandas to a tab delimited file which I then want to be able to ftp to another mssql server location. Since I have not tried this before, I experimented as follows:
\\\\\\\\\\\\\\\\\\\
This works: passing hardcoded variable
\\\\\\\\\\\\\\\\\\\\
from ftplib import FTP
host='myserver'
host_name='group_user_name'
host_pw='hostpassword'
host_dir='myworkingdirectory'
ftp = FTP(host)
ftp.login(user=host_name, passwd =host_pw)
ftp.cwd(host_dir)
filename = 'c:/temp/cigfun.txt'
fn = 'cigfun.txt'
def placeFile():
ftp.storbinary('STOR '+fn, open(filename, 'rb'))
ftp.quit()
placeFile()
\\\\\\\\\\\\\\
This also works: hard coded
\\\\\\\\\\\\\\
from ftplib import FTP
ftp = FTP('myserver')
ftp.login(user='group_user_name', passwd ='hostpassword')
ftp.cwd('myworkingdirectory')
ftp.retrlines('LIST')
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
This does not connect: retrieving data from stored procedure in MSSQL
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
if sql_id == str(5):
sql_cursor.execute("[dbo].[usp_GetProcess_"+seq_action+"] #sql_id = "+ sql_id)
table = sql_cursor.fetchall()
for row in table:
host = "'"+row[1]+"'"
host_name = "'"+row[2]+"'"
host_pw = "'"+row[3]+"'"
host_dir = "'"+row[4]+"'"
host_dir = re.sub('[cd ]', '', host_dir)
ftp_file = row[9]
f_type = row[10]
if f_type == "Tab (txt)":
f_type = '.txt'
elif f_type == "Comma (csv)":
f_type = '.csv'
else: f_type = '.xlsx'
path_file = "'"+ftp_file + f_type+"'"
fn="'"+os.path.basename(path_file)
path_file = path_file.replace(r'\\','/')
from ftplib import FTP
ftp = FTP(host)
ftp.login(user = host_name, passwd = host_pw)
ftp.cwd(host_dir)
\\\\\\\\\\\\\\
throws the following error:
File "C:\Users\A22021\AppData\Local\Continuum\anaconda332\lib\socket.py", line 745, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
gaierror: [Errno 11004] getaddrinfo failed
aside from my awkward efforts at coding (only my 3rd production attempt using Python), can you see anything obvious that would cause the failure? error messages are so cryptic.
I suspect it might have something to do with coded variable assignments, but I am baffled right now. Any constructive advice appreciated.
PLG
If everything works when you hard-code the FTP information, then the issue here is most likely the way you are parsing the query results. You need to see exactly what is getting put into your host* variables. To start, you could just pepper your code with some print statements, but I would highly suggest using the python debugger instead. This probably sounds scary, but it's not that bad. Run it like this:
python -m pdb my_script.py
You can then step through each line of your code by pressing n. So for example, when you hit the line that says for row in table: and you press n you will now be inside that loop and can print(row). This will give you some insight into what is going wrong with the parsing.
Related
I want to read multi big files that exist on centos server with python.I wrote a simple code for that and it's worked but entire file came to a paramiko object (paramiko.sftp_file.SFTPFile) after that I can process line. it has not good performance and I want process file and write to csv piece by piece because process entire file can affect performance. Is there a way to solve the problem?
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(host, port, username, password)
sftp_client = ssh.open_sftp()
remote_file = sftp_client.open(r'/root/bigfile.csv')
try:
for line in remote_file:
#Proccess
finally:
remote_file.close()
Here could solve your problem.
def lazy_loading_ftp_file(sftp_host_conn, filename):
"""
Lazy loading ftp file when exception simple sftp.get call
:param sftp_host_conn: sftp host
:param filename: filename to be downloaded
:return: None, file will be downloaded current directory
"""
import shutil
try:
with sftp_host_conn() as host:
sftp_file_instance = host.open(filename, 'r')
with open(filename, 'wb') as out_file:
shutil.copyfileobj(sftp_file_instance.raw, out_file)
return {"status": "sucess", "msg": "sucessfully downloaded file: {}".format(filename)}
except Exception as ex:
return {"status": "failed", "msg": "Exception in Lazy reading too: {}".format(ex)}
This will avoid reading the whole thing into memory at once.
Reading in chunks will help you here:
import pandas as pd
chunksize = 1000000
for chunk in pd.read_csv(filename, chunksize=chunksize):
process(chunk)
Update:
Yeah, I'm aware that my answer written based on a local file. Just giving example for reading file in chunks.
To answer the question, check out this one:
paramiko.sftp_client.SFTPClient.putfo
Functions for working with remote files using pandas and paramiko (SFTP/SSH). - pass the chunk size as I mentioned above.
I’m having a hard time with this simple script. It’s giving me an error of file or directory not found but the file is there. Script below I’ve masked user and pass plus FTP site
Here is my script
from ftplib import FTP
ftp = FTP('ftp.domain.ca')
pas = str('PASSWORD')
ftp.login(user = 'user', passwd=pas)
ftp.cwd('/public_html/')
filepaths = open('errorstest.csv', 'rb')
for j in filepaths:
print(j)
ftp.delete(str(j))
ftp.quit()
The funny thing tho is if I slight change the script to have ftp.delete() it finds the file and deletes it. So modified to be like this:
from ftplib import FTP
ftp = FTP('ftp.domain.ca')
pas = str('PASSWORD')
ftp.login(user = 'user', passwd=pas)
ftp.cwd('/public_html/')
ftp.delete(<file path>)
ftp.quit()
I’m trying to read this from a csv file. What am I doing wrong?
Whatever you have showed seems to be fine. But could you try this?
from ftplib import FTP
ftp = FTP(host)
ftp.login(username, password)
ftp.cwd('/public_html/')
print(ftp.pwd())
print(ftp.nlst())
with open('errorstest.csv') as file:
for line in file:
if line.strip():
ftp.delete(line.strip())
print(ftp.nlst())
I have a Odoo8 running on my linux server and I need to copy a file from this server to a Windows 10 shared folder with authentication.
I tried to do it programmatically like this:
full_path = "smb://hostname/shared_folder/other_path"
if not os.path.exists(full_path):
os.makedirs(full_path)
full_path = os.path.join(full_path, file_name)
bin_value = stream.decode('base64')
if not os.path.exists(full_path):
try:
with open(full_path, 'wb') as fp:
fp.write(bin_value)
fp.close()
return True
except IOError:
_logger.exception("stream_save writing %s", full_path)
but even if no exception is raised, folders are not created and file is not written.
Then I tried to remove the "smb:" part from the uri and it raised an exception regarding authentication.
I'd like to fix the problem just by using python, possibly avoiding os.system calls or external scripts, but if no other way is possible, then any suggestion is welcome.
I also tried with
"//user:password#hostname"
and
"//domain;user:password#hostname"
both with and without smb
Well, I found it out by myself a way using SAMBA:
First you need to install pysmb (pip install pysmb) then:
from smb.SMBConnection import SMBConnection
conn = SMBConnection(user, password, "my_name", server, domain=domain, use_ntlm_v2 = True)
conn.connect(ip_server)
conn.createDirectory(shared_folder, sub_directory)
file_obj = open(local_path_file,'rb')
conn.storeFile(shared_folder, sub_directory+"/"+filename, file_obj)
file_obj.close()
in my case sub_directory is a whole path, thus I need to create each folder one by one (createDirectory works only this way) and everytime I need to check if the directory does not already exists because otherwise createDirectory raise an exception.
I hope my solution could be useful for others.
If anybody find a better solution, please answer...
quick and simple:
I have the following function, works well if i specify the file name.
import os
import ftplib
def ftpcon(self, host, port, login, pwd, path):
ftp = ftplib.FTP()
ftp.connect(host, port, 20)
try:
ftp.login(login, pwd)
ftp.cwd(path)
for files in ftp.nlst():
if files.endswith('.doc') or files.endswith('.DOC'):
ftp.retrbinary('RETR ' + files, open(file, 'wb').write)
print files
But when i use the for loop with ftp.nlst() to try to match an specific type of file, i receive the error:
coercing to Unicode: need string or buffer, type found
Since im not sure if this is the best way to do it, what could the "correct" way to download a file ?
Maybe try:
from ftplib import FTP
server = FTP("ip/serveradress")
server.login("user", "password")
server.retrlines("LIST") # Will show a FTP content list.
server.cwd("Name_Of_Folder_in_FTP_to_browse_to") # browse to folder containing your file for DL
then:
server.sendcmd("TYPE i") # ready for file transfer
server.retrbinary("RETR %s"%("FILENAME+EXT to DL"), open("DESTINATIONPATH+EXT", "wb").write) # this will transfer the selected file...to selected path/file
believe this is as correct as serves..
u can set server.set_debuglevel(0) to (1) or (2) for more detailed description while logged in to server.
I am able to download files from the FTP using the ftplib in Python, but this is like I hard code the name the name of the file(R.pdf) and this downloads only (R.pdf), is there a way to download all files in the FTP with the extension .PDF to my local system using Python. I am able to do this in Shell by just using *.pdf
Replace host, user and password with your credentials,
and 'public_html/soleil' with the address of the directory in which are the PDF files you want to be downloaded,
in the following code and it should be OK I think.
from ftplib import *
from os import listdir
from os.path import getsize
ftp_dt = FTP(host,user,password)
ftp_pi = FTP(host,user,password)
print '\n- Ouverture de connection et logging : OK'
ftp_dt.cwd('public_html/soleil')
ftp_pi.cwd('public_html/soleil')
def func(content, li = [0], la = [], si = [0], memname = ['']):
if name!=memname[0]:
memname[0],li[0:1],la[:],si[0:1] = name,[0],[],[0]
li[0] = li[0] + 1
si[0] = si[0] + len(content)
la.append(str(len(content)))
if li[0]%8==0:
print ' '.join(la) +\
' total: '+str(li[0])+' chunks, '+str(si[0])+' bytes'
la[:] = []
f.write(content)
li_files = []
for name in ftp_dt.nlst():
try:
ftp_dt.size(name)
if name not in ('.','..') and name[-4:]=='.pdf':
li_files.append(name)
except:
pass
if li_files:
for name in li_files:
print '\n- Downloading '+name
with open('E:\\PDF\\DOWNS\\'+name,'wb') as f:
ftp_pi.retrbinary('RETR '+name,func)
if getsize('E:\\PDF\\DOWNS\\'+name)==ftp_dt.size(name):
print ' OK ! Download of complete '+repr(name)+' SUCCEEDED'
else:
print ' FAILURE !! : '+name+' only partially downloaded'
else:
print '\nThere is no PDF file in this FTP directory'
ftp_dt.quit()
ftp_pi.quit()
Two connexions ftp_dt and ftp_pi are defined for “Data Transfers“ and “Protocol Interpretation“ because FTP protocol is based on two channels, one for the commands and the other for..... guess what ?
The func() function is used as callback in the fonction retrbinary()
It could be just
def func(content):
f.write()
but I played a bit with the possibilities of default variables of a function.
One thing I don’t understand well: how can this code work while the reference f in func() is only defined in the text of code after the definition of func() . But I tested it and it works !
I don't have access to an FTP server I can try this but a cursory look at the documentation indicates that this is not possible.
You can, however, obtain a list of files on the remote end with the dir or nlst commands and then fetch each file in a loop.
use two python modules glob and wget. Your snippet could look like this
import glob
import wget
list_to_download = glob.glob(url+'*.pdf')
for file in list_to_download:
wget.download(file)