I am trying to FTP a file from a windows box running python 3.3 to a linux box running Ubuntu 12.04.
I have been able to FTP the file, but the file on the remote server is always empty even though the local file has about 8k in it. Its a small text file.
I have tried a few suggestions from both Google and the SE network. All the different tests are commented out. Those have failed with various errors and/or similar results as the uncommented one. (0 byte file)
I also read in the txt file to ensure I was FTPing the right file.
Code is below. Any help would be greatly appreciated:
filename = 'Music.txt'
#write data to file
file = open('Music.txt', 'w+')
for music in musicList:
file.write(music+'\n')
file.close()
localfile = open(filename, 'r')
#print(localfile)
#for line in localfile:
# print(line)
#ftp file to webserver for consumption
ftp = FTP(host) # connect to host webdev (local)
ftp.login(username, password)
print(ftp.getwelcome())
ftp.cwd('projects/home/musicCat/data')
ftp.retrbinary("RETR Music.txt", localfile.write)
#ftp.storlines("STOR Music.txt", open('Music.txt'))
#ftp.storbinary("STOR Music.txt", localfile.write)
#ftp.storbinary("STOR Music.txt", file(file, "rb"))
#ftp.storbinary('STOR' + localfile.name, open(file.name, 'wb').write)
#ftp.storlines("STOR Music.txt", open(localfile, 'r'))
#ftp.storlines('STOR Music.txt', file)
print(ftp.retrlines('LIST'))
ftp.quit()
Interpreter Output:
220 (vsFTPd 2.3.5)
-rw-r--r-- 1 1000 1000 0 Dec 27 21:14 Music.txt
226 Directory send OK.
>>>
BUMP: Anyone have any ideas? I have yet to be able to successfully transfer the file via ftp. I'm debating about using python 2.7 and paramiko.
I'm sure you're looking for a cross-platform pure python approach, but sometimes it's easiest to just use python to execute an external utility via subprocess.call()
http://docs.python.org/3.3/library/subprocess.html
I had similar issues establishing a SSH tunnel from Windows, and really did not want to switch to 2.7 to use paramiko. I ended up just calling plink.exe from my script.
For ftp on Windows, you can use the built-in utility ftp
Transfers files to and from a computer running an FTP server service
(sometimes called a daemon). Ftp can be used interactively.
FTP [-v] [-d] [-i] [-n] [-g] [-s:filename] [-a] [-A] [-x:sendbuffer] [-r:recvbuffer] [-b:asyncbuffers] [-w:windowsize] [host]
-v Suppresses display of remote server responses.
-n Suppresses auto-login upon initial connection.
-i Turns off interactive prompting during multiple file
transfers.
-d Enables debugging.
-g Disables filename globbing (see GLOB command).
-s:filename Specifies a text file containing FTP commands; the
commands will automatically run after FTP starts.
-a Use any local interface when binding data connection.
-A login as anonymous.
-x:send sockbuf Overrides the default SO_SNDBUF size of 8192.
-r:recv sockbuf Overrides the default SO_RCVBUF size of 8192.
-b:async count Overrides the default async count of 3
-w:windowsize Overrides the default transfer buffer size of 65535.
host Specifies the host name or IP address of the remote
host to connect to.
Notes:
- mget and mput commands take y/n/q for yes/no/quit.
- Use Control-C to abort commands.
Related
I'm currently utilising Paramiko and SCPClient in Python to transfer a directory from one server to another as a means of backup. This works well however I do not want it to copy hidden files (.file_name) or symbolic links. Is this possible?
Unfortunately rsync isn't an option for me as it's not available on either of the remote servers I connect to. My script is below (sensitive info replaced with dummy data). Note I need to connect to a jump host before being able to connect to target_1 or target_2.
import os
import shutil
import time
import paramiko
from scp import SCPClient
#set up ssh variables
j_host = '00.00.00.00'
target_host_1 = '00.00.00.001'
target_host_2 = '00.00.00.002'
port_no = 22
username = ''
passw = ''
#set up temporary folder on local machine to store files
path = "/local_path/"
os.mkdir(path)
#create SSH Client for jump server
jump_host=paramiko.SSHClient()
jump_host.set_missing_host_key_policy(paramiko.AutoAddPolicy())
jump_host.connect(j_host, username=username, password=passw)
#set up channel to connect to 1 via jump server
jump_host_transport_1 = jump_host.get_transport()
src_addr = (j_host, port_no)
dest_addr_1 = (target_host_1, port_no)
jump_host_channel_1 = jump_host_transport_1.open_channel("direct-tcpip", dest_addr_1, src_addr)
#set up channel to connect to 2 via jump server
jump_host_transport_2 = jump_host.get_transport()
dest_addr_2 = (target_host_2, port_no)
jump_host_channel_2 = jump_host_transport_2.open_channel("direct-tcpip", dest_addr_2, src_addr)
#function which sets up target server, either 1 or 2
def create_SSHClient(server, port, user, password, sock):
target=paramiko.SSHClient()
target.set_missing_host_key_policy(paramiko.AutoAddPolicy())
target.connect(server, port, user, password, sock=sock)
return target
#invoke above function to set up connections for 1 & 2
ssh_1 = create_SSHClient(target_host_1, port_no, username, passw, jump_host_channel_1)
ssh_2 = create_SSHClient(target_host_2, port_no, username, passw, jump_host_channel_2)
#delete old files in backup folder
command = "rm -rf /filepath/{*,.*}"
stdin, stdout, stderr = ssh_2.exec_command(command)
lines = stdout.readlines()
#print(lines)
#pause to ensure old directory is cleared
time.sleep(5)
#SCPCLient takes a paramiko transport as an argument, sets up file transfer connection
scp_1 = SCPClient(ssh_1.get_transport())
scp_2 = SCPClient(ssh_2.get_transport())
#get files from 1, store on local machine, put on 2
scp_1.get('/filepath/.', '/target_folder_local/', recursive=True)
scp_2.put('/target_folder_local/.', '/filepath/', recursive=True)
#remove temporary folder
shutil.rmtree(path)
#close connections
ssh_1.close()
ssh_2.close()
jump_host.close()
There's no API in SCPClient to skip hidden files or symbolic links.
For upload, it's easy, if you copy the SCPClient's code and modify it as you need. See the os.walk loop in _send_recursive function.
If you do not want to modify the SCPClient's code, you will have to iterate the files on your own, calling SCPClient.put for each. It will be somewhat less efficient, as it will start new SCP server for each file.
For download, you might be able to modify the SCPClient code to respond with non-zero code to C commands fed by the server for the files you do not want to download.
Check the _recv_file function. There where name is resolved, check for names or attributes of files you are not interested in downloading and do chan.send('\x01') and exit the function.
Though why do you want to use SCP? Use SFTP. It is much better suited for custom rules you need.
Paramiko does not have recursive SFTP transfer functionality (But pysftp does, see pysftp vs. Paramiko). But you won't be able to use it anyway, for the same reason you cannot use it with SCP. For your specific needs.
But check my answer to Python pysftp get_r from Linux works fine on Linux but not on Windows. It shows a simple recursive SFTP download code. Just modify it slightly to skip the files you do not want to download.
Something like
if (not S_ISLNK(mode)) and (not entry.filename.startswith(".")):
(see Checking if a file on SFTP server is a symbolic link, and deleting the symbolic link, using Python Paramiko/pysftp)
I'm a software tester, trying to verify that the log on a remote QNX (a BSD variant) machine will contain the correct entries after specific actions are taken. I am able to list the contents of the directory in which the log resides, and use that information in the command to read (really want to use tail -n XX <file>) the file. So far, I always get a "(No such file or directory)" when trying to read the file.
We are using Froglogic Squish for automated testing, because the Windows UI (that interacts with the server piece on QNX) is built using Qt extensions for standard Windows elements. Squish uses Python 2.7, so I am using Python 2.7.
I am using paramiko for the SSH connection to the QNX server. This has worked great for sending commands to the simulator piece that also runs on the QNX server.
So, here's the code. Some descriptive names have been changed to avoid upsetting my employer.
import sys
import time
import select
sys.path.append(r"C:\Python27\Lib\site-packages")
sys.path.append(r"C:\Python27\Lib\site-packages\pip\_vendor")
import paramiko
# Import SSH configuration variables
ssh_host = 'vvv.xxx.yyy.zzz'
thelog_dir = "/logs/the/"
ssh_user = 'un'
ssh_pw = 'pw'
def execute_Command(fullCmd):
outptLines = []
#
# Try to connect to the host.
# Retry a few times if it fails.
#
i = 1
while True:
try:
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(ssh_host, 22, ssh_user, ssh_pw)
break
except paramiko.AuthenticationException:
log ("Authentication failed when connecting to %s" % ssh_host)
return 1
except:
log ("Could not SSH to %s, waiting for it to start" % ssh_host)
i += 1
time.sleep(2)
# If we could not connect within time limit
if i == 30:
log ("Could not connect to %s. Giving up" % ssh_host)
return 1
# Send the command (non-blocking?)
stdin, stdout, stderr = ssh.exec_command(fullCmd, get_pty=True)
for line in iter(stdout.readline, ""):
outptLines.append(line)
#
# Disconnect from the host
#
ssh.close()
return outptLines
def get_Latest_Log():
fullCmd = "ls -1 %s | grep the_2" %thelog_dir
files = execute_Command(fullCmd)
theFile = files[-1]
return theFile
def main():
numLines = 20
theLog = get_Latest_Log()
print("\n\nThe latest log is %s\n\n" %theLog)
fullCmd = "cd /logs/the; tail -n 20 /logs/the/%s" %theLog
#fullCmd = "tail -n 20 /logs/the/%s" %theLog
print fullCmd
logLines = execute_Command(fullCmd)
for line in logLines:
print line
if __name__ == "__main__":
# execute only if run as a script
main()
I have tried to read the file using both tail and cat. I have also tried to get and open the file using Paramiko's SFTP client.
In all cases, the response of trying to read the file fails -- despite the fact that listing the contents of the directory works fine. (?!) And BTW, the log file is supposed to be readable by 'world'. Permissions are -rw-rw-r--.
The output I get is:
"C:\Users\xsat086\Documents\paramikoTest>python SSH_THE_MsgChk.py
The latest log is the_20210628_115455_205.log
cd /logs/the; tail -n 20 /logs/the/the_20210628_115455_205.log
(No such file or directory)the/the_20210628_115455_205.log"
The file name is correct. If I copy and paste the tail command into an interactive SSH session with the QNX server, it works fine.
Is it something to do with the 'non-interactive' nature of this method of sending commands? I read that some implementations of SSH are built upon a command that offers a very limited environment. I don't see how that would impact this tail command.
Or am I doing something stupid in this code?
I cannot really explain completely, why you get the results you get.
But in general a corrupted output is a result of enabling and not handling terminal emulation. You enable the terminal emulation using get_pty=True. Remove it. You should not use the terminal emulation, when automating command execution.
Related question:
Is there a simple way to get rid of junk values that come when you SSH using Python's Paramiko library and fetch output from CLI of a remote machine?
i am writing a python script that helps me download files from sftp server to my local folder.
when i run the script it just downloads the blank document
i keep on trying but i am failing
i made a sftp server to test the code and the path specified in remote path is server root directory
i giving code below
import pysftp
myHostname = "192.168.56.1"
myUsername = "new45"
myPassword = "146515"
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
with pysftp.Connection(host=myHostname, username=myUsername, password=myPassword,cnopts=cnopts) as sftp:
# Define the file that you want to download from the remote directory
remoteFilePath = 'C:\\Users\\Simon\\Desktop\\ftp\\testfile.txt'
# Define the local path where the file will be saved
# or absolute "C:\Users\sdkca\Desktop\TUTORIAL.txt"
localFilePath = 'C:\\Users\\Simon\\Desktop\\ftp2\\textfile.txt'
sftp.get(remoteFilePath, localFilePath)
# connection closed automatically at the end of the with-block'''
please tell me whats the error why is blank file is being downloaded
Login to that server using any GUI SFTP client. And observe what path syntax your server is using. It's definitely not C:\Users\Simon\Desktop\ftp\testfile.txt. SFTP uses forward slashes. So it can be like /C:/Users/Simon/Desktop/ftp/testfile.txt. But even something completely different, particularly if the server or your account are chrooted.
You are probably trying to use the path syntax, which you would have used, were you directly on that target Windows system. But your SFTP server presents the files using a different path syntax. You need to adhere to that syntax.
I have a .yml file present in a remote server , I want to make changes on it using python fabric. If it can be done with other python libraries feel free to share.
Thank you
You are trying to edit a line in the middle of a file which is imo is not possible.
What you can do, is making a copy of the remote file on your local machine with the desired values you want to change, and then send it back to the remote server.
from fabric import Connection as connection, task
#task
def executeTask(ctx):
with connection(host=dev_server, user=myuser) as c:
c.put('PATH_TO_YOUR_YML_FILE_LOCALLY', 'PATH_TO_YOUR_REMOTE_YML_FILE')
Don't forget to :
Replacedev_server and myuser with the remote server IP and username on it
put the code above in a file called fabfile.py and you run from your command line fab executeTask
The code above is fabric 2.4 compatible
EDIT:
Because of permissions issue you can do the following :
#task
def executeTask(ctx):
with connection(host=dev_server, user=myuser) as c:
c.put("PATH_TO_YOUR_YML_FILE_LOCALLY") # implicit to remote $HOME
c.sudo("mv YOUR_FILE_NAME YOUR_DESIRED_LOCATION") # again implicitly with a CWD of $HOME
c.sudo("chown root:root YOUR_REMOTE_FILE")
Referenc:
https://github.com/fabric/fabric/issues/1750#issuecomment-406043571
If you just need to change port number you can use sed like this
def change_port(filename):
with cd('/location'):
run('sed -i "s/old_port_number/new_port_number/g" ' +filename)
I am trying to upload a file to an FTP server, I wanted to write a Python script to simplify the process. Currently I ftp to the server and execute these commands:
quote allo file_size_in_bytes
put c:\path\to\file
This is what I have so far, I am not able to get the file to transfer using the put command.
from ftplib import FTP
import os
import time
from subprocess import call
ip = raw_input('Enter ip address: ') # User input for host
print ip # Prints host
filelocation = raw_input('Drag file here: ') # File to upload
print ('This is the local file '+filelocation) # Verify file location
filename = os.path.basename(filelocation) # If a File name is needed
filesize = os.path.getsize(filelocation) # Need file size in bytes for quote allo command
print ('This is the file size in bytes '+str(filesize)) # Verify correct size
time.sleep(2) # Pause to look at screen
ftp = FTP(ip) # Open ftp connection
ftp.login('user','pass') # Login
print ftp.getwelcome() # Verify proper connection
remotepath = ftp.pwd()+filename # If a remote path to ftp server is needed
print ('This is the file path on the processor '+remotepath) # Verify remote path
"\"quote\"" # Need this, not sure why
ftp.sendcmd('allo '+str(filesize)) # quote allo filesize, seems to work
#"\"put\"" # Experimenting, don't know if this is needed
call(['echo "put C:\filelocation" | ftp']) # This doesn't appear to work
time.sleep(5)
ftp.quit()
You are logging in with ftplib, yet you are trying to run an external ftp process for the upload itself. That cannot work as the external ftp process does not know about your ftplib FTP session.
To upload a file using the ftplib, use storbinary or storlines method.
See for example Python Script Uploading files via FTP.