I am sending a file from one Linux VM to another Linux VM using python which is doing great. Files are sent successfully BUT before I send the file, I want to change the file encoding to "UTF-8" and line ending to "Unix/Linux". How to do that?
Below is the piece of code that is sending the file through sftp:
with pysftp.Connection(host=host, username=userName, password=passWord) as sftpVal:
#print(sftpVal.listdir()) #list directories in sftp home
sftpVal.put(source_file_path,'incoming/'+fileName) #(localPath, destinationPath)
One option would be to read the file contents, change them as you need, save them to a temp file, and send the temp file instead. For example:
with open('file_name.txt') as original_file, open('file_to_send.txt', 'w') as send_file: # Open the original and prepared files
content = original_file.read()
content = content.replace('\r\n', '\n') # Change the line endings
content = content.encode('utf-8') # Encode in UTF-8
send_file.write(content)
# Do your file send here but with the prepared file instead.
os.remove('file_to_send.txt') # Optional removal of temp file.
Related
I want to download config file from my router via web scraping. The procedure I want to achieve is this:
Save the config file into disk
Send a factory reset
Load the config file previously downloaded.
So far, I have this code:
with requests.Session() as s: # To login into the modem
pagePostBackUp = 'https://192.168.1.1/goform/BackUp'
s.post(urlLogin, data=loginCredentials, verify=False, timeout=5)
dataBackUp = {'dir': 'admin/','file': 'cmconfig.cfg'}
resultBackUp = s.post(pagePostBackUp, data=dataBackUp, verify=False, timeout=10)
print(resultBackUp.text)
The last line is what I want to save into a file. But, when I try to do it with this code:
f = open('/Users/user/Desktop/file.cfg', 'w')
Throws an error that ascii codec can't encode character. If I save the file with, for example, encode='utf16', differs from what I originally download manually.
So, the question is, How can I save this file with the same encoding the router gives me via web? (As unicode). The content of the file looks like this:
�����g���m��� ������Z������ofpqJ
U\V,.o /����zf��v���~W3=,�D};y�tL�cJ
Change the last line of your code to the following:
with open('/Users/user/Desktop/file.cfg', 'wb') as f:
f.write(resultBackUp.content)
This will treat the payload as data (bytes), not text: the file is opened in binary mode, and the content is taken as-is.
There's no encoding/decoding happening.
SOLVED
It looks like I needed to close the sys.stdout file (which was open for writing) with a
sys.stdout.close()
and then re-open it, but in read mode, with
sys.stdout = open ('/home/myuser/verify_yslog_conf/remote_hostname/hi', 'r')
and then it uploaded correctly. I used a
time.sleep(60)
to cat the file mid-run for troubleshooting. It was empty until the def had finished, which by default closed the "hi" file. So I realized that I needed to close it to fully write the file. thanks.
Code below is meant to check if syslog is running, and if so, comment out any lines with "#" (but NOT #192.168.x.xx or 192.168.x.xxx) from the pulled remote syslog.conf file. After commenting those lines locally, it is supposed to upload the new file to the remote server in the old location. However, all that happens is that my /etc/syslog.conf file gets erased. The file isn't removed, it is just empty.
I know the issue is from the put statement
fabric.operations.put('/home/myuser/verify_yslog_conf/remote_hostname/hi', '/etc/syslog.conf', use_sudo=True)
but nothing looks wrong with it.
def verify():
#lines below are good; keep them
ip1 = '192.168.x.xx'
ip2 = '92.168.x.xxx'
#check if syslog is running
output = sudo("/sbin/service syslog status")
if 'is running...' in output:
#pull the syslog file from the remote server
fabric.operations.get('/etc/syslog.conf')
#read thru the file
fh = open('/home/myuser/verify_yslog_conf/remote_hostname/syslog.conf', 'r')
f = fh.read()
fh.close()
#if the file needs commenting ...
if re.search(r'(#(?!({0}|{1})))'.format(ip1, ip2), f):
#get the file again -- maybe the problem? was advised to do this
fabric.operations.get('/etc/syslog.conf')
#save the file locally for reading and then for writing (r+ failed)
conf = open ('/home/myuser/verify_yslog_conf/remote_hostname/syslog.conf', 'r')
sys.stdout = open ('/home/myuser/verify_yslog_conf/remote_hostname/hi', 'w')
#for every line in the old /etc/syslog.conf
for line in conf:
#if it has an # but not followed by the permitted IPs
if re.search(r'(#(?!({0}|{1})))'.format(ip1, ip2), line):
#comment out the line then print it (does it twice, dont care)
line = "#" + line
sys.stdout.write(line)
sys.stdout.write(line)
conf.close()
#upload the newly commented file to the remote host
fabric.operations.put('/home/myuser/verify_yslog_conf/remote_hostname/hi', '/etc/syslog.conf', use_sudo=True)
else:
print GOOD
else:
print RSYSLOG
Try to sys.stdout.close() immediately prior to the fabric.operations.put command.
It's likely that before the put operation, the data isn't flushed to the file, so it's sending an empty file, but when the script finishes, it automatically flushes (writes) the data which is why the file appears normal on the local system.
I'm appending values to a log file every 6th second. Every 30 sec I'm transferring this log to an FTP server as a file. But instead of transfering the whole file, I just want to append the collected data to the file on my server. I haven't been able to figure out how to open the server file and then append the values.
My code so far:
session = ftplib.FTP(authData[0],authData[1],authData[2])
session.cwd("//"+serverCatalog()+"//") # open server catalog
file = open(fileName(),'rb')
with open(fileName(), 'rb') as f:
f = f.readlines()
for line in f:
collected = line
# In some way open server file, write lines to it
session.storbinary('STOR ' + fileName(), open(fileName(), 'a'), 1024)
file.close()
session.quit()
Instead, do I have to download the server file open and append, then send it back?
Above was my question, the full solution is below:
session.cwd("//"+serverCatalog()+"//") # open server catalog
localfile = open("logfile.txt",'rb')
session.storbinary('APPE serverfile.txt', localfile)
localfile.close()
Use APPE instead of STOR.
Source: http://www.gossamer-threads.com/lists/python/python/22960 (link to web.archive.org)
I have a text file with data such as
b'\x00\x09\x00\xfe'
This was piped into a text file from a TCP socket stream. Call this text file 'stream.txt'. I opened this file with the following code:
f = open("stream.txt", "rb")
bytes_read = f.read()
When I open this file within another Python script, I get a '\' for each '\' in the original file. On top of this, I cannot access the bytes array as such, since it appears to have become a string. That is, 'bytes_read' is now
'b"\\x00\\x09\\x00\\xfe"'
How can I recover this string as a bytes array?
The client code I used to capture this data is the following script:
from socket import *
clientsock = socket(AF_INET, SOCK_STREAM)
clientsock.connect(('1.2.3.4', 2000)) # Open the TCP socket
clientsock.sendall(b'myCommand') # Send a command to the server
data = clientsock.recv(16) # Wait for the response
print(data) # For piping to 'stream.txt'
clientsock.close()
As the data was printed to the terminal, I redirected it to a file:
$ python3 client.py > stream.txt
My goal is to bypass the redirect into text file and pipe directly into a plotter... But first I wanted to get this to work.
Was able to solve this by writing directly to a file. So rather than using 'print(data)', and redirecting to a file, I tried this:
file = open("rawData", "wb")
...
file.write(data)
...
file.close()
Was able to process "rawData" as expected.
this below code is working fine, but the issue is that the transferred file has no spaces , everything is in single line
ftp =FTP('x.x.x.x')
ftp.login('user','password')
filename = "test.txt"
local = open(filename, 'wb')
ftp.cwd('/root/Desktop')
ftp.retrbinary('RETR ' +filename, local.write,1024)
say test.txt had
1)one
2)two
3)three
the test.txt thats copied from the reemote server has
1)one2)two3)three
everything in one line,
any help pls.
To quote the internet:
http://www.cs.toronto.edu/~krueger/csc209h/tut/line-endings.html
Text files created on DOS/Windows machines have different line endings
than files created on Unix/Linux. DOS uses carriage return and line
feed ("\r\n") as a line ending, which Unix uses just line feed ("\n").
It's the writing binary that's causing your problem. It prevents the translation which should be automatic when writing as text. Can you transfer and save as a text file? You can use retrlines instead of retrbinary.
ftp =FTP('x.x.x.x')
ftp.login('user','password')
filename = "test.txt"
local = open(filename, 'w')
ftp.cwd('/root/Desktop')
ftp.retrlines('RETR ' +filename, local.write)