I am trying to tail a log file on a remote server using a Python script. The log file rolls over very fast, I am using the python-sshtail module to tail the log file and capture the output of the tail in a file on my local machine. I was able to capture the log file and save it to a file on my local machine but the my script seems to be writing it twice and data is formatted.
The script is working but not the way I want it to, I should be able to run the script, perform some actions on the servers, tail the logs, save the output to a file on my local machine and kill the script using CTRL-C.
I did write some code and it does work but not the way it should. For now I am using time.sleep to wait for the output to be written to the output file on my local machine.
#!/usr/bin/python3
import time
import sshtail.tailers as t
import sys
import datetime
username = "username"
logfile = "/var/log/file.log"
k = t.paramiko.RSAKey.from_private_key_file('keyfile')
conn = t.SSHTailer(username,logfile, k, verbose=True)
try:
conn.connect()
print(f"Connected to {conn.host}")
print("Tailing the file..")
except:
print("Connection unsuccesful...")
conn.tail()
for line in conn.tail():
print(line)
for line in conn.get_new_lines():
print(line)
x = conn.remote_file_size
print(f"The file size is: {x}")
time.sleep(10)
output_file = str(conn.host)+"_"+str(datetime.datetime.now().strftime("%Y-%m-%d_%H:%M:%S"))+".txt"
with open(output_file, "a") as f:
for line in conn.get_new_lines():
print(line)
f.write(line)
conn.disconnect()
I recommend replacing time.sleep with asyncio.sleep
Related
I'm executing these lines of code on python terminal and it is working but when i execute it on webserver which is xampp i get output 255 in transcribe_text.txt file but when i run through terminal i get audio to text proper conversion. I'm working on windows environment.
import subprocess
out=subprocess.getstatusoutput(['deepspeech', '--model', 'C:/deepspeech/deepspeech-0.6.0-models/output_graph.pb', '--lm', 'C:/deepspeech/deepspeech-0.6.0-models/lm.binary', '--trie', 'C:/deepspeech/deepspeech-0.6.0-models/trie', '--audio', 'C:/deepspeech/audio/8455-210777-0068.wav'])
#open and read the file after the appending:
print(str(out))
f = open("transcribe_text.txt", "a")
f.write(str(out))
f.close()
#open and read the file after the appending:
f = open("transcribe_text.txt", "r")
print(f.read())
Any solution Please, Thanks
I want to edit a line in a text file in a Linux server using python. The process involves following steps
Telnet to the server (using telnetlib)
Go to the required directory
open the text file in the directory
set or unset the flag (YES or NO) of the variable in the text file based on the requirement
save the file and exit
I'm able to automate until step 2. However, I'm stuck at step 3 through 5.
I tried to mimic the steps I follow manually (using vim editor). But I'm not able to perform the 'ESC', replace and ':wq!' steps. Is there an alternative procedure to edit the file or any ways to improve upon mimicking the manual process
I have added my code here
host = input("Enter the IP address:")
port = input("Enter the port:")
tn = telnetlib.Telnet(host,port)
tn.write(b'\n')
tn.read_until(b"login:")
tn.write(b"admin" + b'\n')
tn.read_until(b"Password:")
tn.write(b"admin" + b'\n')
tn.write(b"version" + b'\n')
tn.write(b"path/to/file/" + b'\n')
# OPEN THE FILE and SET or RESET THE FLAG and CLOSE
with in_place.InPlace('filename.txt') as file:
for line in file:
line = line.replace('line_to_change', 'changed_data')
file.write(line)
print('Task executed')
I tried using the in-place library to set the flag but the programme is looking for the file in my local machine rather in the server. So it throws an error message indicating that the file is not present.
If you are able to connect to your remote server, the rest should work as follows:
with open('path/to/file','r') as fr:
data = fr.readlines() # returns list of lines
changed_data = ["changed_data\n" if line=="line_to_change\n" else line
for line in data]
with open('path/to/file','w') as fw:
for line in changed_data:
fw.write(line) # write the lines back to the back
In my code, user uploads file which is saved on server and read using the server path. I'm trying to delete the file from that path after I'm done reading it. But it gives me following error instead:
An error occurred while reading file. [WinError 32] The process cannot access the file because it is being used by another process
I'm reading file using with, and I've tried f.close() and also f.closed but its the same error every time.
This is my code:
f = open(filePath)
with f:
line = f.readline().strip()
tempLst = line.split(fileSeparator)
if(len(lstHeader) != len(tempLst)):
headerErrorMsg = "invalid headers"
hjsonObj["Line No."] = 1
hjsonObj["Error Detail"] = headerErrorMsg
data['lstErrorData'].append(hjsonObj)
data["status"] = True
f.closed
return data
f.closed
after this code I call the remove function:
os.remove(filePath)
Edit: using with open(filePath) as f: and then trying to remove the file gives the same error.
Instead of:
f.closed
You need to say:
f.close()
closed is just a boolean property on the file object to indicate if the file is actually closed.
close() is method on the file object that actually closes the file.
Side note: attempting a file delete after closing a file handle is not 100% reliable. The file might still be getting scanned by the virus scanner or indexer. Or some other system hook is holding on to the file reference, etc... If the delete fails, wait a second and try again.
Use below code:
import os
os.startfile('your_file.py')
To delete after completion:
os.remove('your_file.py')
This
import os
path = 'path/to/file'
with open(path) as f:
for l in f:
print l,
os.remove(path)
should work, with statement will automatically close the file after the nested block of code
if it fails, File could be in use by some external factor. you can use Redo pattern.
while True:
try:
os.remove(path)
break
except:
time.sleep(1)
There is probably an application that is opening the file; check and close the application before executing your code:
os.remove(file_path)
Delete files that are not used by another application.
SOLVED
It looks like I needed to close the sys.stdout file (which was open for writing) with a
sys.stdout.close()
and then re-open it, but in read mode, with
sys.stdout = open ('/home/myuser/verify_yslog_conf/remote_hostname/hi', 'r')
and then it uploaded correctly. I used a
time.sleep(60)
to cat the file mid-run for troubleshooting. It was empty until the def had finished, which by default closed the "hi" file. So I realized that I needed to close it to fully write the file. thanks.
Code below is meant to check if syslog is running, and if so, comment out any lines with "#" (but NOT #192.168.x.xx or 192.168.x.xxx) from the pulled remote syslog.conf file. After commenting those lines locally, it is supposed to upload the new file to the remote server in the old location. However, all that happens is that my /etc/syslog.conf file gets erased. The file isn't removed, it is just empty.
I know the issue is from the put statement
fabric.operations.put('/home/myuser/verify_yslog_conf/remote_hostname/hi', '/etc/syslog.conf', use_sudo=True)
but nothing looks wrong with it.
def verify():
#lines below are good; keep them
ip1 = '192.168.x.xx'
ip2 = '92.168.x.xxx'
#check if syslog is running
output = sudo("/sbin/service syslog status")
if 'is running...' in output:
#pull the syslog file from the remote server
fabric.operations.get('/etc/syslog.conf')
#read thru the file
fh = open('/home/myuser/verify_yslog_conf/remote_hostname/syslog.conf', 'r')
f = fh.read()
fh.close()
#if the file needs commenting ...
if re.search(r'(#(?!({0}|{1})))'.format(ip1, ip2), f):
#get the file again -- maybe the problem? was advised to do this
fabric.operations.get('/etc/syslog.conf')
#save the file locally for reading and then for writing (r+ failed)
conf = open ('/home/myuser/verify_yslog_conf/remote_hostname/syslog.conf', 'r')
sys.stdout = open ('/home/myuser/verify_yslog_conf/remote_hostname/hi', 'w')
#for every line in the old /etc/syslog.conf
for line in conf:
#if it has an # but not followed by the permitted IPs
if re.search(r'(#(?!({0}|{1})))'.format(ip1, ip2), line):
#comment out the line then print it (does it twice, dont care)
line = "#" + line
sys.stdout.write(line)
sys.stdout.write(line)
conf.close()
#upload the newly commented file to the remote host
fabric.operations.put('/home/myuser/verify_yslog_conf/remote_hostname/hi', '/etc/syslog.conf', use_sudo=True)
else:
print GOOD
else:
print RSYSLOG
Try to sys.stdout.close() immediately prior to the fabric.operations.put command.
It's likely that before the put operation, the data isn't flushed to the file, so it's sending an empty file, but when the script finishes, it automatically flushes (writes) the data which is why the file appears normal on the local system.
I have a Python program where the initiation script looks like this:
if __name__ == "__main__":
main(sys.argv[1:])
To run this, I have to use Shell or Terminal like this:
myscript somefile.xml
The script accepts a file and then does all the rest of the work.
Now, I am trying to run this program on a web server.
SO I use a HTML Form to submit the file to this script.
In my Python script, I am doing like this:
....
elif req.form.has_key("filename"):
item=req.form["filename"]
if item.file:
req.write("I GO HERE")
myscript.main(item)
....
As you can see here, I am trying to send the file directly to the "main" function.
Is this the right way to do?
I dont get any script error, but the Python script is not producing the expected results.
Any help?
Thanks
Write the uploaded file contents to a temporary file (using tempfile.mkstemp()) and pass the filename of the temporary file into main() wrapped in a list.
For example (untested):
import os
import tempfile
fd, temp_filename = tempfile.mkstemp()
try:
with os.fdopen(fd, "wb") as f:
# Copy file data to temp file
while True:
chunk = item.file.read(100000)
if not chunk: break
f.write(chunk)
# Call script's main() function
myscript.main([temp_filename])
finally:
os.remove(temp_filename)