I have set up syslog-ng to tail a file and forward each new event to python script which will process it.
My syslog-ng.conf looks like this -
source s_src {
file("/var/log/xyz.log" flags(no_parse);
};
destination d_dest {
program("python -u /home/user1/processlog.py" flush_lines(1) flags(no_multi_line));
};
log { source(s_src); destination(d_dest); };
And processlog.py just contains
#!/usr/bin/python
import sys
f1 = open('success.txt', 'a')
while 0 < 1:
try:
line = sys.stdin.readline()
f1.write(line)
except Exception, e:
f = open('/tmp/error.txt','ab')
f.write(e)
f.close()
exit(0)
This script works prefectly fine from command line. Takes each input and writes to success.txt.
Syslog-ng also starts but does not forward the event to python program above. It starts the program though.
ps -ef| grep processlog
root 6242 6236 0 13:00 ? 00:00:00 /bin/sh -c python -u /home/user1/processlog.py
root 6244 6242 0 13:00 ? 00:00:00 python -u /home/user1/processlog.py
I have checked all permissions too. But whenever new event happens in xyz.log, it is not getting forwarded to python script, which i am testing via writing to success.txt
Any leads are highly appreciated.
Try to run syslog-ng in foreground mode with enabled debug messages
syslog-ng -Fedtv
Perhaps you will get information about why the program destination doesn't get the messages.
By the way, is it possible that the syslog-ng read this file once?
In this case you should delete the persist file (var/syslog-ng.persist) to force the syslog-ng to read this file from beginning.
Related
I am currently working on a small Python Application which turns a Raspberry Pi 4B into a 48 Channel Audio-Recorder. Basics work, but during Recording, I need a log file which tells me when recording started, which ALSA warnings occurred and when recording stopped.
The recorder can be started with this terminal command:
pi#raspberrypi:~ $ rec -q -t caf --endian little --buffer 96000 -c 48 -b 24 /home/pi/myssd-one/Aufnahmen/test.caf 2>&1 | tee /home/pi/myssd-one/Aufnahmen/logging.log
this records audio in the test.caf file and writes ALSA warnings to logging.log
So far so good.
The Python Program (which should run on a touchscreen with GUI so recording can easily started and stopped) takes care of variable audio-filenames (date-time-stamp) and controls an LED to show that recording is running.
This part of the code takes care of switching on and off:
#!/usr/bin/env python
from tkinter import *
import shlex
import os
import subprocess
import tkinter.font
import datetime
from gpiozero import LED
import RPi.GPIO as GPIO
GPIO.setmode(GPIO.BCM)
GPIO.setup(11, GPIO.OUT)
def ledToggle():
if led.is_lit:
led.off()
my_env = os.environ.copy()
my_env['AUDIODRIVER'] = 'alsa'
my_env['AUDIODEV'] = 'hw:KTUSB,0'
ledButton["text"] = "Turn Recorder on"
print ("recorder stops")
subprocess.Popen(['sudo', 'pkill', '-SIGINT', 'rec'], env = my_env, shell = FALSE, stdout=subprocess.PIPE)
else:
led.on()
my_env = os.environ.copy()
my_env['AUDIODRIVER'] = 'alsa'
my_env['AUDIODEV'] = 'hw:KTUSB,0'
ledButton["text"] = "Turn Recorder off"
print ("recorder starts")
##reference statement command line: "rec -q -t caf --endian little --buffer 96000 -c 48 -b 24 /home/pi/myssd-one/Aufnahmen/test.caf 2>&1 | tee /home/pi/myssd-one/Aufnahmen/logging.log"
command_line = shlex.split("rec '-q' '-t' 'caf' '--buffer' '96000' '-c 48' '-b 24' '/home/pi/myssd-one/Aufnahmen/test.caf' '"2>&1 | tee"' '/home/pi/myssd-one/Aufnahmen/logging.log'")
p1 = subprocess.Popen(command_line, env = my_env, shell = False, stdout=subprocess.PIPE)
I am trying to move the original command line statement into the subprocess.Popen command, to no success yet. The part where routing to the log file is done, fails. It looks as the initiating sox-application 'rec' tries to interpret it as part of its own parameter list, instead of interpreting it as a redirection of stdout and stderr to the log file. I appreciate some guidance in this issue.
Variable Filenames for audio files is already done, but for simplicity taken out of this code snippet.
Thanks Mark, I dived into this command line along your hint that it only can run with shell=True and this implied that it had to be written as a full statement without separating commas and escape quotes. Now it works. Actually, the shlex.split() becomes obsolete.
Attempting to run a Python script on boot on Raspberry Pi 3B+ 1GB RAM, Raspbian, with a SunFounder 10" Touch Screen, - .log file returns "Bad display name'
Python script is 100% functional when run via Terminal / executable script / Thonny etc. Attempted to run at boot first via rc.local - created a service, enabled service, daemon-reload... etc. Did not work.
Tried to run as crontab, same result - .log output from crontab shows "Bad display name". Thought it was lack of Display Environment imported and declared within the Python script, so I added that - but on boot returns the same result.
This is the Python Script I'm using
#!/usr/bin/env python3
import RPi.GPIO as GPIO
import os
import sys
import webbrowser
import time
import subprocess
from pynput import keyboard
from Xlib.display import Display
#GPIO Readout
GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)
#GPIO Header Setup
header = 2
GPIO.setup(header, GPIO.IN)
#Omxplayer Commands
Loop = 'omxplayer -b --loop --no-osd -o hdmi /home/pi/Videos/PlanetEarth.mp4 > /dev/null'
Donation = 'omxplayer -b --no-osd -o hdmi /home/pi/Videos/Cartoon.mp4 > /dev/null'
KillPlayer = 'pkill omxplayer.bin'
KillForm = 'pkill chromium'
#Set Display Environment
new_env = dict(os.environ)
new_env['DISPLAY'] = ':0.0'
#Form Handling Required Below
#If Donation is successful, Stop Looping Video, Open Form in Chromium, Wait 60 seconds, Close Chromium, Restart Loop
def PullDownSuccess():
subprocess.Popen(KillPlayer, env=new_env, shell=True)
time.sleep(2)
webbrowser.open('<url>')
time.sleep(60)
subprocess.Popen(KillForm, env=new_env, shell=True)
time.sleep(2)
subprocess.Popen(Loop, env=new_env, shell=True)
#Inception
subprocess.Popen(Loop, env=new_env, shell=True)
#Terminate Loop with Escape Key or Manually Initiate Donation Success
def on_press(key):
if key == keyboard.Key.ctrl:
PullDownSuccess()
if key == keyboard.Key.esc:
subprocess.Popen(KillPlayer, shell=True)
#Keyboard Listener Module
with keyboard.Listener(
on_press=on_press) as listener:
listener.join()
#Donation Successful Do:
while True:
header_state = GPIO.input(header)
if header_state == GPIO.HIGH:
PullDownSuccess()
I am currently attempting to run this script on Boot via crontab with this line:
#reboot (/bin/sleep 10; /usr/bin/python3 /home/pi/Custom_Scripts/<script>.py > /home/pi/Custom_Scripts/cronjoblog 2>&1)
The error log file for crontab returns the following:
raise error.DisplayNameError(display)
Xlib.error.DisplayNameError: Bad display name ""
This error only exists when attempting to run script on boot up. Is the Display overriding the boot display permissions on Boot Up? What is keeping the script from running on the Display on boot up, but not when remotely executed? Thank you for your consideration.
Update: Still no solution. Display environment returns ":0.0' ... so far I have tried to remove
> /dev/null from #Omxplayer Commands
Replacing crontab startup line to:
DISPLAY=":0" /usr/bin/python3 /home/pi/Custom_Scripts/<script>.py
and
DISPLAY=":0.0" /usr/bin/python3 /home/pi/Custom_Scripts/<script>.py
And any possible combination of these.
Confirmed the script is not waiting for any background processes as I have added delay (time.sleep) up to 30 seconds, as well as returning IP addresses, etc.
Returns either Bad Display Name still OR "Can't connect to display ":0": b'Invalid MIT-MAGIC-COOKIE-1 key"
Still looking for a solution if anyone has one.
EDIT:
Fixed using /LXDE-pi/autostart. Answer below.
Try to add DISPLAY environment variable before calling your script:
DISPLAY=":0" /usr/bin/python3 /home/pi/Custom_Scripts/<script>.py
DISPLAY="/dev/null" /usr/bin/python3 /home/pi/Custom_Scripts/<script>.py
Have you tried a longer sleep?
If you try to get your ip adress during your startup script with (for example) :
import socket
def get_ip_address():
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(("8.8.8.8", 80))
return s.getsockname()[0]
you will get an errror like:
File "/usr/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
error: [Errno 101] Network is unreachable
This is because at reboot network is not necessarily available. In order to make it available
see https://raspberrypi.stackexchange.com/questions/45769/how-to-wait-for-networking-on-login-after-reboot :
sudo raspi-config
then select option 3 Boot Options
then select option 4 Wait for network at boot
Once you have done that networking should not be an issue at reboot.
You get also check for other ways to enforce network setup before running your script take a look at :
https://askubuntu.com/questions/3299/how-to-run-cron-job-when-network-is-up
If there are unforeseen ramifications for using this method, I will update this thread accordingly. I fixed this issue just by adding two lines to the autostart file which resides in /etc/sdg/lxsession/LXDE-pi/autostart.
sudo nano /etc/xgd/lxsession/LXDE-pi/autostart
Add these two lines, do not modify the existing code.
sleep 5
#/usr/bin/python3 /home/pi/Custom_Scripts/<script>.py
My autostart file looks like this:
#lxpanel --profile LXDE-pi
#pcmanfm --desktop --profile LXDE-pi
#xscreensaver -no-splash
point-rpi
sleep 5
#usr/bin/python3 /home/pi/Custom_Scripts/<script>/py
Ctrl + O to write file
sudo reboot
I'm writing a command line tool (let's call it interactive_rm), that is supposed to read file paths line by line from STDIN so that it can be used with Unix pipes like in the following example:
$ find . | interactive_rm
I'm currently reading each path from STDIN like this:
def _parse_stdin():
for line in sys.stdin:
yield prepare_line(line)
Now the problem... Before the tool removes a path it should ask the user for confirmation. To do so I would use input() like this:
for path in _parse_stdin():
print('Do you want to delete this path: [y/n]\n' + path)
answer = input()
if answer == 'y':
delete(path)
But this doesn't work, because STDIN is already occupied by the pipe, so the input() function is skipped or I end up with an "EOFError: EOF when reading a line".
Does anyone know a solution to this?
When stdin is redirected, it needs to reopen the terminal to be able to read from it, e.g.:
from __future__ import print_function
import os
def main():
tty = os.open("/dev/tty", os.O_RDONLY)
while True:
r = os.read(tty, 1024)
if not r: # Wait for Ctrl-D.
break
print("----", r)
if __name__ == "__main__":
main()
And run it like the following to test it:
python t.py < /dev/null
Note that the terminal may be unavailable. For example, if the command is run through a ssh session without allocating a terminal as ssh -T ... command.
So I have this python3 script that does a lot of automated testing for me, it takes roughly 20 minutes to run, and some user interaction is required. It also uses paramiko to ssh to a remote host for a separate test.
Eventually, I would like to hand this script over to the rest of my team however, it has one feature missing: evidence collection!
I need to capture everything that appears on the terminal to a file. I have been experimenting with the Linux command 'script'. However, I cannot find an automated method of starting script, and executing the script.
I have a command in /usr/bin/
script log_name;python3.5 /home/centos/scripts/test.py
When I run my command, it just stalls. Any help would be greatly appreciated!
Thanks :)
Is a redirection of the output to a file what you need ?
python3.5 /home/centos/scripts/test.py > output.log 2>&1
Or if you want to keep the output on the terminal AND save it into a file:
python3.5 /home/centos/scripts/test.py 2>&1 | tee output.log
I needed to do this, and ended up with a solution that combined pexpect and ttyrec.
ttyrec produces output files that can be played back with a few different player applications - I use TermTV and IPBT.
If memory serves, I had to use pexpect to launch ttyrec (as well as my test's other commands) because I was using Jenkins to schedule the execution of my test, and pexpect seemed to be the easiest way to get a working interactive shell in a Jenkins job.
In your situation you might be able to get away with using just ttyrec, and skip the pexpect step - try running ttyrec -e command as mentioned in the ttyrec docs.
Finally, on the topic of interactive shells, there's an alternative to pexpect named "empty" that I've had some success with too - see http://empty.sourceforge.net/. If you're running Ubuntu or Debian you can install empty with apt-get install empty-expect
I actually managed to do it in python3, took a lot of work, but here is the python solution:
def record_log(output):
try:
with open(LOG_RUN_OUTPUT, 'a') as file:
file.write(output)
except:
with open(LOG_RUN_OUTPUT, 'w') as file:
file.write(output)
def execute(cmd, store=True):
proc = Popen(cmd.encode("utf8"), shell=True, stdout=PIPE, stderr=PIPE)
output = "\n".join((out.decode()for out in proc.communicate()))
template = '''Command:\n====================\n%s\nResult:\n====================\n%s'''
output = template % (cmd, output)
print(output)
if store:
record_log(output)
return output
# SSH function
def ssh_connect(start_message, host_id, user_name, key, stage_commands):
print(start_message)
try:
ssh.connect(hostname=host_id, username=user_name, key_filename=key, timeout=120)
except:
print("Failed to connect to " + host_id)
for command in stage_commands:
try:
ssh_stdin, ssh_stdout, ssh_stderr = ssh.exec_command(command)
except:
input("Paused, because " + command + " failed to run.\n Please verify and press enter to continue.")
else:
template = '''Command:\n====================\n%s\nResult:\n====================\n%s'''
output = ssh_stderr.read() + ssh_stdout.read()
output = template % (command, output)
record_log(output)
print(output)
I wrote a python script to run a command called "gtdownload" on a bunch of files with multiprocessing. The function "download" is where I am having trouble with.
#/usr/bin/env python
import os, sys, subprocess
from multiprocessing import Pool
def cghub_dnld_file(file1, file2, file3, np):
<open files>
<read in lines>
p = Pool(int(np))
map_args = [(line.rstrip(),name_lines[i].rstrip(),bar_lines[i].rstrip()) for i, line in enumerate(id_lines)]
p.map(download_wrapper,map_args)
def download(id, name, bar):
<check if file has been downloaded, if not download>
<.....>
link = "https://cghub.ucsc.edu/cghub/data/analysis/download/" + id
dnld_cmd = "gtdownload -c ~/.cghub.key --max-children 4 -vv -d " + link + " > gt.out 2>gt.err"
subprocess.call(dnld_cmd,shell=True)
def download_wrapper(args):
return download(*args)
def main():
<read in arguments>
<...>
cghub_dnld_file(file1,file2,file3,threads)
if __name__ == "__main__":
main()
If this file does not exist in the database, gtdownload would quit which also kills my python job with the following error:
Traceback (most recent call last):
File "/rsrch1/rists/djiao/bin/cghub_dnld.py", line 102, in <module>
main()
File "/rsrch1/rists/djiao/bin/cghub_dnld.py", line 98, in main
cghub_dnld_file(file1,file2,file3,threads)
File "/rsrch1/rists/djiao/bin/cghub_dnld.py", line 22, in cghub_dnld_file
p.map(download_wrapper,map_args)
File "/rsrch1/rists/apps/x86_64-rhel6/anaconda/lib/python2.7/multiprocessing/pool.py", line 250, in map
return self.map_async(func, iterable, chunksize).get()
File "/rsrch1/rists/apps/x86_64-rhel6/anaconda/lib/python2.7/multiprocessing/pool.py", line 554, in get
raise self._value
OSError: [Errno 2] No such file or directory
The actual error message from gtdownload :
Welcome to gtdownload-3.8.5a.
Ready to download
Communicating with GT Executive ...
Headers received from the client: 'HTTP/1.1 100 Continue
HTTP/1.1 404 Not Found
Date: Tue, 29 Jul 2014 18:49:57 GMT
Server: Apache/2.2.15 (Red Hat and CGHub)
Strict-Transport-Security: max-age=31536000
X-Powered-By: PHP/5.3.3
Content-Length: 669
Connection: close
Content-Type: text/xml
'
Error: You have requested to download a uuid which either does not exist within the system, or has not yet reached the 'live' state. The requested action will not be performed. Please double check the supplied uuid or contact thelpdesk for further assistance.
I would like the script to skip the one that does not exist and start gtdownload on the next one. I tried to output the stderr of subprocess.call to a pipe and see if there is the "error" keyword. But it seems it stops at the exact subprocess.call command. Same thing with os.system.
I made a MCV case without the multiprocessing and subprocess did not kill the main process at all. Looks like multiprocessing messes things up although I had it run with 1 thread just for testing.
#!/usr/bin/env python
import subprocess
#THis is the id that gtdownload had problem with
id = "df1e073f-4485-4d5f-8659-cd8eaac17329"
link = "https://cghub.ucsc.edu/cghub/data/analysis/download/" + id
dlnd_cmd = "gtdownload -c ~/.cghub.key --max-children 4 -vv -d " + link + " > gt.out 2>gt.err"
print dlnd_cmd
subprocess.call(dlnd_cmd,shell=True)
print "done"
Clearly multiprocessing conflicts subprocess.call but it is not clear to me why.
What is the best way to avoid the failure of subprocess killing the main process?
Handle the exception in some appropriate way and move on, of course.
try:
subprocess.call(dlnd_cmd)
except OSError as e:
print 'failed to download: {!r}'.format(e)
However, this may not be appropriate here. The kinds of exceptions that subprocess.call raises are usually not transient things that you can just log and work around; if it's not working now, it will continue to not work forever until you fix the underlying problem (a bug in your script, or gtdownload not being installed right, or whatever).
For example, if the code you showed us is your actual code:
dlnd_cmd = "gtdownload -c ~/.cghub.key --max-children 4 -vv -d " + link + " > gt.out 2>gt.err"
subprocess.call(dlnd_cmd)
… then this is guaranteed to raise an OSError for the reason explained in dano's answer: call (without shell=True) will try to take that entire string—spaces, shell-redirection, etc.—as the name of an executable program to find on your $PATH. And there is no such program. So it will raise an OSError(errno.ENOENT). (Which is exactly what you're seeing.) Just logging that doesn't do you any good; it's a good thing that your entire process is exiting, so you can debug that problem.
subprocess.call should not kill the main process. Something else must be wrong with your script or your conclusions about the script's behaviour is wrong. Did you try printing some trace output after the subprocess call?
You have to use shell=True to use subprocess.call with a string for an argument (and with shell redirection):
subprocess.call(dlnd_cmd, shell=True)
Without shell=True, subprocess tries to treat your entire command string like a single executable name, which of course doesn't exist, and leads to the No such file or directory exception.
See this answer for more info on when to use a string vs. when to use a sequence with subprocess.