Develop Reference

How to write Pcap packets in FIFO using Scapy (pcapwriter)

I'm French, sorry if my english isn't perfect !
Before starting, if you want to try my code, you can download a pcap sample file here : https://wiki.wireshark.org/SampleCaptures?action=AttachFile&do=get&target=ipv4frags.pcap
I succeed to open pcap file, read packets and write them to another file with this code :
# Python 3.6
# Scapy 2.4.3
from scapy.utils import PcapReader, PcapWriter
import time
i_pcap_filepath = "inputfile.pcap" # pcap to read
o_filepath = "outputfile.pcap" # pcap to write
i_open_file = PcapReader(i_pcap_filepath) # opened file to read
o_open_file = PcapWriter(o_filepath, append=True) # opened file to write
while 1:
# I will have EOF exception but anyway
time.sleep(1) # in order to see packet
packet = i_open_file.read_packet() # read a packet in file
o_open_file.write(packet) # write it
So now I want to write in a FIFO and see the result in a live Wireshark window.
To do that, I just create a FIFO :
$ mkfifo /my/project/location/fifo.fifo
and launch Wireshark application on it : $ wireshark -k -i /my/project/location/fifo.fifo
I change my filepath in my Python script : o_filepath = "fifo.fifo" # fifo to write
But I have a crash ... Here is the traceback :
Traceback (most recent call last):
File "fifo.py", line 25, in <module>
o_open_file = PcapWriter(o_pcap_filepath, append=True)
File "/home/localuser/.local/lib/python3.6/site-packages/scapy/utils.py", line 1264, in __init__
self.f = [open, gzip.open][gz](filename, append and "ab" or "wb", gz and 9 or bufsz) # noqa: E501
OSError: [Errno 29] Illegal seek
Wireshark also give me an error ("End of file on pipe magic during open") : wireshark error
I don't understand why, and what to do. Is it not possible to write in FIFO using scapy.utils library ? How to do then ?
Thank you for your support,
Nicos44k

Night was useful because I fix my issue this morning !
I didn't undestand the traceback yesterday but it give me in reality a big hint : we have a seek problem.
Wait ... There is no seek in FIFO file !!!
So we cannot set "append" parameter to true.
I changed with : o_open_file = PcapWriter(o_filepath)
And error is gone.
However, packets were not showing in live...
To solve this problem, I needed to force FIFO flush with : o_open_file.flush()
Remember that you can download a pcap sample file here : https://wiki.wireshark.org/SampleCaptures?action=AttachFile&do=get&target=ipv4frags.pcap
So here is the full code :
# Python 3.6
# Scapy 2.4.3
from scapy.utils import PcapReader, PcapWriter
import time
i_pcap_filepath = "inputfile.pcap" # pcap to read
o_filepath = "fifo.fifo" # pcap to write
i_open_file = PcapReader(i_pcap_filepath) # opened file to read
o_open_file = PcapWriter(o_filepath) # opened file to write
while 1:
# I will have EOF exception but anyway
time.sleep(1) # in order to see packet
packet = i_open_file.read_packet() # read a packet in file
o_open_file.write(packet) # write it
o_open_file.flush() # force buffered data to be written to the file
Have a good day !
Nicos44k

throwing exception SerialException: device reports readiness to read but returned no data

My raspberry pi is connected to microcontroller over serial pin. I am trying to read the data from the serial port. The script reads the data for few seconds. However, it terminates throwing following exception
serial.serialutil.SerialException: device reports readiness to read but returned no data (device disconnected?)
I have used following python code
#!/usr/bin/python
import serial
import time
serialport = serial.Serial("/dev/ttyAMA0", 115200, timeout=.5)
while 1:
response = serialport.readlines(None)
print response
time.sleep(.05)
serialport.close()

Here is the code you should be using if you are seriously trying to just transfer and print a file:
for line in serialport.readlines().split('\n'):
print line
------------------------------------------------------------
I believe you are having problems because you are using readlines(None) instead of readline() Readline() reads it a line at a time, and will wait for each one. If reading a whole file it will be slower than readlines. But readlines() expects a whole file all at once. It is obviously not waiting for your serial transfer speed.
--------------------------------------------------
My data-logging loop receives a line every two minutes and writes it to a file. It could easily just print each line like you show in the OP.
readine() waits for each line. I have tested it to wait up to 30 minutes between lines with no problems by altering the program on the Nano.
import datetime
import serial
ser = serial.Serial("/dev/ttyUSB0",9600) --/dev/ACM0 is fine
while True :
linein = ser.readline()
date = str(datetime.datetime.now().date())
date = date[:10]
time = str(datetime.datetime.now().time())
time = time[:8]
outline = date + tab + time + tab + linein
f = open("/home/pi/python/today.dat","a")
f.write(outline)
f.close()
Maybe changing to this approach would be better for you.

Syslog-ng not flushing output to python program

I have set up syslog-ng to tail a file and forward each new event to python script which will process it.
My syslog-ng.conf looks like this -
source s_src {
file("/var/log/xyz.log" flags(no_parse);
};
destination d_dest {
program("python -u /home/user1/processlog.py" flush_lines(1) flags(no_multi_line));
};
log { source(s_src); destination(d_dest); };
And processlog.py just contains
#!/usr/bin/python
import sys
f1 = open('success.txt', 'a')
while 0 < 1:
try:
line = sys.stdin.readline()
f1.write(line)
except Exception, e:
f = open('/tmp/error.txt','ab')
f.write(e)
f.close()
exit(0)
This script works prefectly fine from command line. Takes each input and writes to success.txt.
Syslog-ng also starts but does not forward the event to python program above. It starts the program though.
ps -ef| grep processlog
root 6242 6236 0 13:00 ? 00:00:00 /bin/sh -c python -u /home/user1/processlog.py
root 6244 6242 0 13:00 ? 00:00:00 python -u /home/user1/processlog.py
I have checked all permissions too. But whenever new event happens in xyz.log, it is not getting forwarded to python script, which i am testing via writing to success.txt
Any leads are highly appreciated.

Try to run syslog-ng in foreground mode with enabled debug messages
syslog-ng -Fedtv
Perhaps you will get information about why the program destination doesn't get the messages.
By the way, is it possible that the syslog-ng read this file once?
In this case you should delete the persist file (var/syslog-ng.persist) to force the syslog-ng to read this file from beginning.

Outputting with Willie bot doesn't work, but print does

I am trying to return the output of a command to the IRC channel using Willie bot.
My code seems to work outputting my variable line by line, but for some reason once I utilize Willie bots say command to output to IRC it doesn't output anything.
Here is my code:
from willie import module
import subprocess
import urllib2
import os
#module.commands('splint')
def splint(bot, trigger):
bot.reply('I will process your request now!')
page = urllib2.urlopen(trigger.group(2))
page_content = page.read();
with open('codeToCheck.c', 'w') as code:
code.write(page_content)
command = 'splint "C:\Users\Justin\Desktop\codeToCheck.c"'
output = subprocess.Popen(command,shell=True, stdout=subprocess.PIPE,stderr=subprocess.PIPE).communicate()[0]
bot.say('I have saved the file successfully. Outputting:')
for i in output.splitlines():
bot.say(i)
bot.say(output)
Using my little test code here I have determined it works with print:
import subprocess,os
output = subprocess.Popen(["splint", "C:\cygwin64\home\Justin\codeToCheck.c"], stdout=subprocess.PIPE).communicate()[0]
command = 'splint "C:\Users\Justin\Desktop\codeToCheck.c"'
output = subprocess.Popen(command,shell=True, stdout=subprocess.PIPE,stderr=subprocess.PIPE).communicate()[0]
for i in output.splitlines():
print i
print 'I have saved the file successfully. Outputting:'
This is what the irc output looks like for my code:
<Fogest> .splint http://pastebin.com/raw.php?i=8cB7DdnQ
<Fogbot> Fogest: I will process your request now!
<Fogbot> I have saved the file successfully. Outputting:
There should be output, but there is nothing. Am I doing something wrong here? Running my test file (the test code I show on this post) via the command line I get the following output like I should:
$ python test.py
Splint 3.1.2 --- 25 Aug 2010
Finished checking --- no warnings
I have saved the file successfully. Outputting:

I switched the code to use the following instead:
process = subprocess.Popen(command,shell=True, stderr=subprocess.PIPE)
output = process.stderr.read()
The problem is now resolved.

how to prevent failure of subprocess stopping the main process in python

I wrote a python script to run a command called "gtdownload" on a bunch of files with multiprocessing. The function "download" is where I am having trouble with.
#/usr/bin/env python
import os, sys, subprocess
from multiprocessing import Pool
def cghub_dnld_file(file1, file2, file3, np):
<open files>
<read in lines>
p = Pool(int(np))
map_args = [(line.rstrip(),name_lines[i].rstrip(),bar_lines[i].rstrip()) for i, line in enumerate(id_lines)]
p.map(download_wrapper,map_args)
def download(id, name, bar):
<check if file has been downloaded, if not download>
<.....>
link = "https://cghub.ucsc.edu/cghub/data/analysis/download/" + id
dnld_cmd = "gtdownload -c ~/.cghub.key --max-children 4 -vv -d " + link + " > gt.out 2>gt.err"
subprocess.call(dnld_cmd,shell=True)
def download_wrapper(args):
return download(*args)
def main():
<read in arguments>
<...>
cghub_dnld_file(file1,file2,file3,threads)
if __name__ == "__main__":
main()
If this file does not exist in the database, gtdownload would quit which also kills my python job with the following error:
Traceback (most recent call last):
File "/rsrch1/rists/djiao/bin/cghub_dnld.py", line 102, in <module>
main()
File "/rsrch1/rists/djiao/bin/cghub_dnld.py", line 98, in main
cghub_dnld_file(file1,file2,file3,threads)
File "/rsrch1/rists/djiao/bin/cghub_dnld.py", line 22, in cghub_dnld_file
p.map(download_wrapper,map_args)
File "/rsrch1/rists/apps/x86_64-rhel6/anaconda/lib/python2.7/multiprocessing/pool.py", line 250, in map
return self.map_async(func, iterable, chunksize).get()
File "/rsrch1/rists/apps/x86_64-rhel6/anaconda/lib/python2.7/multiprocessing/pool.py", line 554, in get
raise self._value
OSError: [Errno 2] No such file or directory
The actual error message from gtdownload :
Welcome to gtdownload-3.8.5a.
Ready to download
Communicating with GT Executive ...
Headers received from the client: 'HTTP/1.1 100 Continue
HTTP/1.1 404 Not Found
Date: Tue, 29 Jul 2014 18:49:57 GMT
Server: Apache/2.2.15 (Red Hat and CGHub)
Strict-Transport-Security: max-age=31536000
X-Powered-By: PHP/5.3.3
Content-Length: 669
Connection: close
Content-Type: text/xml
'
Error: You have requested to download a uuid which either does not exist within the system, or has not yet reached the 'live' state. The requested action will not be performed. Please double check the supplied uuid or contact thelpdesk for further assistance.
I would like the script to skip the one that does not exist and start gtdownload on the next one. I tried to output the stderr of subprocess.call to a pipe and see if there is the "error" keyword. But it seems it stops at the exact subprocess.call command. Same thing with os.system.
I made a MCV case without the multiprocessing and subprocess did not kill the main process at all. Looks like multiprocessing messes things up although I had it run with 1 thread just for testing.
#!/usr/bin/env python
import subprocess
#THis is the id that gtdownload had problem with
id = "df1e073f-4485-4d5f-8659-cd8eaac17329"
link = "https://cghub.ucsc.edu/cghub/data/analysis/download/" + id
dlnd_cmd = "gtdownload -c ~/.cghub.key --max-children 4 -vv -d " + link + " > gt.out 2>gt.err"
print dlnd_cmd
subprocess.call(dlnd_cmd,shell=True)
print "done"
Clearly multiprocessing conflicts subprocess.call but it is not clear to me why.

What is the best way to avoid the failure of subprocess killing the main process?
Handle the exception in some appropriate way and move on, of course.
try:
subprocess.call(dlnd_cmd)
except OSError as e:
print 'failed to download: {!r}'.format(e)
However, this may not be appropriate here. The kinds of exceptions that subprocess.call raises are usually not transient things that you can just log and work around; if it's not working now, it will continue to not work forever until you fix the underlying problem (a bug in your script, or gtdownload not being installed right, or whatever).
For example, if the code you showed us is your actual code:
dlnd_cmd = "gtdownload -c ~/.cghub.key --max-children 4 -vv -d " + link + " > gt.out 2>gt.err"
subprocess.call(dlnd_cmd)
… then this is guaranteed to raise an OSError for the reason explained in dano's answer: call (without shell=True) will try to take that entire string—spaces, shell-redirection, etc.—as the name of an executable program to find on your $PATH. And there is no such program. So it will raise an OSError(errno.ENOENT). (Which is exactly what you're seeing.) Just logging that doesn't do you any good; it's a good thing that your entire process is exiting, so you can debug that problem.

subprocess.call should not kill the main process. Something else must be wrong with your script or your conclusions about the script's behaviour is wrong. Did you try printing some trace output after the subprocess call?

You have to use shell=True to use subprocess.call with a string for an argument (and with shell redirection):
subprocess.call(dlnd_cmd, shell=True)
Without shell=True, subprocess tries to treat your entire command string like a single executable name, which of course doesn't exist, and leads to the No such file or directory exception.
See this answer for more info on when to use a string vs. when to use a sequence with subprocess.