I'm trying to write a script to get the following outputs to a folder (YYYYMMDDHHMMSS = current date and time) using a Linux command in Python, with the ID's in a configutation file
1234_YYYYMMDDHHMMSS.txt
12345_YYYYMMDDHHMMSS.txt
12346_YYYYMMDDHHMMSS.txt
I have a config file with the list of ID's
id1 = 1234
id2 = 12345
id3 = 123456
I want to be able to loop through these in python and incorporate them into a linux command.
Currently, my linux commands are hardcoded in python as such
import subprocess
import datetime
now = datetime.datetime.now()
subprocess.call('autorep -J 1234* -q > /home/test/output/1234.txt', shell=True)
subprocess.call('autorep -J 12345* -q > /home/test/output/12345.txt', shell=True)
subprocess.call('autorep -J 123456* -q > /home/test/output/123456.txt', shell=True)
print now.strftime("%Y%m%d%H%M%S")
The datetime is defined, but doesn't do anything currently, except print it to the console, when I want to incorporate it into the output txt file. However, I want to be able to write a loop to do something like this
subprocess.call('autorep -J id1* -q > /home/test/output/123456._now.strftime("%Y%m%d%H%M%S").txt', shell=True)
subprocess.call('autorep -J id2* -q > /home/test/output/123456._now.strftime("%Y%m%d%H%M%S").txt', shell=True)
subprocess.call('autorep -J id3* -q > /home/test/output/123456._now.strftime("%Y%m%d%H%M%S").txt', shell=True)
I know that I need to use ConfigParser and currently have been this piece written which simply prints the ID's from the configuration file to the console.
from ConfigParser import SafeConfigParser
import os
parser = SafeConfigParser()
parser.read("/home/test/input/ReportConfig.txt")
def getSystemID():
for section_name in parser.sections():
print
for key, value in parser.items(section_name):
print '%s = %s' % (key,value)
print
getSystemID()
But as mentioned in the beggining of the post, my goal is to be able to loop through the ID's, and incorporate them into my linux command while adding the datetime format to the end of the file. I'm thinking all I need is some kind of while loop in the above function in order to get the type of output I want. However, I'm not sure how to call the ID's and the datetime into a linux command.
So far you have most of what you need, you are just missing a few things.
First, I think using ConfigParser is overkill for this. But it's simple enough so lets continue with it. Lets change getSystemID to a generator returning your IDs instead of printing them out, its just a one line change.
parser = SafeConfigParser()
parser.read('mycfg.txt')
def getSystemID():
for section_name in parser.sections():
for key, value in parser.items(section_name):
yield key, value
With a generator we can use getSystemID in a loop directly, now we need to pass this on to the subprocess call.
# This is the string of the current time, what we add to the filename
now = datetime.datetime.now().strftime('%Y%m%d%H%M%S')
# Notice we can iterate over ids / idnumbers directly
for name, number in getSystemID():
print name, number
Now we need to build the subprocess call. The bulk of your problem above was knowing how to format strings, the syntax is described here.
I'm also going to make two notes on how you use subprocess.call. First, pass a list of arguments instead of a long string. This helps python know what arguments to quote so you don't have to worry about it. You can read about it in the subprocess and shlex documentation.
Second, you redirect the output using > in the command and (as you noticed) need shell=True for this to work. Python can redirect for you, and you should use it.
To pick up where I left off above in the foor loop.
for name, number in getSystemID():
# Make the filename to write to
outfile = '/home/test/output/{0}_{1}.txt'.format(number, now)
# open the file for writing
with open(outfile, 'w') as f:
# notice the arguments are in a list
# stdout=f redirects output to the file f named outfile
subprocess.call(['autorep', '-J', name + '*', '-q'], stdout=f)
You can insert the datetime using Python's format instruction.
For example, you could create a new file with the 1234 prefix and the datime stamp like this:
new_file = open("123456.{0}".format(datetime.datetime.now()), 'w+')
I am not sure if I understood what your are looking for, but I hope this helps.
Related
I have a simple shell script script.sh:
echo "ubuntu:$1" | sudo chpasswd
I need to open the script, read it, insert the argument, and save it as a string like so: 'echo "ubuntu:arg_passed_when_opening" | sudo chpasswd' using Python.
All the options suggested here actually execute the script, which is not what I want.
Any suggestions?
You would do this the same way that you read any text file, and we can use sys.argv to get the argument passed when running the python script.
Ex:
import sys
with open('script.sh', 'r') as sfile:
modified_file_contents = sfile.read().replace('$1', sys.argv[1])
With this method, modified_file_contents is a string containing the text of the file, but with the specified variable replaced with the argument passed to the python script when it was run.
I have two questions about my script, how can I get it to output to the file I requested. I ask this because it loops infinitely and when I cancel the script and show the file it is empty. Also, how can I use the variables assigned if I must cancel the script to input anything? Thanks!
import subprocess
import datetime
#open results file and assign to results variable, add append rights
results = open("results.txt", "a")
#Run until stopped
while 1:
#split the blah variable by line
#Run tshark command 100 times, then restart script. Assign to blah variable
blah = subprocess.check_output(["tshark -i mon0 -f \"subtype probe-req\" -T fields -e wlan.sa -e wlan_mgt.ssid -c 20"], shell=True)
splitblah = blah.split("\n")
#repeat for each line, ignore first line since it contains headers
for value in splitblah[:-1]:
#split each line by tab delimiter
splitvalue = value.split("\t")
#Assign variables to split fields
MAC = str(splitvalue[1])
SSID = str(splitvalue[2])
time = str(datetime.datetime.now())
#write and format output to results file
Results.write(MAC+" "+SSID+" "+time+"\r\n")
You should put a condition in your while statement, or the program will (indeed) never stop.
Also, datas are not necessarily written on the disk immediately after someFileObject.write function call, you need to call someFileObject.flush to insure that.
Having some issues calling awk from within Python. Normally, I'd do the following to call the command in awk from the command line.
Open up command line, in admin mode or not.
Change my directory to awk.exe, namely cd R\GnuWin32\bin
Call awk -F "," "{ print > (\"split-\" $10 \".csv\") }" large.csv
My command is used to split up the large.csv file based on the 10th column into a number of files named split-[COL VAL HERE].csv. I have no issues running this command. I tried to run the same code in Python using subprocess.call() but I'm having some issues. I run the following code:
def split_ByInputColumn():
subprocess.call(['C:/R/GnuWin32/bin/awk.exe', '-F', '\",\"',
'\"{ print > (\\"split-\\" $10 \\".csv\\") }\"', 'large.csv'],
cwd = 'C:/R/GnuWin32/bin/')
and clearly, something is running when I execute the function (CPU usage, etc) but when I go to check C:/R/GnuWin32/bin/ there are no split files in the directory. Any idea on what's going wrong?
As I stated in my previous answer that was downvoted, you overprotect the arguments, making awk argument parsing fail.
Since there was no comment, I supposed there was a typo but it worked... So I suppose that's because I should have strongly suggested a full-fledged python solution, which is the best thing to do here (as stated in my previous answer)
Writing the equivalent in python is not trivial as we have to emulate the way awk opens files and appends to them afterwards. But it is more integrated, pythonic and handles quoting properly if quoting occurs in the input file.
I took the time to code & test it:
def split_ByInputColumn():
# get rid of the old data from previous runs
for f in glob.glob("split-*.csv"):
os.remove(f)
open_files = dict()
with open('large.csv') as f:
cr = csv.reader(f,delimiter=',')
for r in cr:
tenth_row = r[9]
filename = "split-{}.csv".format(tenth_row)
if not filename in open_files:
handle = open(filename,"wb")
open_files[filename] = (handle,csv.writer(handle,delimiter=','))
open_files[filename][1].writerow(r)
for f,_ in open_files.values():
f.close()
split_ByInputColumn()
in detail:
read the big file as csv (advantage: quoting is handled properly)
compute the destination filename
if filename not in dictionary, open it and create csv.writer object
write the row in the corresponding dictionary
in the end, close file handles
Aside: My old solution, using awk properly:
import subprocess
def split_ByInputColumn():
subprocess.call(['awk.exe', '-F', ',',
'{ print > ("split-" $10 ".csv") }', 'large.csv'],cwd = 'some_directory')
Someone else posted an answer (and then subsequently deleted it), but the issue was that I was over-protecting my arguments. The following code works:
def split_ByInputColumn():
subprocess.call(['C:/R/GnuWin32/bin/awk.exe', '-F', ',',
'{ print > (\"split-\" $10 \".csv\") }', 'large.csv'],
cwd = 'C:/R/GnuWin32/bin/')
I have a bash script which goes out,does some DNS queries and returns an IP address value which I assign as a variable $external
I want to take this variable from the bash script and feed it into python and do some subnet intelligence but I'm not sure how to handle the data after this and pass it back to bash.
I can see that the $external variable is being passed from bash into python ok but this is where I'm not sure what to do next. (thanks Farhan.K for assisting me with what I have already)
python3 <<END
import ipaddress
ipsub = {"10.10.10.0/24": "Firewall-Denver", "10.10.20.0/25": "FirewallNewYork"}
iplist = [$external]
ipfirewall = []
for i in ipsub:
for j in iplist:
if ipaddress.ip_address(j) in ipaddress.ip_network(i):
ipfirewall.append([j,ipsub[i]])
END
The following would write it to a file:
with open('output.txt', 'w') as file:
file.writelines('\t'.join(i) + '\n' for i in ipfirewall)
But how to I pass it back to bash in the same format?
Thanks in advance for your advice and assitance.
You can do this by writing the intended result to Python's standard output, and capturing the result in a variable.
#!/bin/bash
external="'10.10.10.1','10.10.20.1'"
result="$(python3 <<END
import ipaddress
ipsub = {"10.10.10.0/24": "Firewall-Denver", "10.10.20.0/25": "FirewallNewYork"}
iplist = [$external]
ipfirewall = []
for i in ipsub:
for j in iplist:
if ipaddress.ip_address(j) in ipaddress.ip_network(i):
ipfirewall.append([j,ipsub[i]])
print(ipfirewall)
END
)"
echo "$result"
In the above script, I pass $external into the Python heredoc, run your process, and print the result in Python. The $() construct captures the standard output of the Python script into a Bash variable.
For the more generic version of this question, see How to assign a heredoc value to a variable in Bash?
I am writing a python script that reads a line/string, calls Unix, uses grep to search a query file for lines that contain the string, and then prints the results.
from subprocess import call
for line in infilelines:
output = call(["grep", line, "path/to/query/file"])
print output
print line`
When I look at my results printed to the screen, I will get a list of matching strings from the query file, but I will also get "1" and "0" integers as output, and line is never printed to the screen. I expect to get the lines from the query file that match my string, followed by the string that I used in my search.
call returns the process return code.
If using Python 2.7, use check_output.
from subprocess import check_output
output = check_output(["grep", line, "path/to/query/file"])
If using anything before that, use communicate.
import subprocess
process = subprocess.Popen(["grep", line, "path/to/query/file"], stdout=subprocess.PIPE)
output = process.communicate()[0]
This will open a pipe for stdout that you can read with communicate. If you want stderr too, you need to add "stderr=subprocess.PIPE" too.
This will return the full output. If you want to parse it into separate lines, use split.
output.split('\n')
I believe Python takes care of line-ending conversions for you, but since you're using grep I'm going to assume you're on Unix where the line-ending is \n anyway.
http://docs.python.org/library/subprocess.html#subprocess.check_output
The following code works with Python >= 2.5:
from commands import getoutput
output = getoutput('grep %s path/to/query/file' % line)
output_list = output.splitlines()
Why would you want to execute a call to external grep when Python itself can do it? This is extra overhead and your code will then be dependent on grep being installed. This is how you do simple grep in Python with "in" operator.
query=open("/path/to/query/file").readlines()
query=[ i.rstrip() for i in query ]
f=open("file")
for line in f:
if "line" in query:
print line.rstrip()
f.close()