How would I do file reading loop in python? I'm trying to convert my bash script to python but have never written python before. FYI, the reason I am read reading the file after a successful command competition is to make sure it reads the most recent edit (say if the URLs were reordered).
Thanks!
#!/bin/bash
FILE=$1
declare -A SUCCESS=()
declare -A FAILED=()
for (( ;; )); do
# find a new link
cat "$FILE" > temp.txt
HASNEW=false
while read; do
[[ -z $REPLY || -n ${SUCCESS[$REPLY]} || -n ${FAILED[$REPLY]} ]] && continue
HASNEW=true
break
done < temp.txt
[[ $HASNEW = true ]] || break
# download
if axel --alternate --num-connections=6 "$REPLY"; then
echo
echo "Succeeded at $DATETIME downloading following link $REPLY"
echo "$DATETIME Finished: $REPLY" >> downloaded-links.txt
echo
SUCCESS[$REPLY]=.
else
echo
echo "Failed at $DATETIME to download following link $REPLY"
echo "$DATETIME Failed: $REPLY" >> failed-links.txt
FAILED[$REPLY]=.
fi
# refresh file
cat "$FILE" > temp.txt
while read; do
[[ -z ${SUCCESS[REPLY]} ]] && echo "$REPLY"
done < temp.txt > "$FILE"
done
This is what I've got so far which is working, and I can't figure out how to make it read the top line of the file after every successful execution of the axel line like the bash script does. I'm open to other options on the subprocess call such as threading, but I'm not sure how to make that work.
#!/usr/bin/env python
import subprocess
from optparse import OptionParser
# create command line variables
axel = "axel --alternate --num-connections=6 "
usage = "usage: %prog [options] ListFile.txt"
parser = OptionParser(usage=usage)
parser.add_option("-s", "--speed", dest="speed",
help="speed in bits per second i.e. 51200 is 50kps", metavar="speedkbps")
(opts, args) = parser.parse_args()
if args[0] is None:
print "No list file given\n"
parser.print_help()
exit(-1)
list_file_1 = args[0]
try:
opts.speed
except NoSpeed:
with open(list_file_1, 'r+') as f:
for line in f:
axel_call = axel + "--max-speed=" + opts.speed + " " + line
# print ("speed option set line send to subprocess is: " + axel_call)
subprocess.call(axel_call, shell=True)
else:
with open(list_file_1, 'r+') as f:
for line in f:
axel_call = axel + line
# print ("no speed option set line send to subprocess is:" + axel_call)
subprocess.call(axel_call, shell=True)
Fully Pythonic way to read a file is the following:
with open(...) as f:
for line in f:
<do something with line>
The with statement handles opening and closing the file, including if an exception is raised in the inner block. The for line in f treats the file object f as an iterable, which automatically uses buffered IO and memory management so you don't have to worry about large files.
There should be one -- and preferably only one -- obvious way to do it.
A demonstration of using a loop to read a file is shown at http://docs.python.org/tutorial/inputoutput.html#reading-and-writing-files
Related
I have a requirement to fetch the count the occurrence of '|' in each line of a file then match the count with given inputcount, needs to throw exception when the count is wrong.
Say if the inputcount=3 and the file has following content
s01|test|aaa|hh
S02|test|bbb
so3|test|ccc|oo
then exception should get thrown on executing the line 2 and it should exit the file.
Tried below Awk command to fetch the count for each lines, but I was not sure how to compare and throw the exception, when it not matches
awk ' {print (split($0,a,"\|")-1) }' test.dat
Can anyone please help me with it?
You may use this awk:
awk -v inputcount=3 -F '\\|' 'NF && NF != inputcount+1 {exit 1}' file &&
echo "good" || echo "bad"
Details:
-F '\\|' sets | as input field separator
NF != inputcount+1 will return true if any line doesn't have inputcount pipe delimiters.
$ inputcount=3
$ awk -v c="$inputcount" 'gsub(/\|/,"&") != c{exit 1}' file
$ echo $?
1
As you also tagged the post with python I will write a python answer that could be a simple script.
The core is:
with open(filename) as f:
for n, line in enumerate(f):
if line.count("|") != 3:
print(f"Not valid file at line {n + 1}")
Than you can add some boilerplate:
import fileinput
import sys
with fileinput.input() as f:
for n, line in enumerate(f):
if line.count("|") != 3:
print(f"Not valid file at line {n + 1}")
sys.exit(1)
And with fileinput you can accept almost any sort of input: see Piping to a python script from cmd shell
Maybe try
awk -F '[|]' -v cols="$inputcount" 'NF != cols+1 {
print FILENAME ":" FNR ":" $0 >"/dev/stderr"; exit 1 }' test.dat
The -F argument says to split on this delimiter; the number of resulting fields NF will be one more than there are delimiters, so we scream and die when that number is wrong.
I'm beginning with bash and I'm executing a script :
$ ./readtext.sh ./InputFiles/applications.txt
Here is my readtext.sh code :
#!/bin/bash
filename="$1"
counter=1
while IFS=: true; do
line=''
read -r line
if [ -z "$line" ]; then
break
fi
echo "$line"
python3 ./download.py \
-c ./credentials.json \
--blobs \
"$line"
done < "$filename"
I want to print the string ("./InputFiles/applications.txt") in a python file, I used sys.argv[1] but this line gives me -c. How can I get this string ? Thank you
It is easier for you to pass the parameter "$1" to the internal command python3.
If you don't want to do that, you can still get the external command line parameter with the trick of /proc, for example:
$ cat parent.sh
#!/usr/bin/bash
python3 child.py
$ cat child.py
import os
ext = os.popen("cat /proc/" + str(os.getppid()) + "/cmdline").read()
print(ext.split('\0')[2:-1])
$ ./parent.sh aaaa bbbb
['aaaa', 'bbbb']
Note:
the shebang line in parent.sh is important, or you should execute ./parent.sh with bash, else you will get no command line param in ps or /proc/$PPID/cmdline.
For the reason of [2:-1]: ext.split('\0') = ['bash', './parent.sh', 'aaaa', 'bbbb', ''], real parameter of ./parent.sh begins at 2, ends at -1.
Update: Thanks to the command of #cdarke that "/proc is not portable", I am not sure if this way of getting command line works more portable:
$ cat child.py
import os
ext = os.popen("ps " + str(os.getppid()) + " | awk ' { out = \"\"; for(i = 6; i <= NF; i++) out = out$i\" \" } END { print out } ' ").read()
print(ext.split(" ")[1 : -1])
which still have the same output.
This is the python file that you can use in ur case
import sys
file_name = sys.argv[1]
with open(file_name,"r") as f:
data = f.read().split("\n")
print("\n".join(data))
How to use sys.argv
How to use join method inside my python code
My goal is to compare two data one is from text file and one is from directory and after comparing it this is will notify or display in the console what are the data that is not found for example:
ls: /var/patchbundle/rpms/:squid-2.6.STABLE21-7.el5_10.x86_64.rpm NOT FOUND!
ls: /var/patchbundle/rpms/:tzdata-2014j-1.el5.x86_64.rpm
ls: /var/patchbundle/rpms/:tzdata-java-2014j-1.el5.x86_64.rpm
ls: /var/patchbundle/rpms/:wireshark-1.0.15-7.el5_11.x86_64.rpm
ls: /var/patchbundle/rpms/:wireshark-gnome-1.0.15-7.el5_11.x86_64.rpm
ls: /var/patchbundle/rpms/:yum-updatesd-0.9-6.el5_10.noarch.rpm NOT FOUND
It must be like that. So Here's my python code.
import package, sys, os, subprocess
path = '/var/tools/tools/newrpms.txt'
newrpms = open(path, "r")
fds = newrpms.readline()
def checkrc(rc):
if(rc != 0):
sys.exit(rc)
cmd = package.Errata()
for i in newrpms:
rc = cmd.execute("ls /var/patchbundle/rpms/ | grep %newrpms ")
if ( != 0):
cmd.logprint ("%s not found !" % i)
checkrc(rc)
sys.exit(0)
newrpms.close
Please see the shell script. This script its executing file but because I want to use another language that's why Im trying python
retval=0
for i in $(cat /var/tools/tools/newrpms.txt)
do
ls /var/patchbundle/rpms/ | grep $i
if [ $? != 0 ]
then
echo "$i NOT FOUND!"
retval=255
fi
done
exit $retval
Please see my Python code. What is wrong because it is not executing like the shell executing it.
You don't say what the content of "newrpms.txt" is; you say the script is not executing how you want - but you don't say what it is doing; I don't know what package or package.Errata are, so I'm playing guess-the-problem; but lots of things are wrong.
if ( != 0): is a syntax error. If {empty space} is not equal to zero?
cmd.execute("ls /var/patchbundle/rpms/ | grep %newrpms ") is probably not doing what you want. You can't put a variable in a string in Python like that, and if you could newrpms is the file handle not the current line. That should probably be ...grep %s" % (i,)) ?
The control flow is doing:
Look in this folder, try to find files
Call checkrc()
Only quit with an error if the last file was not found
newrpms.close isn't doing anything, it would need to be newrpms.close() to call the close method.
You're writing shell-script-in-Python. How about:
import os, sys
retval=0
for line in open('/var/tools/tools/newrpms.txt'):
rpm_path = '/var/patchbundle/rpms/' + line.strip()
if not os.path.exists(rpm_path):
print rpm_path, "NOT FOUND"
retval = 255
else:
print rpm_path
sys.exit(retval)
Edited code slightly, and an explanation:
The code is almost a direct copy of the shell script into Python. It loops over every line in the text file, and calls line.strip() to get rid of the newline character at the end. It builds rpm_path which will be something like "/var/patchbundle/rpms/:tzdata-2014j-1.el5.x86_64.rpm".
Then it uses sys.path.exists() which tests if a file exists and returns True if it does, False if it does not, and uses that test to set the error value and print the results like the shell script prints them. This replaces the "ls ... | grep " part of your code for checking if a file exists.
I have a Python program (below) and when I run it I get the following error:
% python SSH_Prog.py
About to connect...
stderr: ["bash: -c: line 0: unexpected EOF while looking for matching `''\n", 'bash: -c: line 1: syntax error: unexpected end of file\n']
pwd: []
stderr: ['watch: no process found\n']
pwd: []
^CTraceback (most recent call last):
File "SSH_Prog.py", line 32, in <module>
time.sleep(3)
KeyboardInterrupt
I think it is to do with escape sequence probably, and the "\n" character from stdin, but I lack the experience to deal with it.
Here's the program:
import os
import sys
import time
import paramiko
#from ssh import SSHClient
# Define remote machine
host="<ip>"
user="<usrnm>"
passw="<passw>"
client = paramiko.SSHClient()
#client.load_system_host_keys()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
# Try SSH connection, catch exception
#if not
print('About to connect...')
client.connect(host, username=user, password=passw)
# ForLoop to iterate through the interactions
for x in range(10):
xx = str(x)
# Commands to execute on local machine
f = os.popen3('tshark -i eth0 -f snmp -F pcapng -w ~/Desktop/traf_logs/n'+(xx))
# commands to execute on remote machine
stdin, stdout, stderr = client.exec_command("watch -n 0.1 'ps -p $(pgrep -d"," -x snmpd) -o rss= | awk '\''{ i += $1 } END { print i }'\'' >> ~/Desktop/mem_logs/mem_"+(xx)+";")
print "stderr: ", stderr.readlines()
print "pwd: ", stdout.readlines()
g = os.popen3('snmpget -v 2c -c communitystring <ip> sysContact.0')
time.sleep(3)
stdin, stdout, stderr = client.exec_command('killall watch;')
print "stderr: ", stderr.readlines()
print "pwd: ", stdout.readlines()
ff = os.popen3('killall tshark')
# terminate connection
client.close()
exit(0)
Do you have any idea to fix it?
Regards.
Your first exec_command looks like this:
stdin, stdout, stderr = client.exec_command("watch -n 0.1 'ps -p $(pgrep -d"," -x snmpd) -o rss= | awk '\''{ i += $1 } END { print i }'\'' >> ~/Desktop/mem_logs/mem_"+(xx)+";")
In other words, the first argument is:
"watch -n 0.1 'ps -p $(pgrep -d"
And your second argument is:
" -x snmpd) -o rss= | awk '\''{ i += $1 } END { print i }'\'' >> ~/Desktop/mem_logs/mem_"+(xx)+";"
If you fire up bash in a terminal and type that first argument (without the quotes), followed by a newline and a ^D, it'll tell you this:
> -bash: unexpected EOF while looking for matching `''
-bash: syntax error: unexpected end of file
Which is exactly what you're getting back from Paramiko.
And the second error is just killall telling you that there is no process named watch, because your first command never started one.
If you just replace the "," with a space, that'll solve that problem… but without knowing why you thought you wanted a "," there, I'm not sure it'll do what you actually were intending to do.
I'm also not sure what the '\'' is supposed to do. Why do you want to triple-quote the arguments to awk, or why you're doing something so complicated when it ought to be equivalent to just { print $1 }, or why you're explicitly asking ps for multiple columns just to use awk to pick out the first one, or…
On Unix-like systems I use this script, which I'd like some help on porting to Python for execution on Windows hosts:
#!/bin/bash
SENTINEL_FILENAME='__sentinel__'
SENTINEL_MD5_CHECKSUM=''
SENTINEL_SHA_CHECKSUM=''
function is_directory_to_be_flattened() {
local -r directory_to_consider="$1"
local -r sentinel_filepath="${directory_to_consider}/${SENTINEL_FILENAME}"
if [ ! -f "${sentinel_filepath}" ]; then
return 1
fi
if [[
"$(
md5 "${sentinel_filepath}" \
| awk '{ print $NF }' 2> /dev/null
)" \
== "${SENTINEL_MD5_CHECKSUM}"
&& \
"$(
shasum -a 512 "${sentinel_filepath}" \
| awk '{ print $1 }' 2> /dev/null
)" \
== "${SENTINEL_SHA_CHECKSUM}"
]]; then
return 0
else
return 1
fi
}
function conditionally_flatten() {
local -r directory_to_flatten="$1"
local -r flatten_into_directory="$2"
if is_directory_to_be_flattened "${directory_to_flatten}"; then
if [ ! -d "${flatten_into_directory}" ]; then
mkdir -v "${flatten_into_directory}"
fi
for file_to_move in $(find ${directory_to_flatten} -type f -maxdepth 1); do
mv \
-v \
-n \
"${file_to_move}" \
"${flatten_into_directory}"
done
fi
}
function flatten_directory() {
local -r directory_to_flatten="$1"
local -r descend_depth="$2"
local -r flattened_directory="${directory_to_flatten}/__flattened__"
if [ ! -d "${directory_to_flatten}" ]; then
printf "The argument '%s' does not seem to be a directory.\n" \
"${directory_to_flatten}" \
>&2
return
fi
find "${directory_to_flatten}" \
-type d \
-maxdepth "${descend_depth}" \
| \
while read directory_path; do
conditionally_flatten \
"${directory_path}" \
"${flattened_directory}"
done
}
n_arguments="$#"
if [ "${n_arguments}" -eq 1 ]; then
flatten_directory "$1" '1' # maybe use a constant, not a "magic #" here?
else
echo usage: "$0" /path/to/directory/to/flatten
fi
unset is_directory_to_be_flattened
unset conditionally_flatten
unset flatten_directory
How would you port this to Win Python? I am a beginner in both Python and Bash scripting..
Feel free to upgrade my implementation as you port it if you find it lacking in any way too, with a justification please. This is not "Code Review" but a "thumbs up/thumbs down" on my effort in Bash would give me a sense of whether I am improving or I should change the way I study altogether...
Here we go, my attempt in Python: (criticise it hard if need be, it's the only way for me to learn!)
#!/usr/bin/env python2.7
import sys
import os
import shutil
SENTINEL_FILENAME=''
SENTINEL_MD5_CHECKSUM=''
SENTINEL_SHA_CHECKSUM=''
DEFAULT_DEPTH = 1
FLATTED_DIRECTORY_NAME = '__flattened__'
def is_directory_to_be_flattened(directory_to_consider):
sentinel_location = os.path.join(directory_to_consider, SENTINEL_FILENAME)
if not os.path.isfile(sentinel_location):
return False
import hashlib
with open(sentinel_location) as sentinel_file:
file_contents = sentinel_file.read()
return (hashlib.md5(file_contents).hexdigest() == SENTINEL_MD5_CHECKSUM
and hashlib.sha512(file_contents).hexdigest() == SENTINEL_SHA_CHECKSUM)
def flatten(directory, depth, to_directory, do_files_here):
if depth < 0:
return
contained_filenames = [f for f in os.listdir(directory)]
if do_files_here:
for filename in contained_filenames:
if filename == SENTINEL_FILENAME:
continue
filepath = os.path.join(directory, filename)
if not os.path.isfile(filepath):
continue
file_to = os.path.join(to_directory, filename)
if not os.path.isdir(to_directory):
os.makedirs(to_directory)
if not os.path.isfile(file_to):
print "Moving: '{}' -> '{}'".format(filepath, file_to)
shutil.move(filepath, file_to)
else:
sys.stderr.write('Error: {} exists already.\n'.format(file_to))
next_depth = depth - 1
for subdirectory in (d for d in contained_filenames if os.path.isdir(d)):
if is_directory_to_be_flattened(subdirectory):
flatten(subdirectory, next_depth, to_directory, True)
def flatten_directory(to_flatten, depth):
to_directory = os.path.join(to_flatten, FLATTED_DIRECTORY_NAME)
if not os.path.isdir(to_flatten):
sys.stderr.write(
'The argument {} does not seem to be a directory.\n'.format(
to_flatten))
return
flatten(to_flatten, depth, to_directory, False)
def main():
if len(sys.argv) == 2:
flatten_directory(sys.argv[1], DEFAULT_DEPTH)
else:
print 'usage: {} /path/to/directory/to/flatten'.format(sys.argv[0])
if __name__ == '__main__':
main()
Although it's obvious from the code, the intent is:
Start at a given directory
Descend up to a certain depth
Consider subdirectories and move all files therein if and only if:
The directory contains a "sentinel file" with a given filename
The sentinel file is actually a sentinel file, not just a file renamed to the same name
Collate files in a __flattened__ directory under the directory in which the search started
Most file dealing functions in Python are in the module "os" - therein you will find
os.rename (for renaming or moving a directoruy entry), os.listdir - which gives you a listing of filenames in the directory, passed as first arg, os.walk - to recursively walk through a directory structure, os.path.walk, to do the same, but with a callback, os.path.exists, os.path.isdir, os.mkdir, are others that might be handy.
For a "quick and dirty" translation you might also cehck "os.system". which allows you to execute a shell command just like it was typed in the shell, and os.popen - which allows access to stdin and stdout of said process. A more carefull translation, tough, would require using anothe module: "subprocess" which can give one full control of a shell command executed as sub process (although if you need find, for example, it won't be available on windows)
Other moduless of interest are sys (sys.argv are the arguments passed to the script) and shutil (with things like copy, rmtree and such)
Your script does some error checking, and it is trivial, given the above funcion names in "os" and basic Python to add them - but a short "just do it" script in Python could be just:
import os, sys
dir_to_flatten = sys.argv[1]
for dirpath, dirnames, filenames in os.walk(dir_to_flatten):
for filename in filenames:
try:
os.rename(os.path.join(dirpath, filename), os.path.join(dir_to_flatten, filename))
except OSError:
print ("Could not move %s " % os.path.join(dirpath, filename))