Develop Reference

Rename multiple folder names based on Text file

I would like to rename folders name in specific path based on text list I will provide.
for example , I have folders list structure as following :
/home/XXX-01/$file1
/home/XXX-12/$file2
/home/XXX-23/$file66
/home/XXX-32/$file44
/home/XXX-123/$file65
and rand.txt file with details what folder name need to be change to , for example
XXX-22
XXX-33
XXX-55
XXX-4321
XXX-24456
the final folder structure would be like
/home/XXX-22/$file1
/home/XXX-33/$file2
/home/XXX-55/$file66
/home/XXX-4321/$file44
/home/XXX-24456/$file65
thank you

Using GNU awk version 4.0.2
awk 'NR==FNR { # Process the list of directories (direcs.txt)
map[NR]=$0 # Create an array indexed by the number record and with the directory as the value
}
NR!=FNR { # Process the output of the find command
newpath=gensub("(/home/)(.*)(/.*$)","\\1"map[NR]"\\3",$0); # Create the new path using the entry in the map array
newdir=gensub("(/home/)(.*)(/.*.txt$)","\\1"map[num],$0) # Create the directory to create.
print "mkdir "newdir # Print the mkdir command
print "mv "$0" "newpath # Print the command to execute
}' direcs.txt <(find /home -regextype posix-extended -regex "^.*[[:digit:]]+.*$")
One liner:
awk 'NR==FNR {map[NR]=$0} NR!=FNR { newpath=gensub("(/home/)(.*)(/.*.txt$)","\\1"map[NR]"\\3",$0);newdir=gensub("(/home/)(.*)(/.*.txt$)","\\1"map[num],$0);print "mkdir "newdir;print "mv "$0" "newpath }' direcs.txt <(find /home -regextype posix-extended -regex "^.*[[:digit:]]+.*$")
Once you are happy that the commands look as expected, execute by piping through to bash/sh and so
awk 'NR==FNR {map[NR]=$0} NR!=FNR { newpath=gensub("(/home/)(.*)(/.*.txt$)","\\1"map[NR]"\\3",$0);newdir=gensub("(/home/)(.*)(/.*.txt$)","\\1"map[num],$0);print "mkdir "newdir;print "mv "$0" "newpath }' direcs.txt <(find /home/robbo -regextype posix-extended -regex "^.*[[:digit:]]+.*$") | bash

search strings in multiple textfile

I have thousands of text files on my disk.
I need to search for them in terms of selected words.
Currently, I use:
grep -Eri 'text1|text2|text3|textn' dir/ > results.txt
The result is saved to a file: results.txt
I would like the result to be saved to many files.
results_text1.txt, results_text2.txt, results_textn.txt
Maybe someone has encountered some kind of script eg in python?

One solution might be to use a bash for loop.
for word in text1 text2 text3 textn; do grep -Eri '$word' dir/ > results_$word.txt; done
You can run this directly from the command line.

By using combination of "sed" and "xargs"
echo "text1,text2,text3,textn" | sed "s/,/\n/g" | xargs -I{} sh -c "grep -ir {} * > result_{}"

One way (using Perl because it's easier for regex and one-liner).
Sample data:
% mkdir dir dir/dir1 dir/dir2
% echo -e "text1\ntext2\nnope" > dir/file1.txt
% echo -e "nope\ntext3" > dir/dir1/file2.txt
% echo -e "nope\ntext2" > dir/dir1/file3.txt
Search:
% find dir -type f -exec perl -ne '/(text1|text2|text3|textn)/ or next;
$pat = $1; unless ($fh{$pat}) {
($fn = $1) =~ s/\W+/_/ag;
$fn = "results_$fn.txt";
open $fh{$pat}, ">>", $fn;
}
print { $fh{$pat} } "$ARGV:$_"' {} \;
Content of results_text1.txt:
dir/file1.txt:text1
Content of results_text2.txt:
dir/dir2/file3.txt:text2
dir/file1.txt:text2
Content of results_text3.txt:
dir/dir1/file2.txt:text3
Note:
you need to put the pattern inside parentheses to capture it. grep doesn't allow one to do this.
the captured pattern is then filtered (s/\W+/_/ag means to replace nonalphanumeric characters with underscore) to ensure it's safe as part of a filename.

Converting a kshell script to python

Can anyone guide me? I'm trying to migrate a kshell script to python.
If anyone could guide me on how to proceed on this task. Do I really need the function? I'm trying to make more concise easy to read. Thank you.
here is the code:
#!/bin/ksh
DB="BATCH_JOB_STAT"
LINE=$DB
export $DB
NoFile()
{
# no file
COLOR="clear"
LINE=$DB" NO BATCH_JOB ERROR FLAG FOUND"
echo $LINE
$BB $BBDISP "status ${MACHINE}.BATCH_JOB_STAT $COLOR `date` $LINE"
exit 1
}
MultiFile()
{
fail_flag=0
# Check more than One file
files=$(ls /opt/rh/flag/*)
for file in $files
do
if [ -f $file ] ; then
## echo "# Error. Flag File found: $file"
fail_flag=1
fi
done
## echo "-- Fail flag is : $fail_flag"
if [ $fail_flag -eq 0 ]; then
# no FAIL status means GOOD
COLOR="green"
LINE=$DB" OK"
echo $LINE
else
COLOR="red"
LINE=$DB" BATCH_JOB ERROR FLAG FILE FOUND. <P> -- $file -- </P> <P> Support <A HREF=http://johndoe/support.htm> Tech </A>"
echo $LINE
fi
$BB $BBDISP "status ${MACHINE}.BATCH_JOB_STAT $COLOR `date` $LINE"
}
# MAIN
file_count=$(ls -l /opt/rh/flag 2>/dev/null |grep -v total |wc -l)
case "$file_count" in
0) NoFile ;;
*) MultiFile ;;
esac
exit 0

I suggest you to see shutil for high-level operations on files in Python. And of course, sys and os (which are used by the first).

how to import python variable to bash

I have two scripts:
Python:
if canceled==True:
os.environ["c"] = "0"
if canceled==False:
os.environ["c"] = "1"
Bash:
kill_jobs()
{
pkill -TERM -P "$BASHPID"
echo $c
if [ $c == "0" ]
then
echo -e "OPERATIONS CANCELED"
elif [ $c == "1" ]
then
echo -e "OPERATIONS FINISHED"
fi
}
trap kill_jobs EXIT
How can I do to pass a python variable to bash script ?
(My level in bash is near to 0 ...)
Thanks
Edit: Now I have this error: (in french)
[: == : opérateur unaire attendu

Or you can try:
os.environ["c"] = "value"
You can set it this way, I guess
Refer

The python script should end with:
print c
Then you use it in bash with:
c=$(python_script)

Bash to Python: flatten directory tree

On Unix-like systems I use this script, which I'd like some help on porting to Python for execution on Windows hosts:
#!/bin/bash
SENTINEL_FILENAME='__sentinel__'
SENTINEL_MD5_CHECKSUM=''
SENTINEL_SHA_CHECKSUM=''
function is_directory_to_be_flattened() {
local -r directory_to_consider="$1"
local -r sentinel_filepath="${directory_to_consider}/${SENTINEL_FILENAME}"
if [ ! -f "${sentinel_filepath}" ]; then
return 1
fi
if [[
"$(
md5 "${sentinel_filepath}" \
| awk '{ print $NF }' 2> /dev/null
)" \
== "${SENTINEL_MD5_CHECKSUM}"
&& \
"$(
shasum -a 512 "${sentinel_filepath}" \
| awk '{ print $1 }' 2> /dev/null
)" \
== "${SENTINEL_SHA_CHECKSUM}"
]]; then
return 0
else
return 1
fi
}
function conditionally_flatten() {
local -r directory_to_flatten="$1"
local -r flatten_into_directory="$2"
if is_directory_to_be_flattened "${directory_to_flatten}"; then
if [ ! -d "${flatten_into_directory}" ]; then
mkdir -v "${flatten_into_directory}"
fi
for file_to_move in $(find ${directory_to_flatten} -type f -maxdepth 1); do
mv \
-v \
-n \
"${file_to_move}" \
"${flatten_into_directory}"
done
fi
}
function flatten_directory() {
local -r directory_to_flatten="$1"
local -r descend_depth="$2"
local -r flattened_directory="${directory_to_flatten}/__flattened__"
if [ ! -d "${directory_to_flatten}" ]; then
printf "The argument '%s' does not seem to be a directory.\n" \
"${directory_to_flatten}" \
>&2
return
fi
find "${directory_to_flatten}" \
-type d \
-maxdepth "${descend_depth}" \
| \
while read directory_path; do
conditionally_flatten \
"${directory_path}" \
"${flattened_directory}"
done
}
n_arguments="$#"
if [ "${n_arguments}" -eq 1 ]; then
flatten_directory "$1" '1' # maybe use a constant, not a "magic #" here?
else
echo usage: "$0" /path/to/directory/to/flatten
fi
unset is_directory_to_be_flattened
unset conditionally_flatten
unset flatten_directory
How would you port this to Win Python? I am a beginner in both Python and Bash scripting..
Feel free to upgrade my implementation as you port it if you find it lacking in any way too, with a justification please. This is not "Code Review" but a "thumbs up/thumbs down" on my effort in Bash would give me a sense of whether I am improving or I should change the way I study altogether...
Here we go, my attempt in Python: (criticise it hard if need be, it's the only way for me to learn!)
#!/usr/bin/env python2.7
import sys
import os
import shutil
SENTINEL_FILENAME=''
SENTINEL_MD5_CHECKSUM=''
SENTINEL_SHA_CHECKSUM=''
DEFAULT_DEPTH = 1
FLATTED_DIRECTORY_NAME = '__flattened__'
def is_directory_to_be_flattened(directory_to_consider):
sentinel_location = os.path.join(directory_to_consider, SENTINEL_FILENAME)
if not os.path.isfile(sentinel_location):
return False
import hashlib
with open(sentinel_location) as sentinel_file:
file_contents = sentinel_file.read()
return (hashlib.md5(file_contents).hexdigest() == SENTINEL_MD5_CHECKSUM
and hashlib.sha512(file_contents).hexdigest() == SENTINEL_SHA_CHECKSUM)
def flatten(directory, depth, to_directory, do_files_here):
if depth < 0:
return
contained_filenames = [f for f in os.listdir(directory)]
if do_files_here:
for filename in contained_filenames:
if filename == SENTINEL_FILENAME:
continue
filepath = os.path.join(directory, filename)
if not os.path.isfile(filepath):
continue
file_to = os.path.join(to_directory, filename)
if not os.path.isdir(to_directory):
os.makedirs(to_directory)
if not os.path.isfile(file_to):
print "Moving: '{}' -> '{}'".format(filepath, file_to)
shutil.move(filepath, file_to)
else:
sys.stderr.write('Error: {} exists already.\n'.format(file_to))
next_depth = depth - 1
for subdirectory in (d for d in contained_filenames if os.path.isdir(d)):
if is_directory_to_be_flattened(subdirectory):
flatten(subdirectory, next_depth, to_directory, True)
def flatten_directory(to_flatten, depth):
to_directory = os.path.join(to_flatten, FLATTED_DIRECTORY_NAME)
if not os.path.isdir(to_flatten):
sys.stderr.write(
'The argument {} does not seem to be a directory.\n'.format(
to_flatten))
return
flatten(to_flatten, depth, to_directory, False)
def main():
if len(sys.argv) == 2:
flatten_directory(sys.argv[1], DEFAULT_DEPTH)
else:
print 'usage: {} /path/to/directory/to/flatten'.format(sys.argv[0])
if __name__ == '__main__':
main()
Although it's obvious from the code, the intent is:
Start at a given directory
Descend up to a certain depth
Consider subdirectories and move all files therein if and only if:
The directory contains a "sentinel file" with a given filename
The sentinel file is actually a sentinel file, not just a file renamed to the same name
Collate files in a __flattened__ directory under the directory in which the search started

Most file dealing functions in Python are in the module "os" - therein you will find
os.rename (for renaming or moving a directoruy entry), os.listdir - which gives you a listing of filenames in the directory, passed as first arg, os.walk - to recursively walk through a directory structure, os.path.walk, to do the same, but with a callback, os.path.exists, os.path.isdir, os.mkdir, are others that might be handy.
For a "quick and dirty" translation you might also cehck "os.system". which allows you to execute a shell command just like it was typed in the shell, and os.popen - which allows access to stdin and stdout of said process. A more carefull translation, tough, would require using anothe module: "subprocess" which can give one full control of a shell command executed as sub process (although if you need find, for example, it won't be available on windows)
Other moduless of interest are sys (sys.argv are the arguments passed to the script) and shutil (with things like copy, rmtree and such)
Your script does some error checking, and it is trivial, given the above funcion names in "os" and basic Python to add them - but a short "just do it" script in Python could be just:
import os, sys
dir_to_flatten = sys.argv[1]
for dirpath, dirnames, filenames in os.walk(dir_to_flatten):
for filename in filenames:
try:
os.rename(os.path.join(dirpath, filename), os.path.join(dir_to_flatten, filename))
except OSError:
print ("Could not move %s " % os.path.join(dirpath, filename))