Related
The following Perl script (my.pl) can read from either the file in the command line arguments or from standard input (STDIN):
while (<>) {
print($_);
}
perl my.pl will read from standard input, while perl my.pl a.txt will read from a.txt. This is very handy.
Is there an equivalent in Bash?
The following solution reads from a file if the script is called with a file name as the first parameter $1 and otherwise from standard input.
while read line
do
echo "$line"
done < "${1:-/dev/stdin}"
The substitution ${1:-...} takes $1 if defined. Otherwise, the file name of the standard input of the own process is used.
Perhaps the simplest solution is to redirect standard input with a merging redirect operator:
#!/bin/bash
less <&0
Standard input is file descriptor zero. The above sends the input piped to your bash script into less's standard input.
Read more about file descriptor redirection.
Here is the simplest way:
#!/bin/sh
cat -
Usage:
$ echo test | sh my_script.sh
test
To assign stdin to the variable, you may use: STDIN=$(cat -) or just simply STDIN=$(cat) as operator is not necessary (as per #mklement0 comment).
To parse each line from the standard input, try the following script:
#!/bin/bash
while IFS= read -r line; do
printf '%s\n' "$line"
done
To read from the file or stdin (if argument is not present), you can extend it to:
#!/bin/bash
file=${1--} # POSIX-compliant; ${1:--} can be used either.
while IFS= read -r line; do
printf '%s\n' "$line" # Or: env POSIXLY_CORRECT=1 echo "$line"
done < <(cat -- "$file")
Notes:
- read -r - Do not treat a backslash character in any special way. Consider each backslash to be part of the input line.
- Without setting IFS, by default the sequences of Space and Tab at the beginning and end of the lines are ignored (trimmed).
- Use printf instead of echo to avoid printing empty lines when the line consists of a single -e, -n or -E. However there is a workaround by using env POSIXLY_CORRECT=1 echo "$line" which executes your external GNU echo which supports it. See: How do I echo "-e"?
See: How to read stdin when no arguments are passed? at stackoverflow SE
I think this is the straightforward way:
$ cat reader.sh
#!/bin/bash
while read line; do
echo "reading: ${line}"
done < /dev/stdin
--
$ cat writer.sh
#!/bin/bash
for i in {0..5}; do
echo "line ${i}"
done
--
$ ./writer.sh | ./reader.sh
reading: line 0
reading: line 1
reading: line 2
reading: line 3
reading: line 4
reading: line 5
The echo solution adds new lines whenever IFS breaks the input stream. #fgm's answer can be modified a bit:
cat "${1:-/dev/stdin}" > "${2:-/dev/stdout}"
The Perl loop in the question reads from all the file name arguments on the command line, or from standard input if no files are specified. The answers I see all seem to process a single file or standard input if there is no file specified.
Although often derided accurately as UUOC (Useless Use of cat), there are times when cat is the best tool for the job, and it is arguable that this is one of them:
cat "$#" |
while read -r line
do
echo "$line"
done
The only downside to this is that it creates a pipeline running in a sub-shell, so things like variable assignments in the while loop are not accessible outside the pipeline. The bash way around that is Process Substitution:
while read -r line
do
echo "$line"
done < <(cat "$#")
This leaves the while loop running in the main shell, so variables set in the loop are accessible outside the loop.
Perl's behavior, with the code given in the OP can take none or several arguments, and if an argument is a single hyphen - this is understood as stdin. Moreover, it's always possible to have the filename with $ARGV.
None of the answers given so far really mimic Perl's behavior in these respects. Here's a pure Bash possibility. The trick is to use exec appropriately.
#!/bin/bash
(($#)) || set -- -
while (($#)); do
{ [[ $1 = - ]] || exec < "$1"; } &&
while read -r; do
printf '%s\n' "$REPLY"
done
shift
done
Filename's available in $1.
If no arguments are given, we artificially set - as the first positional parameter. We then loop on the parameters. If a parameter is not -, we redirect standard input from filename with exec. If this redirection succeeds we loop with a while loop. I'm using the standard REPLY variable, and in this case you don't need to reset IFS. If you want another name, you must reset IFS like so (unless, of course, you don't want that and know what you're doing):
while IFS= read -r line; do
printf '%s\n' "$line"
done
More accurately...
while IFS= read -r line ; do
printf "%s\n" "$line"
done < file
Please try the following code:
while IFS= read -r line; do
echo "$line"
done < file
I combined all of the above answers and created a shell function that would suit my needs. This is from a Cygwin terminal of my two Windows 10 machines where I had a shared folder between them. I need to be able to handle the following:
cat file.cpp | tx
tx < file.cpp
tx file.cpp
Where a specific filename is specified, I need to used the same filename during copy. Where input data stream has been piped through, then I need to generate a temporary filename having the hour minute and seconds. The shared mainfolder has subfolders of the days of the week. This is for organizational purposes.
Behold, the ultimate script for my needs:
tx ()
{
if [ $# -eq 0 ]; then
local TMP=/tmp/tx.$(date +'%H%M%S')
while IFS= read -r line; do
echo "$line"
done < /dev/stdin > $TMP
cp $TMP //$OTHER/stargate/$(date +'%a')/
rm -f $TMP
else
[ -r $1 ] && cp $1 //$OTHER/stargate/$(date +'%a')/ || echo "cannot read file"
fi
}
If there is any way that you can see to further optimize this, I would like to know.
#!/usr/bin/bash
if [ -p /dev/stdin ]; then
#for FILE in "$#" /dev/stdin
for FILE in /dev/stdin
do
while IFS= read -r LINE
do
echo "$#" "$LINE" #print line argument and stdin
done < "$FILE"
done
else
printf "[ -p /dev/stdin ] is false\n"
#dosomething
fi
Running:
echo var var2 | bash std.sh
Result:
var var2
Running:
bash std.sh < <(cat /etc/passwd)
Result:
root:x:0:0::/root:/usr/bin/bash
bin:x:1:1::/:/usr/bin/nologin
daemon:x:2:2::/:/usr/bin/nologin
mail:x:8:12::/var/spool/mail:/usr/bin/nologin
Two principle ways:
Either pipe the argument files and stdin into a single stream and process that like stdin (stream approach)
Or redirect stdin (and argument files) into a named pipe and process that like a file (file approach)
Stream approach
Minor revisions to earlier answers:
Use cat, not less. It's faster and you don't need pagination.
Use $1 to read from first argument file (if present) or $* to read from all files (if present). If these variables are empty, read from stdin (like cat does)
#!/bin/bash
cat $* | ...
File approach
Writing into a named pipe is a bit more complicated, but this allows you to treat stdin (or files) like a single file:
Create pipe with mkfifo.
Parallelize the writing process. If the named pipe is not read from, it may block otherwise.
For redirecting stdin into a subprocess (as necessary in this case), use <&0 (unlike what others have been commenting, this is not optional here).
#!/bin/bash
mkfifo /tmp/myStream
cat $* <&0 > /tmp/myStream & # separate subprocess (!)
AddYourCommandHere /tmp/myStream # process input like a file,
rm /tmp/myStream # cleaning up
File approach: Variation
Create named pipe only if no arguments are given. This may be more stable for reading from files as named pipes can occasionally block.
#!/bin/bash
FILES=$*
if echo $FILES | egrep -v . >&/dev/null; then # if $FILES is empty
mkfifo /tmp/myStream
cat <&0 > /tmp/myStream &
FILES=/tmp/myStream
fi
AddYourCommandHere $FILES # do something ;)
if [ -e /tmp/myStream ]; then
rm /tmp/myStream
fi
Also, it allows you to iterate over files and stdin rather than concatenate all into a single stream:
for file in $FILES; do
AddYourCommandHere $file
done
The following works with standard sh (tested with Dash on Debian) and is quite readable, but that's a matter of taste:
if [ -n "$1" ]; then
cat "$1"
else
cat
fi | commands_and_transformations
Details: If the first parameter is non-empty then cat that file, else cat standard input. Then the output of the whole if statement is processed by the commands_and_transformations.
The code ${1:-/dev/stdin} will just understand the first argument, so you can use this:
ARGS='$*'
if [ -z "$*" ]; then
ARGS='-'
fi
eval "cat -- $ARGS" | while read line
do
echo "$line"
done
Reading from stdin into a variable or from a file into a variable.
Most examples in the existing answers use loops that immediately echo each of line as it is read from stdin. This might not be what you really want to do.
In many cases you need to write a script that calls a command which only accepts a file argument. But in your script you may want to support stdin also. In this case you need to first read full stdin and then provide it as a file.
Let's see an example. The script below prints the certificate details of a certificate (in PEM format) that is passed either as a file or via stdin.
# print-cert script
content=""
while read line
do
content="$content$line\n"
done < "${1:-/dev/stdin}"
# Remove the last newline appended in the above loop
content=${content%\\n}
# Keytool accepts certificate only via a file, but in our script we fix this.
keytool -printcert -v -file <(echo -e $content)
# Read from file
cert-print mycert.crt
# Owner: CN=....
# Issuer: ....
# ....
# Or read from stdin (by pasting)
cert-print
#..paste the cert here and press enter
# Ctl-D
# Owner: CN=....
# Issuer: ....
# ....
# Or read from stdin by piping to another command (which just prints the cert(s) ). In this case we use openssl to fetch directly from a site and then print its info.
echo "" | openssl s_client -connect www.google.com:443 -prexit 2>/dev/null \
| sed -n -e '/BEGIN\ CERTIFICATE/,/END\ CERTIFICATE/ p' \
| cert-print
# Owner: CN=....
# Issuer: ....
# ....
This one is easy to use on the terminal:
$ echo '1\n2\n3\n' | while read -r; do echo $REPLY; done
1
2
3
I don't find any of these answers acceptable. In particular, the accepted answer only handles the first command line parameter and ignores the rest. The Perl program that it is trying to emulate handles all the command line parameters. So the accepted answer doesn't even answer the question.
Other answers use Bash extensions, add unnecessary 'cat' commands, only work for the simple case of echoing input to output, or are just unnecessarily complicated.
However, I have to give them some credit, because they gave me some ideas. Here is the complete answer:
#!/bin/sh
if [ $# = 0 ]
then
DEFAULT_INPUT_FILE=/dev/stdin
else
DEFAULT_INPUT_FILE=
fi
# Iterates over all parameters or /dev/stdin
for FILE in "$#" $DEFAULT_INPUT_FILE
do
while IFS= read -r LINE
do
# Do whatever you want with LINE here.
echo $LINE
done < "$FILE"
done
As a workaround, you can use the stdin device in the /dev directory:
....| for item in `cat /dev/stdin` ; do echo $item ;done
With...
while read line
do
echo "$line"
done < "${1:-/dev/stdin}"
I got the following output:
Ignored 1265 characters from standard input. Use "-stdin" or "-" to tell how to handle piped input.
Then decided with for:
Lnl=$(cat file.txt | wc -l)
echo "Last line: $Lnl"
nl=1
for num in `seq $nl +1 $Lnl`;
do
echo "Number line: $nl"
line=$(cat file.txt | head -n $nl | tail -n 1)
echo "Read line: $line"
nl=$[$nl+1]
done
Use:
for line in `cat`; do
something($line);
done
I have a shell script that needs to take an .so and get all its prefixes, where a prefix is the part of the name, up until the ".so" part + the next part up until the ".".
Example: for 'example.so.1' we'll have the following prefixes: 'example.so', 'example.so.1'
I have a python(3) code that does it, and I want to get a bash equivalent.
Bash wrapper for the Python source:
#!/bin/bash
dst='/tmp'
for src in 'example.so' 'example.so.1' 'example.so.1.2' 'example.so.1.2.3'; do
python3 -c "
import os, sys, itertools as it, re;
so_path = os.path.abspath(sys.argv[1]);
dst = sys.argv[2];
so = os.path.basename(so_path);
so_name = so.split('.')[0];
regex = '\.\w+';
for suffix in it.accumulate(re.findall(regex, so)):
dst_so = os.path.join(dst, so_name + suffix)
print('src: {}. dst: {}'.format(so_path, dst_so))
" "${src}" "${dst}";
done
This is my tryout in bash using awk (it's not complete and only prints the source. I keep tweaking it, but can't get it do exactly what I want):
#!/bin/bash
dst='/tmp'
delimiter='.'
for src in 'example.so' 'example.so.1' 'example.so.1.2' 'example.so.1.2.3'; do
for nubmer_of_delimiters in `seq $(echo ${src} | grep ${delimiter} | wc -l)`; do
echo ${src} :: ${src} | awk -F. '{print $nubmer_of_delimiters}';
done
done
What would be the best way to achieve this? (I'm guessing awk, though I did try to use a bit og cut, sed, etc.
The bash code must run on clean ubuntu18 with no extra installs
Looks like you should be able to just shave in a loop.
given f=example.so.1.2.3, try
$: while [[ "$f" =~ [.]so[.] ]]; do echo "$f"; f=${f%.*}; done; echo "$f"
example.so.1.2.3
example.so.1.2
example.so.1
example.so
If you want the smaller ones first, pass it through a sort.
$: { while [[ "$f" =~ [.]so[.] ]]
> do echo "$f"
> f=${f%.*}
> done
> echo "$f"
> } | sort
example.so
example.so.1
example.so.1.2
example.so.1.2.3
I am new to python and working on some xyz project where i am taking the day-1 dated report, fetching the data and redirecting it into another file on linux machine
here is my code.
#!/usr/bin/python
import os
cur_date = os.popen("date -d '-1 day' '+%Y%m%d'").read()
print (cur_date)
os.system('zgrep "919535144580" /var/tmp/comp?/emse_revres_rdc.log.%s* | grep -v "RI" | cut -d "|" -f 9,10,23,24,26 | sort | uniq -c | sort -nr >> /var/tmp/Andy/test.txt'%cur_date)
it is printing below error.
20180731
**gzip: /var/tmp/comp?/emse_revres_rdc.log.20180731.gz: No such file or directory
sh: line 1: 1: command not found**
but when i am executing the same in shell it is running absolutely fine or if i manually give the date and run the above, it runs successfully.
Please provide your suggestions on the same.
The * has nothing to do with the problem; the string you're substituting with %s ends with a newline, and that newline is what breaks your code.
When you use os.popen('...').read(), you get the entire output of ... -- including the trailing newline, which shell command substitutions implicitly trim.
The best answer would be to rewrite your logic in Python, but the easy answer here is to use such a command substitution, which also avoids trying to pass values into a script via string substitution (which is a fast route to shell-injection security bugs):
shell_script = r'''
cur_date=$(date -d '-1 day' '+%Y%m%d')
zgrep "919535144580" /var/tmp/comp?/emse_revres_rdc.log."$cur_date"* \
| grep -v "RI" \
| cut -d "|" -f 9,10,23,24,26 \
| sort \
| uniq -c \
| sort -nr \
>> /var/tmp/Andy/test.txt
'''
os.system(shell_script)
That said, if you're just going for the shortest change possible, put the following before your original code's os.system() call:
cur_date = cur_date.rstrip('\n')
Basically I want to take as input text from a file, remove a line from that file, and send the output back to the same file. Something along these lines if that makes it any clearer.
grep -v 'seg[0-9]\{1,\}\.[0-9]\{1\}' file_name > file_name
however, when I do this I end up with a blank file.
Any thoughts?
Use sponge for this kind of tasks. Its part of moreutils.
Try this command:
grep -v 'seg[0-9]\{1,\}\.[0-9]\{1\}' file_name | sponge file_name
You cannot do that because bash processes the redirections first, then executes the command. So by the time grep looks at file_name, it is already empty. You can use a temporary file though.
#!/bin/sh
tmpfile=$(mktemp)
grep -v 'seg[0-9]\{1,\}\.[0-9]\{1\}' file_name > ${tmpfile}
cat ${tmpfile} > file_name
rm -f ${tmpfile}
like that, consider using mktemp to create the tmpfile but note that it's not POSIX.
Use sed instead:
sed -i '/seg[0-9]\{1,\}\.[0-9]\{1\}/d' file_name
try this simple one
grep -v 'seg[0-9]\{1,\}\.[0-9]\{1\}' file_name | tee file_name
Your file will not be blank this time :) and your output is also printed to your terminal.
You can't use redirection operator (> or >>) to the same file, because it has a higher precedence and it will create/truncate the file before the command is even invoked. To avoid that, you should use appropriate tools such as tee, sponge, sed -i or any other tool which can write results to the file (e.g. sort file -o file).
Basically redirecting input to the same original file doesn't make sense and you should use appropriate in-place editors for that, for example Ex editor (part of Vim):
ex '+g/seg[0-9]\{1,\}\.[0-9]\{1\}/d' -scwq file_name
where:
'+cmd'/-c - run any Ex/Vim command
g/pattern/d - remove lines matching a pattern using global (help :g)
-s - silent mode (man ex)
-c wq - execute :write and :quit commands
You may use sed to achieve the same (as already shown in other answers), however in-place (-i) is non-standard FreeBSD extension (may work differently between Unix/Linux) and basically it's a stream editor, not a file editor. See: Does Ex mode have any practical use?
One liner alternative - set the content of the file as variable:
VAR=`cat file_name`; echo "$VAR"|grep -v 'seg[0-9]\{1,\}\.[0-9]\{1\}' > file_name
Since this question is the top result in search engines, here's a one-liner based on https://serverfault.com/a/547331 that uses a subshell instead of sponge (which often isn't part of a vanilla install like OS X):
echo "$(grep -v 'seg[0-9]\{1,\}\.[0-9]\{1\}' file_name)" > file_name
The general case is:
echo "$(cat file_name)" > file_name
Edit, the above solution has some caveats:
printf '%s' <string> should be used instead of echo <string> so that files containing -n don't cause undesired behavior.
Command substitution strips trailing newlines (this is a bug/feature of shells like bash) so we should append a postfix character like x to the output and remove it on the outside via parameter expansion of a temporary variable like ${v%x}.
Using a temporary variable $v stomps the value of any existing variable $v in the current shell environment, so we should nest the entire expression in parentheses to preserve the previous value.
Another bug/feature of shells like bash is that command substitution strips unprintable characters like null from the output. I verified this by calling dd if=/dev/zero bs=1 count=1 >> file_name and viewing it in hex with cat file_name | xxd -p. But echo $(cat file_name) | xxd -p is stripped. So this answer should not be used on binary files or anything using unprintable characters, as Lynch pointed out.
The general solution (albiet slightly slower, more memory intensive and still stripping unprintable characters) is:
(v=$(cat file_name; printf x); printf '%s' ${v%x} > file_name)
Test from https://askubuntu.com/a/752451:
printf "hello\nworld\n" > file_uniquely_named.txt && for ((i=0; i<1000; i++)); do (v=$(cat file_uniquely_named.txt; printf x); printf '%s' ${v%x} > file_uniquely_named.txt); done; cat file_uniquely_named.txt; rm file_uniquely_named.txt
Should print:
hello
world
Whereas calling cat file_uniquely_named.txt > file_uniquely_named.txt in the current shell:
printf "hello\nworld\n" > file_uniquely_named.txt && for ((i=0; i<1000; i++)); do cat file_uniquely_named.txt > file_uniquely_named.txt; done; cat file_uniquely_named.txt; rm file_uniquely_named.txt
Prints an empty string.
I haven't tested this on large files (probably over 2 or 4 GB).
I have borrowed this answer from Hart Simha and kos.
This is very much possible, you just have to make sure that by the time you write the output, you're writing it to a different file. This can be done by removing the file after opening a file descriptor to it, but before writing to it:
exec 3<file ; rm file; COMMAND <&3 >file ; exec 3>&-
Or line by line, to understand it better :
exec 3<file # open a file descriptor reading 'file'
rm file # remove file (but fd3 will still point to the removed file)
COMMAND <&3 >file # run command, with the removed file as input
exec 3>&- # close the file descriptor
It's still a risky thing to do, because if COMMAND fails to run properly, you'll lose the file contents. That can be mitigated by restoring the file if COMMAND returns a non-zero exit code :
exec 3<file ; rm file; COMMAND <&3 >file || cat <&3 >file ; exec 3>&-
We can also define a shell function to make it easier to use :
# Usage: replace FILE COMMAND
replace() { exec 3<$1 ; rm $1; ${#:2} <&3 >$1 || cat <&3 >$1 ; exec 3>&- }
Example :
$ echo aaa > test
$ replace test tr a b
$ cat test
bbb
Also, note that this will keep a full copy of the original file (until the third file descriptor is closed). If you're using Linux, and the file you're processing on is too big to fit twice on the disk, you can check out this script that will pipe the file to the specified command block-by-block while unallocating the already processed blocks. As always, read the warnings in the usage page.
The following will accomplish the same thing that sponge does, without requiring moreutils:
shuf --output=file --random-source=/dev/zero
The --random-source=/dev/zero part tricks shuf into doing its thing without doing any shuffling at all, so it will buffer your input without altering it.
However, it is true that using a temporary file is best, for performance reasons. So, here is a function that I have written that will do that for you in a generalized way:
# Pipes a file into a command, and pipes the output of that command
# back into the same file, ensuring that the file is not truncated.
# Parameters:
# $1: the file.
# $2: the command. (With $3... being its arguments.)
# See https://stackoverflow.com/a/55655338/773113
siphon()
{
local tmp file rc=0
[ "$#" -ge 2 ] || { echo "Usage: siphon filename [command...]" >&2; return 1; }
file="$1"; shift
tmp=$(mktemp -- "$file.XXXXXX") || return
"$#" <"$file" >"$tmp" || rc=$?
mv -- "$tmp" "$file" || rc=$(( rc | $? ))
return "$rc"
}
There's also ed (as an alternative to sed -i):
# cf. http://wiki.bash-hackers.org/howto/edit-ed
printf '%s\n' H 'g/seg[0-9]\{1,\}\.[0-9]\{1\}/d' wq | ed -s file_name
You can use slurp with POSIX Awk:
!/seg[0-9]\{1,\}\.[0-9]\{1\}/ {
q = q ? q RS $0 : $0
}
END {
print q > ARGV[1]
}
Example
This does the trick pretty nicely in most of the cases I faced:
cat <<< "$(do_stuff_with f)" > f
Note that while $(…) strips trailing newlines, <<< ensures a final newline, so generally the result is magically satisfying.
(Look for “Here Strings” in man bash if you want to learn more.)
Full example:
#! /usr/bin/env bash
get_new_content() {
sed 's/Initial/Final/g' "${1:?}"
}
echo 'Initial content.' > f
cat f
cat <<< "$(get_new_content f)" > f
cat f
This does not truncate the file and yields:
Initial content.
Final content.
Note that I used a function here for the sake of clarity and extensibility, but that’s not a requirement.
A common usecase is JSON edition:
echo '{ "a": 12 }' > f
cat f
cat <<< "$(jq '.a = 24' f)" > f
cat f
This yields:
{ "a": 12 }
{
"a": 24
}
Try this
echo -e "AAA\nBBB\nCCC" > testfile
cat testfile
AAA
BBB
CCC
echo "$(grep -v 'AAA' testfile)" > testfile
cat testfile
BBB
CCC
I usually use the tee program to do this:
grep -v 'seg[0-9]\{1,\}\.[0-9]\{1\}' file_name | tee file_name
It creates and removes a tempfile by itself.
I am using a python script which takes in few arguments:
pipeline.py -t reg.bed -n WT -s run -ID 567
Inside the python script, i am using awk script which should take in the arguments passed above:
For instance, it should take the argument by name, ID = 567 and print it in a file:
awk '{print "567", $0}' input.txt > output.txt
Could anyone help me, how i can pass the string "567" to the awk command.
Thanks in advance.
use awk's -v option
awk -v id=567 '{print id, $0}' input.txt > output.txt
Although this awk script is so simple, surely you should implement it in python.
Special attention should be paid to the cmd format inside os.system(quote shell cmd in python script).
I'll give an example to illustrate
I have a file named bNormal1.bedgraph.
I want to extract the lines that satisfy my condition:
(NR==1 && $1== "track")||($1=="chr1" && $2>220138080 && $3 < 220138189)
And I want to use variables replacing some arguments:
chrom="chr1" $2=220138080 $3= 220138189
This is the right python cmd:
import os
chrom="chr1"
start=220138080
end=220138189
bedgraphfile="bNormal1.bedgraph"
os.system('awk \'(NR==1 && $1== "track")||($1=="%s" && $2> %d && $3 < %d) {{print $0}} \' %s'%(chrom,start,end,bedgraphfile))**
cmd output:
track type=bedGraph name=bNormal1 visibility=full color=54,54,54
chr1 220138087 220138187 0.226285
0
Full content of the query file bNormal1.bedgraph:
track type=bedGraph name=bNormal1 visibility=full color=54,54,54
chr1 220138087 220138187 0.226285
chr1 220138229 220138314 0.226285