I get the below error while trying to execute a shell script,
$'\r': command not found: line 2:
Please suggest a solution for the same.
Below are the intial lines used in the script,
#!/bin/sh
if [[ $# -lt 1 ]]; then
echo "ERROR Environment argument missing <dev,test,qa,prod>"
export RC=50
exit $RC
fi
Your problem is that the file has Windows line endings. This can be caused by editing a file in Windows and trying to run it on a non-Windows system.
You can fix this problem using dos2unix to convert the line endings:
dos2unix ConstruedTermsXMLGenerator.sh
The corresponding utility to convert in the other direction is unix2dos.
Some systems have fromdos and todos.
You can use sed -i 's/\r$//' scriptname.sh
Replace the scriptname with actual script name.
I used notepad++ to convert the line endings.
Edit > EOL Conversion > UNIX/OSX Format
I had the same error and what I did was to transform the characters '\r' to '\n'. using this line:
tr '\r' '\n' < oldfile.sh > newfile.sh
mv newfile.sh oldfile.sh
chmod +x oldfile.sh
./oldfile.sh
I think you could also delete the '\r' characters by using:
tr -d '\r' < oldfile.sh > newfile.sh
tr is the command trasnform, and the -d is delete the following character.
I think the shell actually doesn't like '\r' character.
I had this exact issue when creating a .sh file on a Mac (unix) and executing it in Linux.
Turns out that I had to set FileZilla FTP settings to 'Binary' transfer type:
"Settings>Transfers>File Types>Default transfer type" to "Binary" (instead of "Auto")
I got a different error message when running your script under /bin/sh, but when I switched to /bin/bash, it worked fine:
$ cat foo.sh
#!/bin/sh
if [[ $# -lt 1 ]];
then echo "ERROR Environment argument missing"
RC=50
exit $RC
fi
$ sh foo.sh
foo.sh: 6: [[: not found
$ bash foo.sh
ERROR Environment argument missing
You've built in a bashism. This may or may not be a big deal for your organization. If you want to keep using bash-specific features, change the shebang line to #!/bin/bash and see if that helps.
You can just replace '\r' with '' (nothing), this will do the trick.
If you have any GUI editor (Notepad++) recommended directly find/replace.
If you are only doing it on Linux server then use something like:
sed -i 's/old-text/new-text/g' input.txt
But Since '\r' comes only if you have edited it in windows as it's a new line character (\r\n), so first method should work.
Related
So I know there are plenty of answers on stack overflow for ignoring errors in a bash script. None of them seem to work for the source command though.
I have tried the tried and true source ../bin/activate || true
I have tried setting set -e before running the command
I have tried source ../bin/activate 2>&1 /dev/null
I have tried setting set +e before running the command. This did not work either.
but on every run-through of this code, I receive
run.01: line 12: ../bin/activate: No such file or directory
The context for this problem is that I'm creating a simple bash script that runs some python code. The user is instructed for how to create a specific virtual environment and this line will automatically activate it if they set it up correctly, otherwise, this should ignore failing to activate running and continue running the commands in whatever environment is currently activated.
# Try virtual environment
source ../bin/activate || true
## Run.
code="../src-01/driver.v01.py"
## --------------------
graphInputFile="undirected_graph_01.inp"
graphType="undirected"
srcColId="0"
desColId="1"
degreeFind="2"
outFile="count.undirected.num.nodes.out"
python $code -inpGraphFile $graphInputFile -graphFormat $graphType -colSrcId $srcColId -colDesId $desColId -degreeFind $degreeFind -output_file $outFile
The python command should execute regardless of whether or not the source ../bin/activate command succeeds or not. I'm a little a loss for why none of these solutions are working and am currently under the assumption that source might do something different than a normal command given the circumstances.
EDIT:
I added the shebang #!/bin/bash -x to my file as requested but this did not do anything.
I this is my exact terminal output when I run this script.
Lucas-Macbook:test-01 lucasmachi$ sh run.01
run.01: line 14: ../bin/activate: No such file or directory
Lucas-Macbook:test-01 lucasmachi$
Where run.01 is the name of the bash script.
Also to clarify, the code I showed is not censored. that is the entire script (except now with the mentioned shebang at the top.)
This is a bug in Bash versions before 4.0 (macOS is stuck on 3.2).
Given this script:
#!/bin/bash
set -e
echo "Running $BASH_VERSION"
source "does not exist" || true
echo "Continuing"
Running on macOS will say:
macos$ ./myscript
Running 3.2.57(1)-release
./myscript: line 3: does not exist: No such file or directory
While on a modern system with an updated bash version, you get the expected behavior:
debian$ ./myscript
Running 5.0.11(1)-release
./myscript: line 3: does not exist: No such file or directory
Continuing
If you need to support macOS and Bash 3.2, run set +e first to disable errexit, and optionally re-enable it afterwards.
You could add a validation in your code to check whether the file exists before sourcing it :
[[ -f "../bin/activate" ]] && source ../bin/activate
Using the -f flag before a path will return true if the path file exists, and false if it doesn't.
The one liner syntax for bash if statements is as follow :
[[ Condition that returns true or false ]] && Exec if true
I ran into this issue with gitlab pipelines. None of the above worked for me, but combining the answers did:
[[ -f "${ENVIRONMENT}_test" ]] && source "${ENVIRONMENT}_test" || true
This gets me to where I need to go, hopefully it will help someone in the future.
CONTEXT
I am working on a simulation cluster.
In order to make as flexible as possible (working with different simulation soft) , we created a python file that parse a config file defining environment variables, and command line to start the simulation. This command is launched through SLURM sbatch command (shell $COMMAND)
ISSUE
From python, all Environment variables are enrolled reading the config file
I have issue with variable COMMAND that is using other environment variables (displayed as shell variable)
For example
COMMAND = "fluent -3ddp -n$NUMPROCS -hosts=./hosts -file $JOBFILE"
os.environ['COMMAND']=COMMAND
NUMPROCS = "32"
os.environ['NUMPROCS']=NUMPROCS
[...]
exe = Popen(['sbatch','template_document.sbatch'], stdout=PIPE, stderr=PIPE)
sbatch distribute COMMAND to all simulation nodes as COMMAND being a command line
COMMAND recalls other saved env. variables. Shell interprets it strictly as text... Which makes the command line fails. it is strictly as a string using $ not variable for example :
'fluent -3ddp -n$NUMPROCS -hosts=./hosts -file $JOBFILE'
SOLUTION I AM LOOKING FOR
I am looking for a simple solution
Solution 1: A 1 to 3 python command lines to evaluate the COMMAND as shell command to echo
Solution 2: A Shell command to evaluate the variables within the "string" $COMMAND as a variable
At the end the command launched from within sbatch should be
fluent -3ddp -n32 -hosts=./hosts -file /path/to/JOBFILE
You have a few options:
Partial or no support for bash's variable substitution, e.g. implement some python functionality to reproduces bash's $VARIABLE syntax.
Reproduce all of bash's variable substitution facilities which are supported in the config file ($VARIABLE, ${VARIABLE}, ${VARIABLE/x/y}, $(cmd) - whatever.
Let bash do the heavy lifting, for the cost of performance and possibly security, depending on your trust of the content of the config files.
I'll show the third one here, since it's the most resilient (again, security issues notwithstanding). Let's say you have this config file, config.py:
REGULAR = "some-text"
EQUALS = "hello = goodbye" # trap #1: search of '='
SUBST = "decorated $REGULAR"
FANCY = "xoxo${REGULAR}xoxo"
CMDOUT = "$(date)"
BASH_A = "trap" # trap #2: avoid matching variables like BASH_ARGV
QUOTES = "'\"" # trap #3: quoting
Then your python program can run the following incantation:
bash -c 'source <(sed "s/^/export /" config.py | sed "s/[[:space:]]*=[[:space:]]*/=/") && env | grep -f <(cut -d= -f1 config.py | grep -E -o "\w+" | sed "s/.*/^&=/")'
which will produce the following output:
SUBST=decorated some-text
CMDOUT=Thu Nov 28 12:18:50 PST 2019
REGULAR=some-text
QUOTES='"
FANCY=xoxosome-textxoxo
EQUALS=hello = goodbye
BASH_A=trap
Which you can then read with python, but note that the quotes are now gone, so you'll have to account for that.
Explanation of the incantation:
bash -c 'source ...instructions... && env | grep ...expressions...' tells bash to read & interpret the instructions, then grep the environment for the expressions. We're going to turn the config file into instructions which modify bash's environment.
If you try using set instead of env, the output will be inconsistent with respect to quoting. Using env avoids trap #3.
Instructions: We're going to create instructions for the form:
export FANCY="xoxo${REGULAR}xoxo"
so that bash can interpret them and env can read them.
sed "s/^/export /" config.py prefixes the variables with export.
sed "s/[[:space:]]*=[[:space:]]*/=/" converts the assignment format to syntax that bash can read with source. Using s/x/y/ instead of s/x/y/g avoids trap #1.
source <(...command...) causes bash to treat the output of the command as a file and run its lines, one by one.
Of course, one way to avoid this complexity is to have the file use bash syntax to begin with. If that were the case, we would use source config.sh instead of source <(...command...).
Expressions: We want to grep the output of env for patterns like ^FANCY=.
cut -d= -f1 config.py | grep -E -o "\w+" finds the variable names in config.py.
sed "s/.*/^&=/" turns variable names like FANCY to grep search expressions such as ^FANCY=. This is to avoid trap #2.
grep -f <(...command...) gets grep to treat the output of the command as a file containing one search expression in each line, which in this case would be ^FANCY=, ^CMDOUT= etc.
EDIT
Since you actually want to just pass this environment to another bash command rather than use it in python, you can actually just have python run this:
bash -c 'source <(sed "s/^/export /" config.py | sed "s/[[:space:]]*=[[:space:]]*/=/") && $COMMAND'
(assuming that COMMAND is specified in the config file).
It seems I have not explained well enough the issue, but your 3rd solution seems replying to my expectations... though so far I did not manage to adapt it
Based on your 3rd solution BASH, I will make it more straight :
Let's say I have got following after running python, and this that cannot be modified
EXPORT COMMAND='fluent -3ddp -n$NUMPROCS -hosts=./hosts -file $JOBFILE'
EXPORT JOBFILE='/path/to/jobfile'
EXPORT NUMPROCS='32'
EXPORT WHATSOEVER='SPECIFIC VARIABLE TO SIMULATION SOFTWARE'
I wish to execute the following from the slurm batch file (bash), using $COMMAND / $JOBFILE /$NUMPROCS
fluent -3ddp -n32-hosts=./hosts -file /path/to/jobfile
Please note : I have backup solution in python - I managed to substitute $VARIABLE by its value - based on the assumption $VARIABLE is not composed by another $variable... using regex substitution... just it looks so many lines to have what seemed to me simple request
I'm attempting to create a .sh file to batch a number of runs of a neural network on Python whilst on holidays.
At the moment I have been calling this from the command line:
python neural_network_trainer.py [args]
I now have a .sh script written:
#!/bin/bash
python neural_network_trainer.py [args]
# Repeated with varied args
That I am attempting to call in the same terminal as the original command line was running:
./august_hols.sh
I get the following error:
File "/data/Python-3.6.9/lib/python3.6/site.py", line 177
file=sys.stderr)
^
SyntaxError: invalid syntax
Where the Python install is in /data (for reasons).
Running which on the command line reports the correct Python directory set via an alias in ~/.bashrc:
alias python=/data/Python-3.6.9/bin/python3
But running which between the Bash shebang and the first python call reports /bin/python.
I've attempted to set the alias again at the start of the .sh script to no avail. I'm scratching my head as this is exact process I have used elsewhere, albeit not on this precise PC. I can copy the exact command from the top of the bash file into the terminal and it runs fine, try and call ./august_hols.sh and get the above Python error.
Where is Bash getting that path from, and why is it not using my expected route through ~/.bashrc?
Bash sub-shell does not inherit alias in the main shell
You can source the script (run in the main shell), instead of execute it (run in the sub-shell)
source script.sh
EDIT:
Solution 2:
Run bash as the login shell so ~/.bashrc is executed, so your alias is loaded before your script.
The subshell needs to be interactive to enable alias, because alias is enabled by default only for interactive shell, but script is non-interactive by default.
bash --login -i script.sh
Solution 3:
Similar to above, except alias is enabled explicitly
bash --login -O expand_aliases script.sh
Have you tried:
python=/data/Python-3.6.9/bin/python3 ./[your_bash].sh
In your .sh
Do this
#!/usr/bin/env bash
export PATH=/data/Python-3.6.9/bin:$PATH
exec python neural_network_trainer.py "$#"
Aliases are tricky.
A maybe more nasty solution
mapfile < <(declare -p | grep -m 1 BASH_ALIASES) && bash script.sh "${MAPFILE[#]}"
within your script you will need
shopt -s expand_aliases
eval $1
echo ${BASH_ALIASES[python]}
python --version
How about this:
#!/bin/bash
/data/Python-3.6.9/bin/python3 neural_network_trainer.py [args]
# Repeated with varied args
If I use the following command in the Git Bash, it works fine. The Output from the curl are write into the file output.txt
curl -k --silent "https://gitlab.myurl.com/api/v4/groups?page=1&per_page=1&simple=yes&private_token=mytoken&all?page=1&per_page=1" > output.txt
Python Code:
import subprocess, shlex
command = shlex.split("curl -k --silent https://gitlab.myurl.com/api/v4/groups?page=1&per_page=1&simple=yes&private_token=mytoken&all?page=1&per_page=1 > output.txt")
subprocess.Popen(command)
The Python code write nothing in my file "output.txt".
How can I write in the output.txt or get the Output direct in Python?
You cannot use redirection directly with subprocess, because it is a shell feature. Use check_output:
import subprocess
command = ["curl", "-k", "--silent", "https://gitlab.myurl.com/api/v4/groups?page=1&per_page=1&simple=yes&private_token=mytoken&all?page=1&per_page=1"]
output = subprocess.check_output(command)
You could use the command in another way in order to write to output.txt:
curl -k --silent "https://gitlab.myurl.com/api/v4/groups?page=1&per_page=1&simple=yes&private_token=mytoken&all?page=1&per_page=1" --output output.txt
Also you should consider that output.txt might not be saved in the directory you expect so I would also advice you to name output.txt in another way, a unique one, then update the locate linux command database (see updatedb command) and then search the file with locate.
PS: all this make sense when you need the writing to output.txt (your question accepts this situation too, so I hope it helps)
I am running iperf between a set of hosts that are read from a txt file, here's how I am running it:
h1,h2 = net.getNodeByName(node_id_1, node_id_2)
net.iperf((h1, h2))
It runs well and displays the results. But, I want to save the output of iperf result in a separate txt file. Does anyone know how I can apply it on the above code?
In order to store the results of iperf test in a file , add | tee followed by the filename.txt to your command line for example :
iperf -c ipaddress -u -t 10 -i 1 | tee result.txt
Do you already try:
--output test.log
(in newer versions --logfile)
or using
youriperfexpr > test.log
I had this problem as well. Although the manpage specifies "-o" or "--output" to save your output to a file, this does not actually work.
It seems that this was marked as "WontFix":
https://code.google.com/p/iperf/issues/detail?id=24:
Looks like -o/--output existed in a previous version but in not in the
current version. The consensus in yesterday's meeting was that if
--output existed then we should fix it, otherwise people should just use shell redirection and we'll mark this WontFix. So, WontFix.
So maybe just use typescript or ">test.log" as suggested by Paolo
I think the answer is given by Chiara Contoli in here: iperf result in output file
In summary:
h1.cmd('iperf -s > server_output.txt &')
h2.cmd('iperf -t 5 -c ', h1.IP() + ' > client_output.txt &')
Since you are running it on python, another method to save the result is to use popen:
popen( '<command> > <filename>', shell=True)
For example:
popen('iperf -s -u -i 1 > outtest.txt', shell=True)
You can check this for further information:
https://github.com/mininet/mininet/wiki/Introduction-to-Mininet#popen
If you need to save a file in the txt format.
On the client machine run cmd(adm) and after that you need to write this:
cd c:\iperf3
iperf3.exe -c "you server address" -p "port" -P 10 -w 32000 -t 0 >> c:\iperf3\text.txt
(-t 0) - infinity
On the client machine, you will see a black screen in cmd. It's normal. You will see all the process in the server machine. After your test, on the client machine in cmd need push ctrl+ c and after (y).
Your file in directory c:\iperf3\text.txt after that collect all information about this period.
If you push close in cmd this file text.txt will be empty.
Recommended open this file in NotePad or WordPad for the correct view.
Server
iperf3 -s -p -B >> &
Client
iperf3 -p -c <server_ip> -B <client_ip> -t 5 >>
Make sure you kill the iperf process on the server when done