i have start.sh bash script that is running though CRON JOB on ubuntu server
start.sh contains bellow mentioned lines of code
path of start.sh is /home/ubuntu/folder1/folder2/start.sh
#!/bin/bash
crawlers(){
nohup scrapy crawl first &
nohup scrapy crawl 2nd &
wait $!
nohup scrapy crawl 3rd &
nohup scrapy crawl 4th &
wait
}
cd /home/ubuntu/folder1/folder2/
PATH=$PATH:/usr/local/bin
export PATH
python init.py &
wait $!
crawlers
python final.py
my issue is if i run start.sh my myself on command line it outputs in nohup.out file
but when it executes this bash file through cronjob (although scripts are running fine) its not producing nohup.out
how can i get output of this cronjob in nohup.out ?
Why are you using nohup? nohup is a command that tells the running terminal to ignore the hangup signal. cron, however, has no hangup signal, because it is not linked to a terminal session.
In this case, instead of:
nohup scrapy crawl first &
You probably want:
scrapy crawl first > first.txt &
The last example also works in a terminal, but when you close the terminal, the hangup signal (hup) is sent, which ends the program.
Related
nohup scrapy crawl test -o fed.csv &
nohup scrapy crawl test -o feder.csv &
nohup scrapy crawl fullrun -o dez.csv &
Hello, How can I run the following commands in sequence (when the first one finishes,the next one gets executed) on Virtual machine (Ubuntu terminal)
I wasn't sure if to ask this question on askubuntu.com or here, I hope this is correct place to ask
If you want to run commands one by one and only when the previous one completed successfully you can use :
command1 && command2
if your commands are independent of each other then you can use:
command1; command2
I have a python script. Script have selenium with Chrome and go to a website, take data and put in CSV file.
This is a very long work.
I put the script on the server. And run. All work.
But I need script work in the background.
chmod +x createdb.py
nohup python ./createdb.py &
And I see
(env)$ nohup ./createdb.py &
[1] 32257
(env)$ nohup: ignoring input and appending output to 'nohup.out'
Press Enter.
(env)$ nohup ./createdb.py &
[1] 32257
(env)$ nohup: ignoring input and appending output to 'nohup.out'
[1]+ Exit 1 nohup ./createdb.py
Then it runs and immediately writes errors to the file, that Chrome did not start or there was no click.
I want to remind you that if you start without nohup, then everything will work.
What am I doing wrong? How to run a script?
Thank you very much.
You could create a background daemon (service)
You taged Ubuntu 16.04 it means you got systemd, for more information on how to set it up, please visit this link
create a file called <my_service>.system
and put it there: /etc/systemd/system
you systemd unit could look like this:
[Unit]
Description=my service
After=graphical.target
[Service]
Type=simple
WorkingDirectory=/my_dir
ExecStart=python my_script.py
[Install]
WantedBy=multi-user.target
then all you have to do is, reload systemd manage and start your service:
sudo systemctl daemon-reload
sudo systemctl myservice start
You can use the screen command, it works perfectly.
Here is a very good link: https://www.rackaid.com/blog/linux-screen-tutorial-and-how-to/
You can use a simple command, from the env directory:
(env)$ python /path/to/createdb.py > logger.txt 2>&1 &
This will help for storing the program logs in a defined file called "logger.txt"
I have followed a few posts on here trying to run either a python or shell script on my ec2 instance after every boot not just the first boot.
I have tried the:
[scripts-user, always] to /etc/cloud/cloud.cfg file
Added script to ./scripts/per-boot folder
and
adding script to /etc/rc.local
Yes the permissions were changed to 755 for /etc/rc.local
I am attempting to pipe the output of the file into a file located in the /home/ubuntu/ directory and the file does not contain anything after boot.
If I run the scripts (.sh or .py) manually they work.
Any suggestions or request for additional info to help?
So the current solution appears to be a method I wrote off in my initial question post as I may have not performed the setup exactly as outline in the link below...
This link -->
How do I make cloud-init startup scripts run every time my EC2 instance boots?
The link shows how to modify the /etc/cloud/cloud.cfg file to update scripts-user to [scripts-user, always]
Also that link says to add your *.sh file to /var/lib/cloud/scripts/per-boot directory.
Once you reboot your system your script should have executed and you can verify this in: sudo cat /var/log/cloud-init.log
if your script still fails to execute try to erase the instance state of your server with the following command: sudo rm -rf /var/lib/cloud/instance/*
--NOTE:--
It appears print commands from a python script do not pipe (>>) as expected but echo commands pipe easily
Fails to pipe
sudo python test.py >> log.txt
Pipes successfully
echo "HI" >> log.txt
Is this something along the lines that you want?
It copies the script to the instance, connects to the instance, and runs the script right away.
ec2 scp ~/path_to_script.py : instance_name -y && ec2 ssh instance_name -yc "python script_name.py" 1>/dev/null
I read that the use of rc.local is getting deprecated. One thing to try is a line in /etc/crontab like this:
#reboot full-path-of-script
If there's a specific user you want to run the script as, you can list it after #reboot.
I have a python script that I want to execute in the background on my unix server. The catch is that I need the python script to wait for the previous step to finish before moving onto the next task, yet I want my job to continue to run after I exit.
I think I can set up as follows but would like confirmation:
An excerpt of the script looks like this where command 2 is dependent on the output from command 1 since it outputs an edited executable file in same directory. I would like to point out that commands 1 and 2 do not have the nohup/& included.
subprocess.call('unix command 1 with options', shell=True)
subprocess.call('unix command 2 with options', shell=True)
If when I initiate my python script like so:
% nohup python python_script.py &
Will my script run in the background since I explicitly did not put nohup/& in my scripted unix commands but instead ran the python script in the background?
yes, by running your python script with nohup (no hangup), your script won't keel over when the network is severed and the trailing & symbol will run your script in the background.
You can still view the output of your script, nohup will pipe the stdout to the nohop.out file. You can babysit the output in real time by tailing that output file:
$ tail -f nohop.out
quick note about the nohup.out file...
nohup.out The output file of the nohup execution if
standard output is a terminal and if the
current directory is writable.
or append the command with & to run the python script as a deamon and tail the logs.
$ nohup python python_script.py > my_output.log &
$ tail -f my_output.log
You can use nohup
chomd +x /path/to/script.py
nohup python /path/to/script.py &
Or
Instead of closing your terminal, use logout It is not SIGHUP when you do logout thus the shell won't send a SIGHUP to any of its children.children.
I am trying to run a shell script with the nohup command. The shell script takes an array of files, runs a python program on each file in a loop, and appends the output to a file. This works fine on the server, but if I try to use the nohup command it does not work. I have successfully run other programs using nohup on this server, just not this script.
#!/bin/sh
ARRAY=(0010.dat 0020.dat 0030.dat)
rm batch_results.dat
touch batch0.dat
touch batch_results.dat
for file in ${ARRAY[#]}
do
python fof.py $file > /dev/null
python mdisk5.py > ./batch0.dat
tail -1 batch0.dat
tail -1 batch0.dat >> batch_results.dat
done
The program works fine when I run it while staying connected to the server, for example
./batch.sh > /dev/null &
./batch.sh > ./output.txt &
However, when I try to run it with the nohup command,
nohup ./batch.sh > /dev/null &
if I exit the server and come back the output file (batch_results.dat) does not have any data.
I am sure I am missing some simple fix or command in here. Any ideas?
Edit:
The program fof.py produces two files that are used as input for mdisk5.py.
When I exit the server while running nohup, these two files are produced, but only for the first input file '0010.dat'. The output files batch0.dat and batch_results.dat remain empty.
Here's your problem:
#!/bin/sh
sh does not support arrays. Either change your shebang line to invoke a shell that does support arrays, like bash, or use a normal, whitespace separated string of your data files in a like
DAT_FILES="0010.dat 0020.dat 0030.dat"
for file in $DAT_FILES
do
...
done