I need some help running a python script from crontab:
The script looks for subfolders from current path and does something to them, also extracts a zip file located in the same folder of the script into each found subfolder.
When I go with cd /folder/folder then python script.py is all good. But when run it with crontab it runs in users home folder and not where the script is placed.
To overcome this I placed in crontab something like this:
* * * * cd /folder_of_scrpit/ && /python_path/python script.py >> log.txt
and works as needed but feels weird, is there a better way of achieving this?
You can cd in crontab the way you do it. Or you can call os.chdir() in your script. In the latter case you can write the directory in the script or pass it as a command line argument: /python path/python script.py /folder/folder.
Related
I have a python script that contains a "spark-shell -i script.scala" but the cron does not continue to the calling of the scala
python.py
import os
os.system(spark-shell -i script.scala)
script.scala
//pseudo code create folder named foo
cron
* * * * * python python.py
*does not create folder foo
if i run it manually, it will create the folder,
but in cron it does not create the folder, how will i make this able to work?
ps. tried using screen command but i would like to know if there is another way.
experts, i have a 50 nos of python scripts that i want to run one by one on the same folder.But problem is that in the system where i work many people also use it very often.so there is a possibility of misuse of my python scripts.so i want to copy the 50 nos of python scripts to the same folder and delete the python scripts(script1.py,script2.py...) from the same folder after running the master script which content all the running scripts.so i did something like as below
#!/bin/sh
python script1.py
python script2.py
python script3.py
python script4.py
python script5.py
python script6.py
python script7.py
python script8.py
python script9.py
python script10.py
....
but in the above script problem is that if i delete the python scripts only first python script i.e. python script1.py is running not others.I hope experts will help me overcoming this problem.
You are doing it wrong.
You should write a main.py program which will execute your functions one after the other.
On the way that you are currently using only first one is executed as cli is passed to the script1.py and other are ignored if your script do not exit.
So for example if you made a infinite loop error then yeah other will never run.
I would suggest learning python imports and execution.
One way is:
# main.py you have the script1.py and others in same folder.
# you need to have the scripts inside of the functions so you could as well do that in same file.
# so script1 should look like:
def script1_func(): # any arguments that are needed can be passed
# code of your script here
return value # if your script returns some value write it there
# then in main.py
import script1
import script2
import script3
if __name__ == '__main__':
script1.script1_func()
script2.script2_func()
script3.script3_func()
# on that way you can execute them all in sequence. If needed to use paralelism check multithreading.
# if you need to remove them or any other files.
import os
basedir = os.path.abspath(os.path.dirname(__file__))
# this will give you a path to the file and then you just issue delete
if os.path.exists(basedir + "script1.py"):
os.remove(basedir + "script1.py")
else:
print("The file does not exist")
os.rmdir(basedir) # this will delete whole folder
Btw it is better to use better conventions and do not duplicate similar functions.
https://gist.github.com/MarkoShiva/7584151b08a095a1708bdee339f61a65
Other way to do the same is to run a exec on each of the files from bash which might lead to too much load of the machine if you have more scripts then cores.
That is other answer.
So other user suggested you using find command to execute scripts in parallel.
For your exact needs, you can use below solution
I have create 2 sample python files and a shell script to invoke both the scripts.
➜ 65976750 ls
sandex.sh script1.py script2.py
Debug mode execution
➜ 65976750 bash -x sandex.sh
+ mkdir ./sandexec
+ find . -name '*.py' -exec cp '{}' ./sandexec ';'
+ python3 ./sandexec/script1.py
running python script 1
+ python3 ./sandexec/script2.py
Running from script2
+ rm -rf ./sandexec
original script
➜ 65976750 cat sandex.sh
mkdir ./sandexec
find . -name "*.py" -exec cp {} ./sandexec \;
python3 ./sandexec/script1.py
python3 ./sandexec/script2.py
rm -rf "./sandexec"
➜ 65976750 ls
sandex.sh script1.py script2.py
➜ 65976750
I have a python script that queries a database. I run it from the terminal with python3 myscript.py
I've added a cron task for it in my crontab file
*/30 9-17 * * 1-5 python3 /path/to/my/python/script\ directory\ space/myscript.py
The script imports a function in the same directory that parses login info for a database located in database.ini in the same directory. The database.ini is:
[postgresql]
host=my-db-host-1-link.11.thedatabase.com
database=dbname
user=username
password=password
port=10898
But currently cron outputs to the file in my mail folder:
Section postgresql not found in the database.ini file
The section is clearly present in the database.ini file, so what am I missing here?
Instead of running "python3 myscript.py" in the directory where it is present, try running it from some other directory (like home directory). Most likely you will see the same issue.
Note that cron's current-working-directory is different on different systems. So, the safest method is to explicitly switch to the directory where your script is and run the command there:
cd /path/to/my/python/script\ directory\ space/ && python3 myscript.py
Try this:
import os
...
change --> filename=database.ini
for --------> filename=os.path.dirname(__file__)+'/database.ini'
I'm writing a python script which will be placed in a location. I want to execute it just like a command. for ex.
$ find_branch test
where find_branch is a script placed in anywhere in the system.
I would like to know how to achieve this. I can run it on the place where the script is present by chmod u+x on the script and removing the .py from the script
sudo nano /usr/bin/testpyscript
Then inside the script:
#!/usr/bin/python
print("I'm a python script")
Give it x permission:
sudo chmod +x /usr/bin/testpyscript
Now you can use it as a regular command:
bash-4.2$ testpyscript
I'm a python script
It doesn't have to be exactly at /usr/bin, any location that is inside your $PATH will do. Let's say you want it to be located at some folder inside your home directory, you could do something like this:
pwd
/home/brunorb
mkdir somedir
sudo mv /usr/bin/testpyscript somedir/
export PATH=$PATH:/home/brunorb/somedir/
testpyscript # from any folder in the system
I'm a python script
Make sure python has been added to your path and #!/usr/bin/python is located at the top of your script.
Note You could just try adding your script to your /usr/local/bin/ directory and give it the proper permissions.
sudo cp <your script> /usr/local/bin/
You have a number of options on how to achieve this.
Add the location where you put the script to your PATH environment variable, for example in your ~/.bashrc script:
export PATH="${PATH}:/folder/where/you/put/the/script"
Install the script to a location that is already on your path. It does not have to be a system folder like /usr/bin. Many default Bash setups will include ~/bin in your PATH.
Give the full path to your script on the command line:
/folder/where/you/put/the/script/find_branch test
Run the script through Python. This is very similar to option #2:
python /folder/where/you/put/the/script/find_branch test
Create an alias for the script in your environment. In bash you would do something like the following in your ~/.bashrc:
alias find_branch='/folder/where/you/put/the/script/find_branch'
OR
alias find_branch='python /folder/where/you/put/the/script/find_branch'
For options #1, #2, #3 and #5a to work properly, you should have a shebang with the version of python as the first line of the script. Any of the following will do, depending on how you have/want your environment set up:
#!/usr/bin/python
#!/usr/bin/python2
#!/usr/bin/python3
#!/usr/bin/env python
#!/usr/bin/env python2
#!/usr/bin/env python3
Finally, you do not have to remove the .py extension from the script if you do not want to. Many bash scripts have a .sh extension, for example, which does not prevent them from running as-is. You just have to include the extension in the name of the script when you run it.
I am trying to run a python file that takes in one argument (CSV file). It works when i run the script in the terminal but it gives errors when i run it in cron.
This is the line that i run in the terminal:
python nb2.py my_csv_file.csv
And here is my code that i am trying to run in cron:
42 13 * * * /usr/local/bin/python2.7 ~/nb/Development/code/nb2.py ~/nb/Development/code/my_csv_file.csv &> /tmp/June_QB_cat.log
The error says that it cannot find a sqlite table file which is already in the code folder.
Note that when you run the command in the terminal, you're in the ~/nb/Development/code directory already and so your current working directory is that; when you run in it cron, it is not. I would suggest either doing (in your cron job) cd ~/nb/Development/code && python nb2.py my_csv_file.csv &> /tmp/logfile.txt or doing an os.chdir("~/nb/Development/code") as the first step in your code. (Also, I'd suggest doing /home/username instead of ~ just in case you aren't running cronjob as your username at some point, but given the error you get, that sounds like it's not an issue)
You can get the path of a file relative to the current script with
import os.path
relative_path = os.path.join( os.path.dirname(__file__), "sqlitetable" )