I'm rather green in python development but learning nevertheless.
I've written some python codes but they are mostly one-off use and to be run at command line. I still have no idea how to automate and schedule my codes. Lets say I have:
written a python script that pulls some CSV data from API and saves it in /tmp.
written another script to ingest the csv data and transform it into XML per line.
Each time I want to do this, I find myself doing:
$ python getdata.py
$ python converttoxml.py
In shell, I think one can write a wrapper script and cron it. Right? If so, how is this done in the world of python? Bear in mind we have to include all python libraries/ modules used, too.
P.S. developing in Linux environment using PyCharm.
Login to cron using crontab -e
Scroll to bottom at add a line following the following format:
m h dom mon dow command which is minutes, hours, day of month, day of week, command
So if you want to run your command on the hour every hour you would have
* /1 * * * * python path/to/file/getdata.py
whereas if you want it to run at 12 only then you would have
* 12 * * * * python path/to/file/getdata.py
Python files could be treated as executables. You just need to give them executing permissions with chmod +x my_file.py and tell the bash which interpreter should be used to parse the file by adding #!/usr/bin/python as the 1st line of the code file.
Once you've done the two things, you could just run your file using ./my_file.py and the python script should be executed.
From this point on, it's just like any ordinary program and could be used in cron/systemd/whatever other use you need. Once you have a working Python script, it should be treated no differently than any other executable on your system - in this regard, there's no "Python World".
As far as extra modules are concerned - this shouldn't be an issue as Python will still use the same way to resolve library location. It might be necessary to update $PYTHON_PATH to add a library path - but if the library is installed correctly it shouldn't be an issue while for libraries contained within your local directory (only in the case Python fails to find them) you could add:
sys.path.apped(os.path.dirname(__file__)) to the beginning of your script.
Related
I am aware of the difference between import x and from x import yyy, in the latter case one can address x-located functions by bare name, instead of full specification, and this exactly I want to do, I do not want to write full name every time.
Also I know how to redefine default Python dir on Windows so that it starts knowing my developed modules, and this is fine. However how to combine those two things altogether?
I want IDLE to get started knowing all functions from all my modules so I do not need to import them manually.
from * import *
i.e. for all modules in IDLE home directory?
P.S. I saw this question, however it puts solution only for one import and do not scan the whole dir, also this solution with shortcut parameters does not seem beautiful to me.
Are there any more neat ways?
If you start idle with -r file it will first run the file, or with -s, it will first run the file listed in the IDLESTARTUP or PYTHONSTARTUP environmental variables. Then test by starting IDLE in command prompt with, for instance, py -m idlelib -r file. Once this works, you could create on IDLE shortcut on your desktop (drag and drop from Start menu), open Properties, and add either of the above to the end of the 'Target'.
I am new to programming. I think I'm close to solving my issue, but it finally broke my brain and I need help.
I am running a pygame script as a scheduled task in crontabs. I have one code that executes successfully and does what it should. I have another code that executes, but when it does the screen goes blank, usually displaying some lines that I usually see when the Linux boots up, and it just stays stuck there.
I have gone through both codes and absolutely everything is similar and correct. I have #!/usr/bin/env python at the start of each script.
(Elsewhere it was recommended that I give the exact version, because I use dictionaries in my script and apparently crontab could get confused with the dictionary stuff that pygame uses. I don't fully understand, I tried it but it didn't work on my raspberry pi so I don't think that's the solution.)
Each script runs fine in the terminal.
I have set the PATH variable to the one that the python script uses and also set the SHELL to /bin/bash. In the task I also have "export DISPLAY=:0" (e.g. 12 21 * * * export DISPLAY=:0 && ...). I have found this was the magic trick that got the other code to work. Which made me wonder if there is another environment variable that I need to set in the task?
The difference between the first code and the second code: The second code uses pygame.mixer and plays sound files. In the script, the sound files are in dictionaries (e.g. sound = {"word" : "/absolute/path/to/file.wav", "word2" : ...etc} As I said this code runs fine in terminal.
So why does the one script work and not the other. Both use pygame. The other just uses sound, dictionaries instead of strings, and pygame.mixer as well. My reasoning is that there is an issue with crontab getting stuck on one of these things.
I fixed it. I was running it out of root. When I ran the crontab in the user it worked. It must be because the root couldn't get at the files.
Important tips is to add 'import os' and at some point in the script add 'print(os.environ)' to check all the environment variables when it runs in terminal.
Then copy the $PATH variables to the top of the crontab jobs.
Same with $SHELL.
The $DISPLAY variables need to be put in the line, so for example '* * * * * export DISPLAY=:0 && /usr/bin/python /home/user/file.py'
See how I put absolute paths for both the command and the path to file. I also set absolute paths to files in the script, even if the files are in the same folder as the script. I'm not sure if this is necessary I will check that next. I've seen a lot of people struggle with this issue and the advice usually comes down to these things.
I am trying to schedule running a Python module in cron. This question is somewhat similar to this but I think asks for a different use case.
Task
I have a Python script I run in the shell as a module like this:
python -m myscript
It prints a bunch of numbers (via print) and works when I run it from the shell.
Question
I am now trying to run this every minute using a cron job, like so:
*/1 * * * * python -m myscript
Question1: This does not print to the terminal as expected / I don't see any output. Why? (To test if it is running at all, I redirected output to a file, which creates an empty file every minute).
Question2: My thinking was that any command that works when I run it manually in the shell will also work the same way when started via cron. Is that mistaken? E.g do I still have to make the script executable and such?
Question3: Thinking about it, I was not quite sure where cron will direct e.g. a print / stdout command and could not find in the docs. Do I have to / Should I manually specify the output target if I want it to print to a new shell window?
Currently running this on elementary OS but I want to eventually migrate to a Raspberry Pi. Any help is much appreciated!
I am attempting to run a python 3 script every 1 minute using cron on a raspberrypi 3, for testing, where eventually it will just be run once a day.
To start, I made a new cron job using: sudo crontab -e, and typed in the following code for a once a minute job:
*/1 * * * * /home/pi/folder/file.py
Then I saved and closed and waited. My python script emails me text when executed, so I should have seen an email come in. It runs fine (and emails me) when I execute it manually outside of cron.
So, what am I doing wrong with cron for it not to run? And do I need to make the python file executable or something with chmod?
Possible duplicate of Execute python Script on Crontab
EDIT:
Adding comment here since the comment box mangled my formatting.
In your example above it looks like you are just trying to "run" the file. You need to call the python executable, and pass it an argument that points to your file.
From the StackOverflow comment mentioned above look at this crontab entry:
*/2 * * * * /usr/bin/python /home/souza/Documets/Listener/listener.py
Take a look at the first part of the command /usr/bin/python this is pointing to the python executable not just to the .py file you want to run.
I apologize if this is a duplicate, I couldn't find any other examples of this question.
I'm trying to write a program for natural language recognition, and I was reading a blog post where someone had attempted to do something similar and it recommended using these two lines to capture the output.
#capture output of script
./get-language.py | tee preptxt
I can't figure out what this code is attempting to do. I assume it's running the get-language file, but that syntax doesn't look correct to me. Could someone point me in the right direction?
On Unix systems you can run executable files with /path/to/my/executable.
This is nothing python specific.
As . is the current working directory, you are executing the get-language.py script located in the current directory.
However, there are two things you need to do, to get this working for your scripts:
1. add a shebang
This is the first line of your script, it tells the shell which program to use.
To use the python interpreter that is first in the PATH use:
#!/usr/bin/env python
2. Add the permissions to make your script executable:
You need to permit execution of your script. This can be done with chmod:
chmod +x myscript.py
The last part is a so called piping operation.
If you call two programs like this:
$ program_a | program_b
The output (stdout) of program_a is fed into program_b.
That's a shell thing, not Python. . is the current directory; you're running get-language.py from there.
This is not a line of python, but rather a shell script.
It is running a script called get-language.py (which I assume was described elsewhere on that blog and does the actual language processing), and then it is "piping" that output through the unix command line program tee, which stores the output in a file.
Piping takes the output of one program, and uses it as the input of another program.