im working on a small ETL that collects data using webscraping, cleans and manipulates it and sends to a local sqlite3 database.
If i execute the command /virtualenv_path/python /script_path/script.py it runs perfectly, but if i schedule this command with crontab it does not work.
It just does not send any data. However, my log file shows me that the crontab is executing script.py using my venv as expected.
So, what is going on? What should i do to solve this?
I suppose that my script is not incorrect because if i execute without crontab it works flawlessly and even with crontab it does not show any error (as i said, log file suggests that everything is going really well)
this is my repository: https://github.com/raposofrct/wescraping-ETL
there we have ETL folder that contains my script, crontab command that im using and my sqlite database.
thanks for any help or clue that you guys can give me.
Your script is likely working, but it's not putting data into the database file you're looking at. hm_db.sqlite is relative to whatever the current working directory is:
DataBase(dados,create_engine('sqlite:///hm_db.sqlite',echo=False))
That is very unlikely to be the same directory you are in when you run the script manually. Either provide an absolute path or make the path relative to your script directory, e.g.
from pathlib import Path
root_directory = Path(__file__).parent
database_file = root_directory / "hm_db.sqlite"
DataBase(dados, create_engine(f"sqlite:///{database_file}", echo=False))
Alternatively, log os.getcwd() in your existing script to figure out where your cronjob has been storing data.
Related
I'm trying to run a python program using cron.
Initially I tried running a web scraper in python using cron, but I quickly realized that something was wrong.
So I tried breaking down the process to see where the error was.
And I found out that cron actually won't run any python program that is more complicated than a
print("Hello World")
For starters I'm trying to run this code using cron:
import pandas as pd
import os
import sys
sys.stdout = open("/home/pi/testbot/crontask.txt", "w")
df = pd.read_excel('db.xlsx', engine='openpyxl')
print(df)
sys.stdout.close()
I want the program to read a .xlsx file and write the output into a file called crontask.txt.
Now here comes the twist.
When I run the command python3 testbot.py
I get the right result, and content in the crontask.txt.
But when I add * * * * * /usr/bin/python3 /home/pi/testbot/testbot.py >> /home/pi/testbot/log.txt to crontab, I get zero results and zero log-entries in log.txt.
I've tried to make the .py file exeutable with chmod +x testbot.py and change permissions without luck.
I'm starting to wonder if the only way to run such programs is to make a script that runs the program and use cron job on that.
Is that my only solution?
I did solve this one, and as embarrassing as the fault was I still want to post this in case someone else experience the same problem.
As cron doesn't use any spesific paths you have to specify everything in order for it to execute programs that opens any folder/file or other program.
Not just in crontab but also in the programs themself.
In my case it was the df = pd.read_excel('db.xlsx', engine='openpyxl') that caused the error, so when I changed it to df = pd.read_excel('/home/pi/bot/db.xlsx', engine='openpyxl') it worked like a charm.
The same thing goes if you consider using a bash script to open another program. Only direct paths seem to work.
I'm having problems with starting a python script on Raspberry Pi boot. I have read many threads and tried some tricks, however, none of them worked for me.
The file I am trying to execute is named test.py, it just logs a time to another file, when was Pi's startup:
#!/usr/bin/python
import time
f=open('logger.txt','w')
tim=time.strftime("%H:%M:%S")
f.write('Startup on: %s\n'%(tim))
f.close()
It is located in: /home/pi and I modified the privileges to all (777). I tried to add a line to /etc/rc.local file before exit 0, my rc.local looks like that:
python /home/pi/test.py &
exit 0
Nothing happens on the startup. If I write a .sh file with the same function and change the line in rc.local accordingly, everything works fine.
Could anyone please help me, what is that different in running python script on startup? Thank you, Kaki
If you don't specify absolute path, open assume relative path to current working directory.
You'd better to try use absolute path first, before you know where the working directory is.
f = open('/home/pi/logger.txt', 'w')
I'm currently using an open-source SCSS compiler, however, every time I make changes to a scss file, I have to run the compiler manually to compile the output. So, to combat this, I edited a Sublime package that allows people to run commands on file saves and it goes like this:
class CommandOnSave(sublime_plugin.EventListener):
def on_post_save(self, view):
settings = view.settings()
current_file = view.file_name()
if current_file.endswith("scss"):
subprocess.call("php path/to/phpscript", shell=True)
Now, the only issue is that one folder in the path changes with every project, and it would be ideal if there was some way to dynamically execute the php script so I don't have to have a settings file with multiple paths in it to accomplish this.
Does anyone know of an easy way to do this without a lot of server overload? The php script always resides four directories down from the scss file. I've tried ../../../.. but that doesn't work (obviously) since the command isn't looking for the file from the current file's path.
Any help is greatly appreciated!
Python novice here.
I have a Python script that performs some geodatabase management (reconcile/post versions, compress, etc). I have the following line of code in my script:
createLog = open(str(datetime.date.today()) + ".txt", "w")
along each step of the script I add to the text file with the following statements:
createLog.write("Database connections blocked.\n")
When I run the script in my IDE (PyCharm) I get the desired result: A text file with each step written to the .txt file. When I run it in Task Scheduler no .txt file is created and therefore no log. Everything else runs as far as I can tell. I'm able to track edits made to the data.
I have experienced things like this before with task scheduler but have never been able to resolve the problem in the past.
Any ideas?
I think this is a working directory problem. Python's open function opens a file in the current working directory, NOT in the same folder as the script. This is a common misconception! (Which confused me for ages when learning Python...)
So what is a working directory? Well to quote my good friend Wikipedia:
In computing, the working directory of a process is a directory of a hierarchical file system, if any,[1] dynamically associated with each process. When the process refers to a file using a simple file name or relative path (as opposed to a file designated by a full path from a root directory), the reference is interpreted relative to the current working directory of the process. So for example a process with working directory /rabbit-shoes that asks to create the file foo.txt will end up creating the file /rabbit-shoes/foo.txt.
(Source: https://en.wikipedia.org/wiki/Working_directory)
So how is this working directory selected?
Well it is selected by the parent process of that processes! When you run a program from a shell like bash, the shell (the parent process) helpfully sets the working directory of the program you are running (the child process) to the directory you are currently in. (That is, the directory you cd'd to.)
Since your IDE is smart and helpful, it is starting your Python script process and setting the working directory to the same place the script itself is located. The task scheduler is less helpful... I have absolutely no idea what it is setting the working directory to. However if you search your system, I am sure you will find the log file lying about somewhere!
I'm writing a Python application that is executable. It reads and writes to a file. The application uses wx for a GUI and has been given the following permissions:
chmod +x app.py
When I load the application from the terminal like so:
./app.py
The application loads and causes no errors.
However, when I double click the app.py file and click 'execute', everything works perfectly except for the reading and writing of this file. This is a major part of the program and causes errors.
I cannot, for the life of me, understand why this is not working.
I have attempted to set it so that it loads as the root user each time with no luck. I have also been developing as the root user the entire time, so I shouldn't see any issues.
I am using the default Raspbian OS.
It sounds like you need to modify the permissions of the file to be readable/writable/executable by the user you log into the GUI with. Do some reading on Linux File Permissions and see where that takes you.