I've recently been tearing my hear out over this trivial problem, nothing I found online has helped me so far. I am using virtualenv, my project structure :
myproj\
..bin
..tasks.py
..celery_app.py
..projapi
where importFolder is an API folder with a bunch of files in it. proj.py is my script that does all the work and contains imports like
sys.path.append("/abspath/to/projapi")
import projapi
I can source bin/activate and enter my virtualenv just fine, but when I go back a directory and run the celery worker:
celery -A myproj worker --app=myproj.celery_app:app -l info
I'll get an import error for every folder in the myproj directory such as importerror for 'projapi'. It's as if the virtualenv is looking for all files in only the /bin dir (where as my proj is in myproj dir)
How can I fix this? I've tried appending all sorts of system and python paths, both in activate and in my tasks.py, but to no avail.
Probably you are using system wide celery.
Try to install it with pip within virtual env and repeat execution, that would fix your issue.
Also verify that each folder where you .py located contains __init__.py file.
Have you tried changing the celery call itself?
celery -A celery_app worker -E -l INFO
Related
I was trying to systemd a flask app. I tried to write a script like this:
#!/bin/bash
cd /path/to/app
source venv/bin/activate
python start.py
and just ExecStart this script in the .service file. But this doesn't quite work as starting the service errors with
python: command not found
I actually ran into quite a few issues, but eventually resolved with service file:
[Service]
WorkingDirectory=/path/to/app
ExecStart=/path/to/app/venv/bin/python start.py
Without WorkingDirectory, the file paths do not seem to work as the static file can't even be found.
So my question is actually why doesn't script above in the beginning work? The cd took effect, but the venv activate didn't?
This is my fabric script that runs on the jenkins server.
sudo('/home/myjenkins/killit.sh',pty=False)
sudo('/home/myjenkins/makedir.sh',pty=False)
sudo('/home/myjenkins/runit.sh',pty=False)
This kills the old server, creates a virtualenv, installs the requirements and restarts the server.
The problem is the with the script that starts the server - runit.sh :-
nohup /home/myjenkins/devserver/dev/bin/python /home/myjenkins/devserver
/workspace/manage.py runserver --noreload 0:80 >> jenkins.out &
When the jenkins server that starts the server and I navigate to the homepage, it gives me a 404 Page Not Found. It says /static/index.html not found. But the file exists. When I run 'sudo bash runit.sh' and I access the homepage, It works fine.
mkdir -p /home/myjenkins/devserver
cp -rf /home/myjenkins/workspace /home/jenkins/devserver/
cp -f /home/myjenkins/dev_settings.py /home/myjenkins/devserver/workspace/mywebsite/settings.py
cd /home/myjenkins/devserver
virtualenv -p python3 dev
cd /home/myjenkins/devserver/workspace
../dev/bin/pip install -r requirements.txt
Please ask me for more details if you need it.
EDITED 9/2/18
When I start the script from the folder containing manage.py, the server is able to serve the files. But Jenkins was starting the script from the home folder and if I also start the script from the home folder - the server is not able to find the files. look at my comment for more details. It would be great if someone could explain why this happens even though I've specified the full path in the script.
nohup /home/myjenkins/devserver/dev/bin/python /home/myjenkins/devserver
/workspace/manage.py runserver --noreload 0:80 >> jenkins.out &
Okay I figured out the whole deal. My django server was taking the output of the npm build from the wrong folder.
In the settings.py file, the variable STATICFILES_DIRS was set as:-
STATICFILES_DIRS = ('frontend/dist',)
instead of:-
STATICFILES_DIRS = (os.path.join(BASE_DIR,'frontend/dist'),)
Thus, when Jenkins was running the script, it was doing so from the home folder. This made Django's staticfiles finders to look at /home/myjenkins/frontend/dist instead of the relative '../frontend/dist'
I have a python script that queries a database. I run it from the terminal with python3 myscript.py
I've added a cron task for it in my crontab file
*/30 9-17 * * 1-5 python3 /path/to/my/python/script\ directory\ space/myscript.py
The script imports a function in the same directory that parses login info for a database located in database.ini in the same directory. The database.ini is:
[postgresql]
host=my-db-host-1-link.11.thedatabase.com
database=dbname
user=username
password=password
port=10898
But currently cron outputs to the file in my mail folder:
Section postgresql not found in the database.ini file
The section is clearly present in the database.ini file, so what am I missing here?
Instead of running "python3 myscript.py" in the directory where it is present, try running it from some other directory (like home directory). Most likely you will see the same issue.
Note that cron's current-working-directory is different on different systems. So, the safest method is to explicitly switch to the directory where your script is and run the command there:
cd /path/to/my/python/script\ directory\ space/ && python3 myscript.py
Try this:
import os
...
change --> filename=database.ini
for --------> filename=os.path.dirname(__file__)+'/database.ini'
When I move a project folders I have to manually update the project path in the .project file to get the workon command to work. Is it possible to update the path automatically?
According to the docs you can use setvirtualenvproject. This will automatically move you to the project folder if you use the workon command:
bono~$: setvirtualenvproject ~/.virtualenvs/your-virtual-env/ ~/path/to/your/project
Or, as beruic mentioned, it's easier to activate the environment and move to your desired working directory first. Please note that this not always work on my system, but it is a lot easier if it does work for you:
$ workon your-virtual-env
$ cd ~/path/to/your/project
$ setvirtualenvproject
In the future it might also be handy to specify the project path for the virtualenv on creation. You have to specify the project with the -a flag.
The -a option can be used to associate an existing project directory with the new environment.
You can use it something like this:
bono~$: cd ~/your/project
bono~$: mkvirtualenv my-project -a $(pwd)
The next time you use workon you will automatically be moved to your project directory.
Alternative
If you want to automatically detect directory changes and set the correct virtualenvwrapper then and there you can have a look at this post. It's a bit too expansive to go into detail here, but I think you can find what you're looking for there if that's what you meant.
You can just activate your virtual environment, go to the folder you want as project folder and call setvirtualenvproject:
$ workon [your_project]
$ cd [desired_project_folder]
$ setvirtualenvproject
Then the current folder will be set as project folder in the current virtualenv.
When I was developing and testing my project, I used to use virtualenvwrapper to manage the environment and run it:
workon myproject
python myproject.py
Of course, once I was in the right virtualenv, I was using the right version of Python, and other corresponding libraries for running my project.
Now, I want to use Supervisord to manage the same project as it is ready for deployment. The question is what is the proper way to tell Supervisord to activate the right virtualenv before executing the script? Do I need to write a separate bash script that does this, and call that script in the command field of Supervisord config file?
One way to use your virtualenv from the command line is to use the python executable located inside of your virtualenv.
for me i have my virtual envs in .virtualenvs directory. For example
/home/ubuntu/.virtualenvs/yourenv/bin/python
no need to workon
for a supervisor.conf managing a tornado app i do:
command=/home/ubuntu/.virtualenvs/myapp/bin/python /usr/share/nginx/www/myapp/application.py --port=%(process_num)s
Add your virtualenv/bin path to your supervisord.conf's environment:
[program:myproj-uwsgi]
process_name=myproj-uwsgi
command=/home/myuser/.virtualenvs/myproj/bin/uwsgi
--chdir /home/myuser/projects/myproj
-w myproj:app
environment=PATH="/home/myuser/.virtualenvs/myproj/bin:%(ENV_PATH)s"
user=myuser
group=myuser
killasgroup=true
startsecs=5
stopwaitsecs=10
First, run
$ workon myproject
$ dirname `which python`
/home/username/.virtualenvs/myproject/bin
Add the following
environment=PATH="/home/username/.virtualenvs/myproject/bin"
to the related supervisord.conf under [program:blabla] section.