I am trying to do auto-deployment of a Python Flask application using Jenkins and then run it by using shell command on a Raspberry Pi server.
Here are some background info,
Before using Jenkins, my deployment and execution process was manual described below:
FTP to the directory where my Python scripts and Python venv are located
Replace Flask application scripts using FTP
Activate virtual environment to of Python(3.5) through the terminal on Raspberry Pi ("./venv/bin/activate")
Run myFlaskApp.py by executing "python myFlaskApp.py" in terminal
Now I have integrated Jenkins with the deployment/execution process described below:
Code change pushed to github
Jenkins automatically pulls from github
Jenkins deploy files to specified directories by executing shell commands
Jenkins then activates virtual environment and run myFlaskApp.py by bashing a .sh script in the shell terminal.
Now the problem that I am having is on step 4, because a Flask app has to always be alive, my Jenkins will never "finish building successfully", it will always be in a loading state as the Flask app is running on the shell terminal Jenkins is using.
Now my question:
What is the correct approach that I should be taking in order to activate myFlaskApp.py with Jenkins after deploying the files while not causing it to be "locked down" by the build process?
I have read up about Docker, SubShell and the Linux utility "Screen". Will any of these tools be useful to assist me in my situation right now and which approach should I be taking?
The simple and robust solution (in my opinion) is to use Supervisor which is available in Debian as supervisor package. It allows you do make a daemon from script like your app, it can spawn multiple processes, watch if app doesn't crash and if it does it can start it again.
Note about virtualenv - you don't need to activate venv to use it. You just need to point appropriate Python executable (your_venv/bin/python) instead of default one. For example:
$ ./venv/bin/python myFlaskApp.py
You need to create these files for deployment over jenkins.
Code can be found: https://github.com/ishwar6/django_ci_cd
This will work for both flask as well as django.
initial-setup.sh - This file is the first file to look at when setting up this project. It installs the required packages to make this project work such as Nginx, Jenkins, Python etc. Refer to the youtube video to see how and when it is used.
Jenkinsfile - This file contains the definition of the stages in the pipeline. The stages in this project's pipeline are Setup Python Virtual Environment, Setup gunicorn service and Setup Nginx. The stages in this pipeline just does two things. First it makes a file executable and then runs the file. The file carries out the commands that is described by the stage description.
envsetup.sh - This file sets up the python virtual environment, installs the python packages and then creates log files that will be used by Nginx.
gunicorn.sh - This file runs some Django management commands like migration commands and static files collection commands. It also sets up the gunicorn service that will be running the gunicorn server in the background.
nginx.sh - This file sets up Nginx with a configuration file that points Nginx to the gunicorn service that is running our application. This allows Nginx serve our application. I have followed a digital ocean article to setup this file. You can go through the video once to replicate sites-available and sites-enabled scanerio.
app.conf - This is an Nginx server configuration file. This file is used to setup Nginx as proxy server to gunicorn. For this configuration to work, change the value of server_name to the IP address or domain name of your server.
Related
I want to know if there's a way to have windows server 2019 automatically launch django's web server. I also want the launch to be performed at startup and by SYSTEM.
I tried using batch scripts that launch manage.py from venv's python interpreter. When I launch the batch manually (i.e. double click) it works fine and dandy. But it appears that SYSTEM fails in running the script correctly when planning the task.
I made SYSTEM launch another script at startup (a simple python script that creates a txt file from within its own venv) and it works.
If the Django launch sceipt is launched by USER then it works.
The problem is with the launching of django with SYSTEM. I've also tried streamlit and the result is the same.
Do you have any Ideas?
Sample batch script:
cd path\of\managepyfile\
C:\path_to_venv\Scripts\python -m manage.py runserver
We run a similar application (not python) but an application that uses a web server.
We have it setup as a task in task scheduler that when the server starts up, it runs the powershell script that executes a command to start the web server.
Link to setup
However, you could use a web server like IIS and deploy the files to the www folder in the cdrive and run the site as an IIS service.
Setting it up on IIS was a little tricky if you've never used IIS before. Happy to help out as we have deployed our test access tool for one of our apps this way.
I am building CICD through Jenkins.
But there are problems.
It is planning to upload source code first and turn on flask server through batch file.
I wrote a shell script for Jenkins' Build>Execute Shell.
postCommand=/cygdrive/c/workspace/ContactPortal_Flask/run.bat
sshpass -p ${deployPassword} ssh -o StrictHostKeyChecking=no ${deployUser}#${deployServer} ${postCommand}
Here is run.bat file
set FLASK_ENV=development
set path=%path%;C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;C:\develop\instantclient_12_1;C:\develop\Anaconda3;C:\develop\Anaconda3\Library\mingw-w64\bin;C:\develop\Anaconda3\Library\usr\bin;C:\develop\Anaconda3\Library\bin;C:\develop\Anaconda3\Scripts;
set "START=C:\workspace\ContactPortal_Flask\start.bat"
cd C:\workspace\ContactPortal_Flask
python -m flask run
then, The source code upload was successful, and turning on the flask server was also successful, but Jenkins was not marked Success and continued to load.
please help!!
I think the main problem here is that python -m flask run starts the server and will not finish until user hit Ctrl+C.
Since the target system is on Windows, you may want to create custom service and have jenkin start that service at the end instead. For service creation see https://learn.microsoft.com/en-us/troubleshoot/windows-client/deployment/create-user-defined-service And by starting this service (e.g. with NET START <service-name>) jenkin can finish, and flask can start running in the background.
Also for a production system, you may want to consider checking this and pick a proper web server instead of using the builtin web server provided by flask.
There is a python script in my repo that I would like to run whenever I call an API.This Python script merely transfer data from one database to another. The Jenkins server for the project currently is used for builds/pipelines/running tests, I was wondering if I could use this Jenkins service to run this script when i call an API since I found that Jenkins allows you to remotely trigger scripts via REST.
I was wondering it I could use Jenkin's feature of trigger remotely to run this python script in my repo when I need to. The python script is built using a python image in the dockerfile, so docker helps to setup the dependencies/python needed to run the script. the command to run by Jenkins is something like docker build and docker run
Yes you can.
Just setup a pipeline that
Runs in docker (with your image). Have a look at this
Does a git clone of you repository
Runs you python script with something like: sh "python <your script>"
On my Windows 10 machine, I am developing a database manager. Because the backend uses LDAP and the required development libraries are only available for Linux, I want to use Docker to set up an environment with the appropriate libs.
I managed to write a Dockerfile and compose file, that launch the (currently very basic) Django app in a Docker container with all the libs necessary.
I would like to play around with the django-ldapdb package and for that I want to apply the migrations.
When I open PyCharm's terminal and try to execute python manage.py migrate, I get an error telling me that the module ldapdb is not found. I suspect this is because the command does not use the remote Docker interpreter I set up with PyCharm.
The other thing I tried is using PyCharm's dedicated manage.py console. This does not initialize properly. It says the working directory is invalid and needs to be an absolute path, although the path it shows it the absolute path to the project.
I have to admit that I have no idea how this remote interpreter works and I don't see any Docker container running, so I might have not understood something properly here. I even tried running the app using PyCharm's Django run config, which started a container, but still I get the same errors.
I googled a lot, but I couldn't find more infos about remote interpreters nor something solving my issue.
The only way I managed to do this, is by executing the command inside the container.
To get inside a container named contr, use the docker command
docker exec -ti contr /bin/bash
I am having difficulty running Django on my Ubuntu server. I am able to run Django but I don't know how to run it as a service.
Distributor ID: Ubuntu
Description: Ubuntu 10.10
Release: 10.10
Codename: maverick
Here is what I am doing:
I log onto my Ubuntu server
Start my Django process: sudo ./manage.py runserver 0.0.0.0:80 &
Test: Traffic passes and the app displays the right page.
Now I close my terminal window and it all stops. I think I need to run it as a service somehow, but I can't figure out how to do that.
How do I keep my Django process running on port 80 even when I'm not logged in?
Also, I get that I should be linking it through Apache, but I'm not ready for that yet.
Don't use manage.py runserver to run your server on port 80. Not even for development. If you need that for your development environment, it's still better to redirect traffic from 8000 to 80 through iptables than running your django application as root.
In django documentation (or in other answers to this post) you can find out how to run it with a real webserver.
If, for any other reason you need a process to keep running in background after you close your terminal, you can't just run the process with & because it will be run in background but keep your session's session id, and will be closed when the session leader (your terminal) is terminated.
You can circunvent this behaviour by running the process through the setsid utility. See your manpage for setsid for more details.
Anyway, if after reading other comments, you still want to use the process with manage.py, just add "nohup" before your command line:
sudo nohup /home/ubuntu/django_projects/myproject/manage.py runserver 0.0.0.0:80 &
For this kind of job, since you're on Ubuntu, you should use the awesome Ubuntu upstart.
Just specify a file, e.g. django-fcgi, in case you're going to deploy Django with FastCGI:
/etc/init/django-fcgi.conf
and put the required upstart syntax instructions.
Then you can you would be able to start and stop your runserver command simply with:
start runserver
and
stop runserver
Examples of managing the deployment of Django processes with Upstart: here and here. I found those two links helpful when setting up this deployment structure myself.
The problem is that & runs a program in the background but does not separate it from the spawning process. However, an additional issue is that you are running the development server, which is only for testing purposes and should not be used for a production environment.
Use gunicorn or apache with mod_wsgi. Documentation for django and these projects should make it explicit how to serve it properly.
If you just want a really quick-and-dirty way to run your django dev server on port 80 and leave it there -- which is not something I recommend -- you could potentially run it in a screen. screen will create a terminal that will not close even if you close your connection. You can even run it in the foreground of a screen terminal and disconnect, leaving it to run until reboot.
If you are using virtualenv,the sudo command will execute the manage.py runserver command outside of the virtual enviorment context, and you'll get all kind of errors.
To fix that, I did the following:
while working on the virtual env type:
which python
outputs: /home/oleg/.virtualenvs/openmuni/bin/python
then type:
sudo !!
outputs: /usr/bin/python
Then all what's left to do is create a symbolic link between the global python and the python at the virtualenv that you currently use, and would like to run on 0.0.0.0:80
first move the global python folder to a backup location:
mv /usr/bin/python /usr/bin/python.old
/usr/bin/python
that should do it:
ln -s /usr/bin/python /home/oleg/.virtualenvs/openmuni/bin/python
that's it! now you can run sudo python manage.py runserver 0.0.0.0:80 in virtaulenv context!
Keep in mind that if you are using postgres DB on your developement local setup, you'll probably need a root role.
Credit to #ydaniv