Develop Reference

When running Python Flask Server, Jenkins is not successful and continues to load

I am building CICD through Jenkins.
But there are problems.
It is planning to upload source code first and turn on flask server through batch file.
I wrote a shell script for Jenkins' Build>Execute Shell.
postCommand=/cygdrive/c/workspace/ContactPortal_Flask/run.bat
sshpass -p ${deployPassword} ssh -o StrictHostKeyChecking=no ${deployUser}#${deployServer} ${postCommand}
Here is run.bat file
set FLASK_ENV=development
set path=%path%;C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;C:\develop\instantclient_12_1;C:\develop\Anaconda3;C:\develop\Anaconda3\Library\mingw-w64\bin;C:\develop\Anaconda3\Library\usr\bin;C:\develop\Anaconda3\Library\bin;C:\develop\Anaconda3\Scripts;
set "START=C:\workspace\ContactPortal_Flask\start.bat"
cd C:\workspace\ContactPortal_Flask
python -m flask run
then, The source code upload was successful, and turning on the flask server was also successful, but Jenkins was not marked Success and continued to load.
please help!!

I think the main problem here is that python -m flask run starts the server and will not finish until user hit Ctrl+C.
Since the target system is on Windows, you may want to create custom service and have jenkin start that service at the end instead. For service creation see https://learn.microsoft.com/en-us/troubleshoot/windows-client/deployment/create-user-defined-service And by starting this service (e.g. with NET START <service-name>) jenkin can finish, and flask can start running in the background.
Also for a production system, you may want to consider checking this and pick a proper web server instead of using the builtin web server provided by flask.

Azure Post Deployment Script Commands

I have a Django web app on Azure. I'm using Azure's built in continuous deployment, so I don't have any .deployment files within my repo.
However, in order for my application to work correctly, I have to manually install a couple packages after that aren't available via pip. So after every deployment, I have to SSH into the deployment and execute the following 2 commands:
source home/site/wwwroot/pythonenv3.6/bin/activate
sudo apt-get install XXX XXX XXX
I believe I can simply execute this via a post deployment script, but am having a hard time finding any literature explaining exactly how to do so. Is utilizing a post deployment script the best way of doing this? I'd rather not manage a custom docker file if possible.
UPDATE
Here is a screen shot of the packages installed after I manually run the apt-get command:
And here is a screen shot of the packages installed AFTER a re-deploy:
So unless there's a way to run this command from a post-deployment script (and have it execute correctly), my only option will be to create a custom container?
I'm not wild about doing so considering that this is such a small customization... but I may no longer have a choice?
UPDATE
I tried adding "apt-get install XXX XXX" to "PRE_BUILD_COMMAND" in Azure... and I got the following message as a result:
Are there any ideas how to get around this message using a prebuild or post build command?

UPDATE
The command like below.
You must have the wrong path, just enter /home, and then enter the command you want to execute, remember not to add sudo.
PRIVIOUS
First of all, please check this post. The pyodbc in the post cannot be used. Later, I used apt-get install xxx and pip install to solve this problem.
I know you want to use the .sh file, when you publish or re-publish, or when you publish for the first time, execute your apt-get install xxx command.
But after my test, I can tell you that you only need to execute the command once after the portal creates the webapp, or after the first release, to configure the environment without having to execute the .sh file again.
In conclusion:
It is recommended to execute commands such as apt-get install xxx and pip install xxx after the first release using git. Then modify the Readme.md file on github to trigger the redeployment process.
You don’t need to execute the .sh file for every deployment in the future. I replied to your other post. You can execute the .sh file, but there is no need to call it every time after you execute it manually. Because this is a configurable environment variable, the configuration is successful.
Step 1. Install env.
Step 2. Modify readme.md,check deploy status.
we can see my custom .sh command is running.But we no need to do that.
Step 3. Check the webapp after finishing env config .

Deploying a python Flask application with Jenkins and executing it

I am trying to do auto-deployment of a Python Flask application using Jenkins and then run it by using shell command on a Raspberry Pi server.
Here are some background info,
Before using Jenkins, my deployment and execution process was manual described below:
FTP to the directory where my Python scripts and Python venv are located
Replace Flask application scripts using FTP
Activate virtual environment to of Python(3.5) through the terminal on Raspberry Pi ("./venv/bin/activate")
Run myFlaskApp.py by executing "python myFlaskApp.py" in terminal
Now I have integrated Jenkins with the deployment/execution process described below:
Code change pushed to github
Jenkins automatically pulls from github
Jenkins deploy files to specified directories by executing shell commands
Jenkins then activates virtual environment and run myFlaskApp.py by bashing a .sh script in the shell terminal.
Now the problem that I am having is on step 4, because a Flask app has to always be alive, my Jenkins will never "finish building successfully", it will always be in a loading state as the Flask app is running on the shell terminal Jenkins is using.
Now my question:
What is the correct approach that I should be taking in order to activate myFlaskApp.py with Jenkins after deploying the files while not causing it to be "locked down" by the build process?
I have read up about Docker, SubShell and the Linux utility "Screen". Will any of these tools be useful to assist me in my situation right now and which approach should I be taking?

The simple and robust solution (in my opinion) is to use Supervisor which is available in Debian as supervisor package. It allows you do make a daemon from script like your app, it can spawn multiple processes, watch if app doesn't crash and if it does it can start it again.
Note about virtualenv - you don't need to activate venv to use it. You just need to point appropriate Python executable (your_venv/bin/python) instead of default one. For example:
$ ./venv/bin/python myFlaskApp.py

You need to create these files for deployment over jenkins.
Code can be found: https://github.com/ishwar6/django_ci_cd
This will work for both flask as well as django.
initial-setup.sh - This file is the first file to look at when setting up this project. It installs the required packages to make this project work such as Nginx, Jenkins, Python etc. Refer to the youtube video to see how and when it is used.
Jenkinsfile - This file contains the definition of the stages in the pipeline. The stages in this project's pipeline are Setup Python Virtual Environment, Setup gunicorn service and Setup Nginx. The stages in this pipeline just does two things. First it makes a file executable and then runs the file. The file carries out the commands that is described by the stage description.
envsetup.sh - This file sets up the python virtual environment, installs the python packages and then creates log files that will be used by Nginx.
gunicorn.sh - This file runs some Django management commands like migration commands and static files collection commands. It also sets up the gunicorn service that will be running the gunicorn server in the background.
nginx.sh - This file sets up Nginx with a configuration file that points Nginx to the gunicorn service that is running our application. This allows Nginx serve our application. I have followed a digital ocean article to setup this file. You can go through the video once to replicate sites-available and sites-enabled scanerio.
app.conf - This is an Nginx server configuration file. This file is used to setup Nginx as proxy server to gunicorn. For this configuration to work, change the value of server_name to the IP address or domain name of your server.

How do I run Django as a service?

I am having difficulty running Django on my Ubuntu server. I am able to run Django but I don't know how to run it as a service.
Distributor ID: Ubuntu
Description: Ubuntu 10.10
Release: 10.10
Codename: maverick
Here is what I am doing:
I log onto my Ubuntu server
Start my Django process: sudo ./manage.py runserver 0.0.0.0:80 &
Test: Traffic passes and the app displays the right page.
Now I close my terminal window and it all stops. I think I need to run it as a service somehow, but I can't figure out how to do that.
How do I keep my Django process running on port 80 even when I'm not logged in?
Also, I get that I should be linking it through Apache, but I'm not ready for that yet.

Don't use manage.py runserver to run your server on port 80. Not even for development. If you need that for your development environment, it's still better to redirect traffic from 8000 to 80 through iptables than running your django application as root.
In django documentation (or in other answers to this post) you can find out how to run it with a real webserver.
If, for any other reason you need a process to keep running in background after you close your terminal, you can't just run the process with & because it will be run in background but keep your session's session id, and will be closed when the session leader (your terminal) is terminated.
You can circunvent this behaviour by running the process through the setsid utility. See your manpage for setsid for more details.

Anyway, if after reading other comments, you still want to use the process with manage.py, just add "nohup" before your command line:
sudo nohup /home/ubuntu/django_projects/myproject/manage.py runserver 0.0.0.0:80 &

For this kind of job, since you're on Ubuntu, you should use the awesome Ubuntu upstart.
Just specify a file, e.g. django-fcgi, in case you're going to deploy Django with FastCGI:
/etc/init/django-fcgi.conf
and put the required upstart syntax instructions.
Then you can you would be able to start and stop your runserver command simply with:
start runserver
and
stop runserver
Examples of managing the deployment of Django processes with Upstart: here and here. I found those two links helpful when setting up this deployment structure myself.

The problem is that & runs a program in the background but does not separate it from the spawning process. However, an additional issue is that you are running the development server, which is only for testing purposes and should not be used for a production environment.
Use gunicorn or apache with mod_wsgi. Documentation for django and these projects should make it explicit how to serve it properly.
If you just want a really quick-and-dirty way to run your django dev server on port 80 and leave it there -- which is not something I recommend -- you could potentially run it in a screen. screen will create a terminal that will not close even if you close your connection. You can even run it in the foreground of a screen terminal and disconnect, leaving it to run until reboot.

If you are using virtualenv,the sudo command will execute the manage.py runserver command outside of the virtual enviorment context, and you'll get all kind of errors.
To fix that, I did the following:
while working on the virtual env type:
which python
outputs: /home/oleg/.virtualenvs/openmuni/bin/python
then type:
sudo !!
outputs: /usr/bin/python
Then all what's left to do is create a symbolic link between the global python and the python at the virtualenv that you currently use, and would like to run on 0.0.0.0:80
first move the global python folder to a backup location:
mv /usr/bin/python /usr/bin/python.old
/usr/bin/python
that should do it:
ln -s /usr/bin/python /home/oleg/.virtualenvs/openmuni/bin/python
that's it! now you can run sudo python manage.py runserver 0.0.0.0:80 in virtaulenv context!
Keep in mind that if you are using postgres DB on your developement local setup, you'll probably need a root role.
Credit to #ydaniv

Installing and Starting a CherryPy server script as Windows service (using sc.exe)

I am trying to install and start a simple CherryPy server as a Windows service.
Here is the script: (Removed some lines to cut it short. It's fully working when executing manually from the command-line)
app = AdminMediaHandler(django.core.handlers.wsgi.WSGIHandler())
logged_app = TransLogger(app)
server = wsgiserver.CherryPyWSGIServer( ('127.0.0.1', 8632), logged_app, server_name='localhost', numthreads=20 )
try:
server.start()
except KeyboardInterrupt:
server.stop()
I'm using sc.exe to install and start the service. Installation goes fine, but I can't seem to start the service.
The command used is: (note there're spaces in the paths, though I'm handeling this with double-quotes, and the binPath is working when executing its string manually through the command-line)
> sc.exe create "ServiceName" binPath= "\"C:\Path to Python\python.exe\" \"C:\Path to CherryPy Script\cherryserver.py\""
> sc.exe start "ServiceName"
I keep getting this error, no matter if attempting to start the service using sc.exe or through services.msc GUI:
[SC] StartService FAILED 1053:
The service did not respond to the start or control request in a timely fashion.
From what I understand, this is happenning because python.exe doesn't implement the Windows Service API.
I do not wish to create an .exe from the script, using py2exe.
I have found this answer that suggests to install the service using different tools than sc.exe, called srvany.exe & instsrv.exe. However, I can't find them in the Win2K Resource Kit website.
Does anybody know how to install & start this .py as a Windows succesfully?
Does anybody know

CherryPy ships with a module for starting as a Windows service. See this other SO question for instructions on how to install and run it. You'll probably want to switch from your current approach (of passing the Django app directly to the WSGIServer) and use cherrypy.tree.graft instead.

I prefer nssm for installing normal scripts as a service.
You can copy the nssm.exe in the C:\Windows\system32 or C:\Windows\SysWOW64 directory depending on your system. After that you are able to install a Service as follow:
nssm install yourservicename
For a python script you must set the application path to your python.exe and the argument is your script self.
Other common commands for start/stop/edit your service are:
nssm start yourservicename
nssm stop yourservicename
nssm edit yourservicename

I eventually used ServiceInstaller aka SMaster, as stated in this answer. The URL in the given answer is broken, and I couldn't find a working URL. I just had srunner.exe available locally beforehand.
Note there was another obstacle to overcome though, as ServiceInstaller can't handle files with spaces in their paths.
So, I used the old DOS path formatting for service registration.
Instead of registering C:\Program Files\MyApp\python.exe, I registered C:\PROGRA~1\MyApp\python.exe.