For the moment I've created an Python web application running on uwsgi with a frontend created in EmberJS. There is also a small python script running that is controlling I/O and serial ports connected to the beaglebone black.
The system is running on debian, packages are managed and installed via ansible, the applications are updated also via some ansible scripts. With other words, updates are for the moment done by manual work launching the ansible scripts over ssh.
I'm searching now a strategy/method to update my python applications in an easy way and that can also be done by our clients (ex: via webinterface). A good example is the update of a router firmware. I'm wondering how I can use a similar strategy for my python applications.
I checked Yocto where I can build my own linux with but I don't see how to include my applications in those builds, and I don't wont to build a complete image in case of hotfixes.
Anyone who has a similar project and that would like to share with me some useful information to handle some upgrade strategies/methods?
A natural strategy would be to make use of the package manager also used for the rest of the system. The various package managers of Linux distributions are not closed systems. You can create your own package repository containing just your application/scripts and add it as a package source on your target. Your "updater" would work on top of that.
This is also a route you can go when using yocto.
Related
I have a python application that I've created an executable of, using pyinstaller. The entire python interpreter is packaged into the executable with all its pip dependencies.
So now my application can run in environments where python or python modules may not be installed, but there are still some dependencies:
1) MongoDB - This is the database my application uses, and it needs to be installed on a system for it to work of course.
2) Mosquitto - This service is required because the application uses MQTT to receive/send commands.
My current method of handling this is to use a shell script which installs mongodb and mosquitto the first time when my application is deployed somewhere. I just discovered docker, and I was wondering if it is capable of packaging these 'external' dependencies into a docker image?
Is it possible for me to have one standalone "thing" which will run in any environment regardless of whether mongoDB or mosquitto are installed there?
And how exactly would I go about doing this?
(Unrelated but this application is meant to run on a raspberry pi)
If you adopted Docker here:
You'd still have to "separately" run the external services; they couldn't be packaged into a single artifact per se. There's a standard tool called Docker Compose that provides this capability, though, and you'd generally distribute a docker-compose.yml file that describes how to run the set of related containers.
It's unusual to distribute a Docker image as files; instead you'd push your built image to a registry (like Docker Hub, but the major public-cloud providers offer this as a hosted service, there are a couple of independent services, or you can run your own). Docker can then retrieve the image via HTTP.
Docker containers can only be run by root-equivalent users. Since you're talking about installing databases as part of your bringup process this probably isn't a concern for you, but you could run a plain-Python or pyinstallered application as an ordinary user. Anyone who can run any Docker command has unrestricted root-level access on the host.
So my team and I have bought into Docker - it is fantastic for deployment and testing. My real question is how to set up a great developer experience, specifically around writing Python apps, but this question could be generalized to nodejs, Java, etc.
The problem: When writing a Python app, I really like having decent linting/autocomplete functionality, there are some really good editors out there (Atom, VSCode, PyCharm) that provide these, but most really want a Python install on the local disk. The real advantage of Docker is that all of the core language and any project libraries can all be in the container, so reproducing all of that on the host machine just for developing is a pain.
I know that PyCharm pro does support Docker and docker-compose, but I found it quite sluggish and a lot of the test running capabilities were busted. On top of that, I really would like something that I can commit to version control so that the team can share dev setup and people don't have to repeat all of the steps for their own system.
A few Ideas that I had were:
Install an editor (like Atom) in a sidecar Docker container and use X11 forwarding
Use a browser based editor such as https://c9.io/ in a container - this seems most promising
Install some agent in a dev container that could handle autocomplete/linting, etc. and connect to it from a locally running editor - I think this would be the best solution, but I also think that right now it actually doesn't exist.
Has anyone had luck setting up a more productive development environment besides just mounting volumes and editing text?
You should use an 'advanced' IDE like IntelliJ (Pycharm) and configure a remote Python SDK using SSH-Access to your App-Docker-Container (using a shared ssh-key to auth against the app-container with a preinstalled openssh server and preconfigured authorized_keys file).
You can share this SDK information in your project file with all devs, so they wlll have this setup out of the box
1) This will ensure, your IDE knows about all the python libs/symbols available/installed in your docker-container during runtime. It will also enable you to properly debug remotely at the same time
2) This ensures, you have an IDE at your hand including a lot of important additional features like the inspector, 3way duff, search in path.. . hardly any of the Browser-Based IDEs will catch up with Pycharm at this point IMHO
Of course, as already mentioned in the comments, you need to share aka mount your code into the container. On linux, you plainly use host-volume-mounts from your local src folder to the container.
On OSX, you will run into performance issues when using host mounts. You might use something like http://docker-sync.io ( i am biased - there are also a lot of other similar tools )
I know this is an old question, but as I stumbled across it while trying to see what other editors might offer in this space, I would like to point out Visual Studio Code's notion of a Dev Container, which seems to provide the best level of integration I've seen for this so far. I'm hoping to see this turn into an industry trend myself.
Could use x11docker
x11docker allows to run graphical desktop applications (and entire desktops) in Docker Linux containers.
Docker allows to run applications in an isolated container environment. Containers need much less resources than virtual machines for similar tasks.
Docker does not provide a display server that would allow to run applications with a graphical user interface.
x11docker fills the gap. It runs an X display server on the host system and provides it to Docker containers.
Additionally x11docker does some security setup to enhance container isolation and to avoid X security leaks. This allows a sandbox environment that fairly well protects the host system from possibly malicious or buggy software.
https://github.com/mviereck/x11docker
https://github.com/mviereck/x11docker/wiki (extensive! knowledge)
https://dev.to/brickpop/my-dream-come-true-launching-gui-docker-sessions-with-dx11-in-seconds-1a53
So I am creating a local Python script which I plan to export as an executable. However, this script is in need of a MongoDB instances that runs in the background as a service or daemon. How could one possibly include this MongoDB service along with their own ported application?
I have this configuration manually installed on my own computer with a MongoDB database installed as a local Windows service, and Python where my script adds and removes to the database as some events are fired. Is there any possible way to distribute this setup without manual installation of Python and MongoDB?
If you want to include installations of all your utilities, I recommend pynsist. It'll allow you to make a Windows installer that will make your code launchable as an app on the clients system, and include any other files and/or folders that you want.
Py2exe converts python scripts and their dependencies into Windows executable files. It has some limitations, but may work for your application.
You might also get away with not installing mongo, by embedding something like this in your application: https://github.com/Softmotions/ejdb. This may require you to rewrite your data access code.
If you can't or don't do that, then you could have all your clients share a multi-tenant mongo that you host someplace in the cloud.
Finally, if you can't or won't convert your python script to a standalone exe with an embedded database, and you don't want to host a shared mongo instance for your clients, there are legions of software installation makers that make deploying mongo, python, setting up an execution environment, creating services, etc, pretty easy. Some are free, some cost money. A long list can be found here: https://en.m.wikipedia.org/wiki/List_of_installation_software
I want to deploy my python application to my customers. Well, I basically don't know much about python application deployment, but my requirements/questions are
1) The user can install it as long as they can access internet. For mac applications, they are all hosted by apple app store. For chrome extensions, they are hosted by google. My question is, if there's a similar place that are hosting python applications, and it provides updating mechanism? If I have to do it on my own, is there any existing framework stuff for me to do it?
2) My application would be used to read USB device, and act as a http server. I want the install package to be as small as possible, and I also need to package python runtime. What is the package size that I should be expecting? 5M? 10M?
I have sucesfully used pyinstaller for my project
https://github.com/pyinstaller/pyinstaller/wiki
My application is reasonably large, so the installer package is around 100MB which compresses to 60MB. A lot of that is numpy, qt, scipy, and matplotlib.
We use a script to invoke pyinstaller which packages our main script and dependencies into a .app file. https://github.com/Erotemic/ibeis/blob/next/installers.py
If you are installing on a mac, this script in my repo will take a pyinstaller package and bundle it into a dmg.
https://github.com/Erotemic/ibeis/blob/next/_scripts/mac_dmg_builder.sh
If you host your program on your own server you can integrate an auto-update mechanism, but I don't know how to do that exactly. I just host my installers on dropbox.
I have a server that I'd like to use to maintain persistent connections with a set of devices, just so that they can pass simple messages back and forth. It's a trivial task, but selecting a server-side platform has been suprrisingly difficult (especially since I have no administrative privileges - it's a dedicated commercial server).
My best idea so far is to write a TCP server in Python. The Twisted platform seems suitable for the task, and has a lot of good reviews. However, my server has Python 2.7 but not Twisted, and the admins have been reluctant to install it for me.
Is there any way that I can just upload a Twisted package to the server and reference it in my libraries without installing it as a framework?
I'm not sure what you mean by "installing it as a framework". If you are using an OS X server hosting environment, then maybe you're talking about Framework with a Capital F. However, OS X server hosting isn't a very common environment so I'm guessing that's not it.
If you just want to know how to install a Python library in your home directory, then the general answer is:
$ python setup.py install --user
This Just Works™ on Python 2.7 (assuming the package uses distutils, which Twisted does, and you unpack the source .tar.gz and change your working directory to the directory that is the root of the contents of that .tar.gz), so you should be done after that.
Use virtualenv to create your private Python libraries installation.