I am trying to restructure a large project so that I can have one core set of code reused among a web frontend application written in flask and an automated back end that uses luigi tasks.
so I have two clients that I'd like to share code between in order to access a database and perform some automated tasks from either the web or luigi.
What is a good way to keep those three things organized and structured so that I can easily import core modules into either project.
I've had some issues getting Luigi to recognize modules that are parallel to it.
Technically, the only requirement to re-use the code is that the Python files of the shared modules ("core set of code") are found via sys.path of the executing interpreter, e.g. by adding the according directory to the PYTHONPATH environment variable. This enables you to use them as modules. However, you have to assess the changes to your project layout depending on your version control scheme and how you deploy the web application and clients. Usually, I would assume that you have to use the following rules of thumb.
Manage the shared code modules as a separate project. Especially, if the lifecycles of the depending clients and web applications are not tied together.
Treat the shared core modules as a library or framework. I should not rely on implementation details of the web application or the clients. Usually, you need to provide some entry points and configuration hooks.
Change your deployment in a way that the shared core modules are pulled as a dependency of the according client/application. There is a magnitude of possibilites depending on your setup and use case. For example, you could build pip-installable packages using setuptools.
Related
The Current State:
I have some non-negligeble amount of microservices written in python.
Each such microservice has its own yaml configuration file that is located in the git repo. We use dynaconf to read the configuraion.
The Problem:
At first it was fine, the configurations were relatively small and it was easy to maintain them. Time went by, and the configurations went larger. It became annoying to change the configurations and it is bad that they are scattered between different git repos, i.e. not centralized.
I want to use "Externalized Configurations" in order to maintain all the configurations in a single repo and that each microservice will read its portion on startup. I have heard about Spring Boot, but it seems to be way too much and apart from it, it seems that the pip libraries seems to be at beta stage, new and unriliable...
Is there another reccomendation in this particular use case? Or should I proceed with Spring Boot?
You can use Microconfig.IO to manage your configuration with powerful templating and distribution. It's language agnostic as long as your configs are YAML or properties format.
I think you could create a default config file for each service and use the external storage idea of the dynaconf.
One possible solution would be to create a simple system to manage these variables with Redis.
And the DynaConf CLI allows you to make changes on the fly.
My company make a variety of (usually) desktop apps to be used internally to aid the creation/analysis of our datasets.
Recently I created a program which acts as a sort of installation & runtime manager, with the purpose of:
Providing users with a consistent means by which to locate and install applications
Manage execution of installed apps and provide environments for various interpreted languages so the user doesnt need to know/care about python/R
Ensure users are up-to-date with latest versions etc..
The program itself is a desktop GUI which integrates with our in-house software website and grabs applications from a network location.
To run apps, it typically overwrites the users' PATH variable with it's own settings plus what ever the app defines.
Occasionally, this is not enough isolation and a user has a problem running apps.
Is there a way to easily run an app in isolation? Ideally, I could remove all environment variables other than the defaults provided by Windows, but this still leaves the possibility of odd registry values interfering with execution..
The application itself is written in Qt C++
We are developing a distributed application in Python. Right now, we are about to re-organize some of our system components and deploy them on separate servers, so I'm looking to understand more about deployment for an application such as this. We will have several back-end code servers, several database servers (of different types) and possibly several front-end servers.
My question is this: what / which are good deployment patterns for distributed applications (in Python or in general)? How can I manage pushing code to several servers (whose IP's should be parameterized in the deployment system), static files to several front ends, starting / stopping processes in the servers, etc.? We are looking for possibly an easy-to-use solution, but mostly, something that once set-up will get out of our way and let us deploy as painlessly as possible.
To clarify: we are aware that there is no one standard solution for this particular application, but this question is rather more geared towards a guide of best practices for different types / parts of deployment than a single, unified solution.
Thanks so much! Any suggestions regarding this or other deployment / architecture pointers will be very appreciated.
It all depends on your application.
You can:
use Puppet to deploy servers,
use Fabric to remotely connect to the servers and execute specific tasks,
use pip for distributing Python modules (even non-public ones) and install dependencies,
use other tools for specific tasks (such as use boto to work with Amazon Web Services APIs, eg. to start new instance),
It is not always that simple and you will most likely need something customized. Just take a look at your system: it is not so "standard", so do not expect it to be handled in a "standard" way.
I am used to PHP having applications. For example,
c:\xampp\htdocs\app1
c:\xampp\htdocs\app2
can be accessed as
localhost://app1/page.php
localhost://app2/page.php
Things to be noticed:
a directory placed inside the www-root directory maps directly with the URL
when a file/directory is added/removed/changed, the worker processes seamlessly reflect that change (new files are hot-deployed).
I am on the lookout for a mature python web framework. Its for a web API which will be deployed for multiple clients, and each copy will diverge on customization. And our workflow has frequent interaction/revision cycles between us and our clients. Hence the "drag and drop" deployment is a must.
Which python framework enables this? I prefer a lightweight solution (which doesnt impose MVC, ORMs etc)
Related
How to build Twisted servers which are able to do hot code swap in Python?
Python web hosting: Why are server restarts necessary?
fastcgi, cherrypy, and python
No mature python framework that I'm aware of allows you to map urls to python modules, and frankly, for good reason. You can do this with CGI, but it's definitely not the recommended way to deploy python apps. Setting that requirement aside, flask and bottle are both lightweight micro web-frameworks with similar approaches, both allow you to reload automatically when changes are detected (this is only wise during development).
There is no web framework in Python that I know of that lets you do that out of the box, but if you need it it's not to hard to add with a bit of convention over configuration.
Simply pick your web framework of choice in Python and then write a wrapper to the main application that walks a directory or set of directory and auto-registers routes from the modules inside of them. Have your modules do the same thing in their __init__.py files to the other files located with them. Then just set up your WSGI code to autoreload when the WSGI script is updated and your deployment during development simply becomes a two step process - add file then touch dev_app.wsgi. You could even add a real deployment option to this wrapper that walks a set up dev environment like this and generates hard-coded URL-to-function mappings for deployment.
However, all of this work isn't really necessary. Python is not PHP and the way you develop in one doesn't necessarily translate to the other well. If the client wants variable routes, use dynamic routes and give them (or you) an admin interface to control the mapping of content to URL. Use flat files, SQLite, a NoSQL datastore, or the ether to store these mappings and the content. Use a template engine like Jinja2, Mako, Cheetah or Genshi to maintain your general layout. Wrap this all up with an object oriented structure to make extending it easy (or use a functional paradigm if that comes more naturally to you). Or, drop the whole dynamic in production portion and generate flat HTML files a la Jekyll.
CherryPy is a mature web framework that redeploys automatically when changes are detected. The file structure - URL isn't there, but it is a lightweight framework that doesn't impose ORM, MVC, or even a templating engine.
If you are used to PHP, you might want to take a look at the Apache modules mod_python or mod_wsgi (and WSGI in general if you do web development -- which is the Pythonic way).
With those two modules, the Python interpreter gets started every time a request comes in (similar to PHP). Needless to say, this slows things down but you'll always get the result based on your newest code. Depending on your expected traffic numbers, this might or might not be okay for you.
BUT: If you decide to write your own framework, you most probably do not want to write a system that supports "hot-deploying". Even though the reload() command is built-in, it takes more than just that and will get you into a world full of pain.
Well I want to use WEb2Py because it's pretty nice..
I just need to change the working directory to the directory where all my modules/libraries/apps are so i can use them. I want to be able to import my real program when I use the web2py interface/applications. I need to do this instead of putting all my apps and stuff inside the Web2Py folder... I'm trying to give my program a web frontend without putting the program in the Web2Py folder.. Sorry if this is hard to understand .
In any multi-threaded Python program (and not only Python) you should not use os.chdir and you should not change sys.path when you have more than one thread running. It is not safe because it affects other threads. Moreover you should not sys.path.append() in a loop because it may explode.
All web frameworks are multi-threaded and requests are executed in a loop. Some web frameworks do not allow you to install/un-install applications without restarting the web server and therefore IF os.chdir/sys.path.append are only executed at startup then there is no problem.
In web2py we want to be able to install/uninstall applications without restarting the web server. We want apps to be very dynamical (for example define models based on information provided with the http request). We want each app to have its own models folder and we want complete separation between apps so that if two apps need to different versions of the same module, they do not conflict with each other, so we provide APIs to do so (request.folder, local_import).
You can still use the normal os.chdir and sys.path.append but you should do it outside threads (and this is not a web2py specific issue). You can use import anywhere you like as you would in any other Python program.
I strongly suggest moving this discussion to the web2py mailing list.
os.chdir lets you change the OS's working directory, but for your purposes (enabling imports of a bunch of modules &c which are constrained to live in some strange place) it seems better to add the needed directories to sys.path instead.
I had to do this very thing. I have few modules that I wanted to use from my controllers. If you want to be able to use the code that resides in the modules directory in the controller, you can use:
# use this in your controller code
impname = local_import('module_in_modules', reload=True)
# reload true will ensure that it will re load whenever
# there are changes to the module
Jay