I have a directory of python programs, classes and packages that I currently distribute to 5 servers. It seems I'm continually going to be adding more servers and right now I'm just doing a basic rsync over from my local box to the servers.
What would a better approach be for distributing code across n servers?
thanks
I use Mercurial with fabric to deploy all the source code. Fabric's written in python, so it'll be easy for you to get started. Updating the production service is as simple as fab production deploy. Which ends ups doing something like this:
Shut down all the services and put an "Upgrade in Progress" page.
Update the source code directory.
Run all migrations.
Start up all services.
It's pretty awesome seeing this all happen automatically.
First, make sure to keep all code under revision control (if you're not already doing that), so that you can check out new versions of the code from a repository instead of having to copy it to the servers from your workstation.
With revision control in place you can use a tool such as Capistrano to automatically check out the code on each server without having to log in to each machine and do a manual checkout.
With such a setup, deploying a new version to all servers can be as simple as running
$ cap deploy
from your local machine.
While I also use version control to do this, another approach you might consider is to package up the source using whatever package management your host systems use (for example RPMs or dpkgs), and set up the systems to use a custom repository Then an "apt-get upgrade" or "yum update" will update the software on the systems. Then you could use something like "mussh" to run the stop/update/start commands on all the tools.
Ideally, you'd push it to a "testing" repository first, have your staging systems install it, and once the testing of that was signed off on you could move it to the production repository.
It's very similar to the recommendations of using fabric or version control in general, just another alternative which may suit some people better.
The downside to using packages is that you're probably using version control anyway, and you do have to manage version numbers of these packages. I do this using revision tags within my version control, so I could just as easily do an "svn update" or similar on the destination systems.
In either case, you may need to consider the migration from one version to the next. If a user loads a page that contains references to other elements, you do the update and those elements go away, what do you do? You may wish to do something either within your deployment scripting, or within your code where you first push out a version with the new page, but keep the old referenced elements, deploy that, and then remove the referenced elements and deploy that later.
In this way users won't see broken elements within the page.
Related
We have few Zope&Plone projects in our company and until today I was only one single developer developing all changes throught ZMI or ZopeEdit. Our company is growing so I need to start cooperating with others developers which can help me with developing features and solving bugs in projects. This means that is no more possible to use ZMI but every developer needs to make and test own changes without affecting others work and paste own changes to production enviroment using git merge in git repo.
I need to move development to git - this means I need to start tracking all portals files and settings in git.
I think I need to move whole projects from ZODB/ZMI (including templates, scripts, sql methods, properties as portal_properties or portal_javascripts etc.) to filesystem and run git on this file system. In the next step every developer can install own pure Plone instance, pull source code and settings from git, create own branch, make changes, test, commit, push, code review ...
My question is: Is there any way to do this and start well-known rapid development process using git? Supports ZODB something like "live migration" of content/settings to/from filesystem? Is there any way to tel Zope to load some folder with content/settings from filesystem instead only from ZODB?
I know there is something called eggs, but is possible to move all types of files mentioned above to separated egg?
Thank you for your help.
The way your company was following until now was the "Old Way Way" of Plone development, but this was a deprecated and discouraged way to do.
Nowadays ZMI can still be used for "quick and dirty" fixes, but commonly this changes stored in DB must be removed (and moved to real code) as soon as possible. This was already possible on Plone 2.0!
More important: every new Plone release tend to reduce the ZMI powers (for example: until Plone 2.1 you were able to do lot of stuff from ZMI, starting from Plone 2.5 some UI elements where impossible to be modified TTW).
So: the answer to your question is "yes". Plone can (must) read code from filesystem, and this code can be stored on VCS (it can be git svn, ...).
All of those information can be found in the Plone Developer Manual.
Creating a new Plone package for modern Plone? Use mr.bob.
Automatically integrate VCS in your buildout? Use mr.developer
If you are starting today from a project you where developing through ZMI you must probably mode code from ZMI to filesystem.
This can be done manually; it's simpler as you are using Zope External Editor.
There is also a very old add-on (Plone Skin Dump) for flushing skin content to filesystem, but I fear it won't work on recent Plone, thus it was not supporting some stuff like SQL methods (if you are using them).
You can have a look at http://docs.plone.org/develop/
There you can find how to create a package (egg). Its source code can be added to git. You can checkout your git repository using mr.developer during buildout. https://pypi.python.org/pypi/mr.developer/
You can use mr.bob to create a PloneAddon. For MySQL you can use MySQL Python. Add this Packages to your Buildout in the eggs section.
Then you can write your own MySQL Statements in the Addon. I know reimplementation is expensive, but you have more control in the future.
I have several small Python libraries that I wrote with stuff that I find myself wanting over and over again. I think most programmers have something similar. I want to use these libraries from a variety of different machines so I've started keeping this stuff in my DropBox. However, I'd like to be able to use my code on machines on which I can't install DropBox or other cloud storage applications, even in portable form. I can just download the files every time one of them changes (DropBox can provide me a URL for each file in my Public folder), which is only a moderate nuisance. But--and I admit this is a longshot--is there a solution out there that will let me tell Python to load a library from my DropBox via http?
BTW, I'd like to add the whole remove folder to my sys.path, but getting a URL for a folder is complicated, so I'm going to try to walk before I run by starting with individual files.
Yes, it's possible. I think you want the combination of two previous questions:
How to download a file in python over HTTP
How to dynamically load a library in python
So your task basically breaks down into writing a little bit of glue code: download the URL via the first bullet, write it to a local file, and then import that file using the second bullet.
So that's how you'd do that.
BUT - please keep in mind that dynamically downloading and executing code has many potential security downfalls. Will you be doing this over a secure connection? Who else has the ability to manipulate that URL? There are a bunch of security issues inherent in downloading and executing code on the fly. I would ask you to consider going about your solution in a different way, but I'm giving you the answer you're asking for.
As a simple security check, you can establish a known-good hash for your file, and then refuse to import any file other than one that's on the list of known-good hashes. This makes it a pain to update your modules, but gives you a little bit of extra safety.
Don't use DropBox as a Revision control
Pick a real solution like Git
Setup access to the Git repository on one of your servers
Clone the repository to your worker machines and checkout master
Create a develop branch where you put every change you make
Test the changes and when you consider any of them stable, merge it to master
On your worker machines set up a cron job which periodically pulls from master branch of repository (and possibly restarts some Python processes as importing the same module again won't make Python interpreter aware of changes since imported modules are cached)
Enjoy your automatically updated workers :)
Don't feel shame - it happens that even experienced software developers come up with XY problem
We want to use continuous deployment.
We have:
all sources (python) in a local RhodeCode (git) server.
Jenkins for automated testing
SSH connections to the production systems (linux).
a tool which can update servers in one command.
Now something like this should be implemented:
run tests with Jenkins
if there is a failure. Stop, mail developers
If all tests are OK:
deploy
We are long enough in the business to write some scripts to do this.
My questions:
How to you update the version numbers? You could increment them, you could use a timestamp ...
Since we already use Jenkins, I think we do it in a script called by Jenkins. Any reason to do it with a different (better) tool?
My fear: Jenkins becomes a central server for things which are not related to testing (deploy). I think other tools like SaltStack or Ansible should be used for this. Up to now we use Fabric (simple layer above ssh). Maybe we should switch to a central management system before starting with continuous deployment.
Since we already use Jenkins, I think we do it in a script called by
Jenkins. Any reason to do it with a different (better) tool?
To answer your question: No, there aren't any big reasons to not go with Jenkins for deployment.
Pros:
You already know Jenkins (and you probably know some of the quirks)
You don't need to introduce yet another technology
You said that you want to write scripts called by Jenkins, so you can switch easily to a different system later.
Cons:
there might be better tools out there for deployment
Does not tie the best with Change Control tools.
Additional Considerations:
Do not use the same server for prod deployment and continuous build/integration. These are two different tasks performed by two different roles. Therefore two different permission schemes might be employed.
Use permissions wisely. I use two different permissions for my deploy and CI servers. We have 3 Jenkins servers right now.
CI and deploy to uncontrolled environments (Developers can play with these environments)
Deploy to controlled environments. (QA environemnts and upwards)
Deploy to prod (yes, that's the only purpose in live of this server.) with the most restrictive permission scheme.
sandbox, actually there is this forth server for Jenkins admins to play with.
Store your deployable artifacts outside of Jenkins (and you do if I read your question correctly).
So depending on your existing infrastructure and procedure you decide for the tooling. Jenkins won't log you in as long as you keep as much of the logic as possible in scripts that are only executed by Jenkins.
As a fledgling Django developer, I was wondering if it was customary, or indeed possible, to create a site with Django then transfer the complete file structure to a different machine where it would "go live".
Thanks,
~Caitlin
You could use GIT or Mercurial - or other version control system. To put the site structure on a central server. After that you could deploy the site for example with fabric to multiple servers. For deployment process you should consider using for example virtualenv to isolate the project from global python packages and requirements.
Of course that's possible and in fact it's the only way to "go live". You don't want to develop in your live server, do you? And it's true for any platform, not just django.
If I understood your question correctly, you need a system to push your development code to live.
Use a version control system: git, svn, mercurial etc.
Identify environment specific code like setting/config files etc. and have separate instances of them for each environment.
Create a testing/staging/PP environment which has live data or live-like data and deploy your code there before pushing it to live.
To avoid any downtime during deployment process, usually a symbolic link is created which points to the existing code folder. When a new release is to be pushed, a new folder is created with new code, after all other dependencies are done (like setting and database changes) and the sym link is pointed to the new folder.
I commit every time I make some changes that I think might work: I don't do extensive testing before a commit. Also, my commits will soon be automatically pushed to a remote repository. (I'm the only developer, and I have to add features or rewrite parts of the code many times a day.)
I'd like to set up a remote computer to run regression tests automatically whenever I commit anything; and then email me back the differences report.
What's the easiest way to set this up?
All my code is in Python 3. My own system is Windows 7, ActiveState Python, TortoiseHG, and Wing IDE. I can set up the remote computer as either Linux or Windows. The application is all command-line, with text input and output.
Use a continious integration server such as Buildbot or Jenkins and configure it to monitor the repository. Then run the tests using that. Buildbot is written in Python so you should feel right at home with it.
If you feel it's wasteful to make Buildbot or Jenkins poll the repository (even though hg pull uses very few resources when there are no new changesets), then you can configure a changegroup hook in the repository to trigger a build in the CI server.
I would recommend setting up Buildbot. You can have it watch a remote repository (Mercurial is supported) and automatically kick off a build when the repository changes. In your case, a build would just be running your test suite.
Its waterfall display allows you to see which builds failed and when, in relation to commits from the repository. It can even notify you, with the offending commit, when something breaks.
Jenkins is another option, supporting most of the same features. There are even cloud hosting options, like ShiningPanda that can host it for you, and they offer free licensing for open-source projects.