We have a decent set of unit tests on our code and those unit tests run in under 2 minutes. We also use TeamCity to do a build and to run tests after each check in. However, we still get issues where a developer "forgets" to run all the tests before a commit resulting in TeamCity failure which if this check in was done at 6PM may stay broken over night.
"Forgets" is a generic term, there a couple other common reasons why even remembering to run the tests could result in TeamCity failure. Such as.
-> A developer only checks in some of the modified files in his/her workspace.
-> A file was modified outside of eclipse such that eclipse's team synchronize perspective does not detect it as dirty.
How do you deal with this in your organization?
We are thinking of introducing "check in procedure" for developers which will be an automated tool that will automatically run all unit tests and then commit all of the "dirty" files in your workspace. Have you had any experience with such process? Are you aware of any tools which may facilitate this process? Our dev environment is Python using Eclipse's PyDev plugin.
In one of the teams I was working before we had an agreement that anyone who breaks the tests buys bacon sandwiches for the whole team the next morning. Its extreme, but it works perfectly!
I think it is more of a social problem rather than a deficiency of the automated systems.
Yes, you can improve the systems in place, but they will be no match for someone thinking of the implications of their commit, and testing it before they hit commit.
For mercurial you can use hooks, that will run tests and only allow push on success. But this may require a lot of time for a push (but developer has to run those tests anyway).
Or you can just have own set of bash scripts, that will run test and only then run commit command. For example for django and svn commit it could look as simple as this:
./manage.py test && svn commit $#
Or there is another way: if anyone commits code, that doesn't pass test, he pays some sum. Soon people will remember to test, for they won't like the notion of paying money ;-)
TeamCity has some support for pretested commit; if your development team is using a supported IDE, you might look into that.
In my company, we don't worry about it too much - our pattern looks something like this.
(a) each developer has their own project configuration in TeamCity, with roots pointing to their own sandbox. They are allowed to do anything they like here.
(b) the development team has an integration sandbox, where all changes are delivered. A project encapsulates the configurations which monitor this branch in the source control system. Dev Leads get to make up the rules here, but that rule is almost always "it must stay green". I'd have to look at the exact percentage of clean builds - it's not a perfect record, but its high enough that I've never been tempted to insist that the developers be more disciplined about running tests.
(c) the actual deliver comes from a Main stream, which Shall Stay Green (tm). Dev lead is responsible for delivering a clean snapshot of integration to the main stream on a well defined schedule. This project is the one that actually generates the installers that are delivered to testing, the bits that go into escrow, etc.
One reason that you might get a way with a more aggressive policy than we do is that our build cycle is slow - on the order of four hours. If we were an order of magnitude smaller, with a poor success rate, I might argue a different case.
For git you can: http://francoisgaudin.com/2011/02/16/run-django-tests-automatically-before-committing-on-git/
Run Django tests automatically before committing on Git
Since I often forget to run unit tests before committing, I spend a
lot of time looking for the bad commit when I find regressions 3
commits later.
However it’s really easy to automatically run tests before each
commit. In .git/hooks/pre-commit, put :
python manage.py
test exit $?
then chmod 755 this file and it’s done. I really love git :-)
Do not forget to source your virtualenv before committing.
Note that tests are run on your working tree and not the commit
itself, so if you commit only a part of your working tree, it may fail
while your commit passes tests.
Related
This is more of a general question about safety in testing than a specific code problem. Lets say I have feature branch for my git repository, I always run a suite of unit tests before I merge back into develop or master. But these unit tests often take a while (on the order of an hour). So I generally kick off the test and then change branches in my repository so I can work on coding other things...I'm assuming this is safe because the .pyc files are already created?
I recommend that you offload test execution to a proper continuous-integration system such as Jenkins or Travis. Switching out the entire source tree in the middle of a test run is bound to cause weird problems.
Also consider that your test suite likely contains both unit tests and integration tests. Unit tests should be fast! A runtime of 0.1 seconds is a slow unit test. Tests that touch the filesystem, communicate with a database, send packets over the network, and so on are integration tests. You might scale those back to running once or twice a day. See Working Effectively With Legacy Code by Michael Feathers.
If proper CI is not a preferred option for whatever reason and you are OK with some scripting, then you could do a script that would copy a git revision (git export or git new workdir + checkout there ) and execute tests in that location while you keep going further with changes. Worked fine for me with a large java project.
We want to use continuous deployment.
We have:
all sources (python) in a local RhodeCode (git) server.
Jenkins for automated testing
SSH connections to the production systems (linux).
a tool which can update servers in one command.
Now something like this should be implemented:
run tests with Jenkins
if there is a failure. Stop, mail developers
If all tests are OK:
deploy
We are long enough in the business to write some scripts to do this.
My questions:
How to you update the version numbers? You could increment them, you could use a timestamp ...
Since we already use Jenkins, I think we do it in a script called by Jenkins. Any reason to do it with a different (better) tool?
My fear: Jenkins becomes a central server for things which are not related to testing (deploy). I think other tools like SaltStack or Ansible should be used for this. Up to now we use Fabric (simple layer above ssh). Maybe we should switch to a central management system before starting with continuous deployment.
Since we already use Jenkins, I think we do it in a script called by
Jenkins. Any reason to do it with a different (better) tool?
To answer your question: No, there aren't any big reasons to not go with Jenkins for deployment.
Pros:
You already know Jenkins (and you probably know some of the quirks)
You don't need to introduce yet another technology
You said that you want to write scripts called by Jenkins, so you can switch easily to a different system later.
Cons:
there might be better tools out there for deployment
Does not tie the best with Change Control tools.
Additional Considerations:
Do not use the same server for prod deployment and continuous build/integration. These are two different tasks performed by two different roles. Therefore two different permission schemes might be employed.
Use permissions wisely. I use two different permissions for my deploy and CI servers. We have 3 Jenkins servers right now.
CI and deploy to uncontrolled environments (Developers can play with these environments)
Deploy to controlled environments. (QA environemnts and upwards)
Deploy to prod (yes, that's the only purpose in live of this server.) with the most restrictive permission scheme.
sandbox, actually there is this forth server for Jenkins admins to play with.
Store your deployable artifacts outside of Jenkins (and you do if I read your question correctly).
So depending on your existing infrastructure and procedure you decide for the tooling. Jenkins won't log you in as long as you keep as much of the logic as possible in scripts that are only executed by Jenkins.
I'm a freelance editor and tutor as well as a fiction writer and artist looking to transition to the latter on a full-time basis. Naturally, part of that transition involves constructing a website; a dynamic site to which new content in various forms can be added with ease. Now, I've always intended to learn how to program, and I simply haven't the money to hire someone else to do it. So, having had a good experience with my brief dabblings in Python, I decided I'd go with Django for building my site.
I set up a Fedora Virtualbox for a development environment (as I didn't want to jump through hoops to make Windows work) and went to town on some Django tutorials. Everything went swimmingly until life intervened and I didn't touch the project for three weeks. I'm in a position to return to it now, but I've realized two things in the process. First, I'm having to do a fair bit of retracing of my steps just to find where certain files are, and second, I don't know how I'd go about deploying the site after I'm done building it. My intention is to get the cheapest Linode and host off that until some theoretical point in the future where I required more.
I suspect that re: the file organization issue, that's just something I'll become more familiar with over time, though if there are any tricks I should be aware of to simplify the structure of my overall Django development space, I'm eager to know them. However, what about deployment? How viable is it to, with sufficient knowledge, automate the process of pushing the whole file structure of a site with Git? And how can I do that in such a way that it doesn't tamper with the settings of my development environment?
As a Django developer i can assure you that it grows on you and becomes easier to understand the development environment.
You should remember that settings.py is probably going to be where your thoughts will be for quite a while in the start; the good part is that its only once, after you got it up and running you'll only touch settings.py to add new modules or change some configuration but its unlikely.
I believe there are hosts that integrate with git so that should not be a problem since you will probably just git clone your project's url into the host (and not forget to enable/configure wsgi)
To leave the settings.py out of the mess, you will tell git not to track the file with: git rm file; and then when you add your files for commit you do it with git add -u so it refers only to your tracked files.
I'm not sure if i was clear enough. (probably not) But, i hope i could help you in some way.
I commit every time I make some changes that I think might work: I don't do extensive testing before a commit. Also, my commits will soon be automatically pushed to a remote repository. (I'm the only developer, and I have to add features or rewrite parts of the code many times a day.)
I'd like to set up a remote computer to run regression tests automatically whenever I commit anything; and then email me back the differences report.
What's the easiest way to set this up?
All my code is in Python 3. My own system is Windows 7, ActiveState Python, TortoiseHG, and Wing IDE. I can set up the remote computer as either Linux or Windows. The application is all command-line, with text input and output.
Use a continious integration server such as Buildbot or Jenkins and configure it to monitor the repository. Then run the tests using that. Buildbot is written in Python so you should feel right at home with it.
If you feel it's wasteful to make Buildbot or Jenkins poll the repository (even though hg pull uses very few resources when there are no new changesets), then you can configure a changegroup hook in the repository to trigger a build in the CI server.
I would recommend setting up Buildbot. You can have it watch a remote repository (Mercurial is supported) and automatically kick off a build when the repository changes. In your case, a build would just be running your test suite.
Its waterfall display allows you to see which builds failed and when, in relation to commits from the repository. It can even notify you, with the offending commit, when something breaks.
Jenkins is another option, supporting most of the same features. There are even cloud hosting options, like ShiningPanda that can host it for you, and they offer free licensing for open-source projects.
I am starting to work on a hobby project with a Python codebase and I would like to set up some form of continuous integration (i.e. running a battery of test-cases each time a check-in is made and sending nag e-mails to responsible persons when the tests fail) similar to CruiseControl or TeamCity.
I realize I could do this with hooks in most VCSes, but that requires that the tests run on the same machine as the version control server, which isn't as elegant as I would like. Does anyone have any suggestions for a small, user-friendly, open-source continuous integration system suitable for a Python codebase?
We run Buildbot - Trac at work. I haven't used it too much since my codebase isn't part of the release cycle yet. But we run the tests on different environments (OSX/Linux/Win) and it sends emails — and it's written in Python.
One possibility is Hudson. It's written in Java, but there's integration with Python projects:
Hudson embraces Python
I've never tried it myself, however.
(Update, Sept. 2011: After a trademark dispute Hudson has been renamed to Jenkins.)
Second the Buildbot - Trac integration. You can find more information about the integration on the Buildbot website. At my previous job, we wrote and used the plugin they mention (tracbb).
What the plugin does is rewriting all of the Buildbot urls so you can use Buildbot from within Trac. (http://example.com/tracbb).
The really nice thing about Buildbot is that the configuration is written in Python. You can integrate your own Python code directly to the configuration. It's also very easy to write your own BuildSteps to execute specific tasks.
We used BuildSteps to get the source from SVN, pull the dependencies, publish test results to WebDAV, etcetera.
I wrote an X10 interface so we could send signals with build results. When the build failed, we switched on a red lava lamp. When the build succeeded, a green lava lamp switched on. Good times :-)
We use both Buildbot and Hudson for Jython development. Both are useful, but have different strengths and weaknesses.
Buildbot's configuration is pure Python and quite simple once you get the hang of it (look at the epydoc-generated API docs for the most current info). Buildbot makes it easier to define non-testing tasks and distribute the testers. However, it really has no concept of individual tests, just textual, HTML, and summary output, so if you want to have multi-level browsable test output and so forth you'll have to build it yourself, or just use Hudson.
Hudson has terrific support for drilling down from overall results into test suites and individual tests; it also is great for comparing test output between builds, but the distributed (master/slave) stuff is comparatively more complicated because you need a Java environment on the slaves too; also, Hudson is less tolerant of flaky network links between the master and slaves.
So, to get the benefits of both tools, we run a single instance of Hudson, which catches the common test failures, then we do multi-platform regression with Buildbot.
Here are our instances:
Jython Hudson
Jython buildbot
We are using Bitten wich is integrated with trac. And it's python based.
TeamCity has some Python integration.
But TeamCity is:
not open-source
is not small, but rather feature rich
is free for small-mid teams.
I have very good experiences with Travis-CI for smaller code bases.
The main advantages are:
setup is done in less than half a screen of config file
you can do your own installation or just use the free hosted version
semi-automatic setup for github repositories
no account needed on website; login via github
Some limitations:
Python is not supported as a first class language (as of time of writing; but you can use pip and apt-get to install python dependencies; see this tutorial)
code has to be hosted on github (at least when using the official version)