Develop Reference

Queuing and dispatching long simulations

I am developing a Python module to manage the execution of long-baseline simulation processes. Whilst I have had some successes to date, I would like to improve it in order to make it more usable for its intended, er, users.
My code to date uses sys, time, subprocess and the package watchdog; it monitors a folder for new run files, places these in a FIFO queue and executes them simultaneously up to a pre-defined limit. This works, but is a bit clunky, and managing this dynamically is impractical.
I am not a computer scientist or a software engineer; this is why I am asking for help. Despite that, I am confident that I would be able to do the (bulk of the) coding work, but I would like some guidance on:
What are the computer science-y names for some of the concepts I'm describing
What packages are out there for doing some of these things in Python 3.x
What approaches you would use to do these things
What data structures you might use to do this
I appreciate any help you're able to offer, and of course please let me know if there is a better StackExchange site where I could submit this question. Thank you in advance!
Some detail about the simulations:
Entirely based in Windows 10
One of three or four different varieties
Executed from command line with reference to an input file
Simulations either fail instantly or almost instantly, or run for several hours, days or even weeks
Typical batches are up to tens of simulations
Simulations use lots of RAM and an entire CPU
I should also point out that, according to the terms of the simulation software licenses, I am expressly forbidden to use third-party cloud computing platforms.
Here's what I need the module to do:
Simulations dispatched automatically
Simultaneous simulations up to a limit
Queue runs indefinitely
Queue can be added to dynamically (reference in a new file)
Feedback on what's running and what's in the queue
First in, first out queuing
And some extra ideas, from users, about what would be nice to have:
Can dispatch simulations onto multiple computers
Simultaneous simulation limit could be changed dynamically
System of prioritisation
Simulations can be terminated early
If the module crashes, can be restarted without having to rebuild the queue
A browser-based interface where the queue could be viewed / managed

How to restrict the maximum CPU utilization for a python program

I have searched it over internet but found nothing related to it.
I have two large python dictionaries that contains more than 2 Million key value pairs .My computer is continuously showing 100% utilization when I am doing any kind of computation on this data.
Due to this I am not able to perform other task on my system as it hangs frequently.
Is there any way that can restrict the maximum CPU allocation for a python program by writing some code in the python program itself.As I do not want to allow this program to use 100% cpu time.
PS: I am currently using sleep function to restrict it but it looks silly.I am using windows 7.

If you are on a linux/unix platform you can use nice to reduce the priority of your process.
This only helps if it is cpu that is maxed out. If you are waiting on disk/swap I/O for example, nice really won't help.
NICE(1) User Commands NICE(1)
NAME
nice - run a program with modified scheduling priority
SYNOPSIS
nice [OPTION] [COMMAND [ARG]...]
DESCRIPTION
Run COMMAND with an adjusted niceness, which affects process schedul‐
ing. With no COMMAND, print the current niceness. Nicenesses range
from -20 (most favorable scheduling) to 19 (least favorable).
For Windows try the START command

Profiling a long-running Python Server

I have a long-running twisted server.
In a large system test, at one particular point several minutes into the test, when some clients enter a particular state and a particular outside event happens, then this server takes several minutes of 100% CPU and does its work very slowly. I'd like to know what it is doing.
How do you get a profile for a particular span of time in a long-running server?
I could easily send the server start and stop messages via HTTP if there was a way to enable or inject the profiler at runtime?
Given the choice, I'd like stack-based/call-graph profiling but even leaf sampling might give insight.

yappi profiler can be started and stopped at runtime.

There are two interesting tools that came up that try to solve that specific problem, where you might not necessarily have instrumented profiling in your code in advance but want to profile production code in a pinch.
pyflame will attach to an existing process using the ptrace(2) syscall and create "flame graphs" of the process. It's written in Python.
py-spy works by reading the process memory instead and figuring out the Python call stack. It also provides a flame graph but also a "top-like" interface to show which function is taking the most time. It's written in Rust and Python.

Not a very Pythonic answer, but maybe straceing the process gives some insight (assuming you are on a Linux or similar).
Using strictly Python, for such things I'm using tracing all calls, storing their results in a ringbuffer and use a signal (maybe you could do that via your HTTP message) to dump that ringbuffer. Of course, tracing slows down everything, but in your scenario you could switch on the tracing by an HTTP message as well, so it will only be enabled when your trouble is active as well.

Pyliveupdate is a tool designed for the purpose: profiling long running programs without restarting them. It allows you to dynamically selecting specific functions to profiling or stop profiling without instrument your code ahead of time -- it dynamically instrument code to do profiling.
Pyliveupdate have three key features:
Profile specific Python functions' (by function names or module names) call time.
Add / remove profilings without restart programs.
Show profiling results with call summary and flamegraphs.
Check out a demo here: https://asciinema.org/a/304465.

Python parallel processing libraries

Python seems to have many different packages available to assist one in parallel processing on an SMP based system or across a cluster. I'm interested in building a client server system in which a server maintains a queue of jobs and clients (local or remote) connect and run jobs until the queue is empty. Of the packages listed above, which is recommended and why?
Edit: In particular, I have written a simulator which takes in a few inputs and processes things for awhile. I need to collect enough samples from the simulation to estimate a mean within a user specified confidence interval. To speed things up, I want to be able to run simulations on many different systems, each of which report back to the server at some interval with the samples that they have collected. The server then calculates the confidence interval and determines whether the client process needs to continue. After enough samples have been gathered, the server terminates all client simulations, reconfigures the simulation based on past results, and repeats the processes.
With this need for intercommunication between the client and server processes, I question whether batch-scheduling is a viable solution. Sorry I should have been more clear to begin with.

Have a go with ParallelPython. Seems easy to use, and should provide the jobs and queues interface that you want.

There are also now two different Python wrappers around the map/reduce framework Hadoop:
http://code.google.com/p/happy/
http://wiki.github.com/klbostee/dumbo
Map/Reduce is a nice development pattern with lots of recipes for solving common patterns of problems.
If you don't already have a cluster, Hadoop itself is nice because it has full job scheduling, automatic data distribution of data across the cluster (i.e. HDFS), etc.

Given that you tagged your question "scientific-computing", and mention a cluster, some kind of MPI wrapper seems the obvious choice, if the goal is to develop parallel applications as one might guess from the title. Then again, the text in your question suggests you want to develop a batch scheduler. So I don't really know which question you're asking.

The simplest way to do this would probably just to output the intermediate samples to separate files (or a database) as they finish, and have a process occasionally poll these output files to see if they're sufficient or if more jobs need to be submitted.

Safely executing user-submitted python code on the server

I am looking into starting a project which involves executing python code that the user enters via a HTML form. I know this can be potentially lethal (exec), but I have seen it done successfully in at least one instance.
I sent an email off to the developers of the Python Challenge and I was told they are using a solution they came up with themselves, and they only let on that they are using "security features provided by the operating system" and that "the operating system [Linux] provides most of the security you need if you know how to use it."
Would anyone know how a safe and secure way to go about doing this? I thought about spawning a new VM for every submission, but that would have way too much overhead and be pert-near impossible to implement efficiently.

On a modern Linux in addition to chroot(2) you can restrict process further by using clone(2) instead of fork(2). There are several interesting clone(2) flags:
CLONE_NEWIPC (new namespace for semaphores, shared memory, message queues)
CLONE_NEWNET (new network namespace - nice one)
CLONE_NEWNS (new set of mountpoints)
CLONE_NEWPID (new set of process identifiers)
CLONE_NEWUTS (new hostname, domainname, etc)
Previously this functionality was implemented in OpenVZ and merged then upstream, so there is no need for patched kernel anymore.

http://codepad.org/about has implemented such a system successfully (as a public code pasting/running service!)
codepad.org is an online compiler/interpreter, and a simple collaboration tool. It's a pastebin that executes code for you. [...]
How it works
Code execution is handled by a supervisor based on geordi. The strategy is to run everything under ptrace, with many system calls disallowed or ignored. Compilers and final executables are both executed in a chroot jail, with strict resource limits. The supervisor is written in Haskell.
[...]
When your app is remote code execution, you have to expect security problems. Rather than rely on just the chroot and ptrace supervisor, I've taken some additional precautions:
The supervisor processes run on virtual machines, which are firewalled such that they are incapable of making outgoing connections.
The machines that run the virtual machines are also heavily firewalled, and restored from their source images periodically.

If you run the script as user nobody (on Linux), it can write practically nowhere and read no data that has its permissions set up properly. But it could still cause a DoS attack by, for example:
filling up /tmp
eating all RAM
eating all CPU
Furthermore, outside network connections can be opened, etcetera etcetera. You can probably lock all these down with kernel limits, but you are bound to forget something.
So I think that a virtual machine with no access to the network or the real hard drive would be the only (reasonably) safe route. Perhaps the developers of the Python Challenge use KVM which is, in principle, "provided by the operating system".
For efficiency, you could run all submissions in the same VM. That saves you much overhead, and in the worst-case scenario they only hamper each other, but not your server.

Using chroot (Wikipedia) may be part of the solution, e.g. combined with ulimit and some other common (or custom) tools.