I am looking into starting a project which involves executing python code that the user enters via a HTML form. I know this can be potentially lethal (exec), but I have seen it done successfully in at least one instance.
I sent an email off to the developers of the Python Challenge and I was told they are using a solution they came up with themselves, and they only let on that they are using "security features provided by the operating system" and that "the operating system [Linux] provides most of the security you need if you know how to use it."
Would anyone know how a safe and secure way to go about doing this? I thought about spawning a new VM for every submission, but that would have way too much overhead and be pert-near impossible to implement efficiently.
On a modern Linux in addition to chroot(2) you can restrict process further by using clone(2) instead of fork(2). There are several interesting clone(2) flags:
CLONE_NEWIPC (new namespace for semaphores, shared memory, message queues)
CLONE_NEWNET (new network namespace - nice one)
CLONE_NEWNS (new set of mountpoints)
CLONE_NEWPID (new set of process identifiers)
CLONE_NEWUTS (new hostname, domainname, etc)
Previously this functionality was implemented in OpenVZ and merged then upstream, so there is no need for patched kernel anymore.
http://codepad.org/about has implemented such a system successfully (as a public code pasting/running service!)
codepad.org is an online compiler/interpreter, and a simple collaboration tool. It's a pastebin that executes code for you. [...]
How it works
Code execution is handled by a supervisor based on geordi. The strategy is to run everything under ptrace, with many system calls disallowed or ignored. Compilers and final executables are both executed in a chroot jail, with strict resource limits. The supervisor is written in Haskell.
[...]
When your app is remote code execution, you have to expect security problems. Rather than rely on just the chroot and ptrace supervisor, I've taken some additional precautions:
The supervisor processes run on virtual machines, which are firewalled such that they are incapable of making outgoing connections.
The machines that run the virtual machines are also heavily firewalled, and restored from their source images periodically.
If you run the script as user nobody (on Linux), it can write practically nowhere and read no data that has its permissions set up properly. But it could still cause a DoS attack by, for example:
filling up /tmp
eating all RAM
eating all CPU
Furthermore, outside network connections can be opened, etcetera etcetera. You can probably lock all these down with kernel limits, but you are bound to forget something.
So I think that a virtual machine with no access to the network or the real hard drive would be the only (reasonably) safe route. Perhaps the developers of the Python Challenge use KVM which is, in principle, "provided by the operating system".
For efficiency, you could run all submissions in the same VM. That saves you much overhead, and in the worst-case scenario they only hamper each other, but not your server.
Using chroot (Wikipedia) may be part of the solution, e.g. combined with ulimit and some other common (or custom) tools.
Related
I am new to AWS EC2 so that I make this post for some questions.
1) Right now, I am considering running some script on the server. I use two tools usually. One is a software can only be used in Windows. The other is just python. Should I open two instances, one for windows, one for ubuntu? Or just one instance of Windows with Git Bash installed? I want to be cost and performance efficiently.
2) I am not going to use the script very often (usually 2-3 hours per day or 10-12 hours per week). Therefore, is it easy to schedule those jobs automatically across the instances? I mean it can automatically turn off and restart given appropriate time.
3) Some of the script involves web scraping. I am also wondering if it is ok to switch IP address every time I run the script. Mainly, it is for python script.
Thanks.
1) Well, off course, the less instances you have, the less you will pay. Python can run on Windows, I just don't know how tricky it would be to make it work in your case. It all depends on what you are running and what are your management requirements. Those script languages were originally designed for Unix environments, so people usually runs it on those kind of systems, so running it in Windows may be a little unpleasant. Anyway, I don't think you should ask someone else it, you should figure it out yourself what suits you best.
2) AWS doesn't have a scheduler for EC2 (stop, starting, etc, given date/times/recurrence). It's something that I miss on it too. So, to achieve something like this you have some options.
Turning your temporary instance into an auto-scaling group of 1 instance, and scheduling policies to scale it in to zero instances and scale it out to 1 instance again when you want. The problem with this approach is: if you can't be sure how long it will take for your job to be completed, then you have a problem, off course, because those scheduled actions are based in fixed date/times. One solution for this would be the temporary instance itself changing the autoscaling group configuration to zero instances via API when it has finished. (In this case, you would just have a scale out scheduled policy, to launch the instance, leaving the termination of it to be done 'manually', via auto-scaling group configuration handling from inside the temporary instance). But be aware that auto-scaling is very tricky for begginers, and you should go throught the documentation before using it. (For exemple, each time you scale in and out you instances, they're terminated, not just stopped, and you lose every data on it.)
Not using auto-scaling group, having a regular instance, and scheduling all those actions from outside it via API. It could be from your Windows (master) instance. In this case, the master would start the temporary instance via API, which would run its things and then turn itself off when it had finished. Otherwise, the master instance would have to keep polling the temporary one somehow to know when the jobs are done and it can be shutdown from outside.
There are probably more complicated ways for doing this (Elastic Beanstalk crons, maybe).
I think, in this case, the more simple, the better. So, I would stick to the option 2). You will only need to figure how to install and use AWS CLI on Windows and manage IAM credentials and permissions to provide your CLI access enough for it to do what it needs.
3) If you don't assign an Elastic IP to your instance, you will get a different IP each time you stop and start it, so this is, by default, what you want. In auto-scaling, this is the only way, you can't even assign a fixed IP to instances.
I hope I could help you a little bit.
I have been looking at daemons for Linux such as httpd and have also looked at some code that can be used as a skeleton. I have done a fair amount of research and now I want to practice writing it. However, I'm not sure of what can I use a daemon for. Any good examples/ideas that I can try to execute?
I was thinking of using a daemon along with libnotify on Ubuntu to have pop-up notifications of select tweets.
Is this a bad example for implementing a daemon?
Will you even need a daemon for this?
Can this be implemented as a service rather than a daemon?
First: PEP 3143 tries to enumerate all of the fiddly details you have to get right to write a daemon in Python. And it specifies a library that takes care of those details for you.
The PEP was deferred—at least in part because the community felt it was more a responsibility of POSIX or some Linux standards group or something to first define exactly what is essential to being a daemon, before Python could have its own position on how to implement one. But it's still a great guide. However, the reference implementation of that proposed library still lives on, as python-daemon, which you can install from PyPI.
Meanwhile, the really interesting question for this project isn't so much service vs. daemon, as root vs. user. Do you want a single process that keeps track of all users' twitter accounts, and sends notifications to anyone who's logged in? Just a per-user process? Or maybe both, a single process watching all the tweets, then sending notifications via user processes?
Of course you don't really need a daemon or service for this. For example, it could be a GUI app whose main window is a configuration dialog, which keeps running (maybe with a traybar thingy) even when you close the config dialog, and it would work just as well. The question isn't whether you need a daemon, but whether it's more appropriate. Which really is a design choice.
This is my first hack at doing any system-level programming (mostly a LAMPhp, specifically Drupal, web dev up to this point).
Because of availability of a library with a very specific feature, I am using Python for an upcoming project. I need to run, restart as needed, monitor and respond to the output of multiple Python script processes, controlled ideally via a HTTP API from another master program which keeps a database of processes that need to be running, and some metadata about those processes (parameters, pid, etc). I'm planning on building this master program in PHP as I have far more experience in it, hence the want for a nice HTTP API.
Is there some best practice for this type of system? Some initial research lead me to supervisord (which has XML-RPC built in, apparently), but I thought I'd check the wisdom of the masses who've actually been down this road before moving forward with testing.
I can't say I have been down this road, but I am working to go down this road. I would look into the multiprocessing libraries for Python. There are network transparent libraries. A couple of routes you could take with those:
1. Create a process that controls all of the other processes. Make this process a server you can control with your PHP.
2. Determine how to get PHP to communicate to these networked Python processes. They may still need to be launched from a central Python process however.
I have a long-running twisted server.
In a large system test, at one particular point several minutes into the test, when some clients enter a particular state and a particular outside event happens, then this server takes several minutes of 100% CPU and does its work very slowly. I'd like to know what it is doing.
How do you get a profile for a particular span of time in a long-running server?
I could easily send the server start and stop messages via HTTP if there was a way to enable or inject the profiler at runtime?
Given the choice, I'd like stack-based/call-graph profiling but even leaf sampling might give insight.
yappi profiler can be started and stopped at runtime.
There are two interesting tools that came up that try to solve that specific problem, where you might not necessarily have instrumented profiling in your code in advance but want to profile production code in a pinch.
pyflame will attach to an existing process using the ptrace(2) syscall and create "flame graphs" of the process. It's written in Python.
py-spy works by reading the process memory instead and figuring out the Python call stack. It also provides a flame graph but also a "top-like" interface to show which function is taking the most time. It's written in Rust and Python.
Not a very Pythonic answer, but maybe straceing the process gives some insight (assuming you are on a Linux or similar).
Using strictly Python, for such things I'm using tracing all calls, storing their results in a ringbuffer and use a signal (maybe you could do that via your HTTP message) to dump that ringbuffer. Of course, tracing slows down everything, but in your scenario you could switch on the tracing by an HTTP message as well, so it will only be enabled when your trouble is active as well.
Pyliveupdate is a tool designed for the purpose: profiling long running programs without restarting them. It allows you to dynamically selecting specific functions to profiling or stop profiling without instrument your code ahead of time -- it dynamically instrument code to do profiling.
Pyliveupdate have three key features:
Profile specific Python functions' (by function names or module names) call time.
Add / remove profilings without restart programs.
Show profiling results with call summary and flamegraphs.
Check out a demo here: https://asciinema.org/a/304465.
We need to handle massive file uploads without spending resources on an IIS 7 server. To emphasize how light-weight this needs to be, let's say that we need to handle file uploads of sizes that are completely insane, like 100GB uploads, or something that can continue running for an extremely long time without consuming additional resources. Basically we need something that gives us control over the reception of the file from the moment it starts to the moment it ends.
A bit of background:
We're using ColdFusion as the server-side processor, but it has failed us when handling uploads beyond about 1GB and we've exhausted our configuration options. There's a long story behind that, but essentially, if a .cfm page (ColdFusion) is the destination of the file upload and it goes over about 1GB, it gives a 503 error... even if the target file doesn't exist. So clearly too much is going on merely by telling the server that we intend to process the file with a .cfm page.
We suspect that this is due to Java limitations because the server (or really, the workstation in this case) does not show any signs of load on CPU or memory. Since we have limited memory and this website is intended for a lot of concurrent uploads, we can't trust simply raising the virtual machine memory usage, especially because that simply doesn't work currently, even for a single connection... let alone the hundreds of concurrent connections we expect when we go live.
So we're down to writing a specialized solution using CGI that will handle file uploads only. Basically, we need control on the server-side that we don't get with ColdFusion or ASP.NET because those technologies do so many things on their own, behind the scenes, without giving us the control we need. They always end up spending up too many resources one way or the other for an arguably obvious reason; what we're trying to do is completely insane and not the intended function of those technologies. That's why we want a specialized uploader through CGI that bypasses all that ColdFusion/ASP.NET magic that keeps getting in the way, hoping it gives us the control we need.
But before we spent countless hours on this, I figured I'd ask around and see if anyone knows of a proper solution to this problem that might be viable in our case.
The only real restriction here is that it has to be CGI, and it has to run on IIS 7, therefore a Windows "Server" environment. We're fine with it being written in Python, Perl, name it... provided it can run as a CGI, but it has to run as a CGI... unless of course someone has better ideas on how to do this.
So the magic question is; are there CGI solutions out there that already do this or are we stuck with writing it on our own, hoping that the reason no one else has done it already is some other than it being impossible?
Thanks in advance.
You're not going to get reliable multi-GB uploads from a dumb client (eg a browser and standard upload behaviour). Been there, done that, written commercial digital asset management solutions handling huge files.
The key to any degree of reliability in this scenario is chunking - you need to be able to chunk the upload, send each chunk as a discrete file, and re-assemble it server side.
What are your client restrictions though (if any)? Can you use a java applet? Could you even have a client side app ?
One possible starting point for a browser based solution would be the jupload opensource project but there are plenty of others.
You want WebDAV, not CGI. It provides all the nice bits that make file transfers not suck, like resuming and pausing.
the windows TCP stack is limited to 4GB file uploads. Anymore than that is not possible.