I'd like to have a tool in Perl to gather useful statistics for page loads (eg, download time/speed, CDN information, headers, dns lookups, compressions)
Does anyone know if one exists or if there's a place to learn about how to make one?
You might want to try WWW::Mechanize::Timed, which extends the WWW::Mechanize module. The ::Timed features will allow you to collect information on how long your requests take. The underlying ::Mechanize module, which is itself a subclass of LWP::UserAgent, would give you access to your response, including headers, body content, and images. From these you could compute total page "weight", number of requests, etc. This doesn't cover everything YSlow does (exposing the DNS internals underlying gethostbyname would be a good trick!) but I hope it's a place to start, if I've understood your question properly.
You could have the perl CGI (or any perl program) run a few times under the profiler, and scan for commonalities. I haven't seen a web-based interface like this, but if you have control over the perl side of things, the documentation is here:
http://www.perl.com/pub/a/2004/06/25/profiling.html
It basically boils down to running your perl program with -d:DProf and then, after it finishes, running dprofpp in the same directory:
# perl -d:DProf ./foo.pl
# dprofpp
Update:
Yes, this is not the same thing as protocol profiling, as duly noted below, but there aren't may alternatives for perl. If you are trying to find where the perl portion of the slowness is coming from, profiling perl is a good place to start. Products like YSlow will track the pure protocol aspects of it, whether the CGI is perl or php or python.
Personally, I use it to profile my django site, which is in python and flash, and I profile those separately from the protocol portion of the system, which I also use YSlow for.
Also, there are perl plugins for "ddd" which will at least make it graphical:
http://www.gnu.org/software/ddd/
Sorry if this doesn't solve the exact problem, I would like to know if there's a perl interface to collate this as well, but I know this is where I would start looking...
Related
I understand that letting any anonymous user upload any sort of file in general can be dangerous, especially if it's code. However, I have an idea to let users upload custom AI scripts to my website. I would provide the template so that the user could compete with other AI's in an online web game I wrote in Python. I either need a solution to ensure a user couldn't compromise any other files or inject malicious code via their uploaded script or a solution for client-side execution of the game. Any suggestions? (I'm looking for a solution that will work with my Python scripts)
I am in no way associated with this site and I'm only linking it because it tries to achieve what you are getting after: jailing of python. The site is code pad.
According to the about page it is ran under geordi and traps all sys calls with ptrace. In addition to be chroot'ed they are on a virtual machine with firewalls in place to disallow outbound connections.
Consider it a starting point but I do have to chime in on the whole danger thing. Gotta CYA myself. :)
Using PyPy you can create a python sandbox. The sandbox is a separate and supposedly secure python environment where you can execute their scripts. More info here
http://codespeak.net/pypy/dist/pypy/doc/sandbox.html
"In theory it's impossible to do anything bad or read a random file on the machine from this prompt."
"This is safe to do even if script.py comes from some random untrusted source, e.g. if it is done by an HTTP server."
Along with other safeguards, you can also incorporate human review of the code. Assuming part of the experience is reviewing other members' solutions, and everyone is a python developer, don't allow new code to be activated until a certain number of members vote for it. Your users aren't going to approve malicious code.
Yes.
Allow them to script their client, not your server.
PyPy is probably a decent bet on the server side as suggested, but I'd look into having your python backend provide well defined APIs and data formats and have the users implement the AI and logic in Javascript so it can run in their browser. So the interaction would look like: For each match/turn/etc, pass data to the browser in a well defined format, provide a javascript template that receives the data and can implement logic, and provide web APIs that can be invoked by the client (browser) to take the desired actions. That way you don't have to worry about security or server power.
Have an extensive API for the users and strip all other calls upon upload (such as import statements). Also, strip everything that has anything to do with file i/o.
(You might want to do multiple passes to ensure that you didn't miss anything.)
I'm planning to develop a GUI application that uses curses. The idea is to provide an extra interface for a web interface, so that everything on the web site could also be done via the UI.
Basically, it should be platform independent: the user would have to SSH to the server after which the UI would automatically take over.
First of all, is this doable? As far as I understand, it would be platform independent as long as the end-user had the proper terminal software installed. Correct me, if I'm wrong.
I was planning to use Python for this, as it is the language I'm the most proficient in. Python comes with the ncurses library and Urwid, which I've been told, is quite good.
After having a quick test with Urwid, I had some problems. The thing is, I'm quite worried that I won't find answers to the problems that I will encounter down the road because apparently curses UI-s aren't all the rage nowadays. Documentation and examples are thus quite scarce.
In conclusion, should really I embark on this and quit my whining or drop the idea altogether? Any other suggestions?
It's certainly possible, and curses-based applications are still written regularly (e.g. PuDB is only 14 months old) although maybe not very often.
Did you try asking questions on the Urwid mailing list and/or IRC channel?
oh my, wouldn't this be a dream!
i've seen a couple of things out there to varying degrees of success.
Morticious Thrind: http://thrind.xamai.ca/
future death toll: http://f-dt.com/?wptheme=wp-cli
wordpress yadda yadda, this could be as simple as a 960/blueprint CSS, prototype.js, and a oneliner:
//TODO: Implement useful functionality && unit tests && documentation
//TODO: read
try { eval($F(x)); } catch (e) { panic(); }
BUT! this type of thing is pretty radical. i mean- ANYTHING can happen on the canvas of a web-browser these days, but any terminal emulator or lynx serves this purpose with flare.
also be sure to check out: https://stackoverflow.com/questions/472644/javascript-collection-of-one-line-useful-functions
the real question is what sort of software you plan on ncursing (sic,pun,etc.)-- it probably already has some rather useful command-line interfaces (sh).
It can be done but it's a struggle. I would recommend improving the web interface. You can use JavaScript to add keyboard shortcuts, for example, which can be very helpful for a faster workflow (see Gmail's interface, for example).
Is it possible to use multiple languages along side with ruby. For example, I have my application code in Ruby on Rails. I would like to calculate the recommendations and I would like to use python for that. So essentially, python code would get the data and calculate all the stuff and probably get the data from DB, calculate and update the tables.Is it possible and what do you guys think about its adv/disadv
Thanks
If you are offloading work to an exterior process, you may want to make this a webservice (ajax, perhaps) of some sort so that you have some sort of consistent interface.
Otherwise, you could always execute the python script in a subshell through ruby, using stdin/stdout/argv, but this can get ugly quick.
Depending on your exact needs, you can either call out to an external process (using popen, system, etc) or you can setup another mini-web-server or something along those lines and have the rails server communicate with it over HTTP with a REST-style API (or whatever best suits your needs).
In your example, you have a ruby frontend website and then a number-crunching python backend service that builds up recommendation data for the ruby site. A fairly nice solution is to have the ruby site send a HTTP request to the python service when it needs data updating (with a payload of information to identify what it needs doing to what or some such) and then the python backend service can crunch away and update the table which presumably your ruby frontend will automatically pick up the changes of during the next request and display.
I would use the system command
as such
system("python myscript.py")
An easy, quick 'n' dirty solution in case you have python scripts and you want to execute them from inside rails, is this:
%x[shell commands or python path/of/pythonscript.py #{ruby variables to pass on the script}]
or
``shell commands or python path/of/pythonscript.py #{ruby variables to pass on the script}\ (with ` symbol in the beginning and the end).
Put the above inside a controller and it will execute.
For some reason, inside ruby on rails, system and exec commands didn't work for me (exec crashed my application and system doesn't do anything).
is it possible to use python to create flash like browser games? (Actually I want to use it for an economic simulation, but it amounts to the same as a browser game)
Davoud
The answer would be yes, assuming you consider this a good example of what you want to do:
http://pyjs.org/examples/Space.html
This browser-based version of Asteroids was created using Pyjamas, which enables you to write the code in python in one place, and have it run either on the browser, or on the desktop:
http://pyjs.org/
Having recently found Pyjamas, and also preferring to consolidate my code in one language (Python!) and location (instead of having some code server-side, and some browser/client-side, in different languages), it's definitely an exciting technology. Its authors have ported the Google Web Toolkit to Python, a really impressive feat, retaining the expressive power of Python (something like 80,000 lines of Java was shrunk to 8,000 lines of Python). More Pythonistas should know about it. :)
You could use Python to do client side scripting using Silverlight + IronPython. Of course, this requires all your users install Silverlight.
I think you're talking about using Python on the back end, in which case running something on the server side with Python (in which case this Django vs other Python web frameworks SO question is a good general list and may have what you're looking for.
You need to use something that the current browsers support, this means you're stuck with Flash, Java applets or Javascript+HTML if you want your game displayed in a browser.
You can use python on the backend and display pure HTML, if that is enough for your needs.
Yes, but there a a number of ways to get there.
Flash is the client side rendering. You could use Python to generate Flash, or you could use Python to generate some dynamic HTML with Javascript, etc. that was interactive in a similar way.
But the Python will be running on the server. The Flash, ActionScript, HTML, JavaScript, etc. will all be running on the client.
So while the answer to the question is yes, I am going to suggest you might need to do more research and ask a better question.
You could have Python CGI code as a backend and send input in to it through AJAX. Its probably better just use something on the client side for this, though.
I understand that letting any anonymous user upload any sort of file in general can be dangerous, especially if it's code. However, I have an idea to let users upload custom AI scripts to my website. I would provide the template so that the user could compete with other AI's in an online web game I wrote in Python. I either need a solution to ensure a user couldn't compromise any other files or inject malicious code via their uploaded script or a solution for client-side execution of the game. Any suggestions? (I'm looking for a solution that will work with my Python scripts)
I am in no way associated with this site and I'm only linking it because it tries to achieve what you are getting after: jailing of python. The site is code pad.
According to the about page it is ran under geordi and traps all sys calls with ptrace. In addition to be chroot'ed they are on a virtual machine with firewalls in place to disallow outbound connections.
Consider it a starting point but I do have to chime in on the whole danger thing. Gotta CYA myself. :)
Using PyPy you can create a python sandbox. The sandbox is a separate and supposedly secure python environment where you can execute their scripts. More info here
http://codespeak.net/pypy/dist/pypy/doc/sandbox.html
"In theory it's impossible to do anything bad or read a random file on the machine from this prompt."
"This is safe to do even if script.py comes from some random untrusted source, e.g. if it is done by an HTTP server."
Along with other safeguards, you can also incorporate human review of the code. Assuming part of the experience is reviewing other members' solutions, and everyone is a python developer, don't allow new code to be activated until a certain number of members vote for it. Your users aren't going to approve malicious code.
Yes.
Allow them to script their client, not your server.
PyPy is probably a decent bet on the server side as suggested, but I'd look into having your python backend provide well defined APIs and data formats and have the users implement the AI and logic in Javascript so it can run in their browser. So the interaction would look like: For each match/turn/etc, pass data to the browser in a well defined format, provide a javascript template that receives the data and can implement logic, and provide web APIs that can be invoked by the client (browser) to take the desired actions. That way you don't have to worry about security or server power.
Have an extensive API for the users and strip all other calls upon upload (such as import statements). Also, strip everything that has anything to do with file i/o.
(You might want to do multiple passes to ensure that you didn't miss anything.)