Develop Reference

Is it possible to inject python code in Kwargs and how could I prevent this user input

I'm at the moment in the middle of writing my Bachelor thesis and for it creating a database system with Postgres and Flask.
To ensure the safety of my data, I was working on a file to prevent SQL injections, since a user should be able to submit a string via Http request. Since most of my functions which I use to analyze the Http request use Kwargs and a dict based on JSON in the request I was wondering if it is possible to inject python code into those kwargs.
And If so If there are ways to prevent that.
To make it easier to understand what I mean, here are some example requests and code:
def calc_sum(a, b):
c = a + b
return c
#app.route(/<target:string>/<value:string>)
def handle_request(target,value):
if target == 'calc_sum':
cmd = json.loads(value)
calc_sum(**cmd)
example Request:
Normal : localhost:5000/calc_sum/{"a":1, "b":2}
Injected : localhost:5000/calc_sum/{"a":1, "b:2 ): print("ham") def new_sum(a=1, b=2):return a+b":2 }
Since I'm not near my work, where all my code is I'm unable to test it out. And to be honest that my code example would work. But I hope this can convey what I meant.
I hope you can help me, or at least nudge me in the right direction. I've searched for it, but all I can find are tutorials on "who to use kwargs".
Best regards.

Yes you, but not in URL, try to use arguments like these localhost:5000/calc_sum?func=a+b&a=1&b=2
and to get these arguments you need to do this in flask
#app.route(/<target:string>)
def handle_request(target):
if target == 'calc_sum':
func= request.args.get('func')
a = request.args.get('a')
b = request.args.get('b')
result = exec(func)
exec is used to execute python code in strings

How to interact with the UI when testing an application written by kivy?

The application is written by kivy.
I want to test a function via pytest, but in order to test that function, I need to initalize the object first, but the object needs something from the UI when initalizing, but I am at testing phase, so don't know how to retrieve something from the UI.
This is the class which has an error and has been handled
class SaltConfig(GridLayout):
def check_phone_number_on_first_contact(self, button):
s = self.instanciate_ServerMsg(tt)
try:
s.send()
except HTTPError as err:
print("[HTTPError] : " + str(err.code))
return
# some code when running without error
def instanciate_ServerMsg():
return ServerMsg()
This is the helper class which generates the ServerMsg object used by the former class.
class ServerMsg(OrderedDict):
def send(self,answerCallback=None):
#send something to server via urllib.urlopen
This is my tests code:
class TestSaltConfig:
def test_check_phone_number_on_first_contact(self):
myError = HTTPError(url="http://127.0.0.1", code=500,
msg="HTTP Error Occurs", hdrs="donotknow", fp=None)
mockServerMsg = mock.Mock(spec=ServerMsg)
mockServerMsg.send.side_effect = myError
sc = SaltConfig(ds_config_file_missing.data_store)
def mockreturn():
return mockServerMsg
monkeypatch.setattr(sc, 'instanciate_ServerMsg', mockreturn)
sc.check_phone_number_on_first_contact()
I can't initialize the object, it will throw an AttributeError when initialzing since it needs some value from UI.
So I get stuck.
I tried to mock the object then patch the function to the original one, but won't work either since the function itself has has logic related to UI.
How to solve it? Thanks

I made an article about testing Kivy apps together with a simple runner - KivyUnitTest. It works with unittest, not with pytest, but it shouldn't be hard to rewrite it, so that it fits your needs. In the article I explain how to "penetrate" the main loop of UI and this way you can happily go and do with button this:
button = <button you found in widget tree>
button.dispatch('on_release')
and many more. Basically you can do anything with such a test and you don't need to test each function independently. I mean... it's a good practice, but sometimes (mainly when testing UI), you can't just rip the thing out and put it into a nice 50-line test.
This way you can do exactly the same thing as a casual user would do when using your app and therefore you can even catch issues you'd have trouble with when testing the casual way e.g. some weird/unexpected user behavior.
Here's the skeleton:
import unittest
import os
import sys
import time
import os.path as op
from functools import partial
from kivy.clock import Clock
# when you have a test in <root>/tests/test.py
main_path = op.dirname(op.dirname(op.abspath(__file__)))
sys.path.append(main_path)
from main import My
class Test(unittest.TestCase):
def pause(*args):
time.sleep(0.000001)
# main test function
def run_test(self, app, *args):
Clock.schedule_interval(self.pause, 0.000001)
# Do something
# Comment out if you are editing the test, it'll leave the
# Window opened.
app.stop()
def test_example(self):
app = My()
p = partial(self.run_test, app)
Clock.schedule_once(p, 0.000001)
app.run()
if __name__ == '__main__':
unittest.main()
However, as Tomas said, you should separate UI and logic when possible, or better said, when it's an efficient thing to do. You don't want to mock your whole big application just to test a single function that requires communication with UI.

Finally made it, just get things done, I think there must be a more elegant solution. The idea is simple, given the fact that all lines are just simply value assignment except the s.send() statement.
Then we just mock the original object, every time when some errors pop up in the testing phase (since the object lack some values from the UI), we mock it, we repeat this step until the testing method can finally test if the function can handle the HTTPError or not.
In this example, we only need to mock a PhoneNumber class which is lucky, but some times we may need to handle more, so obviously #KeyWeeUsr 's answer is an more ideal choice for the production environment. But I just list my thinking here for somebody who wants a quick solution.
#pytest.fixture
def myHTTPError(request):
"""
Generating HTTPError with the pass-in parameters
from pytest_generate_tests(metafunc)
"""
httpError = HTTPError(url="http://127.0.0.1", code=request.param,
msg="HTTP Error Occurs", hdrs="donotknow", fp=None)
return httpError
class TestSaltConfig:
def test_check_phone_number( self, myHTTPError, ds_config_file_missing ):
"""
Raise an HTTP 500 error, and invoke the original function with this error.
Test to see if it could pass, if it can't handle, the test will fail.
The function locates in configs.py, line 211
This test will run 2 times with different HTTP status code, 404 and 500
"""
# A setup class used to cover the runtime error
# since Mock object can't fake properties which create via __init__()
class PhoneNumber:
text = "610274598038"
# Mock the ServerMsg class, and apply the custom
# HTTPError to the send() method
mockServerMsg = mock.Mock(spec=ServerMsg)
mockServerMsg.send.side_effect = myHTTPError
# Mock the SaltConfig class and change some of its
# members to our custom one
mockSalt = mock.Mock(spec=SaltConfig)
mockSalt.phoneNumber = PhoneNumber()
mockSalt.instanciate_ServerMsg.return_value = mockServerMsg
mockSalt.dataStore = ds_config_file_missing.data_store
# Make the check_phone_number_on_first_contact()
# to refer the original function
mockSalt.check_phone_number = SaltConfig.check_phone_number
# Call the function to do the test
mockSalt.check_phone_number_on_first_contact(mockSalt, "button")

I'm confused about Scrapy Source Codes in shell.py

I have been learning python for about one month, and I was reading scrapy source code during last week.
It's OK for me to understand modules like spiders or crawler. Until I found something interesting in 'shell.py', I was totally confused. Here is the codes.
def _request_deferred(request):
"""Wrap a request inside a Deferred.
This function is harmful, do not use it until you know what you are doing.
This returns a Deferred whose first pair of callbacks are the request
callback and errback. The Deferred also triggers when the request
callback/errback is executed (ie. when the request is downloaded)
WARNING: Do not call request.replace() until after the deferred is called.
"""
# zeal4u: what's the meaning of codes below?
request_callback = request.callback
request_errback = request.errback
def _restore_callbacks(result):
request.callback = request_callback
request.errback = request_errback
return result
d = defer.Deferred()
d.addBoth(_restore_callbacks)
if request.callback:
d.addCallbacks(request.callback, request.errback)
request.callback, request.errback = d.callback, d.errback
return d
So the point is that why it assigns 'request.callback' and 'request.errback' to local vars 'request_callback' and 'request_errback', then assigns vars back to them in next function '_restore_callbacks'?
I know this can't be useless, but what's the real meaning and how it works?
Or should I read some relevant modules to figure it out? Please give me some advice. : )

How to get results out of a Python exec()/eval() call?

I want to write a tool in Python to prepare a simulation study by creating for each simulation run a folder and a configuration file with some run-specific parameters.
study/
study.conf
run1
run.conf
run2
run.conf
The tool should read the overall study configuration from a file including (1) static parameters (key-value pairs), (2) lists for iteration parameters, and (3) some small code snippets to calculate further parameters from the previous ones. The latter are run specific depending on the permutation of the iteration parameters used.
Before writing the run.conf files from a template, I need to run some code like this to determine the specific key-value pairs from the code snippets for that run
code = compile(code_str, 'foo.py', 'exec')
rv=eval(code, context, { })
However, as this is confirmed by the Python documentation, this just leads to a None as return value.
The code string and context dictionary in the example are filled elsewhere. For this discussion, this snippet should do it:
code_str="""import math
math.sqrt(width**2 + height**2)
"""
context = {
'width' : 30,
'height' : 10
}
I have done this before in Perl and Java+JavaScript. There, you just give the code snippet to some evaluation function or script engine and get in return a value (object) from the last executed statement -- not a big issue.
Now, in Python I struggle with the fact that eval() is too narrow just allowing one statement and exec() doesn't return values in general. I need to import modules and sometimes do some slightly more complex calculations, e.g., 5 lines of code.
Isn't there a better solution that I don't see at the moment?
During my research, I found some very good discussions about Pyhton eval() and exec() and also some tricky solutions to circumvent the issue by going via the stdout and parsing the return value from there. The latter would do it, but is not very nice and already 5 years old.

The exec function will modify the global parameter (dict) passed to it. So you can use the code below
code_str="""import math
Result1 = math.sqrt(width**2 + height**2)
"""
context = {
'width' : 30,
'height' : 10
}
exec(code_str, context)
print (context['Result1']) # 31.6
Every variable code_str created will end up with a key:value pair in the context dictionary. So the dict is the "object" like you mentioned in JavaScript.
Edit1:
If you only need the result of the last line in code_str and try to prevent something like Result1=..., try the below code
code_str="""import math
math.sqrt(width**2 + height**2)
"""
context = { 'width' : 30, 'height' : 10 }
lines = [l for l in code_str.split('\n') if l.strip()]
lines[-1] = '__myresult__='+lines[-1]
exec('\n'.join(lines), context)
print (context['__myresult__'])
This approach is not as robust as the former one, but should work for most case. If you need to manipulate the code in a sophisticated way, please take a look at the Abstract Syntax Trees

Since this whole exec() / eval() thing in Python is a bit weird ... I have chose to re-design the whole thing based on a proposal in the comments to my question (thanks #jonrsharpe).
Now, the whole study specification is a .py module that the user can edit. From there, the configuration setup is directly written to a central object of the whole package. On tool runs, the configuration module is imported using the code below
import imp
# import the configuration as a module
(path, name) = os.path.split(filename)
(name, _) = os.path.splitext(name)
(file, filename, data) = imp.find_module(name, [path])
try:
module = imp.load_module(name, file, filename, data)
except ImportError as e:
print(e)
sys.exit(1)
finally:
file.close()

I came across similar needs, and finally figured out a approach by playing with ast:
import ast
code = """
def tf(n):
return n*n
r=tf(3)
{"vvv": tf(5)}
"""
ast_ = ast.parse(code, '<code>', 'exec')
final_expr = None
for field_ in ast.iter_fields(ast_):
if 'body' != field_[0]: continue
if len(field_[1]) > 0 and isinstance(field_[1][-1], ast.Expr):
final_expr = ast.Expression()
final_expr.body = field_[1].pop().value
ld = {}
rv = None
exec(compile(ast_, '<code>', 'exec'), None, ld)
if final_expr:
rv = eval(compile(final_expr, '<code>', 'eval'), None, ld)
print('got locals: {}'.format(ld))
print('got return: {}'.format(rv))
It'll eval instead of exec the last clause if it's an expression, or have all execed and return None.
Output:
got locals: {'tf': <function tf at 0x10103a268>, 'r': 9}
got return: {'vvv': 25}

Python: Networked IDLE/Redo IDLE front-end while using the same back-end?

Is there any existing web app that lets multiple users work with an interactive IDLE type session at once?
Something like:
IDLE 2.6.4
Morgan: >>> letters = list("abcdefg")
Morgan: >>> # now, how would you iterate over letters?
Jack: >>> for char in letters:
print "char %s" % char
char a
char b
char c
char d
char e
char f
char g
Morgan: >>> # nice nice
If not, I would like to create one. Is there some module I can use that simulates an interactive session? I'd want an interface like this:
def class InteractiveSession():
''' An interactive Python session '''
def putLine(line):
''' Evaluates line '''
pass
def outputLines():
''' A list of all lines that have been output by the session '''
pass
def currentVars():
''' A dictionary of currently defined variables and their values '''
pass
(Although that last function would be more of an extra feature.)
To formulate my problem another way: I'd like to create a new front end for IDLE. How can I do this?
UPDATE: Or maybe I can simulate IDLE through eval()?
UPDATE 2: What if I did something like this:
I already have a simple GAE Python chat app set up, that allows users to sign in, make chat rooms, and chat with each other.
Instead of just saving incoming messages to the datastore, I could do something like this:
def putLine(line, user, chat_room):
''' Evaluates line for the session used by chat_room '''
# get the interactive session for this chat room
curr_vars = InteractiveSession.objects.where("chatRoom = %s" % chat_room).get()
result = eval(prepared_line, curr_vars.state, {})
curr_vars.state = curr_globals
curr_vars.lines.append((user, line))
if result:
curr_vars.lines.append(('SELF', result.__str__()))
curr_vars.put()
The InteractiveSession model:
def class InteractiveSession(db.Model):
# a dictionary mapping variables to values
# it looks like GAE doesn't actually have a dictionary field, so what would be best to use here?
state = db.DictionaryProperty()
# a transcript of the session
#
# a list of tuples of the form (user, line_entered)
#
# looks something like:
#
# [('Morgan', '# hello'),
# ('Jack', 'x = []'),
# ('Morgan', 'x.append(1)'),
# ('Jack', 'x'),
# ('SELF', '[1]')]
lines = db.ListProperty()
Could this work, or am I way off/this approach is infeasible/I'm duplicating work when I should use something already built?
UPDATE 3: Also, assuming I get everything else working, I'd like syntax highlighting. Ideally, I'd have some API or service I could use that would parse the code and style it appropriately.
for c in "characters":
would become:
<span class="keyword">for</span> <span class="var">c</span> <span class="keyword">in</span> <span class="string>"characters"</span><span class="punctuation">:</span>
Is there a good existing Python tool to do this?

I could implement something like this pretty quickly in Nevow. Obviously, access would need to be pretty restricted since doing something like this involves allowing access to a Python console to someone via HTTP.
What I'd do is create an Athena widget for the console, that used an instance of a custom subclass of code.InteractiveInterpreter that is common to all users logged in.
UPDATE: Okay, so you have something chat-like in GAE. If you just submit lines to a code.InteractiveInterpreter subclass that looks like this, it should work for you. Note that the interface is pretty similar to the InteractiveSession class you describe:
class SharedConsole(code.InteractiveInterpreter):
def __init__(self):
self.users = []
def write(self, data):
# broadcast output to connected clients here
for user in self.users:
user.addOutput(data)
class ConnectedUser(object):
def __init__(self, sharedConsole):
self.sharedConsole = sharedConsole
sharedConsole.users.append(self) # reference look, should use weak refs
def addOutput(self, data):
pass # do GAE magic to send data to connected client
# this is a hook for submitted code lines; call it from GAE when a user submits code
def gotCommand(self, command):
needsMore = self.sharedConsole.runsource(command)
if needsMore:
pass # tell the client to change the command line to a textarea
# or otherwise add more lines of code to complete the statement

The closest Python interpreter I know of to what you are looking for, in terms of interface, is DreamPie. It has separate input and output areas, much like a chat interface. Also, DreamPie runs all of the code in a subprocess. DreamPie also does completion and syntax coloring, much like IDLE, which means it doesn't just pipe input and output to/from the subprocess -- it has implemented the abstractions which you are looking for.
If you wish to develop a desktop application (not a web-app), I recommend basing your work on DreamPie and just adding the multiple-frontend functionality.
Update: For syntax highlighting (including HTML) see the Pygments project. But that is a completely different question; please ask one question at a time here.

As a proof of concept, you may be able to put something together using sockets and a command-line session.

this is likely possible with the upcoming implimentation of IPython using a 0MQ backend.

I would use ipython and screen. With this method, you would have to create a shared login, but you could both connect to the shared screen session. One downside would be that you would both appear as the same user.