Develop Reference

Context Manager Hackery

I'm trying to make a quick and dirty caching system for Python, using the trick that a context-manager can be made to conditionally skip the code in its context — see Skipping execution of -with- block. I've stumbled upon a weird failure case of this and I was wondering if someone can help understand and fix this.
Before anyone says this, I know what I'm doing is terrible and I shouldn't do it, etc, etc.
Anyway, here is the code for the tricky context manager:
import sys
import inspect
class SkippableContext(object):
def __init__(self,mode=0):
"""
if mode = 0, proceed as normal
if mode = 1, do not execute block
"""
self.mode=mode
def __enter__(self):
if self.mode==1:
print(' ... Skipping Context')
# Do some magic
sys.settrace(lambda *args, **keys: None)
frame = inspect.currentframe(1)
frame.f_trace = self.trace
return 'SET BY TRICKY CONTEXT MANAGER!!'
def trace(self, frame, event, arg):
raise
def __exit__(self, type, value, traceback):
return True
And here is the test code:
print('==== First Pass with skipping disabled ====')
c='not set'
with SkippableContext(mode=0) as c:
print('Should Get into here')
c = 'set in context'
print('c: {}'.format(c))
print('==== Second Pass with skipping enabled ====')
c='not set'
with SkippableContext(mode=1) as c:
print('This code is not printed')
c = 'set in context'
print('c: {}'.format(c))
c='not set'
with SkippableContext(mode=1) as c:
print('This code is not printed')
c = 'set in context'
print('c: {}'.format(c))
print('==== Third Pass: Same as second pass but in a loop ====')
for i in range(2):
c='not set'
with SkippableContext(mode=1) as c: # For some reason, assinging c fails on the second iteration!
print('This code is not printed')
c = 'set in context'
print('c: {}'.format(c))
The output generated by the test code is as expected, except for the very last line, where c is not set:
==== First Pass with skipping disabled ====
Should Get into here
c: set in context
==== Second Pass with skipping enabled ====
... Skipping Context
c: SET BY TRICKY CONTEXT MANAGER!!
... Skipping Context
c: SET BY TRICKY CONTEXT MANAGER!!
==== Third Pass: Same as second pass but in a loop ====
... Skipping Context
c: SET BY TRICKY CONTEXT MANAGER!!
... Skipping Context
c: not set
Why is c not set in the second run of the loop? Is there some hack to fix the bug in this hack?

The awful hack you're using does a lot of things with nasty, subtle consequences. I doubt the author fully understood it (if they did, they wouldn't have used a bare raise, and they wouldn't have tried to pass inspect.currentframe an argument it doesn't take). Incidentally, the incorrect usage of inspect.currentframe causes the code to fail with a TypeError instead of doing what you describe, so for the rest of this answer, I'll assume that call is replaced with sys._getframe(1), which produces the described behavior.
One of the things the hack relies on is setting a local trace function with frame.f_trace = self.trace. This local trace function will raise an exception on the first line inside the with block... or at least, that's what it normally does.
Python calls trace functions when certain trace events happen. One of those trace events is the start of a new source line. Python determines that a new source line has started by checking whether the current bytecode instruction index corresponds to either the first instruction of a line, or an instruction at an index prior to the last instruction executed. You can see that in maybe_call_line_trace in Python/ceval.c.
Python only updates instr_prev, the variable used to determine the last instruction executed, when tracing is active. However, once the local trace function raises an exception, it is automatically deactivated, and instr_prev stops receiving updates.
When the local trace function is set, the next two instructions it could activate on are the STORE_NAME to set c (or STORE_FAST if you put the code in a function), and the LOAD_NAME to load the print function for the next line (or LOAD_GLOBAL if you put the code in a function).
The first time through the loop, it activates on LOAD_NAME, and instr_prev is set to that instruction's index. The local trace function is then disabled, because it raised an exception.
The second time through the loop, instr_prev is still set to the index of the LOAD_NAME, so Python thinks the STORE_NAME marks the beginning of a new line. The local trace function activates on STORE_NAME, and the exception prevents the assignment to c.
You can see the instructions where the local trace function activates by inspecting frame.f_lasti in trace, and comparing the results to the instruction indices in the output of dis.dis. For example, the following variant of your code:
import sys
import inspect
import dis
class SkippableContext(object):
def __enter__(self):
print(' ... Skipping Context')
sys.settrace(lambda *args, **keys: None)
frame = sys._getframe(1)
frame.f_trace = self.trace
return 'SET BY TRICKY CONTEXT MANAGER!!'
def trace(self, frame, event, arg):
print(frame.f_lasti)
raise Exception
def __exit__(self, type, value, traceback):
return True
def f():
for i in range(2):
c='not set'
with SkippableContext() as c:
print('This code is not printed')
c = 'set in context'
print('c: {}'.format(c))
f()
dis.dis(f)
produces the following output:
... Skipping Context
26
c: SET BY TRICKY CONTEXT MANAGER!!
... Skipping Context
24
c: not set
21 0 SETUP_LOOP 64 (to 66)
2 LOAD_GLOBAL 0 (range)
4 LOAD_CONST 1 (2)
6 CALL_FUNCTION 1
8 GET_ITER
>> 10 FOR_ITER 52 (to 64)
12 STORE_FAST 0 (i)
22 14 LOAD_CONST 2 ('not set')
16 STORE_FAST 1 (c)
23 18 LOAD_GLOBAL 1 (SkippableContext)
20 CALL_FUNCTION 0
22 SETUP_WITH 18 (to 42)
24 STORE_FAST 1 (c)
24 26 LOAD_GLOBAL 2 (print)
28 LOAD_CONST 3 ('This code is not printed')
30 CALL_FUNCTION 1
32 POP_TOP
25 34 LOAD_CONST 4 ('set in context')
36 STORE_FAST 1 (c)
38 POP_BLOCK
40 LOAD_CONST 0 (None)
>> 42 WITH_CLEANUP_START
44 WITH_CLEANUP_FINISH
46 END_FINALLY
26 48 LOAD_GLOBAL 2 (print)
50 LOAD_CONST 5 ('c: {}')
52 LOAD_METHOD 3 (format)
54 LOAD_FAST 1 (c)
56 CALL_METHOD 1
58 CALL_FUNCTION 1
60 POP_TOP
62 JUMP_ABSOLUTE 10
>> 64 POP_BLOCK
>> 66 LOAD_CONST 0 (None)
68 RETURN_VALUE
The 26 printed the first time corresponds to the index of the LOAD_GLOBAL, and the 24 printed the second time corresponds to the index of the STORE_FAST.

python how to get BytesIO allocated memory length?

This is the code i am using to test the memory allocation
import pycurl
import io
url = "http://www.stackoverflow.com"
buf = io.BytesIO()
print(len(buf.getvalue())) #here i am getting 0 as length
c = pycurl.Curl()
c.setopt(c.URL, url)
c.setopt(c.CONNECTTIMEOUT, 10)
c.setopt(c.TIMEOUT, 10)
c.setopt(c.ENCODING, 'gzip')
c.setopt(c.FOLLOWLOCATION, True)
c.setopt(c.IPRESOLVE, c.IPRESOLVE_V4)
c.setopt(c.USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0')
c.setopt(c.WRITEFUNCTION, buf.write)
c.perform()
c.close()
print(len(buf.getvalue())) #here length of the dowloaded file
print(buf.getvalue())
buf.close()
How to get the allocated buffer/memory length by BytesIO ?
what am i doing wrong here ? python doesn't allocate fixed buffer length ?

I am not sure what you mean by allocated buffer/memory length, but if you want the length of the user data stored in the BytesIO object you can do
>>> bio = io.BytesIO()
>>> bio.getbuffer().nbytes
0
>>> bio.write(b'here is some data')
17
>>> bio.getbuffer().nbytes
17
But this seems equivalent to the len(buf.getvalue()) that you are currently using.
The actual size of the BytesIO object can be found using sys.getsizeof():
>>> bio = io.BytesIO()
>>> sys.getsizeof(bio)
104
Or you could be nasty and call __sizeof__() directly (which is like sys.getsizeof() but without garbage collector overhead applicable to the object):
>>> bio = io.BytesIO()
>>> bio.__sizeof__()
72
Memory for BytesIO is allocated as required, and some buffering does take place:
>>> bio = io.BytesIO()
>>> for i in range(20):
... _=bio.write(b'a')
... print(bio.getbuffer().nbytes, sys.getsizeof(bio), bio.__sizeof__())
...
1 106 74
2 106 74
3 108 76
4 108 76
5 110 78
6 110 78
7 112 80
8 112 80
9 120 88
10 120 88
11 120 88
12 120 88
13 120 88
14 120 88
15 120 88
16 120 88
17 129 97
18 129 97
19 129 97
20 129 97

io.BytesIO() returns a standard file object which has function tell(). It reports the current descriptor position and does not copy the whole buffer out to compute total size as len(bio.getvalue()) of bio.getbuffer().nbytes. It is a very fast and simple method to get the exact size of used memory in the buffer object.
However, if you preset your buffer, tell() will point at the beginning of the buffer and return 0, but the buffer size is not zero. In this case, you can move the pointer to the end of the buffer seek(0,2), which will report the total buffer size without copying the whole buffer into another chank of the memory.
I posted and recently updated an example code and a more detailed answer here

You can also use tracemalloc to get indirect information about the size of objects, by wrapping memory events in tracemalloc.get_traced_memory()
Do note that active threads (if any) and side effects of your program will affect the output, but it may also be more representative of the real memory cost if many samples are taken, as shown below.
>>> import tracemalloc
>>> from io import BytesIO
>>> tracemalloc.start()
>>>
>>> memory_traces = []
>>>
>>> with BytesIO() as bytes_fh:
... # returns (current memory usage, peak memory usage)
# ..but only since calling .start()
... memory_traces.append(tracemalloc.get_traced_memory())
... bytes_fh.write(b'a' * (1024**2)) # create 1MB of 'a'
... memory_traces.append(tracemalloc.get_traced_memory())
...
1048576
>>> print("used_memory = {}b".format(memory_traces[1][0] - memory_traces[0][0]))
used_memory = 1048870b
>>> 1048870 - 1024**2 # show small overhead
294

Python, os.walk(), pass information back up?

I'm currently attempting to write a simple python program that loops through a bunch of subdirectories finding java files and printing some information regarding the number of times certain keywords are used. I've managed to get this working for the most part. The problem I'm having is printing overall information regarding the higher directories, for example, my current output is as follows:
testcases/part1/testcase2/root_dir:
0 bytes 0 public 0 private 0 try 0 catch
testcases/part1/testcase2/root_dir/folder1:
12586 bytes 19 public 7 private 8 try 22 catch
testcases/part1/testcase2/root_dir/folder1/folder5:
7609 bytes 9 public 2 private 7 try 11 catch
testcases/part1/testcase2/root_dir/folder4:
0 bytes 0 public 0 private 0 try 0 catch
testcases/part1/testcase2/root_dir/folder4/folder2:
7211 bytes 9 public 2 private 4 try 9 catch
testcases/part1/testcase2/root_dir/folder4/folder3:
0 bytes 0 public 0 private 0 try 0 catch
and I want the output to be:
testcases/part1/testcase2/root_dir :
27406 bytes 37 public 11 private 19 try 42 catch
testcases/part1/testcase2/root_dir/folder1 :
20195 bytes 28 public 9 private 15 try 33 catch
testcases/part1/testcase2/root_dir/folder1/folder5 :
7609 bytes 9 public 2 private 7 try 11 catch
testcases/part1/testcase2/root_dir/folder4 :
7211 bytes 9 public 2 private 4 try 9 catch
testcases/part1/testcase2/root_dir/folder4/folder2 :
7211 bytes 9 public 2 private 4 try 9 catch
testcases/part1/testcase2/root_dir/folder4/folder3 :
0 bytes 0 public 0 private 0 try 0 catch
As you can see the lower subdirectories directly provide the information to the higher subdirectories. This is the problem I'm running into. How to efficiently implement this. I have considered storing each print as a string in a list and then printing everything at the very end, but I don't think that would work for multiple subdirectories such as the example provided. This is my code so far:
def lsJava(path):
print()
for dirname, dirnames, filenames in os.walk(path):
size = 0
public = 0
private = 0
tryCount = 0
catch = 0
#Get stats by current directory.
tempStats = os.stat(dirname)
#Print current directory information
print(dirname + ":")
#Print files of directory.
for filename in filenames:
if(filename.endswith(".java")):
fileTempStats = os.stat(dirname + "/" + filename)
size += fileTempStats[6]
tempFile = open(dirname + "/" + filename)
tempString = tempFile.read()
tempString = removeComments(tempString)
public += tempString.count("public", 0, len(tempString))
private += tempString.count("private", 0, len(tempString))
tryCount += tempString.count("try", 0, len(tempString))
catch += tempString.count("catch", 0, len(tempString))
print(" ", size, " bytes ", public, " public ",
private, " private ", tryCount, " try ", catch,
" catch")
The removeComments function simply removes all comments from the java files using a regular expression pattern. Thank you for any help in advance.
EDIT:
The following code was added at the beginning of the for loop:
current_dirpath = dirname
if( dirname != current_dirpath):
size = 0
public = 0
private = 0
tryCount = 0
catch = 0
The output is now as follows:
testcases/part1/testcase2/root_dir/folder1/folder5:
7609 bytes 9 public 2 private 7 try 11 catch
testcases/part1/testcase2/root_dir/folder1:
20195 bytes 28 public 9 private 15 try 33 catch
testcases/part1/testcase2/root_dir/folder4/folder2:
27406 bytes 37 public 11 private 19 try 42 catch
testcases/part1/testcase2/root_dir/folder4/folder3:
27406 bytes 37 public 11 private 19 try 42 catch
testcases/part1/testcase2/root_dir/folder4:
27406 bytes 37 public 11 private 19 try 42 catch
testcases/part1/testcase2/root_dir:
27406 bytes 37 public 11 private 19 try 42 catch

os.walk() takes an optional topdown argument. If you use os.walk(path, topdown=False) it will instead traverse directories bottom-up.
When you first start the loop save off the first element of the tuple (dirpath) as a variable like current_dirpath. As you continue through the loop you can keep a running total of the file sizes in that directory. Then just add a check like if dirpath != current_dirpath, at which point you know you've gone up a directory level, and can reset the totals.

I don't believe you can do this with a single counter, even bottom-up: If a directory A has subdirectories B and C, when you're done with B you need to zero the counter before you descend into C; but when it's time to do A, you need to add the sizes of B and C (but B's count is long gone).
Instead of maintaining a single counter, build up a dictionary mapping each directory (key) to the associated counts (in a tuple or whatever). As you iterate (bottom-up), whenever you are ready to print output for a directory, you can look up all its subdirectories (from the dirname argument returned by os.walk()) and add their counts together.
Since you don't discard the data, this approach can be extended to maintain separate deep and shallow counts, so that at the end of the scan you can sort your directories by shallow count, report the 10 largest counts, etc.

Calculating computational time and memory for a code in python

Can some body help me as how to find how much time and how much memory does it take for a code in python?

Use this for calculating time:
import time
time_start = time.clock()
#run your code
time_elapsed = (time.clock() - time_start)
As referenced by the Python documentation:
time.clock()
On Unix, return the current processor time as a floating
point number expressed in seconds. The precision, and in fact the very
definition of the meaning of “processor time”, depends on that of the
C function of the same name, but in any case, this is the function to
use for benchmarking Python or timing algorithms.
On Windows, this function returns wall-clock seconds elapsed since the
first call to this function, as a floating point number, based on the
Win32 function QueryPerformanceCounter(). The resolution is typically
better than one microsecond.
Reference: http://docs.python.org/library/time.html
Use this for calculating memory:
import resource
resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
Reference: http://docs.python.org/library/resource.html

Based on #Daniel Li's answer for cut&paste convenience and Python 3.x compatibility:
import time
import resource
time_start = time.perf_counter()
# insert code here ...
time_elapsed = (time.perf_counter() - time_start)
memMb=resource.getrusage(resource.RUSAGE_SELF).ru_maxrss/1024.0/1024.0
print ("%5.1f secs %5.1f MByte" % (time_elapsed,memMb))
Example:
2.3 secs 140.8 MByte

There is a really good library called jackedCodeTimerPy for timing your code. You should then use resource package that Daniel Li suggested.
jackedCodeTimerPy gives really good reports like
label min max mean total run count
------- ----------- ----------- ----------- ----------- -----------
imports 0.00283813 0.00283813 0.00283813 0.00283813 1
loop 5.96046e-06 1.50204e-05 6.71864e-06 0.000335932 50
I like how it gives you statistics on it and the number of times the timer is run.
It's simple to use. If i want to measure the time code takes in a for loop i just do the following:
from jackedCodeTimerPY import JackedTiming
JTimer = JackedTiming()
for i in range(50):
JTimer.start('loop') # 'loop' is the name of the timer
doSomethingHere = 'This is really useful!'
JTimer.stop('loop')
print(JTimer.report()) # prints the timing report
You can can also have multiple timers running at the same time.
JTimer.start('first timer')
JTimer.start('second timer')
do_something = 'amazing'
JTimer.stop('first timer')
do_something = 'else'
JTimer.stop('second timer')
print(JTimer.report()) # prints the timing report
There are more use example in the repo. Hope this helps.
https://github.com/BebeSparkelSparkel/jackedCodeTimerPY

Use a memory profiler like guppy
>>> from guppy import hpy; h=hpy()
>>> h.heap()
Partition of a set of 48477 objects. Total size = 3265516 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 25773 53 1612820 49 1612820 49 str
1 11699 24 483960 15 2096780 64 tuple
2 174 0 241584 7 2338364 72 dict of module
3 3478 7 222592 7 2560956 78 types.CodeType
4 3296 7 184576 6 2745532 84 function
5 401 1 175112 5 2920644 89 dict of class
6 108 0 81888 3 3002532 92 dict (no owner)
7 114 0 79632 2 3082164 94 dict of type
8 117 0 51336 2 3133500 96 type
9 667 1 24012 1 3157512 97 __builtin__.wrapper_descriptor
<76 more rows. Type e.g. '_.more' to view.>
>>> h.iso(1,[],{})
Partition of a set of 3 objects. Total size = 176 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 1 33 136 77 136 77 dict (no owner)
1 1 33 28 16 164 93 list
2 1 33 12 7 176 100 int
>>> x=[]
>>> h.iso(x).sp
0: h.Root.i0_modules['__main__'].__dict__['x']

Python Class Stack Frame Limitations

If I have a class of the following format:
class TestClass:
def __init__(self):
self.value = 0
def getNewObject(self):
return TestClass()
Is there a limitation to the amount of times I can call the function? For example:
obj = TestClass()
obj.getNewObject().getNewObject()
Is there a limitation to how many times I can call getNewObject() on the return value of getNewObject()? If so what factors affect this?

I doubt it. One reason that makes me doubt it is that if we have this function:
def test(obj):
obj.getNewObject().getNewObject().getNewObject()
And we disassemble it:
import dis
dis.dis(test)
We get this:
2 0 LOAD_FAST 0 (obj)
3 LOAD_ATTR 0 (getNewObject)
6 CALL_FUNCTION 0
9 LOAD_ATTR 0 (getNewObject)
12 CALL_FUNCTION 0
15 LOAD_ATTR 0 (getNewObject)
18 CALL_FUNCTION 0
21 POP_TOP
22 LOAD_CONST 0 (None)
25 RETURN_VALUE
That's just repetitions of LOAD_ATTR followed by CALL_FUNCTION. I can't imagine that that would require much memory or other resources to manage. As such, there is probably no limit.

There is a recursion limit in Python (adjustable), but that is unrelated. Each call is made after the previous call has completed, so they're all called from the same level of the stack frame (i.e. from your user code). Now, you might hit a line-length limit or something, especially for an interactive Python shell.