I'm writing a script that reads a bunch of files, and then processes the rows from all of those files in parallel.
My problem is that the script behaves strangely if it can't open some of the files. If it's one of the later files in the list, then it processes the earlier files, and reports the exception when it gets to the bad file. However, if it can't open one of the first files in the list, then it processes nothing, and doesn't report an exception.
How can I make the script report all exceptions, no matter where they are in the list?
The key problem seems to be the chunk size of pool.imap(). If the exception occurs before the first chunk is submitted, it fails silently.
Here's a little script to reproduce the problem:
from multiprocessing.pool import Pool
def prepare():
for i in range(5):
yield i+1
raise RuntimeError('foo')
def process(x):
return x
def test(chunk_size):
pool = Pool(10)
n = raised = None
try:
for n in pool.imap(process, prepare(), chunksize=chunk_size):
pass
except RuntimeError as ex:
raised = ex
print(chunk_size, n, raised)
def main():
print('chunksize n raised')
for chunk_size in range(1, 10):
test(chunk_size)
if __name__ == '__main__':
main()
The prepare() function generates five integers, then raises an exception. That generator gets passed to pool.imap() with chunk size from 1 to 10. Then it prints out the chunk size, number of results received, and any exception raised.
chunksize n raised
1 5 foo
2 4 foo
3 3 foo
4 4 foo
5 5 foo
6 None None
7 None None
8 None None
9 None None
You can see that the exception is properly reported until the chunk size increases enough that the exception happens before the first chunk is submitted. Then it silently fails, and no results are returned.
If I run this (I modified it slightly for py2k and py3k cross compatibility) with Python 2.7.13 and 3.5.4 on my own handy system, I get:
$ python2 --version
Python 2.7.13
$ python2 mptest.py
chunksize n raised
1 5 foo
2 4 foo
3 3 foo
4 4 foo
5 5 foo
6 None None
7 None None
8 None None
9 None None
$ python3 --version
Python 3.5.4
$ python3 mptest.py
chunksize n raised
1 5 foo
2 4 foo
3 3 foo
4 4 foo
5 5 foo
6 None foo
7 None foo
8 None foo
9 None foo
I presume the fact that it fails (and hence prints None) for chunk sizes > 5 is not surprising, since no pool process can get six arguments since the generator produced by calling mptest can only be called 5 times.
What does seem surprising is that Python2.7.9 says None for the exceptions for chunk sizes above 5, while Python 3.5 says foo for the exceptions.
This is Issue #28699, fixed in commit 794623bdb2. The fix has apparently been backported to Python 3.5.4, but not to Python 2.7.9, nor apparently to your own Python 3 version.
Related
I'm trying to make a quick and dirty caching system for Python, using the trick that a context-manager can be made to conditionally skip the code in its context — see Skipping execution of -with- block. I've stumbled upon a weird failure case of this and I was wondering if someone can help understand and fix this.
Before anyone says this, I know what I'm doing is terrible and I shouldn't do it, etc, etc.
Anyway, here is the code for the tricky context manager:
import sys
import inspect
class SkippableContext(object):
def __init__(self,mode=0):
"""
if mode = 0, proceed as normal
if mode = 1, do not execute block
"""
self.mode=mode
def __enter__(self):
if self.mode==1:
print(' ... Skipping Context')
# Do some magic
sys.settrace(lambda *args, **keys: None)
frame = inspect.currentframe(1)
frame.f_trace = self.trace
return 'SET BY TRICKY CONTEXT MANAGER!!'
def trace(self, frame, event, arg):
raise
def __exit__(self, type, value, traceback):
return True
And here is the test code:
print('==== First Pass with skipping disabled ====')
c='not set'
with SkippableContext(mode=0) as c:
print('Should Get into here')
c = 'set in context'
print('c: {}'.format(c))
print('==== Second Pass with skipping enabled ====')
c='not set'
with SkippableContext(mode=1) as c:
print('This code is not printed')
c = 'set in context'
print('c: {}'.format(c))
c='not set'
with SkippableContext(mode=1) as c:
print('This code is not printed')
c = 'set in context'
print('c: {}'.format(c))
print('==== Third Pass: Same as second pass but in a loop ====')
for i in range(2):
c='not set'
with SkippableContext(mode=1) as c: # For some reason, assinging c fails on the second iteration!
print('This code is not printed')
c = 'set in context'
print('c: {}'.format(c))
The output generated by the test code is as expected, except for the very last line, where c is not set:
==== First Pass with skipping disabled ====
Should Get into here
c: set in context
==== Second Pass with skipping enabled ====
... Skipping Context
c: SET BY TRICKY CONTEXT MANAGER!!
... Skipping Context
c: SET BY TRICKY CONTEXT MANAGER!!
==== Third Pass: Same as second pass but in a loop ====
... Skipping Context
c: SET BY TRICKY CONTEXT MANAGER!!
... Skipping Context
c: not set
Why is c not set in the second run of the loop? Is there some hack to fix the bug in this hack?
The awful hack you're using does a lot of things with nasty, subtle consequences. I doubt the author fully understood it (if they did, they wouldn't have used a bare raise, and they wouldn't have tried to pass inspect.currentframe an argument it doesn't take). Incidentally, the incorrect usage of inspect.currentframe causes the code to fail with a TypeError instead of doing what you describe, so for the rest of this answer, I'll assume that call is replaced with sys._getframe(1), which produces the described behavior.
One of the things the hack relies on is setting a local trace function with frame.f_trace = self.trace. This local trace function will raise an exception on the first line inside the with block... or at least, that's what it normally does.
Python calls trace functions when certain trace events happen. One of those trace events is the start of a new source line. Python determines that a new source line has started by checking whether the current bytecode instruction index corresponds to either the first instruction of a line, or an instruction at an index prior to the last instruction executed. You can see that in maybe_call_line_trace in Python/ceval.c.
Python only updates instr_prev, the variable used to determine the last instruction executed, when tracing is active. However, once the local trace function raises an exception, it is automatically deactivated, and instr_prev stops receiving updates.
When the local trace function is set, the next two instructions it could activate on are the STORE_NAME to set c (or STORE_FAST if you put the code in a function), and the LOAD_NAME to load the print function for the next line (or LOAD_GLOBAL if you put the code in a function).
The first time through the loop, it activates on LOAD_NAME, and instr_prev is set to that instruction's index. The local trace function is then disabled, because it raised an exception.
The second time through the loop, instr_prev is still set to the index of the LOAD_NAME, so Python thinks the STORE_NAME marks the beginning of a new line. The local trace function activates on STORE_NAME, and the exception prevents the assignment to c.
You can see the instructions where the local trace function activates by inspecting frame.f_lasti in trace, and comparing the results to the instruction indices in the output of dis.dis. For example, the following variant of your code:
import sys
import inspect
import dis
class SkippableContext(object):
def __enter__(self):
print(' ... Skipping Context')
sys.settrace(lambda *args, **keys: None)
frame = sys._getframe(1)
frame.f_trace = self.trace
return 'SET BY TRICKY CONTEXT MANAGER!!'
def trace(self, frame, event, arg):
print(frame.f_lasti)
raise Exception
def __exit__(self, type, value, traceback):
return True
def f():
for i in range(2):
c='not set'
with SkippableContext() as c:
print('This code is not printed')
c = 'set in context'
print('c: {}'.format(c))
f()
dis.dis(f)
produces the following output:
... Skipping Context
26
c: SET BY TRICKY CONTEXT MANAGER!!
... Skipping Context
24
c: not set
21 0 SETUP_LOOP 64 (to 66)
2 LOAD_GLOBAL 0 (range)
4 LOAD_CONST 1 (2)
6 CALL_FUNCTION 1
8 GET_ITER
>> 10 FOR_ITER 52 (to 64)
12 STORE_FAST 0 (i)
22 14 LOAD_CONST 2 ('not set')
16 STORE_FAST 1 (c)
23 18 LOAD_GLOBAL 1 (SkippableContext)
20 CALL_FUNCTION 0
22 SETUP_WITH 18 (to 42)
24 STORE_FAST 1 (c)
24 26 LOAD_GLOBAL 2 (print)
28 LOAD_CONST 3 ('This code is not printed')
30 CALL_FUNCTION 1
32 POP_TOP
25 34 LOAD_CONST 4 ('set in context')
36 STORE_FAST 1 (c)
38 POP_BLOCK
40 LOAD_CONST 0 (None)
>> 42 WITH_CLEANUP_START
44 WITH_CLEANUP_FINISH
46 END_FINALLY
26 48 LOAD_GLOBAL 2 (print)
50 LOAD_CONST 5 ('c: {}')
52 LOAD_METHOD 3 (format)
54 LOAD_FAST 1 (c)
56 CALL_METHOD 1
58 CALL_FUNCTION 1
60 POP_TOP
62 JUMP_ABSOLUTE 10
>> 64 POP_BLOCK
>> 66 LOAD_CONST 0 (None)
68 RETURN_VALUE
The 26 printed the first time corresponds to the index of the LOAD_GLOBAL, and the 24 printed the second time corresponds to the index of the STORE_FAST.
Can someone explain what happened in the second run ? Why did I get a stream of 9's when the code should have given an error?
>>> for __ in range(10): #first run
... print(__)
...
0
1
2
3
4
5
6
7
8
9
This was the second run
>>> for __ in range(10): #second run
... print(_)
...
9
9
9
9
9
9
9
9
9
9
>>> exit()
After this, when I ran the code for the third time, the same code executed as expected and gave the below error. I realize that this question has no practical use. But, I would really like to know why it happened?
NameError: name '_' is not defined
The _ variable is set in the Python interpreter, always holding the last non-None result of any expression statement that has been run.
From the Reserved Classifiers and Identifiers reference:
The special identifier _ is used in the interactive interpreter to store the result of the last evaluation; it is stored in the builtins module.
and from sys.displayhook():
If value is not None, this function prints repr(value) to sys.stdout, and saves value in builtins._. [...] sys.displayhook is called on the result of evaluating an expression entered in an interactive Python session.
Here, that result was 9, from an expression you must have run before the code you shared.
The NameError indicates you restarted the Python interpreter and did not yet run an expression statement yet that produced a non-None value:
>>> _
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name '_' is not defined
>>> 3 * 3
9
>>> _
9
Suppose we have an instance variable in a class.
class MyClass(object):
def __init__():
self.set1 = set()
Is the following operation atomic?
set_shallow_copy = self.set1.copy()
I've tried to search around, but the only information I've found is that reading instance variables are atomic.
(Edit)
I tried to decompile the bytecode (Python 2.7 on Windows), but it's not too useful. It just shows Python bytecode that calls the copy function on set.
9 LOAD_FAST 0 (s)
12 LOAD_ATTR 1 (copy)
15 CALL_FUNCTION 0
18 STORE_FAST 1 (s2)
21 LOAD_CONST 0 (None)
24 RETURN_VALUE
(Edit 2): I wonder if using set2 = set(set1) might be better.
9 LOAD_GLOBAL 0 (set)
12 LOAD_FAST 0 (set1)
15 CALL_FUNCTION 1
18 STORE_FAST 1 (set2)
21 LOAD_CONST 0 (None)
24 RETURN_VALUE
In CPython since version 3.5, the copy is atomic as far as is visible from Python (because, as usual, of the global interpreter lock). No other thread can alter set1 while it is being copied, so you will obtain a copy of some state that the set was in during the manipulations made by (other) threads.
In older versions (like the one tagged here!), the routine that added the elements of the original set to the new (initially empty) copy did not take advantage of the fact that, coming from a set, all the values were unique; as such, it uses == to rediscover that fact. If that comparison is implemented in Python (or by certain C extensions that release the GIL), then the process can be interrupted by other threads (and fail arbitrarily).
Please use automic_set in shared atomic enterprise
Python atomic for shared data types.
https://sharedatomic.top
The module can be used for atomic operations under multiple processs and multiple threads conditions. High performance python! High concurrency, High performance!
atomic api Example with multiprocessing and multiple threads:
You need the following steps to utilize the module:
create function used by child processes, refer to UIntAPIs, IntAPIs, BytearrayAPIs, StringAPIs, SetAPIs, ListAPIs, in each process, you can create multiple threads.
def process_run(a):
def subthread_run(a):
a.array_sub_and_fetch(b'\x0F')
threadlist = []
for t in range(5000):
threadlist.append(Thread(target=subthread_run, args=(a,)))
for t in range(5000):
threadlist[t].start()
for t in range(5000):
threadlist[t].join()
create the shared bytearray
a = atomic_bytearray(b'ab', length=7, paddingdirection='r', paddingbytes=b'012', mode='m')
start processes / threads to utilize the shared bytearray
processlist = []
for p in range(2):
processlist.append(Process(target=process_run, args=(a,)))
for p in range(2):
processlist[p].start()
for p in range(2):
processlist[p].join()
assert a.value == int.to_bytes(27411031864108609, length=8, byteorder='big')
I was wondering how to modify byte code, then recompile that code so I can use it in python as a function? I've been trying:
a = """
def fact():
a = 8
a = 0
"""
c = compile(a, '<string>', 'exec')
w = c.co_consts[0].co_code
dis(w)
which decompiles to:
0 LOAD_CONST 1 (1)
3 STORE_FAST 1 (1)
6 LOAD_CONST 2 (2)
9 STORE_FAST 1 (1)
12 LOAD_CONST 0 (0)
15 RETURN_VALUE
supposing I want to get rid of lines 0 and 3, I call:
x = c.co_consts[0].co_code[6:16]
dis(x)
which results in :
0 LOAD_CONST 2 (2)
3 STORE_FAST 1 (1)
6 LOAD_CONST 0 (0)
9 RETURN_VALUE
my problem is what to do with x, if I try exec x I get an 'expected string without nullbytes and I get the same for exec w,
trying to compile x results in: compile() expected string without null bytes.
I'm not sure what the best way to proceed, except maybe I need to create some kind of code-object, but I'm not sure how, but I'm assuming it must be
possible aka byteplay, python assemblers et al
I'm using python 2.7.10, but I'd like it to be future compatible (Eg python 3) if it's possible.
Update: For sundry reasons I have started writing a Cross-Python-version assembler. See https://github.com/rocky/python-xasm. It is still in very early beta. See also bytecode.
As far as I know there is no other currently-maintained Python assembler. PEAK's Bytecode Disassembler was developed for Python 2.6, and later modified to support early Python 2.7.
It is pretty cool from the documentation. But it relies on other PEAK libraries which might be problematic.
I'll go through the whole example to give you a feel for what you'd have to do. It is not pretty, but then you should expect that.
Basically after modifying the bytecode, you need to create a new types.CodeType object. You need a new one because many of the objects in the code type, for good reason, you can't change. For example the interpreter may have some of these object values cached.
After creating code, you can use this in functions that use a code type which can be used in exec or eval.
Or you can write this to a bytecode file. Alas the code format has changed between Python versions 1.3, 1,5, 2.0, 3.0, 3.8, and 3.10. And by the way so has the optimization and bytecodes. In fact, in Python 3.6 they will be word codes not bytecodes.
So here is what you'd have to do for your example:
a = """
def fact():
a = 8
a = 0
return a
"""
c = compile(a, '<string>', 'exec')
fn_code = c.co_consts[0] # Pick up the function code from the main code
from dis import dis
dis(fn_code)
print("=" * 30)
x = fn_code.co_code[6:16] # modify bytecode
import types
opt_fn_code = types.CodeType(fn_code.co_argcount,
# c.co_kwonlyargcount, Add this in Python3
# c.co_posonlyargcount, Add this in Python 3.8+
fn_code.co_nlocals,
fn_code.co_stacksize,
fn_code.co_flags,
x, # fn_code.co_code: this you changed
fn_code.co_consts,
fn_code.co_names,
fn_code.co_varnames,
fn_code.co_filename,
fn_code.co_name,
fn_code.co_firstlineno,
fn_code.co_lnotab, # In general, You should adjust this
fn_code.co_freevars,
fn_code.co_cellvars)
dis(opt_fn_code)
print("=" * 30)
print("Result is", eval(opt_fn_code))
# Now let's change the value of what's returned
co_consts = list(opt_fn_code.co_consts)
co_consts[-1] = 10
opt_fn_code = types.CodeType(fn_code.co_argcount,
# c.co_kwonlyargcount, Add this in Python3
# c.co_posonlyargcount, Add this in Python 3.8+
fn_code.co_nlocals,
fn_code.co_stacksize,
fn_code.co_flags,
x, # fn_code.co_code: this you changed
tuple(co_consts), # this is now changed too
fn_code.co_names,
fn_code.co_varnames,
fn_code.co_filename,
fn_code.co_name,
fn_code.co_firstlineno,
fn_code.co_lnotab, # In general, You should adjust this
fn_code.co_freevars,
fn_code.co_cellvars)
dis(opt_fn_code)
print("=" * 30)
print("Result is now", eval(opt_fn_code))
When I ran this here is what I got:
3 0 LOAD_CONST 1 (8)
3 STORE_FAST 0 (a)
4 6 LOAD_CONST 2 (0)
9 STORE_FAST 0 (a)
5 12 LOAD_FAST 0 (a)
15 RETURN_VALUE
==============================
3 0 LOAD_CONST 2 (0)
3 STORE_FAST 0 (a)
4 6 LOAD_FAST 0 (a)
9 RETURN_VALUE
==============================
('Result is', 0)
3 0 LOAD_CONST 2 (10)
3 STORE_FAST 0 (a)
4 6 LOAD_FAST 0 (a)
9 RETURN_VALUE
==============================
('Result is now', 10)
Notice that the line numbers haven't changed even though I removed in code a couple of lines. That is because I didn't update fn_code.co_lnotab.
If you want to now write a Python bytecode file from this. Here is what you'd do:
co_consts = list(c.co_consts)
co_consts[0] = opt_fn_code
c1 = types.CodeType(c.co_argcount,
# c.co_posonlyargcount, Add this in Python 3.8+
# c.co_kwonlyargcount, Add this in Python3
c.co_nlocals,
c.co_stacksize,
c.co_flags,
c.co_code,
tuple(co_consts),
c.co_names,
c.co_varnames,
c.co_filename,
c.co_name,
c.co_firstlineno,
c.co_lnotab, # In general, You should adjust this
c.co_freevars,
c.co_cellvars)
from struct import pack
with open('/tmp/testing.pyc', 'w') as fp:
fp.write(pack('Hcc', 62211, '\r', '\n')) # Python 2.7 magic number
import time
fp.write(pack('I', int(time.time())))
# In Python 3.7+ you need to PEP 552 bits
# In Python 3 you need to write out the size mod 2**32 here
import marshal
fp.write(marshal.dumps(c1))
To simplify writing the boilerplate bytecode above, I've added a routine to xasm called write_pycfile().
Now to check the results:
$ uncompyle6 /tmp/testing.pyc
# uncompyle6 version 2.9.2
# Python bytecode 2.7 (62211)
# Disassembled from: Python 2.7.12 (default, Jul 26 2016, 22:53:31)
# [GCC 5.4.0 20160609]
# Embedded file name: <string>
# Compiled at: 2016-10-18 05:52:13
def fact():
a = 0
# okay decompiling /tmp/testing.pyc
$ pydisasm /tmp/testing.pyc
# pydisasm version 3.1.0
# Python bytecode 2.7 (62211) disassembled from Python 2.7
# Timestamp in code: 2016-10-18 05:52:13
# Method Name: <module>
# Filename: <string>
# Argument count: 0
# Number of locals: 0
# Stack size: 1
# Flags: 0x00000040 (NOFREE)
# Constants:
# 0: <code object fact at 0x7f815843e4b0, file "<string>", line 2>
# 1: None
# Names:
# 0: fact
2 0 LOAD_CONST 0 (<code object fact at 0x7f815843e4b0, file "<string>", line 2>)
3 MAKE_FUNCTION 0
6 STORE_NAME 0 (fact)
9 LOAD_CONST 1 (None)
12 RETURN_VALUE
# Method Name: fact
# Filename: <string>
# Argument count: 0
# Number of locals: 1
# Stack size: 1
# Flags: 0x00000043 (NOFREE | NEWLOCALS | OPTIMIZED)
# Constants:
# 0: None
# 1: 8
# 2: 10
# Local variables:
# 0: a
3 0 LOAD_CONST 2 (10)
3 STORE_FAST 0 (a)
4 6 LOAD_CONST 0 (None)
9 RETURN_VALUE
$
An alternate approach for optimization is to optimize at the Abstract Syntax Tree level (AST). The compile, eval and exec functions can start from an AST, or you can dump the AST. You could also write this back out as Python source using the Python module astor
Note however that some kinds of optimization like tail-recursion elimination might leave bytecode in a form that it can't be transformed in a truly faithful way to source code. See my pycon2018 Columbia Lightning Talk for a video I made which eliminates tail recursion in bytecode to get an idea of what I'm talking about here.
If you want to be able to debug and single step bytecode instructions. See my bytecode interpreter and its bytecode debugger.
I have a python script Test.py that runs an R script Test.R below:
import subprocess
import pandas
import pyper
#Run Simple R Code and Print Output
proc = subprocess.Popen(['Path/To/Rscript.exe',
'Path/To/Test.R'],
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = proc.communicate()
print stdout
print stderr
The R script is below:
library("methods")
x <- c(1,2,3,4,5,6,7,8,9,10)
y <- c(2,4,6,8,10,12,14,16,18,20)
data <- data.frame(x,y)
How can I pass the R data frame (or any R object for that matter) to Python? I've had great difficulty getting Rpy2 to work on windows, and I've seen this link to use PypeR but it's using a lot of in-line R code in the Python code and I'd really like to keep the code on separate files (or is this practice considered acceptable?) Thanks.
I've experienced issues getting Rpy2 to work on a mac too and use the same workaround calling R directly from python via subprocess; also agree that keeping files separate helps manage complexity.
First export your data as a .csv from R (again script called through subprocess):
write.table(data, file = 'test.csv')
After that, you can import as a python pandas data frame (Pandas):
import pandas as pd
dat = pd.read_csv('test.csv')
dat
row x y
1 1 2
2 2 4
3 3 6
4 4 8
5 5 10
6 6 12
7 7 14
8 8 16
9 9 18
10 10 20