Develop Reference

Why is multiprocessing module not producing the desired result?

import multiprocessing as mp
import os
def cube(num):
print(os.getpid())
print("Cube is {}".format(num*num*num))
def square(num):
print(os.getpid())
print("Square is {}".format(num*num))
if __name__ == "__main__":
p1 = mp.Process(target = cube, args = (3,))
p2 = mp.Process(target = square, args = (4,))
p1.start()
p2.start()
p1.join()
p2.join()
print("Done")
I was using the multiprocessing module, but I am not able to print any output from a function using that.
I even tried flushing the stdout using the sys module.

Q : "Why is multiprocessing module not producing the desired result?"
Why?
Because it crashes.
The MWE/MCVE-representation of the problem has a wrong code. It crashes & it has nothing to do with the sys.stdout.flush() :
>>> cube( 4 )
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in cube
NameError: global name 'os' is not defined
Solution :
>>> import os # be it in the __main__ or in the def()-ed functions...
>>> cube( 4 )
14165
Cube is 64
and your mp.Process()-based replicas of the python-process instances will stop crashing too.
MCVE that works :
(base) Fri May 29 14:29:33 $ conda activate py3
(py3) Fri May 29 14:34:55 $ python StackOverflow_mp.py
This is ____6745::__main__
This is ____6746::PID
This is ____6747::PID
Cube(__3) is _______27.
Square(__4) is _______16.
Done.
Works.
Q.E.D.
import multiprocessing as mp
import os
import sys
import time
def cube( num ):
print( "This is {0:_>8d}::PID".format( os.getpid() ) )
print( "Cube({0:_>3d}) is {1:_>9d}.".format( num, num*num*num ) )
sys.stdout.flush()
def square( num ):
print( "This is {0:_>8d}::PID".format( os.getpid() ) )
print( "Square({0:_>3d}) is {1:_>9d}.".format( num, num*num ) )
sys.stdout.flush()
if __name__ == "__main__":
print( "This is {0:_>8d}::__main__".format( os.getpid() ) )
p1 = mp.Process( target = cube, args = (3, ) )
p2 = mp.Process( target = square, args = (4, ) )
p1.start()
p2.start()
p1.join()
p2.join()
time.sleep( 1 )
print( "Done.\nWorks.\nQ.E.D." )
I copied and pasted your exact code. But I still didn't get the output from the called functions using the multiprocessing libraries– Kartikeya Agarwal 47 mins ago
So,
- I opened a new Terminal process,
- I copied the conda activate py3 command and
- I hit Enter to let it run, so as to make python3 ecosystem go live.
- I re-launched the proof-of-solution again python StackOverflow_mp.py and
- I hit Enter to let it run
- I saw it working the very same way as it worked last time.
- I doubt the problem is on the provided twice (re)-validated proof-of-solution side, is it?
Q.E.D.
(py3) Fri May 29 19:53:58 $ python StackOverflow_mp.py
This is ___27202::__main__
This is ___27203::PID
Cube(__3) is _______27.
This is ___27204::PID
Square(__4) is _______16.
Done

No module named mem_profile

I'm using this program to measure the time and memory used by two functions and compare which is better for processing a large amount of data. My understanding is that to measure the memory usage we need the mem_profile module, but during the pip install mem_profile it gave me the error No module named mem_profile.
import mem_profile
import random
import time
names = ['Kiran','King','John','Corey']
majors = ['Math','Comps','Science']
print 'Memory (Before): {}Mb'.format(mem_profile.memory_usage_resource())
def people_list(num_people):
results = []
for i in num_people:
person = {
'id':i,
'name': random.choice(names),
'major':random.choice(majors)
}
results.append(person)
return results
def people_generator(num_people):
for i in xrange(num_people):
person = {
'id':i,
'name': random.choice(names),
'major':random.choice(majors)
}
yield person
t1 = time.clock()
people = people_list(10000000)
t2 = time.clock()
# t1 = time.clock()
# people = people_generator(10000000)
# t2 = time.clock()
print 'Memory (After): {}Mb'.format(mem_profile.memory_usage_resource())
print 'Took {} Seconds'.format(t2-t1)
What has caused this error? And are there any alternative packages I could use instead?

1)First import module
pip install memory_profiler
2)include it in your code like this
import memory_profiler as mem_profile
3)change code
mem_profile.memory_usage_psutil() to memory_usage()
4)convert you print statements like this
print('Memory (Before): ' + str(mem_profile.memory_usage()) + 'MB' )
print('Memory (After) : ' + str(mem_profile.memory_usage()) + 'MB')
print ('Took ' + str(t2-t1) + ' Seconds')
5)you will have something like this code:
import memory_profiler as mem_profile
import random
import time
names = ['John', 'Corey', 'Adam', 'Steve', 'Rick', 'Thomas']
majors = ['Math', 'Engineering', 'CompSci', 'Arts', 'Business']
# print('Memory (Before): {}Mb '.format(mem_profile.memory_usage_psutil()))
print('Memory (Before): ' + str(mem_profile.memory_usage()) + 'MB' )
def people_list(num_people):
result = []
for i in range(num_people):
person = {
'id': i,
'name': random.choice(names),
'major': random.choice(majors)
}
result.append(person)
return result
def people_generator(num_people):
for i in range(num_people):
person = {
'id': i,
'name': random.choice(names),
'major': random.choice(majors)
}
yield person
# t1 = time.clock()
# people = people_list(1000000)
# t2 = time.clock()
t1 = time.clock()
people = people_generator(1000000)
t2 = time.clock()
# print 'Memory (After) : {}Mb'.format(mem_profile.memory_usage_psutil())
print('Memory (After) : ' + str(mem_profile.memory_usage()) + 'MB')
# print 'Took {} Seconds'.format(t2-t1)
print ('Took ' + str(t2-t1) + ' Seconds')
Now it work fine i m using python 3.6 and its working without any error.

Was going through the same tutorial and encountered the same problem. But upon further research, I discovered the author of the tutorial used a package called memory_profiler, whose main file he changed to mem_profile, which he imported in the code tutorial.
Just go ahead and do pip install memory_profiler. Copy and rename the file to mem_profile.py in your working directory and you should be fine. If you are on Windows, make sure you install the dependent psutil package as well.
Hope this helps somebody.

Use this for calculating time:
import time
time_start = time.time()
#run your code
time_elapsed = (time.time() - time_start)
As referenced by the Python documentation:
time.time() → float
Return the time in seconds since the epoch as a floating point number.
The specific date of the epoch and the handling of leap seconds is
platform dependent. On Windows and most Unix systems, the epoch is
January 1, 1970, 00:00:00 (UTC) and leap seconds are not counted
towards the time in seconds since the epoch. This is commonly referred
to as Unix time. To find out what the epoch is on a given platform,
look at gmtime(0).
Note that even though the time is always returned as a floating point
number, not all systems provide time with a better precision than 1
second. While this function normally returns non-decreasing values, it
can return a lower value than a previous call if the system clock has
been set back between the two calls.
The number returned by time() may be converted into a more common time
format (i.e. year, month, day, hour, etc…) in UTC by passing it to
gmtime() function or in local time by passing it to the localtime()
function. In both cases a struct_time object is returned, from which
the components of the
calendar date may be accessed as attributes.
Reference: https://docs.python.org/3/library/time.html#time.time
Use this for calculating memory:
import resource
resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
Reference: http://docs.python.org/library/resource.html
Use this if you using python 3.x:
Reference: https://docs.python.org/3/library/timeit.html

Adding to Adebayo Ibro's answer above. Do the following :
In terminal, run $ pip install memory_profiler
In your script, replace import mem_profile with import memory_profiler as mem_profile
In your script, replace all mem_profile.memory_usage_resource() with mem_profile.memory_usage().
Hope this helps!

That module is hand written (not in python packages).
I got this from Corey Schafer's comment in his youtube video.
Just save this code as the module's name:
from pympler import summary, muppy
import psutil
import resource
import os
import sys
def memory_usage_psutil():
# return the memory usage in MB
process = psutil.Process(os.getpid())
mem = process.get_memory_info()[0] / float(2 ** 20)
return mem
def memory_usage_resource():
rusage_denom = 1024.
if sys.platform == 'darwin':
# ... it seems that in OSX the output is different units ...
rusage_denom = rusage_denom * rusage_denom
mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / rusage_denom
return mem

I just encountered the same problem. I solved it installing memory_profiler ($ pip install -U memory_profiler), and them modify the program as follows:
import memory_profiler
...
print('Memory (Before): {}Mb'.format(memory_profiler.memory_usage()))

A couple of python 3.8 updates since time.clock() was removed, and print() has evolved.
Thanks everyone for this discussion, and definitely thanks to Corey Schafer great video.
import memory_profiler as mem_profile
import random
import time
names = ['John', 'Corey', 'Adam', 'Steve', 'Rick', 'Thomas']
majors = ['Math', 'Engineering', 'CompSci', 'Arts', 'Business']
print(f'Memory (Before): {mem_profile.memory_usage()}Mb')
def people_list(num_people):
result = []
for i in range(num_people):
person = {
'id': i,
'name': random.choice(names),
'major': random.choice(majors)
}
result.append(person)
return result
def people_generator(num_people):
for i in range(num_people):
person = {
'id': i,
'name': random.choice(names),
'major': random.choice(majors)
}
yield person
# t1 = time.process_time()
# people = people_list(1000000)
# t2 = time.process_time()
t1 = time.process_time()
people = people_generator(1000000)
t2 = time.process_time()
print(f'Memory (After): {mem_profile.memory_usage()}Mb')
print(f'Took {t2-t1} Seconds')

I went through the same tutorial
the library name is memory_profiler
in order to install, you can use the following
pip install -U memory_profiler
to importing
import memory_profiler
note: the library does not have memory_usage_resource function
but you can use memory_usage with the same functionality.
instead of using clock() function use time() function
import memory_profiler
import random
import time
names=['John', 'Jane', 'Adam','Steve', 'Rick','George','Paul','Bill','Bob']
majors=['Math','Engineering','ComSic','Arts','Stuck Broker']
print ('Memory (Before): {} MB'.format(memory_profiler.memory_usage()))
#using list
def people_list(num_people):
result = []
for i in range(num_people):
person={
'id':i,
'name':random.choice(names),
'major':random.choice(majors)
}
result.append(person)
return result
#using generators
def people_generator(num_people):
for i in range(num_people):
person={
'id':i,
'name':random.choice(names),
'major':random.choice(majors)
}
yield person
# t1=time.time()
# people_list(1000000)
# t2=time.time()
t1=time.time()
people_generator(1000000)
t2=time.time()
print('Memory (After): {} MB'.format(memory_profiler.memory_usage()))
print ('Took {} seconds'.format(t2-t1))

Much simple with sys
import sys
...
print ('Memory (Before): {0}Mb'.format(sys.getsizeof([])))

during the pip install mem_profile it gave me error No module named mem_profile.
by default, pip will download packages from PyPI. No package exists on PyPI named "mem_profile", so of course you will get an error.
for timing blocks of code, the timeit module is what you want to use:
https://docs.python.org/library/timeit.html

Ipython customize prompt to display cell run time

i am wondering how to configure Ipython so that it adds the run time of the last command in milliseconds/seconds to the right command prompt. This could be done in ZSH/Bash shells as illustrated here https://coderwall.com/p/kmchbw
How should I go about doing this?

This is a code snippet that times each statement and prints it right adjusted before the next prompt, and also makes the value accessible by name 'texc'.
# Assumes from __future__ import print_function
from time import time
import blessings # Not a necessary requirement
class ExecTimer(object):
def __init__(self, ip):
self.shell = ip
self.t_pre = time()
self.texc = 0
self.prev_texc = 0
self.term = blessings.Terminal()
def pre_execute(self):
self.t_pre = time()
def post_execute(self):
self.prev_texc = self.texc
self.texc = round(time() - self.t_pre, 4)
print(self.term.bold_blue(
'{} s'.format(self.texc).rjust(self.term.width - 1)
))
# Only add or update user namespace var if it is safe to do so
if 'texc' not in self.shell.user_ns or \
self.shell.user_ns['texc'] == self.prev_texc:
self.shell.push({'texc': self.texc})
else:
pass
def register(self):
self.shell.events.register('pre_execute', self.pre_execute)
self.shell.events.register('post_execute', self.post_execute)
ExecTimer(get_ipython()).register()
To print it above the in-prompt instead, remove the print, and in ipython_config.py set:
c.PromptManager.in_template = '{texc} s\nIn[\\#]: '
or in the same file (startup.py) use
get_ipython().run_line_magic(
'config',
r"PromptManager.in_template = '{texc} s\nIn[\\#]: '"
)

For those who are interested, please refer to this issue opened in Github.
https://github.com/ipython/ipython/issues/5237

Keyboard shortcuts broken running interactive Python Console from a script

You can start an interactive console from inside a script with following code:
import code
# do something here
vars = globals()
vars.update(locals())
shell = code.InteractiveConsole(vars)
shell.interact()
When I run the script like so:
$ python my_script.py
an interactive console opens:
Python 2.7.2+ (default, Jul 20 2012, 22:12:53)
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>>
The console has all globals and locals loaded which is great since I can test stuff easily.
The problem here is that arrows don't work as they normally do when starting an Python console. They simply display escaped characters to the console:
>>> ^[[A^[[B^[[C^[[D
This means that I can't recall previous commands using the up/down arrow keys and I can't edit the lines with the left/right arrow keys either.
Does anyone know why is that and/or how to avoid that?

Check out readline and rlcompleter:
import code
import readline
import rlcompleter
# do something here
vars = globals()
vars.update(locals())
readline.set_completer(rlcompleter.Completer(vars).complete)
readline.parse_and_bind("tab: complete")
shell = code.InteractiveConsole(vars)
shell.interact()

This is the one I use:
def debug_breakpoint():
"""
Python debug breakpoint.
"""
from code import InteractiveConsole
from inspect import currentframe
try:
import readline # noqa
except ImportError:
pass
caller = currentframe().f_back
env = {}
env.update(caller.f_globals)
env.update(caller.f_locals)
shell = InteractiveConsole(env)
shell.interact(
'* Break: {} ::: Line {}\n'
'* Continue with Ctrl+D...'.format(
caller.f_code.co_filename, caller.f_lineno
)
)
For example, consider the following script:
a = 10
b = 20
c = 'Hello'
debug_breakpoint()
a = 20
b = c
c = a
mylist = [a, b, c]
debug_breakpoint()
def bar():
a = '1_one'
b = '2+2'
debug_breakpoint()
bar()
When executed, this file shows to following behavior:
$ python test_debug.py
* Break: test_debug.py ::: Line 24
* Continue with Ctrl+D...
>>> a
10
>>>
* Break: test_debug.py ::: Line 32
* Continue with Ctrl+D...
>>> b
'Hello'
>>> mylist
[20, 'Hello', 20]
>>> mylist.append(a)
>>>
* Break: test_debug.py ::: Line 38
* Continue with Ctrl+D...
>>> a
'1_one'
>>> mylist
[20, 'Hello', 20, 20]

Ctypes callback functions not working with embedded python 3.1 (AMD64)

I try to communicate with a PLC through a DLL (C API interface distributed by the manufacturer of the PLC). I'm using Python 3.1.4 who is embedded as a scripting environment in an other software (x64 - Windows 7).
The callback function bellow doesn't work in this embedded scripting environment (nothing happen). I the trigger for the callback function is generated after that the script has been started and stopped, then the software with the embedded python crashes completely.
The code works fine in a standalone python (3.1.4 MSC v1500 64bit AMD as well)
I have successfully implemented other function of the DLL that doesn't use callbacks in the embedded Python.
Does anyone have I idea what it could be ?
def callback_func(amsaddr,notifHeader,huser):
print('do something')
pass
CMPFUNC = WINFUNCTYPE(None,POINTER(AmsAddr),POINTER(AdsNotificationHeader),c_ulong)
cmp_func = CMPFUNC(callback_func)
netId = AmsNetId((c_ubyte*6)(5,18,18,27,1,1))
plcAddress = AmsAddr(netId,801)
nIndexGroup = c_ulong(0xF021)
nIndexOffset = c_ulong(0x0)
adsNotif = AdsNotificationAttrib(1,4,1000000,1000000)
handle = c_ulong()
huser = c_ulong(10)
ADS_DLL = WinDLL("C:/Program Files/TwinCAT/Ads Api/TcAdsDll/x64/TcAdsDll.dll")
ADS_DLL.AdsSyncAddDeviceNotificationReq.argtypes=[POINTER(AmsAddr),c_ulong,c_ulong,POINTER(AdsNotificationAttrib),CMPFUNC,c_ulong,POINTER(c_ulong)]
ADS_DLL.AdsSyncAddDeviceNotificationReq.restype=c_long
#Function in the DLL with the callback
errCode = ADS_DLL.AdsSyncAddDeviceNotificationReq(pointer(plcAddress),nIndexGroup,nIndexOffset,pointer(adsNotif),cmp_func,huser,pointer(handle))
print('Device Notification error Code : %s' %errCode)
EDIT ^2
I tried a simple ctypes callback and it failed miserably in the embedded python version...The software just hangs and I have to kill it in the taskmanager.
I tried the following code (from the docs):
from ctypes import *
IntArray5 = c_int * 5
ia = IntArray5(5, 1, 7, 33, 99)
libc = cdll.msvcrt #or libc = cdll.msvcr90 --> same problem
qsort = libc.qsort
qsort.restype = None
CMPFUNC = CFUNCTYPE(c_int, POINTER(c_int), POINTER(c_int))
def py_cmp_func(a, b):
print("py_cmp_func", a[0], b[0])
return 0
cmp_func = CMPFUNC(py_cmp_func)
qsort(ia, len(ia), sizeof(c_int), cmp_func)
EDIT ^3
Managed to get some improvements with the use of threading. Only the print() function doesn't print anything in the callback... An other function as os.system('c:/windows/notepad.exe') doesn't work either for example.
from ctypes import *
import threading, queue
import os
IntArray5 = c_int * 5
ia = IntArray5(5, 1, 7, 33, 99)
libc = cdll.msvcrt
qsort = libc.qsort
qsort.restype = None
q = queue.Queue()
CMPFUNC = CFUNCTYPE(None,POINTER(c_int), POINTER(c_int))
def py_cmp_func(a, b):
print("py_cmp_func", a[0], b[0]) #--> doesn't print anything
print('Callback, in thread %s' % threading.current_thread().name) #--> doesn't print anything
q.put('something')
cmp_func = CMPFUNC(py_cmp_func)
t = threading.Thread(target=qsort, args=(ia, len(ia), sizeof(c_int), cmp_func))
t.start()
print(threading.enumerate()) #--> prints [<Thread(Thread-1, started 2068)>, <_MainThread(MainThread, started 2956)>]
t.join()
print(threading.enumerate()) # --> prints [<_MainThread(MainThread, started 2956)>]
print(q.get()) #--> prints 'something'