Develop Reference

Pycharm output adding up when using threading

I was learning about Threads so I made a simple program in sublime text.
import time
from threading import Thread
def func():
for a in range(1, 101):
print(a)
Threads = []
for i in range(25):
t = Thread(target=func)
Threads.append(t)
for i in Threads:
i.start()
for i in Threads:
i.join()
But after a few minutes, I started to get annoyed by the bad quality of autocompletion.
So I switched to Pycharm Edu and something weird happened with output. In cmd it was like this
60
60
97
68
58
59
70
71
74
95
89
68
53
92
91
92
93
99
100
89
96
and in Pycharm the output was
6545
46
47
54
76
775981
66
6555
55
608264
67
48
I don't understand what's going on.

print is in truth two distinct writes to stdout: the text, and then a newline. So print(a) is this:
sys.stdout.write(str(a))
sys.stdout.write('\n')
If now multiple Threads write at the same time, it is similar to this:
sys.stdout.write(str(a))
sys.stdout.write('\n')
sys.stdout.write(str(a))
sys.stdout.write('\n')
Or, sometimes:
sys.stdout.write(str(a))
sys.stdout.write(str(a))
sys.stdout.write('\n')
sys.stdout.write('\n')
So you get two numbers on one line and then two newlines.
The easiest fix is to join the strings & newline before using print:
def func():
for a in range(1, 101):
print(f'{a}\n', end='')
This produces the correct result.
(as to why this didn't happen in CMD: probably just luck)

Regex: How can I check if any process is taking 80% or more CPU in Python

I have CPU process output in a string variable. Now I want to check if any process is occupying 80% or more. Please help me with the regex in Python. command output is like this:CPU utilization for five seconds: 81%/0%; one minute: 30%; five minutes: 26%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
3 6765 291 23247 60.24% 8.07% 1.72% 3 SSH Process
126 22753 14985 1518 4.97% 0.88% 0.43% 0 IP Input
219 424447 390715 1086 1.84% 1.73% 1.66% 0 MMA DP TIMER
70 263766 25673 10274 0.96% 1.02% 1.00% 0 Per-Second Jobs
74 624912 2986593 209 0.80% 2.17% 2.39% 0 Ethernet Msec Ti
125 198586 753349 263 0.80% 0.82% 0.80% 0 IP ARP Retry Age
Output should be something like- "a Process is using 80% or more CPU"

You could use psutil, so you dont have to regexp yourself:
>>> for proc in psutil.process_iter():
... procInfo = proc.as_dict(attrs=['pid', 'name', 'username', 'cpu_percent'])
... if procInfo['cpu_percent'] >= 80:
... print('...')

How to unpickle a python object in Golang

I have a python program wherein I am using Pickle to store the object using the following:
pickle.dump(sample, open( "Pickled_files/sample.p", "wb" ))
I can extract and unpickle this object in Python using the following:
sample_extracted= pickle.load(open( "Pickled_files/sample.p", "rb" ))
However, I need to extract this object in a Golang application. Thus I need to know a way by which objects pickled using Python are extracted in Golang.
Is there a way that this can be achieved? And if yes, I would really appreciate if someone can point me to a sample reference or example.

ogórek (2) is Go library for decoding/encoding Python pickles.
A pickle saved to the file can be loaded in Go as follows:
f, err := os.Open("Pickled_files/sample.p")
d := ogórek.NewDecoder(f)
obj, err := d.Decode()

Pickle is Python specific format. AFAIK there are no pickle-parsers outside of Python. You can try to write one for Go but you will most likely only waste lots of time and mental health. On the other hand that would be an interesting project, indeed.
Anyway, instead of pickling use any language independent format, i.e. xml, json, google's protobuf or even a custom one, whatever suits your needs. Always pick tool for a job, never other way around.

Is there a way that this can be achieved?
Depends on your understanding of this. There are a lot of better options than Pickle -- even in pure Python environments. If you understand this as exchanging data between golang and Python, you should consider the following example:
You can serialize everything in Python, like
import msgpack
import msgpack_numpy as m
m.patch()
import numpy as np
data = {'string': 'Hello World',
'number': 42,
'matrix': np.random.randn(2, 3)}
with open('data.msgp', 'wb') as f:
f.write(msgpack.packb(data, use_bin_type=True))
Reading it is pretty simple
// go get -u github.com/vmihailenco/msgpack
package main
import (
"fmt"
"github.com/vmihailenco/msgpack"
"io/ioutil"
)
func main() {
buf, err := ioutil.ReadFile("data.msgp")
if err != nil {
panic(err)
}
var out map[string]interface{}
err = msgpack.Unmarshal(buf, &out)
if err != nil {
panic(err)
}
for k, v := range out {
fmt.Printf("key[%v] value[%v]\n", k, v)
}
}
This gives you
key[matrix] value[map[data:[145 106 174 12 61 187 235 63 128 225 138 214 167 154 231 191 156 205 144 51 50 116 244 191 251 147 235 33 187 149 251 63 207 56 134 174 206 146 220 63 7 23 246 148 34 235 226 63] type:[60 102 56]
kind:[] nd:true shape:[2 3]]]
key[string] value[[72 101 108 108 111 32 87 111 114 108 100]]
key[number] value[42]
All is left is converting the byte sequences into the object you would like to have.

Getting number of characters through it's memory size in python

[ Python ]
I have a string and I know it's size in memory. I want to get the number of characters (a rough estimate) will it contain.
Actual case, I want to send a report through mail, the content of the mail is exceeding the permitted size is allowed for it. I want to split the mail into multiple according to maximum size. But I don't have a way to co-relate the maximum size to number of characters in the string.
import smtplib
smtp = smtplib.SMTP('server.name')
smtp.ehlo()
max_size = smtp.esmtp_features['size']
message_data = <some string data exceeding the max_size>
# Now, how can I get the number of characters in message_data which exceeds the max_szie
Thanks,

The number of chars in a string is the size in memory in bytes to which you must deduct 37 (python 2.7 / mac os)
import sys
def estimate_chars():
"size in bytes"
s = ""
for idx in range(100):
print idx * 10, sys.getsizeof(s), len(s)
s += '1234567890'
estimate_chars()
result: (chars | bytes | len)
0 37 0
10 47 10
20 57 20
30 67 30
40 77 40
50 87 50
...
960 997 960
970 1007 970
980 1017 980
990 1027 990

Circumventing input-to-action latency in mpl's plt.draw() function

In trying to track down the source of an unacceptable (257ms) runtime of matplotlib's plt.draw() function, I stumbled upon this article: http://bastibe.de/2013-05-30-speeding-up-matplotlib.html. In particular, this quote caught my eye:
"I am using pause() here to update the plot without blocking. The correct way to do this is to use draw() instead..."
Digging further, I found that plt.draw() can be substituted by two commands,
plt.pause(0.001)
fig.canvas.blit(ax1.bbox)
Which take up 256ms and 1ms respectively in my code.
This was abnormal, why would a 1ms pause take 256ms to complete? I took some data, and found the following:
plt.pause(n):
n(s) time(ms) overhead(n(ms)-time(ms))
0.0001 270-246 ~246ms
0.001 270-254 ~253ms
0.01 280-265 ~255ms
0.1 398-354 ~254ms
0.2 470-451 ~251ms
0.5 779-759 ~259ms
1.0 1284-1250 ~250ms
numbers courtesy of rkern's line_profiler
This makes it very clear that plt.pause() is doing more than just pausing the program, and I was correct:
Line # Hits Time Per Hit % Time Line Contents
==============================================================
175 def pause(interval):
176 """
177 Pause for *interval* seconds.
178
179 If there is an active figure it will be updated and displayed,
180 and the GUI event loop will run during the pause.
181
182 If there is no active figure, or if a non-interactive backend
183 is in use, this executes time.sleep(interval).
184
185 This can be used for crude animation. For more complex
186 animation, see :mod:`matplotlib.animation`.
187
188 This function is experimental; its behavior may be changed
189 or extended in a future release.
190
191 """
192 1 6 6.0 0.0 backend = rcParams['backend']
193 1 1 1.0 0.0 if backend in _interactive_bk:
194 1 5 5.0 0.0 figManager = _pylab_helpers.Gcf.get_active()
195 1 0 0.0 0.0 if figManager is not None:
196 1 2 2.0 0.0 canvas = figManager.canvas
197 1 257223 257223.0 20.4 canvas.draw()
198 1 145 145.0 0.0 show(block=False)
199 1 1000459 1000459.0 79.5 canvas.start_event_loop(interval)
200 1 2 2.0 0.0 return
201
202 # No on-screen figure is active, so sleep() is all we need.
203 import time
204 time.sleep(interval)
once again courtesy of rkern's line_profiler
This was a breakthrough, as it was suddenly clear why plt.pause() was able to replace plt.draw(), it had a draw function inside it with that same ~250ms overhead I was getting at the start of my program.
At this point, I decided to profile plt.draw() itself:
Line # Hits Time Per Hit % Time Line Contents
==============================================================
551 def draw():
571 1 267174 267174.0 100.0 get_current_fig_manager().canvas.draw()
Alright, one more step down the rabbit hole:
Line # Hits Time Per Hit % Time Line Contents
==============================================================
57 def draw_wrapper(artist, renderer, *args, **kwargs):
58 769 1798 2.3 0.7 before(artist, renderer)
59 769 242060 314.8 98.5 draw(artist, renderer, *args, **kwargs)
60 769 1886 2.5 0.8 after(artist, renderer)
Unfortunately, this was the point at which my ability to run my profiler through the source code ended, leaving me scratching my head at this next level of draw() function, and why it was being called 769 times.
It turns out, the answer was right in front of me the whole time! That same article, which started this whole obsessive hunt in the first place, was actually created to study the same strange behavior! Their solution: To replace plt.draw() with individual calls to each artist which needed to be updated, rather than every single one!
I hope my chasing of this behavior can help others understand it, though currently I'm stuck with a CGContextRef is NULL error whenever I try to replicate his methods, which seems to be specific to the MacOSX backend...
More info as it comes! Please add any more relevant information in answers below, or if you can help me with my CGContextRef is NULL error.