Develop Reference

Why is assigning whole Blender array so much faster than assigning through iteration?

I have two functions that both generate random noise but one of them takes several seconds to accomplish and another runs in miliseconds. On the hardware level, where does the difference come from? When I use the second function, are instructions executed in parallel? or is the whole array in memory moved to the corresponding memory array where the image is stored?
from random import random
def slow_random():
for i in range(len(bpy.data.images['img'].pixels)):# < pixels are just a list of pixesl color values, where 0 <= value <= 1
bpy.data.images['img'].pixels[i] = random()
def quick_random():
rand = [random() for i in range(len(bpy.data.images['img'].pixels))]
bpy.data.images['img'].pixels[0:] = rand
Here's a link to my blend file. I don't know whether it's safe. I can post a similar question on BlenderStack and upload a file there only in 1.5 hours. download, unpack, and launch blender from here (takes minutes). Open the file, in the script window, you can press Alt+P to run it. Uncomment functions at the end of the list. The slow function is going to simply freeze the application.
I think, #cephner is right in saying that this is due to the implementation of the __setitem__ method. Because it must access the pixel, change its value, and redraw it.

collecting data from running program

I am working on a problem where i should collect data from running program numerous times. For example, every time the program finishes commands, it give certain value t, which is different every time run it. My task is to collect t from N runs of the program. t is going to be different every run. here is the program:
import random
th=0
t=0
tr=0
result=[]
for i in range(7):
i=random.randint(0,1)
result.append(i)
print(result)
a=0
b=len(result)-1
while th<50:
j=random.randint(a,b)
i=j-1
k=j+1
if k<b
k=0
if result[i]==result[k]:
if result[j]!=result[i]:
result[j]==result[i]
th=0
t+=1
else:
th+=1
t+=1
else:
th+=1
t+=1
tr= t-th
print(tr)
print (result)
In this program every run gives you new result. In this generated array there will be obviously every time different arrangement of 0 and 1 and therefore, different t. So resulting t, tr, will be ofcourse different.
I don't know wheter i should do it in new window, or there is a certain function that can do this. Also, if this question is to easy, and there is literature for it, please write what is the name of this kind of problem. thanks :)
btw, im working in python 3.6

See how to make one Python script launch another: you can write a Python script to run the other script and gather the output. You can receive the output as return value from a function call, and tally it just as you would from any function.
Note that your "running program" and your master script need to agree on the form of the information returned: know the data types you're receiving. If you're in doubt, start by having your master script print out what it receives, and the type of each returned value.

how to use the display command of pdb python module?

display expression: prints out the value of an expression each time it gets changed. This is useful for monitoring the value of variables that get changed in loops. So, suppose the following is the code:
for i in range(100):
for j in range(100):
a=f(i,j)
I know something is wrong with the execution of a=f(i,j) for certain values of i and j. Then, how to use the display command from pdb module to find out the values of i and j when it does not work? I suppose when you use display command, it will display the value of i and j automatically, right? Do I need to combine the c command and b command from pdb module also? Many thanks for your time and attention.

display sets a "watch", so that each time execution stops (whether by completing a next, a step, or an until, or breaking on a continue ), if the value has changed, it will print a message showing the old value and the new value.
Since you know something is wrong with your f function, your easiest solution is to put a break on that function, and set display of the inputs inside that scope. Since you have shown us nothing about f, I don't know what the input variables will be called at that level, but it's likely that it won't be "i" and "j", so set the display appropriately.
I find display most useful for when I'm stepping through code that involves loops, using n or s or c. It keeps track of variables for me, and saves me from having to print the variables I'm interested in. If you know your problem is in f, you'll have to step through the code there yourself, and check all the variables at all the interesting statements. If you find yourself checking a variable repeatedly, that's where you use display.

Free up memory by deleting numpy arrays

I have written a fatigue analysis program with a GUI. The program takes strain information for unit loads for each element of a finite element model, reads in a load case using np.genfromtxt('loadcasefilename.txt') and then does some fatigue analysis and saves the result for each element in another array.
The load cases are about 32Mb as text files and there are 40 or so which get read and analysed in a loop. The loads for each element are interpolated by taking slices of the load case array.
The GUI and fatigue analysis run in separate threads. When you click 'Start' on the fatigue analysis it starts the loop over the load cases in the fatigue analysis.
This brings me onto my problem. If I have a lot of elements, the analysis will not finish. How early it quits depends on how many elements there are, which makes me think it might be a memory problem. I've tried fixing this by deleting the load case array at the end of each loop (after deleting all the arrays which are slices of it) and running gc.collect() but this has not had any success.
In MatLab, I'd use the 'pack' function to write the workspace to disk, clear it, and then reload it at the end of each loop. I know this isn't good practice but it would get the job done! Can I do the equivalent in Python somehow?
Code below:
for LoadCaseNo in range(len(LoadCases[0]['LoadCaseLoops'])):#range(1):#xxx
#Get load case data
self.statustext.emit('Opening current load case file...')
LoadCaseFilePath=LoadCases[0]['LoadCasePaths'][LoadCaseNo][0]
#TK: load case paths may be different
try:
with open(LoadCaseFilePath):
pass
except Exception as e:
self.statustext.emit(str(e))
LoadCaseLoops=LoadCases[0]['LoadCaseLoops'][LoadCaseNo,0]
LoadCase=np.genfromtxt(LoadCaseFilePath,delimiter=',')
LoadCaseArray=np.array(LoadCases[0]['LoadCaseLoops'])
LoadCaseArray=LoadCaseArray/np.sum(LoadCaseArray,axis=0)
#Loop through sections
for SectionNo in range(len(Sections)):#range(100):#xxx
SectionCount=len(Sections)
#Get section data
Elements=Sections[SectionNo]['elements']
UnitStrains=Sections[SectionNo]['strains'][:,1:]
Nodes=Sections[SectionNo]['nodes']
rootdist=Sections[SectionNo]['rootdist']
#Interpolate load case data at this section
NeighbourFind=rootdist-np.reshape(LoadCase[0,1:],(1,-1))
NeighbourFind[NeighbourFind<0]=1e100
nearest=np.unravel_index(NeighbourFind.argmin(), NeighbourFind.shape)
nearestcol=int(nearest[1])
Distance0=LoadCase[0,nearestcol+1]
Distance1=LoadCase[0,nearestcol+7]
MxLow=LoadCase[1:,nearestcol+1]
MxHigh=LoadCase[1:,nearestcol+7]
MyLow=LoadCase[1:,nearestcol+2]
MyHigh=LoadCase[1:,nearestcol+8]
MzLow=LoadCase[1:,nearestcol+3]
MzHigh=LoadCase[1:,nearestcol+9]
FxLow=LoadCase[1:,nearestcol+4]
FxHigh=LoadCase[1:,nearestcol+10]
FyLow=LoadCase[1:,nearestcol+5]
FyHigh=LoadCase[1:,nearestcol+11]
FzLow=LoadCase[1:,nearestcol+6]
FzHigh=LoadCase[1:,nearestcol+12]
InterpFactor=(rootdist-Distance0)/(Distance1-Distance0)
Mx=MxLow+(MxHigh-MxLow)*InterpFactor[0,0]
My=MyLow+(MyHigh-MyLow)*InterpFactor[0,0]
Mz=MzLow+(MzHigh-MzLow)*InterpFactor[0,0]
Fx=-FxLow+(FxHigh-FxLow)*InterpFactor[0,0]
Fy=-FyLow+(FyHigh-FyLow)*InterpFactor[0,0]
Fz=FzLow+(FzHigh-FzLow)*InterpFactor[0,0]
#Loop through section coordinates
for ElementNo in range(len(Elements)):
MaterialID=int(Elements[ElementNo,1])
if Materials[MaterialID]['curvefit'][0,0]!=3:
StrainHist=UnitStrains[ElementNo,0]*Mx+UnitStrains[ElementNo,1]*My+UnitStrains[ElementNo,2]*Fz
elif Materials[MaterialID]['curvefit'][0,0]==3:
StrainHist=UnitStrains[ElementNo,3]*Fx+UnitStrains[ElementNo,4]*Fy+UnitStrains[ElementNo,5]*Mz
EndIn=len(StrainHist)
Extrema=np.bitwise_or(np.bitwise_and(StrainHist[1:EndIn-1]<=StrainHist[0:EndIn-2] , StrainHist[1:EndIn-1]<=StrainHist[2:EndIn]),np.bitwise_and(StrainHist[1:EndIn-1]>=StrainHist[0:EndIn-2] , StrainHist[1:EndIn-1]>=StrainHist[2:EndIn]))
Extrema=np.concatenate((np.array([True]),Extrema,np.array([True])),axis=0)
Extrema=StrainHist[np.where(Extrema==True)]
del StrainHist
#Do fatigue analysis
self.statustext.emit('Analysing load case '+str(LoadCaseNo+1)+' of '+str(len(LoadCases[0]['LoadCaseLoops']))+' - '+str(((SectionNo+1)*100)/SectionCount)+'% complete')
del MxLow,MxHigh,MyLow,MyHigh,MzLow,MzHigh,FxLow,FxHigh,FyLow,FyHigh,FzLow,FzHigh,Mx,My,Mz,Fx,Fy,Fz,Distance0,Distance1
gc.collect()

There's obviously a retain cycle or other leak somewhere, but without seeing your code, it's impossible to say more than that. But since you seem to be more interested in workarounds than solutions…
In MatLab, I'd use the 'pack' function to write the workspace to disk, clear it, and then reload it at the end of each loop. I know this isn't good practice but it would get the job done! Can I do the equivalent in Python somehow?
No, Python doesn't have any equivalent to pack. (Of course if you know exactly what set of values you want to keep around, you can always np.savetxt or pickle.dump or otherwise stash them, then exec or spawn a new interpreter instance, then np.loadtxt or pickle.load or otherwise restore those values. But then if you know exactly what set of values you want to keep around, you probably aren't going to have this problem in the first place, unless you've actually hit an unknown memory leak in NumPy, which is unlikely.)
But it has something that may be better. Kick off a child process to analyze each element (or each batch of elements, if they're small enough that the process-spawning overhead matters), send the results back in a file or over a queue, then quit.
For example, if you're doing this:
def analyze(thingy):
a = build_giant_array(thingy)
result = process_giant_array(a)
return result
total = 0
for thingy in thingies:
total += analyze(thingy)
You can change it to this:
def wrap_analyze(thingy, q):
q.put(analyze(thingy))
total = 0
for thingy in thingies:
q = multiprocessing.Queue()
p = multiprocessing.Process(target=wrap_analyze, args=(thingy, q))
p.start()
p.join()
total += q.get()
(This assumes that each thingy and result is both smallish and pickleable. If it's a huge NumPy array, look into NumPy's shared memory wrappers, which are designed to make things much easier when you need to share memory directly between processes instead of passing it.)
But you may want to look at what multiprocessing.Pool can do to automate this for you (and to make it easier to extend the code to, e.g., use all your cores in parallel). Notice that it has a maxtasksperchild parameter, which you can use to recycle the pool processes every, say, 10 thingies, so they don't run out of memory.
But back to actually trying to solve things briefly:
I've tried fixing this by deleting the load case array at the end of each loop (after deleting all the arrays which are slices of it) and running gc.collect() but this has not had any success.
None of that should make any difference at all. If you're just reassigning all the local variables to new values each time through the loop, and aren't keeping references to them anywhere else, then they're just going to get freed up anyway, so you'll never have more than 2 at a (brief) time. And gc.collect() only helps if there are reference cycles. So, on the one hand, it's good news that these had no effect—it means there's nothing obviously stupid in your code. On the other hand, it's bad news—it means that whatever's wrong isn't obviously stupid.
Usually people see this because they keep growing some data structure without realizing it. For example, maybe you vstack all the new rows onto the old version of giant_array instead of onto an empty array, then delete the old version… but it doesn't matter, because each time through the loop, giant_array isn't 5*N, it's 5*N, then 10*N, then 15*N, and so on. (That's just an example of something stupid I did not long ago… Again, it's hard to give more specific examples while knowing nothing about your code.)

help!for function

I have post the similar question before,but this time the problem is different,I got stuck with the following code..can anyone help with it?thanks in advance
I have fixed mu code as suggested,thanks
from numpy import *
#vs,fs,rs are all m*n matrixs,got initial values in,i.e vs[0],fs[0],rs[0] are known
#want use this foor loop to update them
vs=zeros((10,3))
vs[0]=([1,2,3])
fs=zeros((10,3))
fs[0]=([2,3,4])
vs=zeros((10,3))
vs[0]=([3,4,5])
for i in range(5):
#start looping..
vs[i+1]=vs[i]+fs[i]
fs[i+1]=(rs[i]-re[i])
rs[i+1]=rs[i]+vs[i]
print vs,fs,rs
then this code gives vs,fs,rs in different i,but not update each rows of rs,fs,vs and return me a single array of rs,fs,vs (fully updated). whats the problem here?..what should I add?thanks

Put your inizialization outside the loop! Right now, you're resetting the arrays to all zeros each time through the loop, over and over, which is absurd. You also appear to have a typo -- you set vs twice and rs never -- so I've tried to guess what you meant.
from numpy import *
#vs,fs,rs are all m*n matrixs,got initial values in,i.e vs[0],fs[0],rs[0] are known
#want use this foor loop to update them
vs=zeros((10,3))
vs[0]=([1,2,3])
fs=zeros((10,3))
fs[0]=([2,3,4])
rs=zeros((10,3))
rs[0]=([3,4,5])
for i in range(5):
#start looping..
vs[i+1]=vs[i]+fs[i]
fs[i+1]=rs[i]-re[i]
rs[i+1]=rs[i]+vs[i]
print vs,fs,rs

You do not start looping where the comment indicates it, but at the for i in range(5): line. Everything in the indented block (the "body of the for-loop") is done repeatedly for each i. So setting vs and fs to zero is done repeatedly, each time deleting what was calculated before. These initializations should be done before the for.
Also vs is initialized twice while rs isn't initialized at all, probably that's a typo and it should look like this:
vs=zeros((10,3))
vs[0]=([1,2,3])
fs=zeros((10,3))
fs[0]=([2,3,4])
rs=zeros((10,3))
rs[0]=([3,4,5])
for i in range(5):
#start looping..
...

I don't know exactly what you want to print. If you want to print every matrix every time it is updated, then you're fine. But if you want to print the matrices after all the updates have been completed, then you should bring that print statement out of the for loop.
This and what Alex and sth have said should fix your code fully