Develop Reference

Pymel: How do I close For loop after deleting null groups?

As I continue to study For Loops: I've run into some annoying errors. The problem is the script does exactly what I want it to. It deletes the null groups under the demo joints: but unlike other loops I've made for renaming which can be closed with a transform flag in the cmds.ls command: cmds.listRelatives doesn't allow a transform flag to close out the loop. You run the script by simply clicking Build Examples then hitting Delete Waste Groups
I've tried every flag according to the Maya documentation: but nothing seems to be closing the loop. I dont know if I need another variable, or a combination of some flags: or if I am using the wrong type of wording: but ideally what I would like this script to do is simply close out the loop so I dont get the error Error: No object matches name: curve
'''
import DS_wasteGroup_cleanerDemo
reload (DS_wasteGroup_cleanerDemo)
DS_wasteGroup_cleanerDemo.gui()
'''
import re
import maya.cmds as cmds
import maya.mel as mel
if cmds.window("renameWin", exists =True):
cmds.deleteUI("renameWin", window = True)
myWindow = cmds.window("renameWin",t='DS_wasteGroup_cleanerDemo',w=200, h=500, toolbox=True)
column = cmds.columnLayout(adj=True)
def gui():
cmds.button( label="Build Examples", c = buildExamples)
cmds.separator( w=200, h=3)
cmds.button( label="Delete Waste Groups", c = deleteWasteGrp)
cmds.separator( w=200, h=9)
cmds.setParent('..')
cmds.showWindow(myWindow)
def buildExamples(*args):
cmds.group(n='exampleGroup1',world=True,empty=True)
cmds.joint(n='demoJoint1')
cmds.group(n='curve1',world=True,empty=True)
cmds.parent('curve1','demoJoint1')
cmds.joint(n='demoJoint2')
cmds.parent('demoJoint2','exampleGroup1')
cmds.group(n='curve2',world=True,empty=True)
cmds.parent('curve2','demoJoint2')
cmds.joint(n='demoJoint3')
cmds.parent('demoJoint3','exampleGroup1')
cmds.group(n='curve3',world=True,empty=True)
cmds.parent('curve3','demoJoint3')
cmds.joint(n='demoJoint4')
cmds.parent('demoJoint4','exampleGroup1')
cmds.group(n='curve4',world=True,empty=True)
cmds.parent('curve4','demoJoint4')
cmds.joint(n='demoJoint5')
cmds.parent('demoJoint5','exampleGroup1')
cmds.group(n='curve5',world=True,empty=True)
cmds.parent('curve5','demoJoint5')
def deleteWasteGrp(*args):
grpList = cmds.listRelatives('demoJoint*',p=True,f=True)
for name in grpList:
print(grpList)
cmds.delete('curve*')
My apologies if I'm posting simple questions. I do write Python scripts to automate the most tedious tasks in rigging: but my knowledge is only intermediate. I want to learn more python so my scripts arent so clunky and brute forced: as well as the fact that I need them to be more adaptable to various types of characters: so any resources that dumb all this down would also be appreciated. Thank you for your help.

The error is correct, because the very first time the for loop executes, all "curve" obects are deleted, then in the next iteration, the same command does not find any curve objects because they are already deleted. If you place the delete command outside the for loop, the error should disappear.

Honestly I would take a whole different approach as you're hard-coding everything which could easily lead to disaster. When I mean hard-code, I mean you're trying to parent, let's say, "demoJoint2" to an object. This is bad because why are you assuming that "demoJoint2" even exists? If you create an object with a specific name that already exists, Maya will auto-rename the new object, and now you're referencing the wrong one right off the bat! Instead when you create your objects, capture their names in a variable then work with that, otherwise you'll be constantly shooting yourself in the foot.
Here's the same script with an approach I would try instead:
import maya.cmds as cmds
def gui():
if cmds.window("renameWin", exists=True):
cmds.deleteUI("renameWin", window=True)
myWindow = cmds.window("renameWin", t="DS_wasteGroup_cleanerDemo", w=200, h=500, toolbox=True)
column = cmds.columnLayout(adj=True)
cmds.button(label="Build Examples", c=buildExamples)
cmds.separator(w=200, h=3)
cmds.button(label="Delete Waste Groups", c=deleteWasteGrp)
cmds.separator(w=200, h=9)
cmds.setParent("..")
cmds.showWindow(myWindow)
def buildExamples(*args):
root = cmds.group(n="exampleGroup1", world=True, empty=True)
for i in range(5): # Loop to the amount of joints you want to create.
jnt = cmds.createNode("joint", name="demoJoint" + str(i + 1)) # Use `i` to help name the object.
jnt = cmds.parent(jnt, root)[0] # Parenting changes its long name, so recapture the joint in a variable.
crv = cmds.group(n="curve" + str(i + 1), world=True, empty=True) # Create empty group.
cmds.parent(crv, jnt) # Parent curve to joint.
def deleteWasteGrp(*args):
jnts = cmds.ls("demoJoint*", long=True, type="joint") # Get all `demoJoints`.
children = cmds.listRelatives(jnts, f=True, children=True, type="transform") or [] # Get all of their children, and only get transform types.
curves = [obj for obj in children if obj.split("|")[-1].startswith("curve")] # Don't assume we got the right objects. Run a final loop to collect any object that starts with `curve`. Need to use split as we're looping through long names but need to check its short name.
if curves: # `cmds.delete` will error if this list is empty, so don't assume.
cmds.delete(curves) # Delete all curves at once.
gui()
Now I can hit the build button as much as I want with no issues, and delete all the curves when pressing the delete button.
A few more notes:
Notice in buildExamples I'm using a loop to create all the objects instead of reusing redundant code that does the same thing. You could even have a spinbox in your gui that defines how many joints it creates now, where as before it wasn't possible because the count was hard-coded.
cmds.listRelatives does have a way to filter objects by transforms by setting parameter type="transform". In fact you'll see many commands have this same parameter (again start checking docs).
cmds.listRelatives('demoJoint*',p=True,f=True) was grabbing the joint's parent, not its children. The docs clearly state this.
Running cmds.delete('curve*') is going to delete ALL objects with names that start with curve, and since you're running this in a loop it's trying to do this multiple times.
maya.cmds is not pymel. There's a whole separate module called pymel.
If you're unsure with any parts of the code try adding in a print statement to see what it's doing.

I feel like you're going about this whole process a bit wrong, and I would love to elaborate if you're interested, but for now here is a fix for your loop situation:
def deleteWasteGrp(*args):
curveList = cmds.ls('curve*',transforms=True)
try:
cmds.delete(curveList)
print('Deleted the following objects: {}'.format(curveList))
except Exception as e:
cmds.warning('Cleanup failed: {}'.format(e))
The cmds.delete method accepts a list parameter, which in your case is the easiest way to get the job done. Keep in mind that when you delete a parent object, you also delete its children, so depending on your circumstances deleting objects can be order-specific.
Throwing any "likely to fail" calls in a try/except clause is generally a good idea, as it lets you handle the error gracefully. Be careful, however, to not suppress it and just move on -- you at the very least need to alert the user adequately.
Lastly, your buildExamples method will most likely fail if you run it more than once. Because you are addressing objects by string literals (hard coded names) instead of keeping track of their actual names (and full path). You will likely see this error eventually:
# Error: ValueError: file <maya console> line ??: More than one object matches name: demoJoint1 #
Edit: Some elaborations as requested
The commands cmds.group and cmds.joint return a string value indicating the actual name of the object created (in create mode). It's usually a good idea of storing this value in case Maya decides to name your object slightly differently than what you are expecting, usually when there is a naming clash. Eg:
print cmds.group(name='test', world=True, empty=True)
# Returns: test
print cmds.group(name='test', world=True, empty=True)
# Returns: test1
Example of how to capture object names as you create them. I've concatenated your five identical(ish) calls to create joints and curves in this loop:
import maya.cmds as cmds
topGroupName = 'exampleGroup'
actualTopGroupName = None
# Create top level group
actualTopGroupName = cmds.group(n=topGroupName, world=True, empty=True)
# Loop through 5 times and do the following:
for i in range(5):
# PS: hash character in name indicates "next available number"
cmds.select(clear=True)
jnt = cmds.joint(n='demoJoint#')
crv = cmds.group(n='curve#',world=True,empty=True)
cmds.parent(crv, jnt)
cmds.parent(jnt, actualTopGroupName)
Example of how to narrow down which objects to search for with cmds.ls:
topGroupName = 'exampleGroup'
print cmds.ls('|{}*|*|curve*'.format(topGroupName))
# Returns: [u'curve1', u'curve2', u'curve3', u'curve4', u'curve5']
# The string .format() expression is just a dynamic way of writing this:
# |exampleGroup*|*|curve*
Vertical pipes (|) indicate levels in a hierarchy, similar to how slashes (/) work in URLs. And asterisks/wildcards (*) indicate "any character, or none".
Hope this helps you along your way a little bit.

pickle.dump() fails saving a complex object, but no exceptions thrown

I am trying to save a very complex object using pickle. The object's structure is omitted as it is quite complicated, yet it doesn't require as much memory (~5MB).
nX = 20
nY = 50
model = ComplexGridModel(nX, nY) # nX, nY number or elements in X, Y directions
# I don't think it's important but "model" is being use like this:
analyzer = OtherComplexObject(*args, **kwargs)
analyzer.model = model
start = time()
[...] # loop operations on model and analyzer
end = time()
# now the critical part
print("Finished in {:.2f} min".format(end/60 - start/60) ) # print elapsed time
with open('filename.pickle', 'wb') as file:
pickle.dump(model, file, protocol=pickle.HIGHEST_PROTOCOL) # here crushes or sth like that, don't know
The above code creates the file filename.pickle, but it's empty with size 0 KB. When I comment out the pickle part everything works flawlessly, but I still want to save model.
There are 3 mindblown facts that I can't explain:
Everything works as expected, until I change nX > 24. So, everything works fine if e.g. nX=20.
No Errors are thrown, neither in my own code, nor during pickle.dump().
When the bug happens, even the line previous to pickle.dump() i.e. print(...) doesn't execute and so do the next ones. Seems variables are deleted, as well. My Spyder editor displays them in the variable explorer but when I try to access them from the console it throws NameError: name 'variable_name' not defined.
I know it's a drug and I would surely suspect my code being faulty; but with no Errors I can't help it. If my code was at fault wouldn't it at least throw an Error prior to pickle.dump()?.

Can you delete or replace a python logging message?

I'm looking for a way to replace or delete the last message wrote by python's logging module. The goal is to log a change in variables once it occurs. If the variable changes again, the old log message should be deleted and the new one printed instead.
Hi,
I am using pythons's logging module for a deep learning project I'm currently working on. As some GPUs just don't have enough memory to support the default batch size during training and there is no apparent connection between batch size and actual memory usage that could be used for calculations beforehand, I'm catching the runtime error once it occurs and decrease the batch size by one.
This process can be repeated quite a bit and I'm always logging which batch size did not work and which will be the next one tried. Instead of having 10-30 of these messages (or more) I'd like to simply delete the last one and replace it with the newer one instead.
I've already checked out the python logging documentation, stumbled upon the LogRecord object, but upon trying to deal with it, it seems this object does not actually keep a record of all logs, but rather saves some more info on one specific log instead.
If there is simply no way to do this, I will look into some kind of bundling solution as described here: Python logging: bundle reoccurring messages
The code below shows the log message I'm looking to replace.
Any help is greatly apreciated.
training_not_successful = True
while training_not_successful:
try:
model.run_training(global_settings['epochs'],
train_loader,
test_loader,
global_settings['checkpoint_output_path'],
model_name,
global_settings['best_net_criterion'])
training_not_successful = False
except MemoryError:
logging.warning("Ran out of CUDA memory using batch size " + str(batch_size) +
". Trying again with batch size " + str(batch_size-1))
batch_size -= 1
train_loader, test_loader = get_train_test_loaders(
train_dataset_list,
test_dataset_list,
value_counts,
batch_size
)

I believe (correct me if I'm wrong) that the logging module does not allow to supress newlines, meaning that it's simply not possible to do something like that.
It is possible to do it with a print though:
import shutil
def display(variable, rewritable=False):
columns, lines = shutil.get_terminal_size(fallback=(80, 20))
text = str(variable)
filled = text + ((columns - len(text)) * ' ')
print(filled, end='\r' if rewritable else '\n')
if __name__ == "__main__":
from random import random
from time import sleep
for i in range(10):
display(f"x = {random()}", True)
sleep(1)
display(f"x = 0.0") # test if old value is overwritten completely
display("Done!")
Tested this on linux, but it should work everywhere. (the shutil.get_terminal_size function)
It's not mandatory, but very nice when entire line is overwritten as opposed to only the part that's changed.
The key is character \r - it returns the cursor to the end of the line, that's it. Now you can start writing from the front again, overwriting the line if it has anything else, which is exactly what you want.
Display function is simple, but I'll explain it anyway:
First line gets terminal size, what we need is the width of the line, so we can pad the text with spaces and fill entire line with spaces, to completely overwrite previous line no matter what it had.
Then we convert our variable to a string.
After that, it's just simple math, our string takes n characters, so the rest should be spaces, so we add width - n spaces to final string, and then we print it - entire line is overwritten.
rewritable flag allows to control when the variable should be rewritten next time you call display.
While this is not what you want, as it does not use logging module, since there's no way (that I know of) to make logging module to print \r instead of \n, I think this is a good enough substitute, that could be used if it turns out that you can indeed do this with logging module.

Multiprocessing to speed up for loop

Just trying to learn and I"m wondering if multiprocessing would speed
up this for loop ,.. trying to compare
alexa_white_list(1,000,000 lines) and
dnsMISP (can get up to 160,000 lines)
Code checks each line in dnsMISP and looks for it in alexa_white_list.
if it doesn't see it, it adds it to blacklist.
Without mp_handler function the code works fine but it takes
around 40-45 minutes. For brevity, I've omitted all the other imports and
the function that pulls down and unzips the alexa white list.
The below gives me the following error -
File "./vetdns.py", line 128, in mp_handler
p.map(dns_check,dnsMISP,alexa_white_list)
NameError: global name 'dnsMISP' is not defined
from multiprocessing import Pool
def dns_check():
awl = []
blacklist = []
ctr = 0
dnsMISP = open(INPUT_FILE,"r")
dns_misp_lines = dnsMISP.readlines()
dnsMISP.close()
alexa_white_list = open(outname, 'r')
alexa_white_list_lines = alexa_white_list.readlines()
alexa_white_list.close()
print "converting awl to proper format"
for line in alexa_white_list_lines:
awl.append(".".join(line.split(".")[-2:]).strip())
print "done"
for host in dns_misp_lines:
host = host.strip()
host = ".".join(host.split(".")[-2:])
if not host in awl:
blacklist.append(host)
file_out = open(FULL_FILENAME,"w")
file_out.write("\n".join(blacklist))
file_out.close()
def mp_handler():
p = Pool(2)
p.map(dns_check,dnsMISP,alexa_white_list)
if __name__ =='__main__':
mp_handler()
If I label it as global etc I still get the error. I'd appreciate any
suggestions!!

There's no need for multiprocessing here. In fact this code can be greatly simplified:
def get_host_form_line(line):
return line.strip().split(".", 1)[-1]
def dns_check():
with open('alexa.txt') as alexa:
awl = {get_host_from_line(line) for line in alexa}
blacklist = []
with open(INPUT_FILE, "r") as dns_misp_lines:
for line in dns_misp_lines:
host = get_host_from_line(line)
if host not in awl:
blacklist.append(host)
with open(FULL_FILENAME,"w") as file_out:
file_out.write("\n".join(blacklist))
Using a set comprehension to create your Alexa collection has the advantage of being O(1) lookup time. Sets are similar to dictionaries. They are pretty much dictionaries that only have keys with no values. There is some additional overhead in memory and the initial creation time will likely be slower since the values you put in to a set need to be hashed and hash collisions dealt with but the increase in performance you gain from the faster look ups should make up for it.
You can also clean up your line parsing. split() takes an additional parameter that will limit the number of times the input is split. I'm assuming your lines look something like this:
http://www.something.com and you want something.com (if this isn't the case let me know)
It's important to remember that the in operator isn't magic. When you use it to check membership (is an element in the list) what it's essentially doing under the hood is this:
for element in list:
if element == input:
return True
return False
So every time in your code you did if element in list your program had to iterate across each element until it either found what you were looking for or got to the end. This was probably the biggest bottleneck of your code.

You tried to read a variable named dnsMISP to pass as an argument to Pool.map. It doesn't exist in local or global scope (where do you think it's coming from?), so you got a NameError. This has nothing to do with multiprocessing; you could just type a line with nothing but:
dnsMISP
and have the same error.

How to split python script in parts and to import the parts in a loop?

First, sorry for my stupid title :) And here is my problem.. Actually it's not a problem. Everything works, but I want to have better structure...
I have a python script with a loop "looped" each second.
In the loop there are many many IFs. Is it possible to put each IF in a separate file and then to include it in the loop? So this way every time the loop is "looped", all the IFs will be passed, too..
There are too many conditions in my script and all of them are different generally from the otheres so I want to have some kind of folder with modules - mod_wheather.py, mod_sport.py, mod_horoscope.py, etc..
Thanks in advance. I hope I wrote everything understandable..
EDIT:
Here is a structural example of what I have now:
while True:
if condition=='news':
#do something
if condition=='sport':
#so something else
time.sleep(1)
It will be good if I can have something like this:
while True:
import mod_news
import mod_sport
time.sleep(1)
And these IFs from the first example to be separated in files mod_news.py, mod_sport.py...

perhaps you wonder how to work with your own modules in general.
make one file named 'weather.py' and have it contain the appropriate if-statements like:
""" weather.py - conditions to check """
def check_all(*args, **kwargs):
""" check all conditions """
if check_temperature(kwargs['temperature']):
... your code ...
def check_temperature(temp):
-- perhaps some code including temp or whatever ...
return temp > 40
same for sport.py, horoscope.py etc
then your main script would look like:
import time, weather, sport, horoscope
kwargs = {'temperature':30}
condition = 'weather'
while True:
if condition == 'weather':
weather.check_all(**kwargs)
elif condition == 'sport':
sport.check_all()
elif condition == 'horoscope':
horoscope.check_all()
time.sleep(1)
edit: edited according to the edit in your question. Note that I suggest importing all modules only one time, at the beginning of the script, and using its functions. This is better than executing code by importing. But if you insist, you could use reload(weather), which actually performs a reload including code execution. But I cannot stress too much that using functions of external modules is a better way to go!

Put them in functions in separate files and then Import them:
"""thing1.py
A function to demonstrate
"""
def do_things(some_var):
print("Doing things with %s" % (some_var))
``
"""thing2.py
Demonstrates the same thing with a condition
"""
def do_things(some_var):
if len(some_var) < 10:
print("%s is < 10 characters long" % (some_var))
else:
print("too long")
``
"""main_program.py"""
import thing1, thing2
myvar = "cats"
thing1.do_things(myvar)
thing2.do_things(myvar)

I believe you are looking for some kind of PHP-like include() or C prepocessor #include. You would have a file such as the included.py below:
a = 2
print "ok"
and another file which has the following code:
for i in values:
import included
and you want the result to be equivalent to
for i in values:
a = 2
print "ok"
Is it what you are looking for? If so... no, it is not possible. Once Python imports a module, the code of the module is executed and following imports of the same mode only retrieve the already imported instance of the module. The code of a module is not executed everytime it is imported.
I can invent some crazy ways of doing it (let us say, file.read() + eval(), or calling reload() in an imported module.) but it would be a bad idea anyway. I bet we can think of a better solution to your real problem :)

Perhaps all you need is to call functions in your loop; and have those functions in other modules, which you import as needed.
while true:
if condition:
from module_a import f
f()
if condition2
from module_b import g
g()
Though the above is legal Python, and so answers your question, you should in practice write all the imports at the top of your file.

You could import the needed modules if they're needed, for example:
if condition:
import weather
... do something
However I'm not sure if that's what you really want.

I have a python script with a loop "looped" each second. In the loop
there are many many IFs.
Then you must optimize the repeatedly executed tests. Suppose there are 50 IFs blocks in your code and that in a turn of the for-loop, the N th condition is True: that means that the N-1 other conditions must be tested before the N th is tested and triggers the execution of the corresponding code.
It would be preferable to do so:
# to_include.py
def func_weather(*args,**kwargs):
# code
return "I'm the weather"
def func_horoscope(*args,**kwargs):
# code
return "Give me your birth'date"
def func_gastronomy(*args,**kwargs):
# code
return 'Miam crunch'
def func_sports(*args,**kwargs):
# code
return 'golf, swimming and canoeing in the resort station'
didi = {'weather':func_weather, 'horoscope':func_horoscope,
'gastronomy':func_gastronomy, 'sports':func_sports}
and the main module:
# use_to_include.py
import to_include
x = 'sports'
y = to_include.didi[x]()
# instead of
# if x =='weather' : y = func_weather()
# elif x=='horoscope' : y = func_horoscope()
# elif x=='gastronomy': y = func_gastronomy()
# elif x=='sports' : y = func_sports()
print y
result
golf, swimming and canoeing in the resort station