Develop Reference

Can you delete or replace a python logging message?

I'm looking for a way to replace or delete the last message wrote by python's logging module. The goal is to log a change in variables once it occurs. If the variable changes again, the old log message should be deleted and the new one printed instead.
Hi,
I am using pythons's logging module for a deep learning project I'm currently working on. As some GPUs just don't have enough memory to support the default batch size during training and there is no apparent connection between batch size and actual memory usage that could be used for calculations beforehand, I'm catching the runtime error once it occurs and decrease the batch size by one.
This process can be repeated quite a bit and I'm always logging which batch size did not work and which will be the next one tried. Instead of having 10-30 of these messages (or more) I'd like to simply delete the last one and replace it with the newer one instead.
I've already checked out the python logging documentation, stumbled upon the LogRecord object, but upon trying to deal with it, it seems this object does not actually keep a record of all logs, but rather saves some more info on one specific log instead.
If there is simply no way to do this, I will look into some kind of bundling solution as described here: Python logging: bundle reoccurring messages
The code below shows the log message I'm looking to replace.
Any help is greatly apreciated.
training_not_successful = True
while training_not_successful:
try:
model.run_training(global_settings['epochs'],
train_loader,
test_loader,
global_settings['checkpoint_output_path'],
model_name,
global_settings['best_net_criterion'])
training_not_successful = False
except MemoryError:
logging.warning("Ran out of CUDA memory using batch size " + str(batch_size) +
". Trying again with batch size " + str(batch_size-1))
batch_size -= 1
train_loader, test_loader = get_train_test_loaders(
train_dataset_list,
test_dataset_list,
value_counts,
batch_size
)

I believe (correct me if I'm wrong) that the logging module does not allow to supress newlines, meaning that it's simply not possible to do something like that.
It is possible to do it with a print though:
import shutil
def display(variable, rewritable=False):
columns, lines = shutil.get_terminal_size(fallback=(80, 20))
text = str(variable)
filled = text + ((columns - len(text)) * ' ')
print(filled, end='\r' if rewritable else '\n')
if __name__ == "__main__":
from random import random
from time import sleep
for i in range(10):
display(f"x = {random()}", True)
sleep(1)
display(f"x = 0.0") # test if old value is overwritten completely
display("Done!")
Tested this on linux, but it should work everywhere. (the shutil.get_terminal_size function)
It's not mandatory, but very nice when entire line is overwritten as opposed to only the part that's changed.
The key is character \r - it returns the cursor to the end of the line, that's it. Now you can start writing from the front again, overwriting the line if it has anything else, which is exactly what you want.
Display function is simple, but I'll explain it anyway:
First line gets terminal size, what we need is the width of the line, so we can pad the text with spaces and fill entire line with spaces, to completely overwrite previous line no matter what it had.
Then we convert our variable to a string.
After that, it's just simple math, our string takes n characters, so the rest should be spaces, so we add width - n spaces to final string, and then we print it - entire line is overwritten.
rewritable flag allows to control when the variable should be rewritten next time you call display.
While this is not what you want, as it does not use logging module, since there's no way (that I know of) to make logging module to print \r instead of \n, I think this is a good enough substitute, that could be used if it turns out that you can indeed do this with logging module.

How to link interactive problems (w.r.t. CodeJam)?

I'm not sure if it's allowed to seek for help(if not, I don't mind not getting an answer until the competition period is over).
I was solving the Interactive Problem (Dat Bae) on CodeJam. On my local files, I can run the judge (testing_tool.py) and my program (<name>.py) separately and copy paste the I/O manually. However, I assume I need to find a way to make it automatically.
Edit: To make it clear, I want every output of x file to be input in y file and vice versa.
Some details:
I've used sys.stdout.write / sys.stdin.readline instead of print / input throughout my program
I tried running interactive_runner.py, but I don't seem to figure out how to use it.
I tried running it on their server, my program in first tab, the judge file in second. It's always throwing TLE error.
I don't seem to find any tutorial to do the same either, any help will be appreciated! :/

The usage is documented in comments inside the scripts:
interactive_runner.py
# Run this as:
# python interactive_runner.py <cmd_line_judge> -- <cmd_line_solution>
#
# For example:
# python interactive_runner.py python judge.py 0 -- ./my_binary
#
# This will run the first test set of a python judge called "judge.py" that
# receives the test set number (starting from 0) via command line parameter
# with a solution compiled into a binary called "my_binary".
testing_tool.py
# Usage: `testing_tool.py test_number`, where the argument test_number
# is 0 for Test Set 1 or 1 for Test Set 2.
So use them like this:
python interactive_runner.py python testing_tool.py 0 -- ./dat_bae.py

python autopy problems/confusion

so im trying to make a bot script that when a certain hex color is on a certain pixel it will execute some code to move the mouse,click etc. and i have it to where it takes a screenshot every 1 second to the same png file and updates the png file's pic. i have the hex color for the pixel cords print to the console so i can see if its updating or not. it never updates it just stays the same. ive tried writing this script many ways and sadly i only have one version to show you but hopefully you will understand what i was trying to accomplish. im on python 2.7 btw. thank you all for your time!!!!
import autopy
from time import sleep
color_grabber = hex(autopy.bitmap.Bitmap.open("screen1.png").get_color(292,115))
def color_checker():
global color_grabber
color_grabber = color_grabber
return
def mouse_move_click():
autopy.mouse.smooth_move(433,320)
autopy.mouse.click()
def screen_grab():
autopy.bitmap.capture_screen().save("screen1.png")
def the_ifs(mouse_move_click):
if color_checker == "0xffcb05":
mouse_move_click()
while 1==1:
sleep(1)
screen_grab()
color_checker()
the_ifs(mouse_move_click)
print color_grabber

from autopy.mouse import LEFT_BUTTON
autopy.mouse.click(LEFT_BUTTON)
autopy.mouse.toggle(True, LEFT_BUTTON)
autopy.mouse.toggle(False, LEFT_BUTTON)
I see the need to do this in other people's code, but I don't understand why want to use the up and down after the click.In fact when I test on Windows 7, click is effective, but is not very correct, feel more like the down to my operation

I believe your problem is how you're using color_grabber. Saying color_grabber = color_grabber does nothing. What's happening in your code is that when you run it, after the imports, you define the value of color_grabber to be the color in your image. Then your while loop executes and in that loop you call color_checker. This function brings in the variable color_grabber from the global namespace and then you set that variable equal to itself. You're not re-executing the command you used to define color_grabber in the first place. You're just storing the color value back into itself so clearly its not going to change.
You also have a problem in how you're calling your mouse_move_click function. You don't want to pass in the function name, as that isn't really necessary. However, you also performed the check color_checker == "0xffcb05" which was comparing your function (the function itself, not the returned value) to the hex code. That doesn't do you any good. You want to compare the color. The solution is to pass the color into the_ifs and use that color to compare to the hex code. I should note though that you don't need to make the_ifs into its own function. Just put that if statement in your while loop. I left it how you had it though.
What you want is something like this.
import autopy
from time import sleep
def color_checker():
color_grabber = hex(autopy.bitmap.Bitmap.open("screen1.png").get_color(292,115))
return color_grabber
def mouse_move_click():
autopy.mouse.smooth_move(433,320)
autopy.mouse.click()
def screen_grab():
autopy.bitmap.capture_screen().save("screen1.png")
def the_ifs(color):
if color == "0xffcb05":
mouse_move_click()
while 1==1:
sleep(1)
screen_grab()
color = color_checker()
the_ifs(color)
print color
Note that I have not run this code myself so I can't guarantee it works, but I believe it should.

3dsmax python add float script to sub animations

I have the next setup: I have a sphere which has a morpher modifier. This morpher modifier has a certain amount of channels filled with morph targets aka sub animations. Now I want to add a controller to each of these subanimations, more specifically a controller with a float script. I have the code snippet that should work but when I go to the curve editor, the morph channels/ sub animations did not change controller, nor is the value of their controller changed.
import MaxPlus
target = MaxPlus.INode.GetINodeByName('Sphere001')
#Retrieve the morpher modifier
mod = target.GetModifier(0)
#ID of a float script controller
id = MaxPlus.Class_ID(1233584870,1911625032)
#Create float controller
float_co = MaxPlus.Factory.CreateFloatController(id)
#Retrieve the first morph channel / sub animation
sub = mod.GetSubAnim(1)
#Controller is assigned to the sub animation
sub.AssignController(float_co,1)
#Basic test which assigns 20 to the sub animation
float_co.ParameterBlock.Script.Value = '20'
When I add a wrong value to the script, for example:
float_co.ParameterBlock.Script.Value = '=20'
I receive an error and the usual window when you manually add a controller to an object or node. However the strange thing is that at the top of the window: the name of the object to which it is connected, does not show. See figure for clarification:
Can someone tell me what I'm doing wrong? Thank you!

I solved it by using the ugly way:
import MaxPlus
test = MaxPlus.FPValue()
success = MaxPlus.Core.EvalMAXScript(string_with_command,test)
This is used twice: first to create the float script controller and a second time to add the script to the controller. Be careful if anyone wants to try this, the script for the controller needs to be a string. Do not use
x as string
with the expressing you want to use as script for the float script controller since maxscript will evaluate x on the timeframe you are currently in 3ds max and will convert this value to a string. This value will be used as script for all time frames which clearly is not what you want. A small hack I used was:
script_value_example = '"amax #(0, ($sphere.position.x - cube.position.y))"'
This is still a string for python and maxscript will see the " " and will interpret it as a string. The other way around doesn't work, maxscript does not interpret ' ' as string.
Maybe this will help someone in the future. Also if someone knows the proper way to do it using the code in the question, please leave a reply, I'm interested to know.

Accessing global variables from inside a module

I wrote some python code to control a number of USB (electrical relays and temperature sensors) and RS232 (vacuum gauges) devices. From within this main script (e.g., myscript.py), I would like to import a module (e.g., exp_protocols.py) where I define different experimental protocols, i.e. a series of instructions to open or close relays, read temperature and pressure values, with some simple flow control thrown in (e.g. "wait until temperature exceeds 200 degrees C").
My initial attempt looked like this:
switch_A = Relay('A')
switch_B = Relay('B')
gauge_1 = Gauge('1')
global switch_A
global switch_B
global gauge_1
from exp_protocols import my_protocol
my_protocol()
with exp_protocols.py looking like this:
def my_protocol():
print 'Pressure is %.3f mbar.' % gauge_1.value
switch_A.close()
switch_B.open()
This outputs a global variable error, because exp_protocols.my_protocol cannot access the objects defined in myscript.py.
It seems, from reading the answers to earlier questions here, that I could (should?) create all my Relay and Gauge variable in another module, e.g., myconfig.py, and then import myconfig both in myscript.py and exp_protocols? But if I do that, won't my Relay and Gauge objects be created twice (thus trying to open serial ports already active, etc.)?
What would be the best (most Pythonic) way to achieve this kind of inter-module communication?
Thanks in advance.

No matter how many times you import myconfig, python only imports the module once. After the first import, future import statements just grab another reference to the module.
Globals should only be used if these are static bits of data. Your function would be more generic if it took the variables as parameters:
def my_protocol(switch_A, switch_B, gauge_1):
print 'Pressure is %.3f mbar.' % gauge_1.value
switch_A.close()
switch_B.open()
modules could use it with many combinations of data. Suppose you have blocks of switches in a list (and I'm just making this up because I have no idea how you configure your data...), you could process them all with the same function:
import exp_protocols
switch_blocks = [
[Relay('1-A'), Relay('1-B'), Gauge('1-1')],
[Relay('2-A'), Relay('2-B'), Gauge('2-1')],
]
for switch1, switch2, gauge in switch_blocks:
exp_protocols.my_protocol(switch1, switch2, gauge)