Develop Reference

Calling a multiple python scripts from python with predefined environment

Probably related to globals and locals in python exec(), Python 2 How to debug code injected by the exec block and How to get local variables updated, when using the `exec` call?
I am trying to develop a test framework for our desktop applications which uses click bot like functions.
My goal was to enable non-programmers to write small scripts which could work as a test script. So my idea is to structure the test scripts by files like:
tests-folder
| -> 01-first-test.py
| -> 02-second-test.py
| ... etc
| -> fixture.py
And then just execute these scripts in alphabetical order. However, I also wanted to have fixtures which would define functions, classes, variables and make them available to the different scripts without having the scripts to import that fixture explicitly. If that works, I could also have that approach for 2 or more directory levels.
I could get it working-ish with some hacking around, but I am not entirely convinced. I have a test_sequence.py which looks like this:
from pathlib import Path
from copy import deepcopy
from my_module.test import Failure
def run_test_sequence(test_dir: str):
error_occured = False
fixture = {
'some_pre_defined_variable': 'this is available in all scripts and fixtures',
'directory_name': test_dir,
}
# Check if fixture.py exists and load that first
fixture_file = Path(dir) / 'fixture.py'
if fixture_file.exists():
with open(fixture_file.absolute(), 'r') as code:
exec(code.read(), fixture, fixture)
# Go over all files in test sequence directory and execute them
for test_file in sorted(Path(test_dir).iterdir()):
if test_file.name == 'fixture.py':
continue
# Make a deepcopy, so scripts cannot influence one another
fixture_copy = deepcopy(fixture)
fixture_copy.update({
'some_other_variable': 'this is available in all scripts but not in fixture'
})
try:
with open(test_file.absolute(), 'r') as code:
exec(code.read(), fixture_locals, fixture_locals)
except Failure:
error_occured = True
return error_occured
This iterates over all files in the directory tests-folder and executes them in order (with fixture.py first). It also makes the local variables, functions and classes from fixture.py available to each test-script.
A test script could then just be arbitrary code that will be executed and if it raises my custom Failure exception, this will be noted as a failed test.
The whole sequence is started with a script that does
from my_module.test_sequence import run_test_sequence
if __name__ == '__main__':
exit(run_test_sequence('tests-folder')
This mostly works.
What it cannot do, and what leaves me unsatisfied with this approach:
I cannot debug the scripts itself. Since the code is loaded as string and then interpreted, breakpoints inside the test scripts are not recognized.
Calling fixture functions behaves weird. When I define a function in fixture.py like:
from my_hello_module import print_hello
def printer():
print_hello()
I will receive a message during execution that print_hello is undefined because the variables/modules/etc. in the scope surrounding printer are lost.
Stacktraces are useless. On failure it shows the stacktrace but of course only shows my line which says `exec(...)' and the insides of that function, but none of the code that has been loaded.
I am sure there are other drawbacks, that I have not found yet, but these are the most annoying ones.
I also tried to find a solution through __import__ but I couldn't get it to inject my custom locals or globals into the imported script.
Is there a solution, that I am too inexperienced to find or another builtin Python function that actually does, what I am trying to do? Or is there no way to achieve this and I should rather have each test-script import the fixture and file/directory names from the test-scripts itself. I want those scripts to have as few dependencies and pythony code as possible. Ideally they are just:
from my_module.test import *
click(x, y, LEFT)
write('admin')
press('tab')
write('password')
press('enter')
if text_on_screen('Login successful'):
succeed('Test successful')
else:
fail('Could not login')
Additional note: I think I had the debugger working when I still used execfile, but it is not available in python3 environments.

pyqt5 how to open new window and reopen old

Hello
I do not know what to do, the window (register.py) opens very well from login.py, but the window (login.py) does not not open since register.py.
What to do?
register.py
https://hastebin.com/oyoxoyemak.rb
login.py
https://hastebin.com/tanuhigome.rb
error code
error code if i remove
screen = app.primaryScreen()
size = screen.size()
print('Size: %d x %d' % (size.width(), size.height()))
rect = screen.availableGeometry()
print('Available: %d x %d' % (rect.width(), rect.height()))
self.window.move((rect.width() / 2) - 230, (rect.height() / 2) - 230)
error code 2

Both issues are related to the scope.
app is a local variable in both scripts, which exists only in the scope of the script that is running (the if __name__ == "__main__": line).
So, if you run login.py, app is available in its scope, but if you run register.py app exists only for it, but not in the login.py scope (since it was never created "there").
Since primaryScreen is a static function, you don't need a reference to the application instance (which you could get through QtWidgets.QApplication.instance(), anyway):
Just change that line to:
screen = QtWidgets.QApplication.primaryScreen()
In the second issue, the problem is similar: since you're running login.py, MainWindow_Register is never declared in register.py.
I have not a "simple" solution for that, as your approach is a bit confused.
First of all, it seems like you're trying to implent your program starting from the output of pyuic, and if that's the case you should really avoid it: write your own code and use the pyuic generated files as suggested in the documentation.
Then, whenever you have to face multiple related windows, it's better to avoid calling themselves "recursively", and use a single window (or better, a separate object, even a subclass of QApplication) as a "manager". This will make everything easier programmatically, avoiding redundant code, while decreasing the possibility of bugs.
In your case, you should probably always use the login window as a "starting point", then show the register one whenever necessary.

Apply function decorator on print function across all files without having to import and/or reapply?

Edit: My first attempt at asking this might be a bit unfocused/poorly worded here's a better explanation of what I'm trying to do:
I'm trying to modify the default behavior of the print function for the entire environment python is running in without having to modify each file that's being run.
I'm attempting to decorate the print function (I know there are many ways to do this such as overriding it but that's not really the question I'm asking) so I can have it print out some debugging information and force it to always flush. I did that like so:
def modify_print(func):
# I made this so that output always gets flushed as it won't by default
# within the environment I'm using, I also wanted it to print out some
# debugging information, doesn't really matter much in the context of this
# question
def modified_print(*args,**kwargs):
return func(f"some debug prefix: ",flush=True,*args,**kwargs)
return modified_print
print = modify_print(print)
print("Hello world") # Prints "some debug prefix Hello World"
However what I'm trying to do is modify this behavior throughout my entire application. I know I can manually decorate/override/import the print function in each file however I'm wondering if there is some way I can globally configure my python environment to decorate this function everywhere. The only way I can think to do this would be to edit the python source code and build the modified version.
EDIT:
Here's the behavior I wanted implemented, thank you Match for your help.
It prints out the line number and filename everywhere you call a print function within your python environment. This means you don't have to import or override anything manually in all of your files.
https://gist.github.com/MichaelScript/444cbe5b74dce2c01a151d60b714ac3a
import site
import os
import pathlib
# Big thanks to Match on StackOverflow for helping me with this
# see https://stackoverflow.com/a/48713998/5614280
# This is some cool hackery to overwrite the default functionality of
# the builtin print function within your entire python environment
# to display the file name and the line number as well as always flush
# the output. It works by creating a custom user script and placing it
# within the user's sitepackages file and then overwriting the builtin.
# You can disable this behavior by running python with the '-s' flag.
# We could probably swap this out by reading the text from a python file
# which would make it easier to maintain larger modifications to builtins
# or a set of files to make this more portable or to modify the behavior
# of more builtins for debugging purposes.
customize_script = """
from inspect import getframeinfo,stack
def debug_printer(func):
# I made this so that output always gets flushed as it won't by default
# within the environment I'm running it in. Also it will print the
# file name and line number of where the print occurs
def debug_print(*args,**kwargs):
frame = getframeinfo(stack()[1][0])
return func(f"{frame.filename} : {frame.lineno} ", flush=True,*args,**kwargs)
return debug_print
__builtins__['print'] = debug_printer(print)
"""
# Creating the user site dir if it doesn't already exist and writing our
# custom behavior modifications
pathlib.Path(site.USER_SITE).mkdir(parents=True, exist_ok=True)
custom_file = os.path.join(site.USER_SITE,"usercustomize.py")
with open(custom_file,'w+') as f:
f.write(customize_script)

You can use usercustomize script from the site module to achieve something like this.
First, find out where your user site-packages directory is:
python3 -c "import site; print(site.USER_SITE)"
/home/foo/.local/lib/python3.6/site-packages
Next, in that directory, create a script called usercustomize.py - this script will now be run first whenever python is run.
One* way to replace print is to override the __builtins__ dict and replace it with a new method - something like:
from functools import partial
old_print = __builtins__['print']
__builtins__['print'] = partial(old_print, "Debug prefix: ", flush=True)
Drop this into the usercustomize.py script and you should see all python scripts from then on being overridden. You can temporarily disable calling this script by calling python with the -s flag.
*(Not sure if this is the correct way of doing this - there may be a better way - but the main point is that you can use usercustomize to deliver whatever method you choose).

There's no real reason to define a decorator here, because you are only intending to apply it to a single, predetermined function. Just define your modified print function directly, wrapping it around __builtins__.print to avoid recursion.
def print(*args, **kwargs):
__builtins.__print(f"some debug prefix: ", flush=True, *args, **kwargs)
print("Hello world") # Prints "some debug prefix Hello World"
You can use functools.partial to simplify this.
import functools
print = functools.partial(__builtins.__print, f"some debug prefix: ", flush=True)

Maya: Defer a script until after VRay is registered?

I'm trying to delay a part of my pipeline tool (which runs during the startup of Maya) to run after VRay has been registered.
I'm currently delaying the initialization of the tool in a userSetup.py like so:
def run_my_tool():
import my_tool
reload(my_tool)
mc.evalDeferred("run_my_tool()")
I've tried using evalDeferred within the tool to delay the execution of the render_settings script, but it keeps running before VRay has been registered. Any thoughts on how to create a listener for the VRay register event, or what event that is? Thanks!
EDIT:
Made a new topic to figure out how to correctly use theodox's condition/scriptJob commands suggestion here.

Uiron over at tech-artists.com showed me how to do this properly. Here's a link to the thread
Here's the post by uiron:
"don't pass the python code as string unless you have to. Wherever a python callback is accepted (that's not everywhere in Maya's api, but mostly everywhere), try one of these:
# notice that we're passing a function, not function call
mc.scriptJob(runOnce=True, e=["idle", myObject.myMethod], permanent=True)
mc.scriptJob(runOnce=True, e=["idle", myGlobalFunction], permanent=True)
# when in doubt, wrap into temporary function; remember that in Python you can
# declare functions anywhere in the code, even inside other functions
open_file_path = '...'
def idle_handler(*args):
# here's where you solve the 'how to pass the argument into the handler' problem -
# use variable from outer scope
file_manip_open_fn(open_file_path)
mc.scriptJob(runOnce=True, e=["idle", idle_handler], permanent=True)
"

reload the currently running python script

In python ,There is a reload method to reload an imported module , but is there a method to reload the currently running script without restarting it, this would be very helpful in debugging a script and changing the code live while the script is running. In visual basic I remember a similar functionality was called "apply code changes", but I need a similar functionality as a function call like "refresh()" which will apply the code changes instantly.
This would work smoothly when an independent function in the script is modified and we need to immediately apply the code change without restarting the script.
Inshort will:
reload(self)
work?

reload(self) will not work, because reload() works on modules, not live instances of classes. What you want is some logic external to your main application, which checks whether it has to be reloaded. You have to think about what is needed to re-create your application state upon reload.
Some hints in this direction:
Guido van Rossum wrote once this: xreload.py; it does a bit more than reload() You would need a loop, which checks for changes every x seconds and applies this.
Also have a look at livecoding which basically does this. EDIT: I mistook this project for something else (which I didn't find now), sorry.
perhaps this SO question will help you

Perhaps you mean something like this:
import pdb
import importlib
from os.path import basename
def function():
print("hello world")
if __name__ == "__main__":
# import this module, but not as __main__ so everything other than this
# if statement is executed
mainmodname = basename(__file__)[:-3]
module = importlib.import_module(mainmodname)
while True:
# reload the module to check for changes
importlib.reload(module)
# update the globals of __main__ with the any new or changed
# functions or classes in the reloaded module
globals().update(vars(module))
function()
pdb.set_trace()
Run the code, then change the contents of function and enter c at the prompt to run the next iteration of the loop.

test.py
class Test(object):
def getTest(self):
return 'test1'
testReload.py
from test import Test
t = Test()
print t.getTest()
# change return value (test.py)
import importlib
module = importlib.import_module(Test.__module__)
reload(module)
from test import Test
t = Test()
print t.getTest()

Intro
reload is for imported modules. Documentation for reload advises against reloading __main__.
Reloading sys, __main__, builtins and other key modules is not recommended.
To achieve similar behavior on your script you will need to re-execute the script. This will - for normal scripts - also reset the the global state. I propose a solution.
NOTE
The solution I propose is very powerful and should only be used for code you trust. Automatically executing code from unknown sources can lead to a world of pain. Python is not a good environment for soapboxing unsafe code.
Solution
To programmatically execute a python script from another script you only need the built-in functions open, compile and exec.
Here is an example function that will re-execute the script it is in:
def refresh():
with open(__file__) as fo:
source_code = fo.read()
byte_code = compile(source_code, __file__, "exec")
exec(byte_code)
The above will in normal circumstances also re-set any global variables you might have. If you wish to keep these variables you should check weather or not those variables have already been set. This can be done with a try-except block covering NameError exceptions. But that can be tedious so I propose using a flag variable.
Here is an example using the flag variable:
in_main = __name__ == "__main__"
first_run = "flag" not in globals()
if in_main and first_run:
flag = True

None of these answers did the job properly for me, so I put together something very messy and very non-pythonic to do the job myself. Even after running it for several weeks, I am finding small issues and fixing them. One issue will be if your PWD/CWD changes.
Warning this is very ugly code. Perhaps someone will make it pretty, but it does work.
Not only does it create a refresh() function that properly reloads your script in a manner such that any Exceptions will properly display, but it creates refresh_<scriptname> functions for previously loaded scripts just-in-case you need to reload those.
Next I would probably add a require portion, so that scripts can reload other scripts -- but I'm not trying to make node.js here.
First, the "one-liner" that you need to insert in any script you want to refresh.
with open(os.path.dirname(__file__) + os.sep + 'refresh.py', 'r') as f: \
exec(compile(f.read().replace('__BASE__', \
os.path.basename(__file__).replace('.py', '')).replace('__FILE__', \
__file__), __file__, 'exec'))
And second, the actual refresh function which should be saved to refresh.py in the same directory. (See, room for improvement already).
def refresh(filepath = __file__, _globals = None, _locals = None):
print("Reading {}...".format(filepath))
if _globals is None:
_globals = globals()
_globals.update({
"__file__": filepath,
"__name__": "__main__",
})
with open(filepath, 'rb') as file:
exec(compile(file.read(), filepath, 'exec'), _globals, _locals)
def refresh___BASE__():
refresh("__FILE__")
Tested with Python 2.7 and 3.

Take a look at reload. You just need to install the plugin an use reload ./myscript.py, voilà

If you are running in an interactive session you could use ipython autoreload
autoreload reloads modules automatically before entering the execution of code typed at the IPython prompt.
Of course this also works on module level, so you would do something like:
>>>import myscript
>>>myscript.main()
*do some changes in myscript.py*
>>>myscript.main() #is now changed