Develop Reference

Strange Issue Using Logging Module in Python

I seem to be running into a problem when I am logging data after invoking another module in an application I am working on. I'd like assistance in understanding what may be happening here.
To replicate the issue, I have developed the following script...
#!/usr/bin/python
import sys
import logging
from oletools.olevba import VBA_Parser, VBA_Scanner
from cloghandler import ConcurrentRotatingFileHandler
# set up logger for application
dbg_h = logging.getLogger('dbg_log')
dbglog = '%s' % 'dbg.log'
dbg_rotateHandler = ConcurrentRotatingFileHandler(dbglog, "a")
dbg_h.addHandler(dbg_rotateHandler)
dbg_h.setLevel(logging.ERROR)
# read some document as a buffer
buff = sys.stdin.read()
# generate issue
dbg_h.error('Before call to module....')
vba = VBA_Parser('None', data=buff)
dbg_h.error('After call to module....')
When I run this, I get the following...
cat somedocument.doc | ./replicate.py
ERROR:dbg_log:After call to module....
For some reason, my last dbg_h logger write attempt is getting output to the console as well as getting written to my dbg.log file? This only appears to happen AFTER the call to VBA_Parser.
cat dbg.log
Before call to module....
After call to module....
Anyone have any idea as to why this might be happening? I reviewed the source code of olevba and did not see anything that stuck out to me specifically.
Could this be a problem I should raise with the module author? Or am I doing something wrong with how I am using the cloghandler?

The oletools codebase is littered with calls to the root logger though calls to logging.debug(...), logging.error(...), and so on. Since the author didn't bother to configure the root logger, the default behavior is to dump to sys.stderr. Since sys.stderr defaults to the console when running from the command line, you get what you're seeing.
You should contact the author of oletools since they're not using the logging system effectively. Ideally they would use a named logger and push the messages to that logger. As a work-around to suppress the messages you could configure the root logger to use your handler.
# Set a handler
logger.root.addHandler(dbg_rotateHandler)
Be aware that this may lead to duplicated log messages.

reload the currently running python script

In python ,There is a reload method to reload an imported module , but is there a method to reload the currently running script without restarting it, this would be very helpful in debugging a script and changing the code live while the script is running. In visual basic I remember a similar functionality was called "apply code changes", but I need a similar functionality as a function call like "refresh()" which will apply the code changes instantly.
This would work smoothly when an independent function in the script is modified and we need to immediately apply the code change without restarting the script.
Inshort will:
reload(self)
work?

reload(self) will not work, because reload() works on modules, not live instances of classes. What you want is some logic external to your main application, which checks whether it has to be reloaded. You have to think about what is needed to re-create your application state upon reload.
Some hints in this direction:
Guido van Rossum wrote once this: xreload.py; it does a bit more than reload() You would need a loop, which checks for changes every x seconds and applies this.
Also have a look at livecoding which basically does this. EDIT: I mistook this project for something else (which I didn't find now), sorry.
perhaps this SO question will help you

Perhaps you mean something like this:
import pdb
import importlib
from os.path import basename
def function():
print("hello world")
if __name__ == "__main__":
# import this module, but not as __main__ so everything other than this
# if statement is executed
mainmodname = basename(__file__)[:-3]
module = importlib.import_module(mainmodname)
while True:
# reload the module to check for changes
importlib.reload(module)
# update the globals of __main__ with the any new or changed
# functions or classes in the reloaded module
globals().update(vars(module))
function()
pdb.set_trace()
Run the code, then change the contents of function and enter c at the prompt to run the next iteration of the loop.

test.py
class Test(object):
def getTest(self):
return 'test1'
testReload.py
from test import Test
t = Test()
print t.getTest()
# change return value (test.py)
import importlib
module = importlib.import_module(Test.__module__)
reload(module)
from test import Test
t = Test()
print t.getTest()

Intro
reload is for imported modules. Documentation for reload advises against reloading __main__.
Reloading sys, __main__, builtins and other key modules is not recommended.
To achieve similar behavior on your script you will need to re-execute the script. This will - for normal scripts - also reset the the global state. I propose a solution.
NOTE
The solution I propose is very powerful and should only be used for code you trust. Automatically executing code from unknown sources can lead to a world of pain. Python is not a good environment for soapboxing unsafe code.
Solution
To programmatically execute a python script from another script you only need the built-in functions open, compile and exec.
Here is an example function that will re-execute the script it is in:
def refresh():
with open(__file__) as fo:
source_code = fo.read()
byte_code = compile(source_code, __file__, "exec")
exec(byte_code)
The above will in normal circumstances also re-set any global variables you might have. If you wish to keep these variables you should check weather or not those variables have already been set. This can be done with a try-except block covering NameError exceptions. But that can be tedious so I propose using a flag variable.
Here is an example using the flag variable:
in_main = __name__ == "__main__"
first_run = "flag" not in globals()
if in_main and first_run:
flag = True

None of these answers did the job properly for me, so I put together something very messy and very non-pythonic to do the job myself. Even after running it for several weeks, I am finding small issues and fixing them. One issue will be if your PWD/CWD changes.
Warning this is very ugly code. Perhaps someone will make it pretty, but it does work.
Not only does it create a refresh() function that properly reloads your script in a manner such that any Exceptions will properly display, but it creates refresh_<scriptname> functions for previously loaded scripts just-in-case you need to reload those.
Next I would probably add a require portion, so that scripts can reload other scripts -- but I'm not trying to make node.js here.
First, the "one-liner" that you need to insert in any script you want to refresh.
with open(os.path.dirname(__file__) + os.sep + 'refresh.py', 'r') as f: \
exec(compile(f.read().replace('__BASE__', \
os.path.basename(__file__).replace('.py', '')).replace('__FILE__', \
__file__), __file__, 'exec'))
And second, the actual refresh function which should be saved to refresh.py in the same directory. (See, room for improvement already).
def refresh(filepath = __file__, _globals = None, _locals = None):
print("Reading {}...".format(filepath))
if _globals is None:
_globals = globals()
_globals.update({
"__file__": filepath,
"__name__": "__main__",
})
with open(filepath, 'rb') as file:
exec(compile(file.read(), filepath, 'exec'), _globals, _locals)
def refresh___BASE__():
refresh("__FILE__")
Tested with Python 2.7 and 3.

Take a look at reload. You just need to install the plugin an use reload ./myscript.py, voilà

If you are running in an interactive session you could use ipython autoreload
autoreload reloads modules automatically before entering the execution of code typed at the IPython prompt.
Of course this also works on module level, so you would do something like:
>>>import myscript
>>>myscript.main()
*do some changes in myscript.py*
>>>myscript.main() #is now changed

python code runs in standalone function but not as part of class

In Python 3.3.5 I am building a class out of some code that works as standalone functions, but it aborts without an error message when the class version is run. It relies on a 3rd party package pcFastDB, distributed as a pyd.
unit1.py
import sys
import pcFastDB as p3
def get_context():
return p3.pcContext.create()
def get_DB(ctx, dbName):
return p3.pcDB.open(ctx, dbName)
.. more..
can be used as:
ctx = get_context()
db = get_DB(ctx, 'tester')
..do things with db
but
Unit2.py
import pcFastDB as p3
class DBOperation(object):
def __init__(self, DB_name):
'''
Get the context, open the database and fill a channel list
'''
self.ctx = p3.pcContext.create()
print('Context ' + str(self.ctx ))
print(' - opening ' + DB_name)
self.db = p3.pcDB.open(self.ctx, DB_name) #aborts here
print('DB opened successfully...' +str(self.db))
aborts silently when I try to open the DB.
db = DBOperation('tester')
The tester db exists and is in the project folder where I am running it.
There is no exception, no error, it just stops executing. PyDev doesn't like the pyd file very well. I don't get any code insight so debugging is kind of mysterious, but in IPython3 I can see all of the functions and class members, so I know its ok. Unfortunately debugging in IPython3 is an unfathomable mystery to me.
Anyway, I know that I am calling the right functions and I know the functions work, because it all works fine in Unit1, it just doesn't work as a class member. Is this even possible?
I've broken the DB lines out of the init and called them separately, and I've broken ctx out of the class and passed it into the functions but that doesn't work either. It behaves identically when run from within PyDev, IPython3 or the command line.
Any ideas?

Thanks everyone, I've gotten it working by basically rewriting the identical code into a new module. There seems to have been something weird with the file itself. I'm blaming PyDev.
Marc

How can I add a command to the Python interactive shell?

I'm trying to save myself just a few keystrokes for a command I type fairly regularly in Python.
In my python startup script, I define a function called load which is similar to import, but adds some functionality. It takes a single string:
def load(s):
# Do some stuff
return something
In order to call this function I have to type
>>> load('something')
I would rather be able to simply type:
>>> load something
I am running Python with readline support, so I know there exists some programmability there, but I don't know if this sort of thing is possible using it.
I attempted to get around this by using the InteractivConsole and creating an instance of it in my startup file, like so:
import code, re, traceback
class LoadingInteractiveConsole(code.InteractiveConsole):
def raw_input(self, prompt = ""):
s = raw_input(prompt)
match = re.match('^load\s+(.+)', s)
if match:
module = match.group(1)
try:
load(module)
print "Loaded " + module
except ImportError:
traceback.print_exc()
return ''
else:
return s
console = LoadingInteractiveConsole()
console.interact("")
This works with the caveat that I have to hit Ctrl-D twice to exit the python interpreter: once to get out of my custom console, once to get out of the real one.
Is there a way to do this without writing a custom C program and embedding the interpreter into it?
Edit
Out of channel, I had the suggestion of appending this to the end of my startup file:
import sys
sys.exit()
It works well enough, but I'm still interested in alternative solutions.

You could try ipython - which gives a python shell which does allow many things including automatic parentheses which gives you the function call as you requested.

I think you want the cmd module.
See a tutorial here:
http://wiki.python.org/moin/CmdModule

Hate to answer my own question, but there hasn't been an answer that works for all the versions of Python I use. Aside from the solution I posted in my question edit (which is what I'm now using), here's another:
Edit .bashrc to contain the following lines:
alias python3='python3 ~/py/shellreplace.py'
alias python='python ~/py/shellreplace.py'
alias python27='python27 ~/py/shellreplace.py'
Then simply move all of the LoadingInteractiveConsole code into the file ~/py/shellreplace.py Once the script finishes executing, python will cease executing, and the improved interactive session will be seamless.

Debugging a pyQT4 app?

I have a fairly simple app built with pyqt4. I wanted to debug one of the functions connected to one of the buttons in my app. However, when I do the following
python -m pdb app.pyw
> break app.pyw:55 # This is where the signal handling function starts.
things don't quite work like I'd hope. Instead of breaking in the function where I've set the breakpoint and letting me step through it, the debugger enters an infinite loop printing out QCoreApplication::exec: The event loop is already running and I am unable to input anything. Is there a better way to do this?

You need to call QtCore.pyqtRemoveInputHook. I wrap it in my own version of set_trace:
def debug_trace():
'''Set a tracepoint in the Python debugger that works with Qt'''
from PyQt4.QtCore import pyqtRemoveInputHook
# Or for Qt5
#from PyQt5.QtCore import pyqtRemoveInputHook
from pdb import set_trace
pyqtRemoveInputHook()
set_trace()
And when you are done debugging, you can call QtCore.pyqtRestoreInputHook(), probably best when you are still in pdb, and then after you hit enter, and the console spam is happening, keep hitting 'c' (for continue) until the app resumes properly. (I had to hit 'c' several times for some reason, it kept going back into pdb, but after hitting it a few times it resumed normally)
For further info Google "pyqtRemoveInputHook pdb". (Really obvious isn't it? ;P)

I had to use a "next" command at the trace point to get outside of that function first. For that I made a modification of the code from mgrandi:
def pyqt_set_trace():
'''Set a tracepoint in the Python debugger that works with Qt'''
from PyQt4.QtCore import pyqtRemoveInputHook
import pdb
import sys
pyqtRemoveInputHook()
# set up the debugger
debugger = pdb.Pdb()
debugger.reset()
# custom next to get outside of function scope
debugger.do_next(None) # run the next command
users_frame = sys._getframe().f_back # frame where the user invoked `pyqt_set_trace()`
debugger.interaction(users_frame, None)
This worked for me. I found the solution from here : Python (pdb) - Queueing up commands to execute

In my tests, jamk's solution works, while the previous one, although simpler, does not.
In some situations, for reasons that are unclear to me, I've been able to debug Qt without doing any of this.