How can I embed an IPython shell in my code and have it automatically display the line number and function in which it was invoked?
I currently have the following setup to embed IPython shells in my code:
from IPython.frontend.terminal.embed import InteractiveShellEmbed
from IPython.config.loader import Config
# Configure the prompt so that I know I am in a nested (embedded) shell
cfg = Config()
prompt_config = cfg.PromptManager
prompt_config.in_template = 'N.In <\\#>: '
prompt_config.in2_template = ' .\\D.: '
prompt_config.out_template = 'N.Out<\\#>: '
# Messages displayed when I drop into and exit the shell.
banner_msg = ("\n**Nested Interpreter:\n"
"Hit Ctrl-D to exit interpreter and continue program.\n"
"Note that if you use %kill_embedded, you can fully deactivate\n"
"This embedded instance so it will never turn on again")
exit_msg = '**Leaving Nested interpreter'
# Put ipshell() anywhere in your code where you want it to open.
ipshell = InteractiveShellEmbed(config=cfg, banner1=banner_msg, exit_msg=exit_msg)
This allows me to start a full IPython shell anywhere in my code by just using ipshell(). For example, the following code:
a = 2
b = a
ipshell()
starts an IPython shell in the scope of the caller that allows me inspect the values of a and b.
What I would like to do is to automatically run the following code whenever I call ipshell():
frameinfo = getframeinfo(currentframe())
print 'Stopped at: ' + frameinfo.filename + ' ' + str(frameinfo.lineno)
This would always show the context where the IPython shell starts so that I know what file/function, etc. I am debugging.
Perhaps I could do this with a decorator, but all my attemps so far have failed, since I need ipshell() to run within the original context (so that I have access to a and b from the IPython shell).
How can I accomplish this?
You can call ipshell() from within another user-defined function, e.g. ipsh()
from inspect import currentframe
def ipsh():
frame = currentframe().f_back
msg = 'Stopped at {0.f_code.co_filename} and line {0.f_lineno}'.format(frame)
ipshell(msg,stack_depth=2) # Go back one level!
Then use ipsh() whenever you want to drop into the IPython shell.
Explanation:
stack_depth=2 asks ipshell to go up one level when retrieving the namespace for the new IPython shell (the default is 1).
currentframe().f_back() retrieves the previous frame so that you can print the line number and file of the location where ipsh() is called.
Related
Is there any GUI programs that work like this - click button in GUI and it does launch some python code line or multiple lines, i thought about tkinter, but i found none options to make it launch some python code or is there?
Upd - In terminal code will be too complicated to use the code so i need GUI.
There are a few ways to run python code, the way it is required in the OP -:
Using exec or eval to run python code dynamically(NOTE: This would not
return the output but rather directly run the code as if it were
normal code.), Also note that use of exec and eval should generally be avoided but it had to be mentioned here as a possible way through.
exec(object, globals, locals)
eval(expression[, globals[, locals]])
Launching a terminal window(using os.system method of the os module) and using pyautogui to type and run the
command that executes given python script, when a button is
clicked(NOTE: With this method as well you will only be able to see
the output of the script but not fetch it.)
# required imports
import os, pyautogui
import tkinter as tk
from tkinter import filedialog
root = tk.Tk() # initializing the tkinter window.
def ask_for_running_file(event = None) :
filetypes = (("python files", '.py'), ("all files","*.*"))
filename = filedialog.askopenfilename(initialdir = '/', title = "Select file", filetypes = filetypes) # Get the name of the
python file to run.
addr_command = 'start cmd /k "cd "' + filename[ : filename.rindex('/')] # Command used to open the terminal
run_command = ' python \"' + filename[(filename.rindex('/') + 1) : ] + '\"' # The command to be typed in the terminal using pyautogui to execute the python script.
os.system(addr_command) # Runs the command required to open the terminal window.
pyautogui.write(run_command, interval = 0.25) # Writes the command required to execute the python script letter by letter.
NOTE: The interval here can be adjusted as per use case.
pyautogui.press('enter') # Pressing enter to execute the run command.
return
b1 = tk.Button(root, text = 'run file', command = ask_for_running_file) # The button that open a file dialog to choose
#which file to run.
b1.pack() # managing the geometry of the button.
root.mainloop() # starting mainloop
Use subprocess module to get the output of the execution of a
terminal command and then display it using tkinter or any other
module(if required). Note that the subprocess module suggests to use
the run function for most of the use cases so the example provided
below will involve the use of that only. The documentation lists the use of each and every argument and can be used further to modify the below code as pleased.
result = subprocess.run(['python', '\"' + script_name + '\"'],
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell = True)
Once the way of executing the command is figured out, the button that triggers it and the GUI can be designed in any module as pleased, for example the function shown in method 2 can be used with any other GUI module's button's callback function too! And it should work fine keeping in mind to remove the use of tkinter filedialog within the function rest can stay the same.
NOTE:
The second method might not always work if there is a disturbance while pyautogui types in the command, and thus it is not suitable for use cases where background processes are needed to achieve the problem's solution.
Edit: My first attempt at asking this might be a bit unfocused/poorly worded here's a better explanation of what I'm trying to do:
I'm trying to modify the default behavior of the print function for the entire environment python is running in without having to modify each file that's being run.
I'm attempting to decorate the print function (I know there are many ways to do this such as overriding it but that's not really the question I'm asking) so I can have it print out some debugging information and force it to always flush. I did that like so:
def modify_print(func):
# I made this so that output always gets flushed as it won't by default
# within the environment I'm using, I also wanted it to print out some
# debugging information, doesn't really matter much in the context of this
# question
def modified_print(*args,**kwargs):
return func(f"some debug prefix: ",flush=True,*args,**kwargs)
return modified_print
print = modify_print(print)
print("Hello world") # Prints "some debug prefix Hello World"
However what I'm trying to do is modify this behavior throughout my entire application. I know I can manually decorate/override/import the print function in each file however I'm wondering if there is some way I can globally configure my python environment to decorate this function everywhere. The only way I can think to do this would be to edit the python source code and build the modified version.
EDIT:
Here's the behavior I wanted implemented, thank you Match for your help.
It prints out the line number and filename everywhere you call a print function within your python environment. This means you don't have to import or override anything manually in all of your files.
https://gist.github.com/MichaelScript/444cbe5b74dce2c01a151d60b714ac3a
import site
import os
import pathlib
# Big thanks to Match on StackOverflow for helping me with this
# see https://stackoverflow.com/a/48713998/5614280
# This is some cool hackery to overwrite the default functionality of
# the builtin print function within your entire python environment
# to display the file name and the line number as well as always flush
# the output. It works by creating a custom user script and placing it
# within the user's sitepackages file and then overwriting the builtin.
# You can disable this behavior by running python with the '-s' flag.
# We could probably swap this out by reading the text from a python file
# which would make it easier to maintain larger modifications to builtins
# or a set of files to make this more portable or to modify the behavior
# of more builtins for debugging purposes.
customize_script = """
from inspect import getframeinfo,stack
def debug_printer(func):
# I made this so that output always gets flushed as it won't by default
# within the environment I'm running it in. Also it will print the
# file name and line number of where the print occurs
def debug_print(*args,**kwargs):
frame = getframeinfo(stack()[1][0])
return func(f"{frame.filename} : {frame.lineno} ", flush=True,*args,**kwargs)
return debug_print
__builtins__['print'] = debug_printer(print)
"""
# Creating the user site dir if it doesn't already exist and writing our
# custom behavior modifications
pathlib.Path(site.USER_SITE).mkdir(parents=True, exist_ok=True)
custom_file = os.path.join(site.USER_SITE,"usercustomize.py")
with open(custom_file,'w+') as f:
f.write(customize_script)
You can use usercustomize script from the site module to achieve something like this.
First, find out where your user site-packages directory is:
python3 -c "import site; print(site.USER_SITE)"
/home/foo/.local/lib/python3.6/site-packages
Next, in that directory, create a script called usercustomize.py - this script will now be run first whenever python is run.
One* way to replace print is to override the __builtins__ dict and replace it with a new method - something like:
from functools import partial
old_print = __builtins__['print']
__builtins__['print'] = partial(old_print, "Debug prefix: ", flush=True)
Drop this into the usercustomize.py script and you should see all python scripts from then on being overridden. You can temporarily disable calling this script by calling python with the -s flag.
*(Not sure if this is the correct way of doing this - there may be a better way - but the main point is that you can use usercustomize to deliver whatever method you choose).
There's no real reason to define a decorator here, because you are only intending to apply it to a single, predetermined function. Just define your modified print function directly, wrapping it around __builtins__.print to avoid recursion.
def print(*args, **kwargs):
__builtins.__print(f"some debug prefix: ", flush=True, *args, **kwargs)
print("Hello world") # Prints "some debug prefix Hello World"
You can use functools.partial to simplify this.
import functools
print = functools.partial(__builtins.__print, f"some debug prefix: ", flush=True)
Is it possible to save an IPython workspace (defined functions, different kinds of variables, etc) so that it can be loaded later?
This would be a similar function to save.image() in MATLAB or R. Similar questions has been asked before, such as:
Save session in IPython like in MATLAB?
However, since a few years passed, I am wondering if there is a good solution now.
You can use the dill python package:
import dill
filepath = 'session.pkl'
dill.dump_session(filepath) # Save the session
dill.load_session(filepath) # Load the session
To install it:
pip install dill
EDIT: this answer (and gist) has been modified to work for IPython 6
I added a somewhat ad-hoc solution that automates the process of storing/restoring user space variables using the underlying code from IPython's %store magic which is from what I understand what you wanted. See the gist here. Note that this only works for objects that can be pickled.
I can't guarantee its robustness, especially if any of the autorestore mechanisms in IPython change in the future, but it has been working for me with IPython 2.1.0. Hopefully this will at least point you in the right direction.
To reiterate the solution here:
Add the save_user_variables.py script below to your ipython folder (by default $HOME/.ipython). This script takes care of saving user variables on exit.
Add this line to your profile's ipython startup script (e.g., $HOME/.ipython/profile_default/startup/startup.py):
get_ipython().ex("import save_user_variables;del save_user_variables")
In your ipython profile config file (by default $HOME/.ipython/profile_default/ipython_config.py) find the following line:
# c.StoreMagics.autorestore = False
Uncomment it and set it to true. This automatically reloads stored variables on startup. Alternatively you can reload the last session manually using %store -r.
save_user_variables.py
def get_response(quest,default=None,opts=('y','n'),please=None,fmt=None):
try:
raw_input = input
except NameError:
pass
quest += " ("
quest += "/".join(['['+o+']' if o==default else o for o in opts])
quest += "): "
if default is not None: opts = list(opts)+['']
if please is None: please = quest
if fmt is None: fmt = lambda x: x
rin = input(quest)
while fmt(rin) not in opts: rin = input(please)
return default if default is not None and rin == '' else fmt(rin)
def get_user_vars():
"""
Get variables in user namespace (ripped directly from ipython namespace
magic code)
"""
import IPython
ip = IPython.get_ipython()
user_ns = ip.user_ns
user_ns_hidden = ip.user_ns_hidden
nonmatching = object()
var_hist = [ i for i in user_ns
if not i.startswith('_') \
and (user_ns[i] is not user_ns_hidden.get(i, nonmatching)) ]
return var_hist
def shutdown_logger():
"""
Prompts for saving the current session during shutdown
"""
import IPython, pickle
var_hist = get_user_vars()
ip = IPython.get_ipython()
db = ip.db
# collect any variables that need to be deleted from db
keys = map(lambda x: x.split('/')[1], db.keys('autorestore/*'))
todel = set(keys).difference(ip.user_ns)
changed = [db[k] != ip.user_ns[k.split('/')[1]]
for k in db.keys('autorestore/*') if k.split('/')[1] in ip.user_ns]
try:
if len(var_hist) == 0 and len(todel) == 0 and not any(changed): return
if get_response("Save session?", 'n', fmt=str.lower) == 'n': return
except KeyboardInterrupt:
return
# Save interactive variables (ignore unsaveable ones)
for name in var_hist:
obj = ip.user_ns[name]
try:
db[ 'autorestore/' + name ] = obj
except pickle.PicklingError:
print("Could not store variable '%s'. Skipping..." % name)
del db[ 'autorestore/' + name ]
# Remove any previously stored variables that were deleted in this session
for k in todel:
del db['autorestore/'+k]
import atexit
atexit.register(shutdown_logger)
del atexit
Although not so convenient as save.image(), you can use one of the checkpoint/restore applications. If you're using Linux, you might try http://criu.org. I'm using it from time to time to dump my ipython state and restore it later.
In order to dump a shell app with CRIU, you need to find its PID (e.g. pstree -p) and then use something like this (you'll need a second terminal for this; CRIU can't dump stopped jobs):
sudo criu dump -t PID --images-dir ~/tmp/imgs --log-file dump.log -v4 --shell-job
this will write all necessary images to ~/tmp/imgs (remember the --shell-job option). In order to restore the state later to the current terminal (don't forget to hit enter to get the next ipython prompt):
sudo criu restore --images-dir ~/tmp/imgs/ --log-file restore.log -v4 --shell-job
Check out the logs in case of any problems.
Obviously CRIU will work with any app (with some limits, of course). It's just an idea so you can use it for ipython.
You can try
%save name lines
Like if you have input 67 commands and you want to save all of them:
%save myhistory 1-67
you can certainly do this in the ipython notebook.
when the notebook is saved--either manually or by default config--the notebook is persisted as an .ipynb file, which is just a json file (an example in a github gist).
Next time you start the ipython server in the directory where that file resides, the server will detect it.
when you open that notebook in the browser, all of the code & config is there but unexecuted; you can execute the code in every cell by selecting execute all cells from the cells menu.
in addition, you can manually persist snapshots of your notebook, as ipynb_checkpoints, which are stored in a directory of that name preceded by a dot.
and finally, from the file menu option, you can persist your notebook as a pure python source file (.py)
I'm trying to save myself just a few keystrokes for a command I type fairly regularly in Python.
In my python startup script, I define a function called load which is similar to import, but adds some functionality. It takes a single string:
def load(s):
# Do some stuff
return something
In order to call this function I have to type
>>> load('something')
I would rather be able to simply type:
>>> load something
I am running Python with readline support, so I know there exists some programmability there, but I don't know if this sort of thing is possible using it.
I attempted to get around this by using the InteractivConsole and creating an instance of it in my startup file, like so:
import code, re, traceback
class LoadingInteractiveConsole(code.InteractiveConsole):
def raw_input(self, prompt = ""):
s = raw_input(prompt)
match = re.match('^load\s+(.+)', s)
if match:
module = match.group(1)
try:
load(module)
print "Loaded " + module
except ImportError:
traceback.print_exc()
return ''
else:
return s
console = LoadingInteractiveConsole()
console.interact("")
This works with the caveat that I have to hit Ctrl-D twice to exit the python interpreter: once to get out of my custom console, once to get out of the real one.
Is there a way to do this without writing a custom C program and embedding the interpreter into it?
Edit
Out of channel, I had the suggestion of appending this to the end of my startup file:
import sys
sys.exit()
It works well enough, but I'm still interested in alternative solutions.
You could try ipython - which gives a python shell which does allow many things including automatic parentheses which gives you the function call as you requested.
I think you want the cmd module.
See a tutorial here:
http://wiki.python.org/moin/CmdModule
Hate to answer my own question, but there hasn't been an answer that works for all the versions of Python I use. Aside from the solution I posted in my question edit (which is what I'm now using), here's another:
Edit .bashrc to contain the following lines:
alias python3='python3 ~/py/shellreplace.py'
alias python='python ~/py/shellreplace.py'
alias python27='python27 ~/py/shellreplace.py'
Then simply move all of the LoadingInteractiveConsole code into the file ~/py/shellreplace.py Once the script finishes executing, python will cease executing, and the improved interactive session will be seamless.
I am implementing a "breakpoint" system for use in my Python development that will allow me to call a function that, in essence, calls pdb.set_trace();
Some of the functionality that I would like to implement requires me to control pdb from code while I am within a set_trace context.
Example:
disableList = []
def breakpoint(name=None):
def d():
disableList.append(name)
#****
#issue 'run' command to pdb so user
#does not have to type 'c'
#****
if name in disableList:
return
print "Use d() to disable breakpoint, 'c' to continue"
pdb.set_trace();
In the above example, how do I implement the comments demarked by the #**** ?
In other parts of this system, I would like to issue an 'up' command, or two sequential 'up' commands without leaving the pdb session (so the user ends up at a pdb prompt but up two levels on the call stack).
You could invoke lower-level methods to get more control over the debugger:
def debug():
import pdb
import sys
# set up the debugger
debugger = pdb.Pdb()
debugger.reset()
# your custom stuff here
debugger.do_where(None) # run the "where" command
# invoke the interactive debugging prompt
users_frame = sys._getframe().f_back # frame where the user invoked `debug()`
debugger.interaction(users_frame, None)
if __name__ == '__main__':
print 1
debug()
print 2
You can find documentation for the pdb module here: http://docs.python.org/library/pdb and for the bdb lower-level debugging interface here: http://docs.python.org/library/bdb. You may also want to look at their source code.