Develop Reference

reload the currently running python script

In python ,There is a reload method to reload an imported module , but is there a method to reload the currently running script without restarting it, this would be very helpful in debugging a script and changing the code live while the script is running. In visual basic I remember a similar functionality was called "apply code changes", but I need a similar functionality as a function call like "refresh()" which will apply the code changes instantly.
This would work smoothly when an independent function in the script is modified and we need to immediately apply the code change without restarting the script.
Inshort will:
reload(self)
work?

reload(self) will not work, because reload() works on modules, not live instances of classes. What you want is some logic external to your main application, which checks whether it has to be reloaded. You have to think about what is needed to re-create your application state upon reload.
Some hints in this direction:
Guido van Rossum wrote once this: xreload.py; it does a bit more than reload() You would need a loop, which checks for changes every x seconds and applies this.
Also have a look at livecoding which basically does this. EDIT: I mistook this project for something else (which I didn't find now), sorry.
perhaps this SO question will help you

Perhaps you mean something like this:
import pdb
import importlib
from os.path import basename
def function():
print("hello world")
if __name__ == "__main__":
# import this module, but not as __main__ so everything other than this
# if statement is executed
mainmodname = basename(__file__)[:-3]
module = importlib.import_module(mainmodname)
while True:
# reload the module to check for changes
importlib.reload(module)
# update the globals of __main__ with the any new or changed
# functions or classes in the reloaded module
globals().update(vars(module))
function()
pdb.set_trace()
Run the code, then change the contents of function and enter c at the prompt to run the next iteration of the loop.

test.py
class Test(object):
def getTest(self):
return 'test1'
testReload.py
from test import Test
t = Test()
print t.getTest()
# change return value (test.py)
import importlib
module = importlib.import_module(Test.__module__)
reload(module)
from test import Test
t = Test()
print t.getTest()

Intro
reload is for imported modules. Documentation for reload advises against reloading __main__.
Reloading sys, __main__, builtins and other key modules is not recommended.
To achieve similar behavior on your script you will need to re-execute the script. This will - for normal scripts - also reset the the global state. I propose a solution.
NOTE
The solution I propose is very powerful and should only be used for code you trust. Automatically executing code from unknown sources can lead to a world of pain. Python is not a good environment for soapboxing unsafe code.
Solution
To programmatically execute a python script from another script you only need the built-in functions open, compile and exec.
Here is an example function that will re-execute the script it is in:
def refresh():
with open(__file__) as fo:
source_code = fo.read()
byte_code = compile(source_code, __file__, "exec")
exec(byte_code)
The above will in normal circumstances also re-set any global variables you might have. If you wish to keep these variables you should check weather or not those variables have already been set. This can be done with a try-except block covering NameError exceptions. But that can be tedious so I propose using a flag variable.
Here is an example using the flag variable:
in_main = __name__ == "__main__"
first_run = "flag" not in globals()
if in_main and first_run:
flag = True

None of these answers did the job properly for me, so I put together something very messy and very non-pythonic to do the job myself. Even after running it for several weeks, I am finding small issues and fixing them. One issue will be if your PWD/CWD changes.
Warning this is very ugly code. Perhaps someone will make it pretty, but it does work.
Not only does it create a refresh() function that properly reloads your script in a manner such that any Exceptions will properly display, but it creates refresh_<scriptname> functions for previously loaded scripts just-in-case you need to reload those.
Next I would probably add a require portion, so that scripts can reload other scripts -- but I'm not trying to make node.js here.
First, the "one-liner" that you need to insert in any script you want to refresh.
with open(os.path.dirname(__file__) + os.sep + 'refresh.py', 'r') as f: \
exec(compile(f.read().replace('__BASE__', \
os.path.basename(__file__).replace('.py', '')).replace('__FILE__', \
__file__), __file__, 'exec'))
And second, the actual refresh function which should be saved to refresh.py in the same directory. (See, room for improvement already).
def refresh(filepath = __file__, _globals = None, _locals = None):
print("Reading {}...".format(filepath))
if _globals is None:
_globals = globals()
_globals.update({
"__file__": filepath,
"__name__": "__main__",
})
with open(filepath, 'rb') as file:
exec(compile(file.read(), filepath, 'exec'), _globals, _locals)
def refresh___BASE__():
refresh("__FILE__")
Tested with Python 2.7 and 3.

Take a look at reload. You just need to install the plugin an use reload ./myscript.py, voilà

If you are running in an interactive session you could use ipython autoreload
autoreload reloads modules automatically before entering the execution of code typed at the IPython prompt.
Of course this also works on module level, so you would do something like:
>>>import myscript
>>>myscript.main()
*do some changes in myscript.py*
>>>myscript.main() #is now changed

Detect where Python code is running (e.g., in Spyder interpreter vs. IDLE vs. cmd)

Is there a way in Python to detect, within a process, where that process is being executed? I have some code that includes the getpass.getpass() function, which is broken in Spyder, and it's annoying to go back and forth between the command line and the IDE all the time. It would be useful if I could add code like:
if not being run from Spyder:
use getpass
else:
use alternative

Here is the solution I ended up using. After reading Markus's answer, I noticed that Spyder adds half a dozen or so environment variables to os.environ with names like SPYDER_ENCODING, SPYDER_SHELL_ID, etc. Detecting the presence of any of these seems relatively unambiguous, compared to detecting the absence of a variable with as generic a name as 'PYTHONSTARTUP'. The code is simple, and works independently of Spyder's startup script (as far as I can tell):
if any('SPYDER' in name for name in os.environ)
# use alternative
else:
# use getpass
Since the string is at the beginning of each environment variable name, you could also use str.startswith, but it's less flexible, and a little bit slower (I was curious):
>>> import timeit
>>> s = timeit.Timer("[name.startswith('SPYDER') for name in os.environ]", "import os")
>>> i = timeit.Timer("['SPYDER' in name for name in os.environ]", "import os")
>>> s.timeit()
16.18333065883474
>>> i.timeit()
6.156869294143846
The sys.executable method may or may not be useful depending on your installation. I have a couple WinPython installations and a separate Python 2.7 installation, so I was able to check the condition sys.executable.find('WinPy') == -1 to detect a folder name in the path of the executable Spyder uses. Since the warning that shows in IDLE when you try to use getpass is less "loud" than it could be, in my opinion, I ended up also checking the condition sys.executable.find('pythonw.exe') == -1 to make it slightly louder. Using sys.executable only, that method looks like:
if sys.executable.find('pythonw.exe') == sys.executable.find('WinPy') == -1:
# use getpass
else:
# use alternative
But since I want this to work on other machines, and it's much more likely that another user would modify their WinPython installation folder name than that they would rename their IDLE executable, my final code uses sys.executable to detect IDLE and os.environ to detect Spyder, providing a "louder" warning in either case and keeping the code from breaking in the latter.
if any('SPYDER' in name for name in os.environ) \
or 'pythonw.exe' in sys.executable:
password = raw_input('WARNING: PASSWORD WILL BE SHOWN ON SCREEN\n\n' * 3
+ 'Please enter your password: ')
else:
password = getpass.getpass("Please enter your password: ")

By default, Spyder uses a startup scrip, see Preferences -> Console -> Adanced setting. This option is usually set to the scientific_startup.py file that loads pylab et al.
The easiest solution is to just add a global variable to the file and then use that in your if statement, e.g. add this line at the end of scientific_startup.py:
SPYDER_IDE_ACTIVE = True
In your script:
if not 'SPYDER_IDE_ACTIVE' in globals():
use getpass
else:
use alternative
This will work without throwing an error. You can also use exceptions if you like that more.
A second solution would be (if you cannot modify that file for some reason) to just check if the environment variable PYTHONSTARTUP is set. On my machine (using the Anaconda Python stack), it is not set for a regular Python shell. You could do
import os
if not 'PYTHONSTARTUP' in os.environ:
use getpass
else:
use alternative

Spyder provides the option of executing the current editor script in a native system terminal. This would produce identical behavior as if you were running from the command line. To set this up, open the Run Settings dialog by hitting F6. Then select the radio button "Execute in an external System terminal". Now run the script as usual by hitting F5. You should be able to use getpass in the normal fashion with this approach.

You could add env variable when running in Spyder and check it in code.

Invoking shell-command from function in interactive IPython shell

I have just been playing around with IPython. Currently I am wondering how it would be possible to run a shell-command with a python variable within a function. For example:
def x(go):
return !ls -la {go}
x("*.rar")
This gives me "sh: 1: Syntax error: end of file unexpected". Could anybody please give me a clue on how to let my "x"-function invoke ls like "ls -la *.rar"? There are *.rar files in my working directory.
Thank you in advance,
Rainer

If you look at the history command output, you'll see that to call external programs ipython uses _ip.system method.
Hence, this should work for you:
def x(go):
return _ip.system("ls -la {0}".format(go))
However, please note that outside ipython you should probably use subprocess.Popen.

There was a bug in the "!" shell access that made the expansion of "function scoped variables" fail. Your ipython's version might be affected.
You can avoid it by doing yourself the variable expansion:
def x(go):
return get_ipython().getoutput("ls -la {0}".format(go))

While subprocess.Popen is probably the way to go as #jcollado said, just for completeness there is the os.system command to immediately send a command to the shell. However, the subprocess module is almost always a better choice than os.system or os.spawn.
Also, depending on what you are trying to do you may want to use python commands to interact with the operating system rather than passing commands out to a shell. If you want to deal with lists of files for instance, os.walk would likely result in cleaner and more portable code than grabbing the directory list through shell commands. You can look at the documentation for Python's OS module here.

Depending on what you wanted to accomplish, this may be the better way:
In [50]: %alias x ls -la %l
In [51]: x *.rar
-rw-r--r-- 1 dubbaluga users 45254 Apr 4 15:12 schoolbus.rar

Maybe its easier to use Python for this case:
import glob
files = glob.glob('*.rar')

How can I add a command to the Python interactive shell?

I'm trying to save myself just a few keystrokes for a command I type fairly regularly in Python.
In my python startup script, I define a function called load which is similar to import, but adds some functionality. It takes a single string:
def load(s):
# Do some stuff
return something
In order to call this function I have to type
>>> load('something')
I would rather be able to simply type:
>>> load something
I am running Python with readline support, so I know there exists some programmability there, but I don't know if this sort of thing is possible using it.
I attempted to get around this by using the InteractivConsole and creating an instance of it in my startup file, like so:
import code, re, traceback
class LoadingInteractiveConsole(code.InteractiveConsole):
def raw_input(self, prompt = ""):
s = raw_input(prompt)
match = re.match('^load\s+(.+)', s)
if match:
module = match.group(1)
try:
load(module)
print "Loaded " + module
except ImportError:
traceback.print_exc()
return ''
else:
return s
console = LoadingInteractiveConsole()
console.interact("")
This works with the caveat that I have to hit Ctrl-D twice to exit the python interpreter: once to get out of my custom console, once to get out of the real one.
Is there a way to do this without writing a custom C program and embedding the interpreter into it?
Edit
Out of channel, I had the suggestion of appending this to the end of my startup file:
import sys
sys.exit()
It works well enough, but I'm still interested in alternative solutions.

You could try ipython - which gives a python shell which does allow many things including automatic parentheses which gives you the function call as you requested.

I think you want the cmd module.
See a tutorial here:
http://wiki.python.org/moin/CmdModule

Hate to answer my own question, but there hasn't been an answer that works for all the versions of Python I use. Aside from the solution I posted in my question edit (which is what I'm now using), here's another:
Edit .bashrc to contain the following lines:
alias python3='python3 ~/py/shellreplace.py'
alias python='python ~/py/shellreplace.py'
alias python27='python27 ~/py/shellreplace.py'
Then simply move all of the LoadingInteractiveConsole code into the file ~/py/shellreplace.py Once the script finishes executing, python will cease executing, and the improved interactive session will be seamless.

How to save a Python interactive session?

I find myself frequently using Python's interpreter to work with databases, files, etc -- basically a lot of manual formatting of semi-structured data. I don't properly save and clean up the useful bits as often as I would like. Is there a way to save my input into the shell (db connections, variable assignments, little for loops and bits of logic) -- some history of the interactive session? If I use something like script I get too much stdout noise. I don't really need to pickle all the objects -- though if there is a solution that does that, it would be OK. Ideally I would just be left with a script that ran as the one I created interactively, and I could just delete the bits I didn't need. Is there a package that does this, or a DIY approach?

IPython is extremely useful if you like using interactive sessions. For example for your use-case there is the %save magic command, you just input %save my_useful_session 10-20 23 to save input lines 10 to 20 and 23 to my_useful_session.py (to help with this, every line is prefixed by its number).
Furthermore, the documentation states:
This function uses the same syntax as %history for input ranges, then saves the lines to the filename you specify.
This allows for example, to reference older sessions, such as
%save current_session ~0/
%save previous_session ~1/
Look at the videos on the presentation page to get a quick overview of the features.

From Andrew Jones's website (archived):
import readline
readline.write_history_file('/home/ahj/history')

There is a way to do it. Store the file in ~/.pystartup...
# Add auto-completion and a stored history file of commands to your Python
# interactive interpreter. Requires Python 2.0+, readline. Autocomplete is
# bound to the Esc key by default (you can change it - see readline docs).
#
# Store the file in ~/.pystartup, and set an environment variable to point
# to it: "export PYTHONSTARTUP=/home/user/.pystartup" in bash.
#
# Note that PYTHONSTARTUP does *not* expand "~", so you have to put in the
# full path to your home directory.
import atexit
import os
import readline
import rlcompleter
historyPath = os.path.expanduser("~/.pyhistory")
def save_history(historyPath=historyPath):
import readline
readline.write_history_file(historyPath)
if os.path.exists(historyPath):
readline.read_history_file(historyPath)
atexit.register(save_history)
del os, atexit, readline, rlcompleter, save_history, historyPath
and then set the environment variable PYTHONSTARTUP in your shell (e.g. in ~/.bashrc):
export PYTHONSTARTUP=$HOME/.pystartup
You can also add this to get autocomplete for free:
readline.parse_and_bind('tab: complete')
Please note that this will only work on *nix systems. As readline is only available in Unix platform.

If you are using IPython you can save to a file all your previous commands using the magic function %history with the -f parameter, p.e:
%history -f /tmp/history.py

After installing Ipython, and opening an Ipython session by running the command:
ipython
from your command line, just run the following Ipython 'magic' command to automatically log your entire Ipython session:
%logstart
This will create a uniquely named .py file and store your session for later use as an interactive Ipython session or for use in the script(s) of your choosing.

Also, reinteract gives you a notebook-like interface to a Python session.

In addition to IPython, a similar utility bpython has a "save the code you've entered to a file" feature

I had to struggle to find an answer, I was very new to iPython environment.
This will work
If your iPython session looks like this
In [1] : import numpy as np
....
In [135]: counter=collections.Counter(mapusercluster[3])
In [136]: counter
Out[136]: Counter({2: 700, 0: 351, 1: 233})
You want to save lines from 1 till 135 then on the same ipython session use this command
In [137]: %save test.py 1-135
This will save all your python statements in test.py file in your current directory ( where you initiated the ipython).

There is %history magic for printing and saving the input history (and optionally the output).
To store your current session to a file named my_history.py:
>>> %hist -f my_history.py
History IPython stores both the commands you enter, and the results it produces. You can easily go through previous commands with the up- and down-arrow keys, or access your history in more sophisticated ways.
You can use the %history magic function to examine past input and output. Input history from previous sessions is saved in a database, and IPython can be configured to save output history.
Several other magic functions can use your input history, including %edit, %rerun, %recall, %macro, %save and %pastebin. You can use a standard format to refer to lines:
%pastebin 3 18-20 ~1/1-5
This will take line 3 and lines 18 to 20 from the current session, and lines 1-5 from the previous session.
See %history? for the Docstring and more examples.
Also, be sure to explore the capabilities of %store magic for lightweight persistence of variables in IPython.
Stores variables, aliases and macros in IPython’s database.
d = {'a': 1, 'b': 2}
%store d # stores the variable
del d
%store -r d # Refresh the variable from IPython's database.
>>> d
{'a': 1, 'b': 2}
To autorestore stored variables on startup, specifyc.StoreMagic.autorestore = True in ipython_config.py.

The %history command is awesome, but unfortunately it won't let you save things that were %paste 'd into the sesh. To do that I think you have to do %logstart at the beginning (although I haven't confirmed this works).
What I like to do is
%history -o -n -p -f filename.txt
which will save the output, line numbers, and '>>>' before each input (o, n, and p options). See the docs for %history here.

Just putting another suggesting in the bowl:
Spyder
It has History log and Variable explorer. If you have worked with MatLab, then you'll see the similarities.

As far as Linux goes, one can use script command to record the whole session. It is part of util-linux package so should be on most Linux systems . You can create and alias or function that will call script -c python and that will be saved to a typescript file. For instance, here's a reprint of one such file.
$ cat typescript
Script started on Sat 14 May 2016 08:30:08 AM MDT
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print 'Hello Pythonic World'
Hello Pythonic World
>>>
Script done on Sat 14 May 2016 08:30:42 AM MDT
Small disadvantage here is that the script records everything , even line-feeds, whenever you hit backspaces , etc. So you may want to use col to clean up the output (see this post on Unix&Linux Stackexchange) .

there is another option --- pyslice.
in the "wxpython 2.8 docs demos and tools", there is a open source program named "pyslices".
you can use it like a editor, and it also support using like a console ---- executing each line like a interactive interpreter with immediate echo.
of course, all the blocks of codes and results of each block will be recorded into a txt file automatically.
the results are logged just behind the corresponding block of code. very convenient.

In IPython, I first use
In [2]: %hist
to view my past code. I select the chunk I want to save and then paste it into file my_file.py using the %%file magic (short for %%writefile)
In [3]: %%file my_file.py
...: # paste code here
...:
...:
hitting return two times in the end.
To append to file use the option -a: %%file -a my_file.py.
If needed, I can list, edit, etc. the file in the underlying command line using the exclamation mark
In [5]: !ls -l my_file.py
In [6]: !vi my_file.py

Some comments were asking how to save all of the IPython inputs at once. For %save magic in IPython, you can save all of the commands programmatically as shown below, to avoid the prompt message and also to avoid specifying the input numbers.
currentLine = len(In)-1
%save -f my_session 1-$currentLine
The -f option is used for forcing file replacement and the len(IN)-1 shows the current input prompt in IPython, allowing you to save the whole session programmatically.

For those using spacemacs, and ipython that comes with python-layer, save magic creates a lot of unwanted output, because of the constant auto-completion command working in the backround such as:
len(all_suffixes)
';'.join(__PYTHON_EL_get_completions('''len'''))
';'.join(__PYTHON_EL_get_completions('''all_substa'''))
len(all_substantives_w_suffixes)
';'.join(__PYTHON_EL_get_completions('''len'''))
';'.join(__PYTHON_EL_get_completions('''all'''))
';'.join(__PYTHON_EL_get_completions('''all_'''))
';'.join(__PYTHON_EL_get_completions('''all_w'''))
';'.join(__PYTHON_EL_get_completions('''all_wo'''))
';'.join(__PYTHON_EL_get_completions('''all_wor'''))
';'.join(__PYTHON_EL_get_completions('''all_word'''))
';'.join(__PYTHON_EL_get_completions('''all_words'''))
len(all_words_w_logograms)
len(all_verbs)
To avoid this just save the ipython buffer like you normally save any other: spc f s

If you use bpython, all your command history is by default saved to ~/.pythonhist.
To save the commands for later reusage you can copy them to a python script file:
$ cp ~/.pythonhist mycommands.py
Then edit that file to clean it up and put it under Python path (global or virtual environment's site-packages, current directory, mentioning in *.pth, or some other way).
To include the commands into your shell, just import them from the saved file:
>>> from mycommands import *

I'd like to suggest another way to maintain python session through tmux on linux. you run tmux, attach your self to the session you opened (if not attached after opening it directly). execute python and do whatever you are doing on it. then detach from session. detaching from a tmux session does not close the session. the session remains open.
pros of this method:
you can attach to this session from any other device (in case you can ssh your pc)
cons of this method:
this method does not relinquish the resources used by the opened python session until you actually exist the python interpreter.

To save input and output on XUbuntu:
In XWindows, run iPython from the Xfce terminal app
click Terminal in the top menu bar and look for save contents in the dropdown
I find this saves the input and output, going all the way back to when I opened the terminal. This is not ipython specific, and would work with ssh sessions or other tasks run from the terminal window.

You can use built-in function open: I use it in all of my
programs in which I need to store some history (Including Calculator, etc.)
for example:
#gk-test.py or anything else would do
try: # use the try loop only if you haven't created the history file outside program
username = open("history.txt").readline().strip("\n")
user_age = open("history.txt").readlines()[1].strip("\n")
except FileNotFoundError:
username = input("Enter Username: ")
user_age = input("Enter User's Age: ")
open("history.txt", "w").write(f"{username}\n{user_age}")
#Rest of the code is secret! try it your own!
I would thank to all of them who liked my comments! Thank you for reading this!