Develop Reference

How can you open a saved .py file in idle3 without it defaulting to normal text?

I recently started using idle3 to follow along with some course material as recommended by the instructor. Opening a new shell and running python code works just fine, the problem is when i save the file and later try to reopen it to continue on with the course work.
When ever i open the designated .py file, it only loads up in regular text, i can no longer execute new code, the line break doesn't have the ">>>" before each line, it won't even give any output
How can i open a saved file and continue on implementing code?

The basic use of the IDLE is for 2 thing, text editor and interactive console.
The interactive console allow you to write any python code and see it running immediately, great quick calculation and/or testing code until you get the desired result.
The save option (File -> Save) is deceptive however, because it save everything as you see including the >>> so saving as .py just doesn't work, or not without some extra work, namely removing all the >>> and all other undesired stuff, so is more like screenshot of sort.
The text editor mode is, well, just that but specialized for python with its syntax highlight and whatnot, use it to write your modules or whatever else, save it as .py and run the code by pressing F5 in order to test it, which will open a new interactive console with all that code loaded into it or would load it into a previously open one in which case it will reset it first, and that means that anything not in the file will be discarded (but you can scroll up can copy-pasted again if needed)
So for example, in a file you have
def greetings(name):
print("hello",name)
and you open it with the text editor mode of IDLE, simply run (F5) and you now have access to that function in the interactive console
>>> greetings("bob")
hello bob
>>> a=23
>>>
and lets make a new variable with some value
Now lets said we go back to the file and add some more stuff and now it looks like
def greetings(name):
print("hello",name)
def add(a,b):
return a+b
b=42
and run it again, this reset the console, which is indicated a msj like this containing the path of the loaded file
====== RESTART: C:\Users\Copperfield\Documents\StackOverflow\example.py =====
>>>
now we have the greetings and add function but the variable a from before is now loss
>>> a
Traceback (most recent call last):
File "<pyshell#6>", line 1, in <module>
a
NameError: name 'a' is not defined
>>>
but you have b
>>> b
42
>>>
So, use the text editor mode to write all the stuff you want to keep and in the interactive console play with it to test that is indeed the thing you want, and if not just go and edit it some more...

Ignore the rest of the python file

My python scripts often contain "executable code" (functions, classes, &c) in the first part of the file and "test code" (interactive experiments) at the end.
I want python, py_compile, pylint &c to completely ignore the experimental stuff at the end.
I am looking for something like #if 0 for cpp.
How can this be done?
Here are some ideas and the reasons they are bad:
sys.exit(0): works for python but not py_compile and pylint
put all experimental code under def test():: I can no longer copy/paste the code into a python REPL because it has non-trivial indent
put all experimental code between lines with """: emacs no longer indents and fontifies the code properly
comment and uncomment the code all the time: I am too lazy (yes, this is a single key press, but I have to remember to do that!)
put the test code into a separate file: I want to keep the related stuff together
PS. My IDE is Emacs and my python interpreter is pyspark.

Use ipython rather than python for your REPL It has better code completion and introspection and when you paste indented code it can automatically "de-indent" the pasted code.
Thus you can put your experimental code in a test function and then paste in parts without worrying and having to de-indent your code.
If you are pasting large blocks that can be considered individual blocks then you will need to use the %paste or %cpaste magics.
eg.
for i in range(3):
i *= 2
# with the following the blank line this is a complete block
print(i)
With a normal paste:
In [1]: for i in range(3):
...: i *= 2
...:
In [2]: print(i)
4
Using %paste
In [3]: %paste
for i in range(10):
i *= 2
print(i)
## -- End pasted text --
0
2
4
In [4]:
PySpark and IPython
It is also possible to launch PySpark in IPython, the enhanced Python interpreter. PySpark works with IPython 1.0.0 and later. To use IPython, set the IPYTHON variable to 1 when running bin/pyspark:1
$ IPYTHON=1 ./bin/pyspark

Unfortunately, there is no widely (or any) standard describing what you are talking about, so getting a bunch of python specific things to work like this will be difficult.
However, you could wrap these commands in such a way that they only read until a signifier. For example (assuming you are on a unix system):
cat $file | sed '/exit(0)/q' |sed '/exit(0)/d'
The command will read until 'exit(0)' is found. You could pipe this into your checkers, or create a temp file that your checkers read. You could create wrapper executable files on your path that may work with your editors.
Windows may be able to use a similar technique.
I might advise a different approach. Separate files might be best. You might explore iPython notebooks as a possible solution, but I'm not sure exactly what your use case is.

Follow something like option 2.
I usually put experimental code in a main method.
def main ():
*experimental code goes here *
Then if you want to execute the experimental code just call the main.
main()

With python-mode.el mark arbitrary chunks as section - for example via py-sectionize-region.
Than call py-execute-section.
Updated after comment:
python-mode.el is delivered by melpa.
M-x list-packages RET
Look for python-mode - the built-in python.el provides 'python, while python-mode.el provides 'python-mode.
Developement just moved hereto: https://gitlab.com/python-mode-devs/python-mode

I think the standard ('Pythonic') way to deal with this is to do it like so:
class MyClass(object):
...
def my_function():
...
if __name__ == '__main__':
# testing code here
Edit after your comment
I don't think what you want is possible using a plain Python interpreter. You could have a look at the IEP Python editor (website, bitbucket): it supports something like Matlab's cell mode, where a cell can be defined with a double comment character (##):
## main code
class MyClass(object):
...
def my_function():
...
## testing code
do_some_testing_please()
All code from a ##-beginning line until either the next such line or end-of-file constitutes a single cell.
Whenever the cursor is within a particular cell and you strike some hotkey (default Ctrl+Enter), the code within that cell is executed in the currently running interpreter. An additional feature of IEP is that selected code can be executed with F9; a pretty standard feature but the nice thing here is that IEP will smartly deal with whitespace, so just selecting and pasting stuff from inside a method will automatically work.

I suggest you use a proper version control system to keep the "real" and the "experimental" parts separated.
For example, using Git, you could only include the real code without the experimental parts in your commits (using add -p), and then temporarily stash the experimental parts for running your various tools.
You could also keep the experimental parts in their own branch which you then rebase on top of the non-experimental parts when you need them.

Another possibility is to put tests as doctests into the docstrings of your code, which admittedly is only practical for simpler cases.
This way, they are only treated as executable code by the doctest module, but as comments otherwise.

Copy code from IPython without leading triple dots

I'm using IPython Qt Console and when I copy code FROM Ipython it comes out like that:
class notathing(object):
...:
...: def __init__(self):
...: pass
...:
Is there any way to copy them without those leading triple dots and doublecolon?
P.S. I tried both Copy and Copy Raw Text in context menu and it's still the same. OS: Debian Linux 7.2 (KDE).

How about using %hist n to print line n (or a range of lines) without prompts (including line continuations), and doing your copy from that? (Simply scrolling back to that line is nearly as good).
In [1]: def foo():
...: return 1+2
...:
In [6]: %history 1
def foo():
return 1+2

One of the cool features of ipython is session logging. If you enable it, the code you input in your session is logged to a file. It's very useful, I use it all the time.
To make things even niftier for me, I have a shell alias ipy_log_cat, which prints the entire file. You can do something like: ipy_log_cat | tail to get the most recent input lines. (this is also useful for greping session history, etc.). You can also save a few keyboard/mouse strokes by piping it into xclip!

This QTconsole copy regression has been fixed, see https://github.com/ipython/ipython/issues/3206 - I can confirm that the desired behavior is again present in the QtConsole in the Canopy 1.2 GUI and, I suspect, in the ipython egg installable by free users from the Enthought egg repo.

This may be too roundabout for you, but you could use the %save magic function to save the lines in question and then copy them from the save file.

I tend to keep an open gvim window for this kind of things. Paste your class definition as is and then do something like:
:%s/^.*\.://

How to save a Python interactive session?

I find myself frequently using Python's interpreter to work with databases, files, etc -- basically a lot of manual formatting of semi-structured data. I don't properly save and clean up the useful bits as often as I would like. Is there a way to save my input into the shell (db connections, variable assignments, little for loops and bits of logic) -- some history of the interactive session? If I use something like script I get too much stdout noise. I don't really need to pickle all the objects -- though if there is a solution that does that, it would be OK. Ideally I would just be left with a script that ran as the one I created interactively, and I could just delete the bits I didn't need. Is there a package that does this, or a DIY approach?

IPython is extremely useful if you like using interactive sessions. For example for your use-case there is the %save magic command, you just input %save my_useful_session 10-20 23 to save input lines 10 to 20 and 23 to my_useful_session.py (to help with this, every line is prefixed by its number).
Furthermore, the documentation states:
This function uses the same syntax as %history for input ranges, then saves the lines to the filename you specify.
This allows for example, to reference older sessions, such as
%save current_session ~0/
%save previous_session ~1/
Look at the videos on the presentation page to get a quick overview of the features.

From Andrew Jones's website (archived):
import readline
readline.write_history_file('/home/ahj/history')

There is a way to do it. Store the file in ~/.pystartup...
# Add auto-completion and a stored history file of commands to your Python
# interactive interpreter. Requires Python 2.0+, readline. Autocomplete is
# bound to the Esc key by default (you can change it - see readline docs).
#
# Store the file in ~/.pystartup, and set an environment variable to point
# to it: "export PYTHONSTARTUP=/home/user/.pystartup" in bash.
#
# Note that PYTHONSTARTUP does *not* expand "~", so you have to put in the
# full path to your home directory.
import atexit
import os
import readline
import rlcompleter
historyPath = os.path.expanduser("~/.pyhistory")
def save_history(historyPath=historyPath):
import readline
readline.write_history_file(historyPath)
if os.path.exists(historyPath):
readline.read_history_file(historyPath)
atexit.register(save_history)
del os, atexit, readline, rlcompleter, save_history, historyPath
and then set the environment variable PYTHONSTARTUP in your shell (e.g. in ~/.bashrc):
export PYTHONSTARTUP=$HOME/.pystartup
You can also add this to get autocomplete for free:
readline.parse_and_bind('tab: complete')
Please note that this will only work on *nix systems. As readline is only available in Unix platform.

If you are using IPython you can save to a file all your previous commands using the magic function %history with the -f parameter, p.e:
%history -f /tmp/history.py

After installing Ipython, and opening an Ipython session by running the command:
ipython
from your command line, just run the following Ipython 'magic' command to automatically log your entire Ipython session:
%logstart
This will create a uniquely named .py file and store your session for later use as an interactive Ipython session or for use in the script(s) of your choosing.

Also, reinteract gives you a notebook-like interface to a Python session.

In addition to IPython, a similar utility bpython has a "save the code you've entered to a file" feature

I had to struggle to find an answer, I was very new to iPython environment.
This will work
If your iPython session looks like this
In [1] : import numpy as np
....
In [135]: counter=collections.Counter(mapusercluster[3])
In [136]: counter
Out[136]: Counter({2: 700, 0: 351, 1: 233})
You want to save lines from 1 till 135 then on the same ipython session use this command
In [137]: %save test.py 1-135
This will save all your python statements in test.py file in your current directory ( where you initiated the ipython).

There is %history magic for printing and saving the input history (and optionally the output).
To store your current session to a file named my_history.py:
>>> %hist -f my_history.py
History IPython stores both the commands you enter, and the results it produces. You can easily go through previous commands with the up- and down-arrow keys, or access your history in more sophisticated ways.
You can use the %history magic function to examine past input and output. Input history from previous sessions is saved in a database, and IPython can be configured to save output history.
Several other magic functions can use your input history, including %edit, %rerun, %recall, %macro, %save and %pastebin. You can use a standard format to refer to lines:
%pastebin 3 18-20 ~1/1-5
This will take line 3 and lines 18 to 20 from the current session, and lines 1-5 from the previous session.
See %history? for the Docstring and more examples.
Also, be sure to explore the capabilities of %store magic for lightweight persistence of variables in IPython.
Stores variables, aliases and macros in IPython’s database.
d = {'a': 1, 'b': 2}
%store d # stores the variable
del d
%store -r d # Refresh the variable from IPython's database.
>>> d
{'a': 1, 'b': 2}
To autorestore stored variables on startup, specifyc.StoreMagic.autorestore = True in ipython_config.py.

The %history command is awesome, but unfortunately it won't let you save things that were %paste 'd into the sesh. To do that I think you have to do %logstart at the beginning (although I haven't confirmed this works).
What I like to do is
%history -o -n -p -f filename.txt
which will save the output, line numbers, and '>>>' before each input (o, n, and p options). See the docs for %history here.

Just putting another suggesting in the bowl:
Spyder
It has History log and Variable explorer. If you have worked with MatLab, then you'll see the similarities.

As far as Linux goes, one can use script command to record the whole session. It is part of util-linux package so should be on most Linux systems . You can create and alias or function that will call script -c python and that will be saved to a typescript file. For instance, here's a reprint of one such file.
$ cat typescript
Script started on Sat 14 May 2016 08:30:08 AM MDT
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print 'Hello Pythonic World'
Hello Pythonic World
>>>
Script done on Sat 14 May 2016 08:30:42 AM MDT
Small disadvantage here is that the script records everything , even line-feeds, whenever you hit backspaces , etc. So you may want to use col to clean up the output (see this post on Unix&Linux Stackexchange) .

there is another option --- pyslice.
in the "wxpython 2.8 docs demos and tools", there is a open source program named "pyslices".
you can use it like a editor, and it also support using like a console ---- executing each line like a interactive interpreter with immediate echo.
of course, all the blocks of codes and results of each block will be recorded into a txt file automatically.
the results are logged just behind the corresponding block of code. very convenient.

In IPython, I first use
In [2]: %hist
to view my past code. I select the chunk I want to save and then paste it into file my_file.py using the %%file magic (short for %%writefile)
In [3]: %%file my_file.py
...: # paste code here
...:
...:
hitting return two times in the end.
To append to file use the option -a: %%file -a my_file.py.
If needed, I can list, edit, etc. the file in the underlying command line using the exclamation mark
In [5]: !ls -l my_file.py
In [6]: !vi my_file.py

Some comments were asking how to save all of the IPython inputs at once. For %save magic in IPython, you can save all of the commands programmatically as shown below, to avoid the prompt message and also to avoid specifying the input numbers.
currentLine = len(In)-1
%save -f my_session 1-$currentLine
The -f option is used for forcing file replacement and the len(IN)-1 shows the current input prompt in IPython, allowing you to save the whole session programmatically.

For those using spacemacs, and ipython that comes with python-layer, save magic creates a lot of unwanted output, because of the constant auto-completion command working in the backround such as:
len(all_suffixes)
';'.join(__PYTHON_EL_get_completions('''len'''))
';'.join(__PYTHON_EL_get_completions('''all_substa'''))
len(all_substantives_w_suffixes)
';'.join(__PYTHON_EL_get_completions('''len'''))
';'.join(__PYTHON_EL_get_completions('''all'''))
';'.join(__PYTHON_EL_get_completions('''all_'''))
';'.join(__PYTHON_EL_get_completions('''all_w'''))
';'.join(__PYTHON_EL_get_completions('''all_wo'''))
';'.join(__PYTHON_EL_get_completions('''all_wor'''))
';'.join(__PYTHON_EL_get_completions('''all_word'''))
';'.join(__PYTHON_EL_get_completions('''all_words'''))
len(all_words_w_logograms)
len(all_verbs)
To avoid this just save the ipython buffer like you normally save any other: spc f s

If you use bpython, all your command history is by default saved to ~/.pythonhist.
To save the commands for later reusage you can copy them to a python script file:
$ cp ~/.pythonhist mycommands.py
Then edit that file to clean it up and put it under Python path (global or virtual environment's site-packages, current directory, mentioning in *.pth, or some other way).
To include the commands into your shell, just import them from the saved file:
>>> from mycommands import *

I'd like to suggest another way to maintain python session through tmux on linux. you run tmux, attach your self to the session you opened (if not attached after opening it directly). execute python and do whatever you are doing on it. then detach from session. detaching from a tmux session does not close the session. the session remains open.
pros of this method:
you can attach to this session from any other device (in case you can ssh your pc)
cons of this method:
this method does not relinquish the resources used by the opened python session until you actually exist the python interpreter.

To save input and output on XUbuntu:
In XWindows, run iPython from the Xfce terminal app
click Terminal in the top menu bar and look for save contents in the dropdown
I find this saves the input and output, going all the way back to when I opened the terminal. This is not ipython specific, and would work with ssh sessions or other tasks run from the terminal window.

You can use built-in function open: I use it in all of my
programs in which I need to store some history (Including Calculator, etc.)
for example:
#gk-test.py or anything else would do
try: # use the try loop only if you haven't created the history file outside program
username = open("history.txt").readline().strip("\n")
user_age = open("history.txt").readlines()[1].strip("\n")
except FileNotFoundError:
username = input("Enter Username: ")
user_age = input("Enter User's Age: ")
open("history.txt", "w").write(f"{username}\n{user_age}")
#Rest of the code is secret! try it your own!
I would thank to all of them who liked my comments! Thank you for reading this!

Save and run at the same time in Vim

I do a lot of Python quick simulation stuff and I'm constantly saving (:w) and then running (:!!). Is there a way to combine these actions?
Maybe a "save and run" command.

Okay, the simplest form of what you're looking for is the pipe command. It allows you to run multiple cmdline commands on the same line. In your case, the two commands are write `w` and execute current file `! %:p`. If you have a specific command you run for you current file, the second command becomes, e.g. `!python %:p`. So, the simplest answer to you question becomes:
:w | ! %:p
^ ^ ^
| | |--Execute current file
| |--Chain two commands
|--Save current file
One last thing to note is that not all commands can be chained. According to the Vim docs, certain commands accept a pipe as an argument, and thus break the chain...

Option 1:
Write a function similar to this and place it in your startup settings:
function myex()
execute ':w'
execute ':!!'
endfunction
You could even map a key combination to it -- look at the documentation.
Option 2 (better):
Look at the documentation for remapping keystrokes - you may be able to accomplish it through a simple key remap. The following works, but has "filename.py" hardcoded. Perhaps you can dig in and figure out how to replace that with the current file?
:map <F2> <Esc>:w<CR>:!filename.py<CR>
After mapping that, you can just press F2 in command mode.
imap, vmap, etc... are mappings in different modes. The above only applies to command mode. The following should work in insert mode also:
:imap <F2> <Esc>:w<CR>:!filename.py<CR>a
Section 40.1 of the Vim manual is very helpful.

Use the autowrite option:
:set autowrite
Write the contents of the file, if it has been modified, on each :next, :rewind, :last, :first, :previous, :stop, :suspend, :tag, :!, :make, CTRL-] and CTRL-^ command [...]

Here you go:
:nmap <F1> :w<cr>:!%<cr>
Save and run (you have to be in n mode though - just add esc and a for i mode).

Command combination seems to work through the | character, so perhaps something like aliasing :w|!your-command-here to a distinct key combination.

Another possibility:
au BufWriteCmd *.py write | !!
Though this will run every time you save, which might not be what you want.

In Vim, you could simply redirect any range of your current buffer to an external command (be it Bash, the Python interpreter, or you own Python script).
# Redirect whole buffer to 'python'
:%w !python
Suppose your current buffer contains two lines as below,
import numpy as np
print np.arange(12).reshape(3, 4)
then :%w !python will run it, be it saved or not. And print something like below on your terminal,
[[ 0 1 2 3]
[ 4 5 6 7]
[ 8 9 10 11]]
Of course, you could make something persistent, for example, some keymaps.
nnoremap <F8> :.w !python<CR>
vnoremap <F8> :w !python<CR>
The first one, run the current line. The second one, run the visual selection, via the Python interpreter.
#!! Be careful, in Vim ':w!python' and ':.w !python' are very different. The
first write (create or overwrite) a file named 'python' with contents of
current buffer, and the second redirects the selected cmdline range (here dot .,
which mean current line) to external command (here 'python').
For cmdline range, see
:h cmdline-ranges
Not the below one, which concerning normal command, not cmdline one.
:h command-range
This was inspired by Execute current line in Bash from Vim.

This is what I put in my .vimrc file and works like a charm:
nnoremap <leader>r :w<CR>:!!<CR>
Of course you need to run your shell command once before this, so it knows what command to execute.
Example:
:!node ./test.js

I got the following from the Vim tips wiki:
command! -complete=file -nargs=+ shell call s:runshellcommand(<q-args>)
function! s:runshellcommand(cmdline)
botright vnew
setlocal buftype=nofile bufhidden=wipe nobuflisted noswapfile nowrap
call setline(1,a:cmdline)
call setline(2,substitute(a:cmdline,'.','=','g'))
execute 'silent $read !'.escape(a:cmdline,'%#')
setlocal nomodifiable
1
endfunction
But I changed new to vnew on the third line, and then for Python I have the following:
map <F9> :w:Shell python %<cr><c-w>
Hitting F9 saves, runs, and dumps the output into a new vertically split scratch buffer, for easy yanking, saving, etc. It also hits c-w so I only have to press h/c to close it and move back to my code.

Try making it inside the Bash.
In case of a C file called t.c, this is very convenient:
vi t.c && cc t.c -o t && ./t
The and symbols (&&) ensure that one error message breaks the command chain.
For Python this might be even easier:
vi t.py && python t.py

This will work in insert mode too:
" F5 => Save & Run python3 "
nnoremap <F5> :w <CR> :!sh -c 'python3 %' <CR>
inoremap <F5> <Esc> :w <CR> :!sh -c 'python3 %' <CR>
I use it all the time to test stuff that is just too long to retype in the interpreter.

Consider switching to IDLE. F5 does everything.
Consider switching to Komodo IDE. You can define a command so that F5 does everything.