I have a text file which contains lines that writes !python commands with 2 arguments.
For example
!python example.py 12345.mp4 12345.pkl
!python example.py 111.mp4 111.pkl
!python example.py 123.mp4 123.pkl
!python example.py 44441.mp4 44441.pkl
!python example.py 333.mp4 333.pkl
...
The thing is I want to run all those lines in notebook environment (of Microsoft Azure ML Notebook, or Google Colab). When I copy paste only a few lines (around 500) allowed to be pasted in a notebook code block and I have tens of thousands of lines. Is there a way to achieve this?
I have thought of using for loops to reproduce the text file, but I can't run !python commands inside of python for loop as far as i know.
Edit: I also feel like I have to add these mp4 files are in the same folder with the python code and my text containing those lines. So I want to run example.py for all files in a single folder, and with the argument that changes its .mp4 extension to .pkl (because that acts as name of my output from the command). Maybe now a better solution which runs faster can be made.
And my example.py file can be found here:
https://github.com/open-mmlab/mmaction2/blob/90fc8440961987b7fe3ee99109e2c633c4e30158/tools/data/skeleton/ntu_pose_extraction.py
What you are asking seems misdirected. Running the commands specifically in a notebook only makes sense if each command produces some output which you want to display in the notebook; and even then, if there are more than a few, you want to automate things.
Either way, a simple shell script will easily loop over all the files.
#!/bin/sh
for f in *.mp4; do
python example.py "$f" "${f%.mp4}.pki"
done
If you really insist on running the above from a notebook, saving it in a file (say, allmp4) and running chmod +x on that file will let you run it with ! at any time (simply ! ./allmp4).
(The above instructions are OS-dependent; if you are running your notebook on Windows, the commands will be different, and sometimes bewildering to the point where you probably want to remove Windows.)
Equivalently, anything you can put in a script can be run interactively; depending on the exact notebook, you might not have access to a full shell in ! commands, in which case you can get one with sh -c '... your commands ...'. In general, newlines can be replaced with semicolons in shell scripts, though there are a few contexts where newlines translate to just whitespace (like after then and do).
Quite similarly, you can run python -c '... your python code ...' though complex Python code is hard to serialize into a one-liner. But then, your notebook already runs Python, so you can just put your loop in a cell, and run that.
from pathlib import Path
import subprocess
for f in Path(".").glob("*.mp4"):
subprocess.run(
["python", "example.py",
str(f), str(f.with_suffix(".pkl"))],
check=True, text=True)
... though running Python as a subprocess of itself is often inefficient and clumsy; if you can import example and run its main function directly, you have more control (in particular around error handling), and more opportunities to use multiprocessing or other facilities for parallel processing etc. If this requires you to refactor example.py somewhat, perhaps weigh reusability against immediate utility - if the task is a one-off, getting it done quickly might be more important than getting it done right.
while running thousands of python interpreters seems like a really bad design as each interpreter needs (a non-negligable amount of) time to start, you can just remove the explaimation mark and run it using os.system.
import os
with open("file.txt",'r') as f:
for line in f:
command = line.strip()[1:] # remove ! and the \r\n
os.system(command)
which will take a few months to finish if you are starting tens of thousands of python interpreters, you are much better off running all the code inside a single interpreter in multiple processes using multiprocessing if you know what the file does.
I'm thinking similar to atom with the hydrogen plugin?
When doing data science in Python I tend to get stuck in a "first create it in jupyter notebook, then re-create it in an actual python script so we can go into production". Using Atom i've been able to just create the code directly in a python script, but still have the great interactive features of jupyter.
I was really hoping that jupyterlab would be similar, but as far as i can tell, you only get the interactive features in the python notebooks and not in python scripts?
To write/save
%%writefile myfile.py
write/save cell contents into myfile.py (use -a to append). Another alias: %%file myfile.py
To run
%run myfile.py
run myfile.py and output results in the current cell
To load/import
%load myfile.py
load "import" myfile.py into the current cell
For more magic and help
%lsmagic
list all the other cool cell magic commands.
%COMMAND-NAME?
for help on how to use a certain command. i.e. %run?
You might like using the Spyder IDE for this. You can use it to edit and execute jupyter notebooks and python scripts:
https://docs.spyder-ide.org/current/plugins/notebook.html
I want to use the useful ! call-to-shell feature of ipython in regular python script, i.e. I want the following code:
for i in range(5):
!echo {i}
To work in an in-house python module that can be imported (thus simply using ! will not work).
I've tried using subprocess but wasn't able to get the output of the shell run into the Jupyter notebook, where most of our scripts are running at.
So, what I'm looking for is using ! functionality as an IPython import some how..
How can I do the equivalent of ipython notebook and ipython profile from inside a python script? It should be straightforward but I can't track down the right invocation. (Inter alia I blindly tried IPython.start_kernel() and from IPython.extensions import notebook and variants, but had no luck so far.)
In my case, I can't just launch a subprocess and execute ipython notebook: I'm on a weird configuration where I can run python from the Start menu (Windows 7), but not from the command line or from the script. (To be completely clear: I do know the location of the python executable, but am restricted from executing it directly).
After some sleuthing and reflection, I decided the best solution is to let IPython start notebook itself. So the problem is to simulate what happens when calling ipython from the command line:
% ipython notebook <directory>
IPython is started from the command line via a short python script. I import it but make it think it's run as the __main__ module:
import sys, imp
ipython_path = r"/path/to/ipython-script"
sys.argv = [ ipython_path, "notebook" ]
_ipython = imp.load_source('__main__', ipython_path)
To serve a directory different from the current directory, just add it to sys.argv as an additional argument:
sys.argv = [ ipython_path, "notebook", "path/to/notebooks" ]
Where is the commandline ipython script?
On Windows, IPython is launched with the help of ipython_script.py in the Scripts directory (e.g., C:\Python27\Scripts\ipython_script.py). On OS X, it can be launched by the python script Library/Python/2.7/bin/ipython. (I installed IPython via easy_install; I suppose there might be other configurations.) You can track down your python installation like this:
import IPython
import inspect
inspect.getsourcefile(IPython)
This is the path to the module, not the launcher script. The script will be in a nearby directory.
Is there a GUI for IPython that allows me to open/run/edit Python files? My way of working in IDLE is to have two windows open: the shell and a .py file. I edit the .py file, run it, and interact with the results in the shell.
Is it possible to use IPython like this? Or is there an alternative way of working?
When I'm working with python, I usually have two terminal windows open - one with IPython, and the other with a fairly customized Vim.
Two good resources:
http://blog.dispatched.ch/2009/05/24/vim-as-python-ide/
http://dancingpenguinsoflight.com/2009/02/python-and-vim-make-your-own-ide/
Though it sounds like what you want is IPython's magic function %ed/%edit:
An example of what you can do:
In [72]: %ed
IPython will make a temporary file named: c:\docume~1\wjwe312\locals~1\temp\ipython_edit_ar8veu.py
In the file I put:
x = "Hello World"
print 3
After saving and quitting the file:
Editing... done. Executing edited code...
3
Out[72]: "x = 'Hello world'\nprint 3\n"
In [73]: x
Out[73]: 'Hello world'
You can define functions or anything else - just remember that the contents of the file will be executed when you close it.
Another similar workflow is to cd to the directory containing your Python script that you're editing with your favorite editor. Then you can %run the script from within IPython and you'll have access to everything defined in the file. For instance, if you have the following in the file test.py in your /home/myself directory:
class Tester(object):
def __init__(self):
print "hi"
def knightme(name):
print "Hello, Sir ", name
Then you can do the following:
In [42]: cd /home/myself
/home/myself
In [43]: %run test.py # <Tab> autocomplete also works
In [44]: knightme('John')
Hello, Sir John
In [45]: t = Tester()
Hi
Either a mix or one of those workflows should give you something very similar to the way you're used to working in IDLE.
Spyder, previously known as SPyderlib / Spyder2
Pretty lightweight, fast and support almost all features you will ever need to work with a python project. It can edit and run .py files in an embedded IPython instance and then interact with them, set breakpoints, etc.
full-size
Try Spyder, I have spent all day trying to find an IDE which has the functionality of ipython and Spyder just kicks it out of the park..
Autocomplete is top notch right from install, no config files and all that crap, and it has an Ipython terminal in the corner for you to instantly run your code.
big thumbs up
Take a look at DreamPie. Might be what you are looking for.
Personally, I like PyScripter. Unfortunately, it only works on Windows, but also runs perfectly in Wine.
The latest version of IdleX supports IPython within IDLE, as well as the %edit magic. You can run your files from the IDLE editor within the IPython shell many ways, either by F5 (run everything), F9 (run a selection), or Ctrl+Enter (run a subcode).
sudo apt-get install ipython
Once you are done with installing ipython.
Start ipython from terminal (just hit ipython in the ternminal)
To run ravi.py file all you need to do is
%run ravi.py
If you like the work-flow under Matlab, then you probably should try the following two:
1, Try the combination of Spyder and Vim.
Edit python files in Vim (Spyder can reload the file automatically)
Run the code in Spyder (in the same interpreter, which is important for me):
Use F9 to run the current file
Ctrl+F9 to run the selected block
2, Use Vim + conque-shell. (on google code)
Open your preferred Python interpreter in Vim,
e.g., just :ConqueTermSplit python.
then visual select some Python code
press F9 to paste and run it in the Python interpreter buffer.
Note: a few more:
:ConqueTermVSplit python,
:ConqueTerm python
:ConqueTermVSplit rlwrap python
If your interpretor misses readline, you can use rlwrap.
You might like PySlices...
It's kind of a shell/editor hybrid that lets you save your session as special (barely) modified python files called .pyslice files.
It's now part of wxPython, so just install that (v2.8.11 or later) and run "python -m wx.py.PySlices" on the command line to launch it.
That said, I still end up using an external editor for scripts (geany).
I want to suggest excellent plugin for vim that makes two-way integration between Vim and IPython: vim-ipython.
From project page on http://github.com/ivanov/vim-ipython:
Using this plugin, you can send lines or whole files for IPython to execute, and also get back object introspection and word completions in Vim, like what you get with: object? and object. in IPython.
This plugin has one big limitation: it doesn't support python 3 (it's planned).
Personally, I use what #Wayne suggested, a combination of vim and ipython...
However, if you'd prefer a different approach, take a look at spyder.
As of the latest version (1.1) ipython should be fully integrated. If you download an earlier version, things will work fine with ipython as an external shell, but you won't get a few of spyder's nifty features (like viewing all of the currently defined variables in the workspace window).
Spyder is definitely a bit heavyweight, but it's an interesting project.
Another (very, very, new) similar project to take a look at is iep. It will (sort-of) work with ipython as shell, and I'd be willing to be bet that nicer ipython integration will be along before too long. At any rate, iep is essentially a more lightweight alternative to spyder.
Both of these are oriented towards scientific computing, and so have nice integration with things like matplotlib (and thus can automatically run gui main loops in a seperate thread). They're not quite like "normal" IDE's but they may fill the niche you're looking for quite nicely.
You can use the autoreload module in IPython to automatically reload code.
Open jupyter qtconsole or jupyter console and type:
%load_ext autoreload
%autoreload 2
from your_work_file import *
Now every time you save your_work_file.py, it will be automatically reloaded.
Hint: if you want this to happen automatically, put the followinglines in your ipython_config.py file:
c.InteractiveShellApp.extensions = ['autoreload']
c.InteractiveShellApp.exec_lines = ['%autoreload 2']
Try Ptpython. It has much better integration with VIM. You can directly edit in VIM by just pressing V. It also allows browsing your history.. so you can pretty much code in the shell, and incrementally build up your code.
If you are already familiar with ipython, you can check the advantages of ptpython here:
https://www.youtube.com/watch?v=XDgIDslyAFM
I just use the exclamation mark (!) to run vi as a shell command
In [1]: !vi myScript.py
and when done with editing I just quit vi to get back to the Ipython shell.
To run the script one can then use
In [2]: %run myScript.py
as suggested in another answer and not !python ... because the Python version in ipython might be different from the one in the underlying shell.
If you want to dump some code in a file use the magic %%writefile
In [3]:%%writefile myScript.py
...: print("hello")
...:
...:
Be careful because this will overwrite myScript.py. To append use %%writefile -a.