Develop Reference

run all lines with !python commands in notebook

I have a text file which contains lines that writes !python commands with 2 arguments.
For example
!python example.py 12345.mp4 12345.pkl
!python example.py 111.mp4 111.pkl
!python example.py 123.mp4 123.pkl
!python example.py 44441.mp4 44441.pkl
!python example.py 333.mp4 333.pkl
...
The thing is I want to run all those lines in notebook environment (of Microsoft Azure ML Notebook, or Google Colab). When I copy paste only a few lines (around 500) allowed to be pasted in a notebook code block and I have tens of thousands of lines. Is there a way to achieve this?
I have thought of using for loops to reproduce the text file, but I can't run !python commands inside of python for loop as far as i know.
Edit: I also feel like I have to add these mp4 files are in the same folder with the python code and my text containing those lines. So I want to run example.py for all files in a single folder, and with the argument that changes its .mp4 extension to .pkl (because that acts as name of my output from the command). Maybe now a better solution which runs faster can be made.
And my example.py file can be found here:
https://github.com/open-mmlab/mmaction2/blob/90fc8440961987b7fe3ee99109e2c633c4e30158/tools/data/skeleton/ntu_pose_extraction.py

What you are asking seems misdirected. Running the commands specifically in a notebook only makes sense if each command produces some output which you want to display in the notebook; and even then, if there are more than a few, you want to automate things.
Either way, a simple shell script will easily loop over all the files.
#!/bin/sh
for f in *.mp4; do
python example.py "$f" "${f%.mp4}.pki"
done
If you really insist on running the above from a notebook, saving it in a file (say, allmp4) and running chmod +x on that file will let you run it with ! at any time (simply ! ./allmp4).
(The above instructions are OS-dependent; if you are running your notebook on Windows, the commands will be different, and sometimes bewildering to the point where you probably want to remove Windows.)
Equivalently, anything you can put in a script can be run interactively; depending on the exact notebook, you might not have access to a full shell in ! commands, in which case you can get one with sh -c '... your commands ...'. In general, newlines can be replaced with semicolons in shell scripts, though there are a few contexts where newlines translate to just whitespace (like after then and do).
Quite similarly, you can run python -c '... your python code ...' though complex Python code is hard to serialize into a one-liner. But then, your notebook already runs Python, so you can just put your loop in a cell, and run that.
from pathlib import Path
import subprocess
for f in Path(".").glob("*.mp4"):
subprocess.run(
["python", "example.py",
str(f), str(f.with_suffix(".pkl"))],
check=True, text=True)
... though running Python as a subprocess of itself is often inefficient and clumsy; if you can import example and run its main function directly, you have more control (in particular around error handling), and more opportunities to use multiprocessing or other facilities for parallel processing etc. If this requires you to refactor example.py somewhat, perhaps weigh reusability against immediate utility - if the task is a one-off, getting it done quickly might be more important than getting it done right.

while running thousands of python interpreters seems like a really bad design as each interpreter needs (a non-negligable amount of) time to start, you can just remove the explaimation mark and run it using os.system.
import os
with open("file.txt",'r') as f:
for line in f:
command = line.strip()[1:] # remove ! and the \r\n
os.system(command)
which will take a few months to finish if you are starting tens of thousands of python interpreters, you are much better off running all the code inside a single interpreter in multiple processes using multiprocessing if you know what the file does.

Can I run a portion of a python script from command line?

I have a single python script with many thousands of lines of code. I would like to run smaller chunks of the script, say a few hundred lines, to make sure they are working unitarily.
I'm using Anaconda Prompt with Anaconda 3 on Windows 10 to run my code and for some reason, running python and pasting chunks of code into the prompt is very slow. To get around having to wait for the slow paste into Anaconda Prompt, I was thinking it could be beneficial if I could run only just a portion of the code from the command line.
I've considered turning each chunk into a function but the issue is I have to import and that's not from the command line. Also, each really isn't a function per se.
EDIT: A good point was brought up for how to run a function from the command line making the above statement untrue. But again each of these sections isn't exactly a function in my opinion.

If you split into functions, then you can do this to run from the shell.
python -c "import bigfile; print(bigfile.func(arg1, arg2, ...))"

Sure. Assuming you're in a POSIX-style shell and you want to run, say, lines 100 to 400, you can use sed
sed -n '100,400 p' my_python_file.py | python
You can read all about the sed command in this tutorial. If you're on Windows, you can use the Linux subsystem or Cygwin to do the same.

Does Ruby have a version of `python -i`?

I've been looking for a while, but I haven't found anything in Ruby like python's -i flag.
Common behaviour for me if I'm testing something is to run the unfinished python script with a -i flag so that I can see and play around with the values in each variable.
If I try irb <file>, it still terminates at EOF, and, obviously ruby <file> doesn't work either. Is there a command-line flag that I'm missing, or some other way this functionality can be achieved?
Edit: Added an explanation of what kind of functionality I'm talking about.
Current Behaviour in Python
file.py
a = 1
Command Prompt
$ python -i file.py
>>> a
1
As you can see, the value of the variable a is available in the console too.

You can use irb -r ./filename.rb (-r for "require"), which should basically do the same as python -i ./filename.py.
Edit to better answer the refined question:
Actually, irb -r ./filename.rb does the equivalent of running irb and subsequently running
irb(main):001:0> require './filename.rb'. Thus, local variables from filename.rb do not end up in scope for inspection.
python -i ./filename.py seems to do the equivalent of adding binding.irb to the last line of the file and then running it with ruby ./filename.rb. There seems to be no one-liner equivalent to achieve this exact behaviour for ruby.

Is there a command-line flag that I'm missing, or some other way this functionality can be achieved?
Yes, there are both. I'll cover an "other way".
Starting with ruby 2.5, you can put a binding.irb in some place of your code and then the program will go into an interactive console at that point.
% cat stop.rb
puts 'hello'
binding.irb
Then
% ruby stop.rb
hello
From: stop.rb # line 3 :
1: puts 'hello'
2:
=> 3: binding.irb
irb(main):001:0>
It was possible for a long time before, with pry. But now it's in the standard package.

You can use the command irb. When that has started you can load and execute any ruby file with load './filename.rb'

Running pytest inside emacs results in ugly output

If I run pytest inside of Emacs, using M-X compile, $TERM gets set to "dumb", and this seems to confuse pytest's attempt to draw a line of =s across the full width of the screen. It gets the width one too high, resulting in hard to read output with extra folded lines.
Running unset COLUMNS; pytest helps a little, but it's still trying to do some fancy overwriting by issuing raw carriage-returns, and messing that up. I've also tried setting various values for $TERM (ansi, glass-tty, etc). I even tried, unset TERM.
Any way to convince pytest to just produce dumb output and not try to do any fancy output formatting?
I'm also open to ideas on how to pistol-whip emacs into setting the environment correctly :-)
I'm running:
GNU Emacs 22.1.1
Python 3.6.2
pytest version 3.4.0
MacOS 10.12.6 (16G1212)
Terminal Version 2.7.3 (388.1.1)

Try pytest.el, or at least check how it invokes py.test. I use it with Spacemacs (it's included in the Python layer), and have never seen any issues with its terminal handling.

Python open default terminal, execute commands, keep open, AND then allow user-input

I'm wanting to open a terminal from a Python script (not one marked as executable, but actually doing python3 myscript.py to run it), have the terminal run commands, and then keep the terminal open and let the user type commands into it.
EDIT (as suggested): I am primarily needing this for Linux (I'm using Xubuntu, Ubuntu and stuff like that). It would be really nice to know Windows 7/8 and Mac methods, too, since I'd like a cross-platform solution in the long-run. Input for any system would be appreciated, however.
Just so people know some useful stuff pertaining to this, here's some code that may be difficult to come up with without some research. This doesn't allow user-input, but it does keep the window open. The code is specifically for Linux:
import subprocess, shlex;
myFilePathString="/home/asdf asdf/file.py";
params=shlex.split('x-terminal-emulator -e bash -c "python3 \''+myFilePathString+'\'; echo \'(Press any key to exit the terminal emulator.)\'; read -n 1 -s"');
subprocess.call(params);
To open it with the Python interpreter running afterward, which is about as good, if not better than what I'm looking for, try this:
import subprocess, shlex;
myFilePathString="/home/asdf asdf/file.py";
params=shlex.split('x-terminal-emulator -e bash -c "python3 -i \''+myFilePathString+'\'"');
subprocess.call(params);
I say these examples may take some time to come up with because passing parameters to bash, which is being opened within another command can be problematic without taking a few steps. Plus, you need to know to use to quotes in the right places, or else, for example, if there's a space in your file path, then you'll have problems and might not know why.
EDIT: For clarity (and part of the answer), I found out that there's a standard way to do this in Windows:
cmd /K [whatever your commands are]
So, if you don't know what I mean try that and see what happens. Here's the URL where I found the information: http://ss64.com/nt/cmd.html