I have a text file which contains lines that writes !python commands with 2 arguments.
For example
!python example.py 12345.mp4 12345.pkl
!python example.py 111.mp4 111.pkl
!python example.py 123.mp4 123.pkl
!python example.py 44441.mp4 44441.pkl
!python example.py 333.mp4 333.pkl
...
The thing is I want to run all those lines in notebook environment (of Microsoft Azure ML Notebook, or Google Colab). When I copy paste only a few lines (around 500) allowed to be pasted in a notebook code block and I have tens of thousands of lines. Is there a way to achieve this?
I have thought of using for loops to reproduce the text file, but I can't run !python commands inside of python for loop as far as i know.
Edit: I also feel like I have to add these mp4 files are in the same folder with the python code and my text containing those lines. So I want to run example.py for all files in a single folder, and with the argument that changes its .mp4 extension to .pkl (because that acts as name of my output from the command). Maybe now a better solution which runs faster can be made.
And my example.py file can be found here:
https://github.com/open-mmlab/mmaction2/blob/90fc8440961987b7fe3ee99109e2c633c4e30158/tools/data/skeleton/ntu_pose_extraction.py
What you are asking seems misdirected. Running the commands specifically in a notebook only makes sense if each command produces some output which you want to display in the notebook; and even then, if there are more than a few, you want to automate things.
Either way, a simple shell script will easily loop over all the files.
#!/bin/sh
for f in *.mp4; do
python example.py "$f" "${f%.mp4}.pki"
done
If you really insist on running the above from a notebook, saving it in a file (say, allmp4) and running chmod +x on that file will let you run it with ! at any time (simply ! ./allmp4).
(The above instructions are OS-dependent; if you are running your notebook on Windows, the commands will be different, and sometimes bewildering to the point where you probably want to remove Windows.)
Equivalently, anything you can put in a script can be run interactively; depending on the exact notebook, you might not have access to a full shell in ! commands, in which case you can get one with sh -c '... your commands ...'. In general, newlines can be replaced with semicolons in shell scripts, though there are a few contexts where newlines translate to just whitespace (like after then and do).
Quite similarly, you can run python -c '... your python code ...' though complex Python code is hard to serialize into a one-liner. But then, your notebook already runs Python, so you can just put your loop in a cell, and run that.
from pathlib import Path
import subprocess
for f in Path(".").glob("*.mp4"):
subprocess.run(
["python", "example.py",
str(f), str(f.with_suffix(".pkl"))],
check=True, text=True)
... though running Python as a subprocess of itself is often inefficient and clumsy; if you can import example and run its main function directly, you have more control (in particular around error handling), and more opportunities to use multiprocessing or other facilities for parallel processing etc. If this requires you to refactor example.py somewhat, perhaps weigh reusability against immediate utility - if the task is a one-off, getting it done quickly might be more important than getting it done right.
while running thousands of python interpreters seems like a really bad design as each interpreter needs (a non-negligable amount of) time to start, you can just remove the explaimation mark and run it using os.system.
import os
with open("file.txt",'r') as f:
for line in f:
command = line.strip()[1:] # remove ! and the \r\n
os.system(command)
which will take a few months to finish if you are starting tens of thousands of python interpreters, you are much better off running all the code inside a single interpreter in multiple processes using multiprocessing if you know what the file does.
Related
I have a single python script with many thousands of lines of code. I would like to run smaller chunks of the script, say a few hundred lines, to make sure they are working unitarily.
I'm using Anaconda Prompt with Anaconda 3 on Windows 10 to run my code and for some reason, running python and pasting chunks of code into the prompt is very slow. To get around having to wait for the slow paste into Anaconda Prompt, I was thinking it could be beneficial if I could run only just a portion of the code from the command line.
I've considered turning each chunk into a function but the issue is I have to import and that's not from the command line. Also, each really isn't a function per se.
EDIT: A good point was brought up for how to run a function from the command line making the above statement untrue. But again each of these sections isn't exactly a function in my opinion.
If you split into functions, then you can do this to run from the shell.
python -c "import bigfile; print(bigfile.func(arg1, arg2, ...))"
Sure. Assuming you're in a POSIX-style shell and you want to run, say, lines 100 to 400, you can use sed
sed -n '100,400 p' my_python_file.py | python
You can read all about the sed command in this tutorial. If you're on Windows, you can use the Linux subsystem or Cygwin to do the same.
I have several scripts perfectly working by themselves and i would like to create a big pyqt-script which launch those scripts when they are selected.
My only one problem is, every import done on my scripts are not working anymore because of the path. (As the new reference is the pyqt-script and no longer themselves).
I tried to make a movable path but it means changing all my scripts and I would like to be able to "copy/past" my scripts in a file without having to modify them, only the pyqt-script.
I also tried to make a function which open a new terminal with gnome-terminal and launch my script but it is not successful as it keep taking my pyqt-script for reference)
os.system("gnome-terminal --tab -e 'tcsh -c \"python /path/script.py; sleep 10)
Do you have any idea? even if they work in the same terminal.
Thank you for your help.
I apologize if this is a duplicate, I couldn't find any other examples of this question.
I'm trying to write a program for natural language recognition, and I was reading a blog post where someone had attempted to do something similar and it recommended using these two lines to capture the output.
#capture output of script
./get-language.py | tee preptxt
I can't figure out what this code is attempting to do. I assume it's running the get-language file, but that syntax doesn't look correct to me. Could someone point me in the right direction?
On Unix systems you can run executable files with /path/to/my/executable.
This is nothing python specific.
As . is the current working directory, you are executing the get-language.py script located in the current directory.
However, there are two things you need to do, to get this working for your scripts:
1. add a shebang
This is the first line of your script, it tells the shell which program to use.
To use the python interpreter that is first in the PATH use:
#!/usr/bin/env python
2. Add the permissions to make your script executable:
You need to permit execution of your script. This can be done with chmod:
chmod +x myscript.py
The last part is a so called piping operation.
If you call two programs like this:
$ program_a | program_b
The output (stdout) of program_a is fed into program_b.
That's a shell thing, not Python. . is the current directory; you're running get-language.py from there.
This is not a line of python, but rather a shell script.
It is running a script called get-language.py (which I assume was described elsewhere on that blog and does the actual language processing), and then it is "piping" that output through the unix command line program tee, which stores the output in a file.
Piping takes the output of one program, and uses it as the input of another program.
I'm wanting to open a terminal from a Python script (not one marked as executable, but actually doing python3 myscript.py to run it), have the terminal run commands, and then keep the terminal open and let the user type commands into it.
EDIT (as suggested): I am primarily needing this for Linux (I'm using Xubuntu, Ubuntu and stuff like that). It would be really nice to know Windows 7/8 and Mac methods, too, since I'd like a cross-platform solution in the long-run. Input for any system would be appreciated, however.
Just so people know some useful stuff pertaining to this, here's some code that may be difficult to come up with without some research. This doesn't allow user-input, but it does keep the window open. The code is specifically for Linux:
import subprocess, shlex;
myFilePathString="/home/asdf asdf/file.py";
params=shlex.split('x-terminal-emulator -e bash -c "python3 \''+myFilePathString+'\'; echo \'(Press any key to exit the terminal emulator.)\'; read -n 1 -s"');
subprocess.call(params);
To open it with the Python interpreter running afterward, which is about as good, if not better than what I'm looking for, try this:
import subprocess, shlex;
myFilePathString="/home/asdf asdf/file.py";
params=shlex.split('x-terminal-emulator -e bash -c "python3 -i \''+myFilePathString+'\'"');
subprocess.call(params);
I say these examples may take some time to come up with because passing parameters to bash, which is being opened within another command can be problematic without taking a few steps. Plus, you need to know to use to quotes in the right places, or else, for example, if there's a space in your file path, then you'll have problems and might not know why.
EDIT: For clarity (and part of the answer), I found out that there's a standard way to do this in Windows:
cmd /K [whatever your commands are]
So, if you don't know what I mean try that and see what happens. Here's the URL where I found the information: http://ss64.com/nt/cmd.html
I find that when I start the python shell I have a bunch of commands I always type to get into the state I want. It is tiresome to keep re-typing these commands, so I have bundled them into a script. Now I just type:
execfile('script.py')
as soon as I enter the shell, and it goes through all the steps to get me to the state I need to be in.
Now I'd like to take it one step further. How can I get the python shell to automatically run script.py every time I start the shell, so I don't have to keep re-typing even that one line?
Here's a way without having to mess with environment variables:
For example, if I had a script with the following in it called script.py:
#!/usr/bin/env python
print("example")
I could tell python to run this before bringing me to the interpreter with the -i flag.
$ python -i script.py
example
>>>
I think you're looking for the PYTHONSTARTUP environment variable
I'd suggest you to use IPython, if possible. It gives tones of great features, and autoexec is only one of them. But of course, correct answer is mentioned by #mgilston
Create a file called usercustomize.py, and place it in your USER_SITE directory (which you can find as import site; site._script(). This file will be executed every time you start an interpreter.
There's a similar file called sitecustomize.py which is executed anytime anyone starts Python.
This is a Windows solution, but I'm sure a Unix equivalent could be done. I created a file \MyPythonCode\autoexec.py (the last line of which is a print("Autoexec Complete")) and a BAT file MyPython.Bat which has:
cd \myPythonCode
python -i autoexec.py
I just use the command: mypython
to start the Python environment every time, now.