I have been trying to debug a massive PyTorch model in VScode.
The starting point of the code first processes a configuration file, and then runs a subprocess containing the cofigs.
The issue is that after calling subprocess.call functions, the VScode, the code is executed in an external sub-process, which does not allow to use breakpoint.
example code:
def main():
args = parse_args()
cmd = construct_cmd(args)
subprocess.call(cmd, shell=True)
where cmd is the string command to be executed, but after this line, all breakpoints are ignored (probably because a sub-process runs this command.
Any solutions how to solve this?
Related
I'm attempting to run a Linux script through Python's subprocess module. Below is the subprocess command:
result = subprocess.run(['/dir/scripts/0_texts.sh'], shell=True)
print(result)
Here is the 0_texts.sh script file:
cd /dir/TXTs
pylanguagetool text_0.txt > comments_0.txt
The subprocess command executes the script file, writing a new comments_0.txt in the correct directory. However, there's an error in the execution. The comments_0.txt contains an error of "input file is required", and the subprocess result returns returncode=2. When I run the pylanguagetool text_0.txt > comments_0.txt directly in the terminal the command executes properly, with the comments_0.txt written with the proper input file of text_0.txt.
Any suggestions on what I'm missing?
There is some ambiguity here in that it's not obvious which shell is run each time 0_texts.sh is invoked, and whether it has the values you expect of environment variables like PATH, which could result in a different copy of pylanguagetool running from when you call it at the command line.
First I'd suggest removing the shell=True option in subprocess.run, which is only involving another, potentially different shell here. Next I would change subprocess.run(['/dir/scripts/0_texts.sh']) to subprocess.run(['bash', '/dir/scripts/0_texts.sh']) (or whichever shell you wanted to run, probably bash or dash) to remove that source of ambiguity. Finally, you can try using type pylanguagetool in the script, invoking pylanguagetool with its full path, or calling bash /dir/scripts/0_texts.sh from your terminal to debug the situation further.
A bigger-picture issue is, pyLanguageTool is a Python library, so you're almost certainly going to be better off calling its functions from your original Python script directly instead of using a shell script as an intermediary.
A few days ago I was getting issues trying to run Tensorflow models with CUDA enabled and for a long time I couldn't resolve in large because PyCharm displayed completely unhelpful message
"Process finished with exit code -1073740791 (0xC0000409)"
I then launched VSCode and ran the same code in PowerShell and got a nice and extensive error message which allowed me to resolve all issues within half hour (same when running in cmd too). Other outputs are also somewhat different. So this makes me assume that PyCharm runs scripts in some type of its own terminal rather than relying on cmd or PowerShell.
Does anyone know if this is the case?
So this makes me assume that PyCharm runs scripts in some type of its own terminal rather than relying on cmd or PowerShell.
Not necessarily. Just because PyCharm displays a custom error message doesn't mean it doesn't rely on cmd. Here's a Python script that simulates what PyCharm does using the cmd:
# So we can launch a process from Python
import subprocess as sp
# Launches a Python process for a specific file
# `stdout=sp.DEVNULL` suppresses the process output
# so that's why there is no detailed error message
child = sp.Popen(["python", "file.py"], stdout=sp.DEVNULL)
# Start the Python process
process = child.communicate()[0]
# Returns the process exit code
# If the process finishes without errors, it'll return 0
# otherwise, it'll return a "random" value
exit_code = process.returncode
# Displays to stdout the completely unhelpful message
print(f"Process finished with exit code {exit_code} ({hex(exit_code)})")
Either way, here's what PyCharm says:
Initially, the terminal emulator runs with your default system shell, but it supports many other shells such as Windows PowerShell, Command Prompt cmd.exe, sh, bash, zsh, csh, and so on.
I am trying to write a simple python script that will open the Windows cmd line, change to a specified directory, and then input the text: 'Test.exe -blah -blahblah etc...' in order to run my Test.exe executable with my specified parameters from the cmd line.
The code I have so far is the following:
import subprocess
subprocess.Popen(r'C:\Windows\System32\cmd.exe', cwd=r'C:\PythonTestScripts')
This code successfully launches the windows cmd.exe, and changes the directory to the specified cwd above, but I have no idea how to pass text into the cmd window from Python.
I have tried passing it as a string argument within the subprocess.Popen brackets, I have also tried assigning PIPE to the stdin and stdout with not much luck. I am familiar with simple coding from Uni, but I am not familiar with Python's syntax or scripting.
Any help is greatly appreciated.
One of your issues may be that cmd.exe doesn't exit once you've started it.
I was successful passing the /C argument to cmd.exe, which tell it to exit after processing the command.
This works:
from subprocess import Popen,PIPE
my_command = "dir *.log"
with Popen([r"C:\Windows\System32\cmd.exe","/C",my_command], stdout=PIPE, cwd=r'C:\Temp') as proc:
print(proc.stdout.read().decode().replace("\r\n","\n"))
Edit:
The original intent of this question was to find a way to launch an interactive ssh session via a Python script. I'd tried subprocess.call() before and had gotten a Killed response before anything was output onto the terminal. I just assumed this was an issue/limitation with the subprocess module instead of an issue somewhere else.This was found not to be the case when I ran the script on a non-resource limited machine and it worked fine.
This then turned the question into: How can I run an interactive ssh session with whatever resource limitations were preventing it from running?
Shoutout to Charles Duffy who was a huge help in trying to diagnose all of this .
Below is the original question:
Background:
So I have a script that is currently written in bash. It parses the output of a few console functions and then opens up an ssh session based on those parsed outputs.
It currently works fine, but I'd like to expand it's capabilities a bit by adding some flag arguments to it. I've worked with argparse before and thoroughly enjoyed it. I tried to do some flag work in bash, and let's just say it leaves much to be desired.
The Actual Question:
Is it possible to have python to do stuff in a console and then put the user in that console?
Something like using subprocess to run a series of commands onto the currently viewed console? This in contrast to how subprocess normally runs, where it runs commands and then shuts the intermediate console down
Specific Example because I'm not sure if what I'm describing makes sense:
So here's a basic run down of the functionality I was wanting:
Run a python script
Have that script run some console command and parse the output
Run the following command:
ssh -t $correctnode "cd /local_scratch/pbs.$jobid; bash -l"
This command will ssh to the $correctnode, change directory, and then leave a bash window in that node open.
I already know how to do parts 1 and 2. It's part three that I can't figure out. Any help would be appreciated.
Edit: Unlike this question, I am not simply trying to run a command. I'm trying to display a shell that is created by a command. Specifically, I want to display a bash shell created through an ssh command.
Context For Readers
The OP is operating on a very resource-constrained (particularly, it appears, process-constrained) jumphost box, where starting an ssh process as a subprocess of python goes over a relevant limit (on number of processes, perhaps?)
Approach A: Replacing The Python Interpreter With Your Interactive Process
Using the exec*() family of system calls causes your original process to no longer be in memory (unlike the fork()+exec*() combination used to start a subprocess while leaving the parent process running), so it doesn't count against the account's limits.
import argparse
import os
try:
from shlex import quote
except ImportError:
from pipes import quote
parser = argparse.ArgumentParser()
parser.add_argument('node')
parser.add_argument('jobid')
args = parser.parse_args()
remote_cmd_str = 'cd /local_scratch/pbs.%s && exec bash -i' % (quote(args.jobid))
local_cmd = [
'/usr/bin/env', 'ssh', '-tt', node, remote_cmd_str
]
os.execv("/usr/bin/env", local_cmd)
Approach B: Generating Shell Commands From Python
If we use Python to generate a shell command, the shell can invoke that command only after the Python process exited, such that we stay under our externally-enforced process limit.
First, a slightly more robust approach at generating eval-able output:
import argparse
try:
from shlex import quote
except ImportError:
from pipes import quote
parser = argparse.ArgumentParser()
parser.add_argument('node')
parser.add_argument('jobid')
args = parser.parse_args()
remoteCmd = ['cd', '/local_scratch/pbs.%s' % (args.jobid)]
remoteCmdStr = ' '.join(quote(x) for x in remoteCmd) + ' && bash -l'
cmd = ['ssh', '-t', args.correctnode, remoteCmdStr]
print(' '.join(pipes.quote(x) for x in cmd)
To run this from a shell, if the above is named as genSshCmd:
#!/bin/sh
eval "$(genSshCmd "$#")"
Note that there are two separate layers of quoting here: One for the local shell running eval, and the second for the remote shell started by SSH. This is critical -- you don't want a jobid of $(rm -rf ~) to actually invoke rm.
This is in no way a real answer, just an illustration to my comment.
Let's say you have a Python script, test.py:
import argparse
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('myarg', nargs="*")
args = parser.parse_args()
print("echo Hello world! My arguments are: " + " ".join(args.myarg))
So, you create a bash wrapper around it, test.sh
set -e
$(python test.py $*)
and this is what you get:
$ bash test.sh
Hello world! My arguments are:
$ bash test.sh one two
Hello world! My arguments are: one two
What is going on here:
python script does not execute commands. Instead, it outputs the commands bash script will run (echo in this example). In your case, the last command will be ssh blabla
bash executes the output of the python script (the $(...) part), passing on all its arguments (the $* part)
you can use argparse inside the python script; if anything is wrong with the arguments, the message will be put to stderr and will not be executed by bash; bash script will stop because of set -e flag
I have always been able to use Python's subprocess.Popen to run bash scripts without any issues.
However, I am now trying to run a bash script with Popen, and then that bash script is trying to run another script. I did not redirect the output that Popen was getting at all, so all of the output would appear on the terminal. I'm using Ubuntu Linux.
However, when the script that is being called by the bash script finishes, the output on the terminal freezes while the rest of the bash script and the python script continues in the background.
I understand that it might not be the smoothest practice to have a python script run a bash script which also runs a bash script, but I'm hoping to fix this issue. I sense that it is an issue with how I'm running the original bash script inside my python script. Here is the code I'm using:
p = subprocess.Popen(["bash", "myScript.sh", param1, param2, param3])
p.wait()
I originally was using shell=True for the Popen, but that was resulting in the same issue. I also tried removing p.wait() but that also does not resolve the issue.
Any ideas? Should I use a different python method to run the bash script?