I have a python script (myscript.py) that I am running on the linux server like below:
python myscript.py
In my script I am using the system call, example below:
os.system("./myprogram.pl -x 5 > results.out")
When I run myscript.py on the server whenever there is os.system call, the command call is visible on the server. Is there a way to hide all the command calls so that they are not displayed when they are called within the program (myscript.py)? (This is not about hiding the output resulted in the command calls within the program)
You can use a subprocess pipe instead of the os.system command:
import subprocess;
NewPipeObject = subprocess.Popen( [ './myprogram.pl -x 5 > results.out' ],
stdout= subprocess.PIPE,
stderr= subprocess.PIPE );
or you can take advantage of the list format of the arguments for the subprocess pipe (it adds a space between each list element) and present them as the following (which will be easier to generalize):
import subprocess;
NewPipeObject = subprocess.Popen( [ './myprogram.pl',
'-x',
'5',
'>',
'results.out' ],
stdout= subprocess.PIPE,
stderr= subprocess.PIPE );
Either way you're suppressing the outputs from stdout and stderr into the pipe object (NewPipeObject).
Use the subprocess module to create a subprocess so you wont see any ouput from the process unless you read from the PIPE directly.
process = subprocess.Popen(['perl', 'myprogram.pl'], stdout = subprocess.PIPE )
You mean, can you conceal the command from showing up in ps, top and other such tools? No.
On some systems, myprogram.pl can itself hide its command line from appearing in ps; for instance in perl:
$0 = 'new program name';
That edits the ps entry rather than removing it. Since it is set by the new process there is a race condition where someone might see it before it is changed; and it may be possible (certainly possible for the superuser) to get the same info in other ways. There is no secure way, nor is there intended to be, to conceal your command line entirely from other users.
Related
I keep trying to run this piece of code but everytime I print, it seems to net me nothing in regards to what I see with my output.
p = subprocess.run(["powershell.exe", "C://Users//xxxxx//Documents//betterNetstatOut.ps1"], shell=True, capture_output=True, text=True)
output = p.stdout
print(output)
My PowerShell command is a very basic println at this point:
Write-Output 'Hello world'
but running print on my out seems to return an empty string. I also tried running subprocess.Popen() and subprocess.call() and they all seem to return an empty string instead of 'Hello World'. Eventually, I would like to parse many lines and move them to a dataframe but I am stuck on this one line first.
Your PowerShell command likely produced only stderr output, which is why you saw no output given that you only printed p.stdout - also printing p.stderr would surface any stderr output (which typically contains error messages).
Assuming your script file path is correct, the likeliest explanation for receiving only stderr output is that your effective PowerShell execution policy prevents execution of scripts (.ps1 files), which you can bypass with -ExecutionPolicy Bypass in a call to the Windows PowerShell CLI, powershell.exe.
Additionally:
There's no need for double slashes (//) in your script path; while it still works, / is sufficient.
It's better to use the -File parameter rather than -Command (which is implied) for invoking scripts via the PowerShell CLI - see this answer for more information.
For a predictably execution environment and to avoid unnecessary overhead from loading profiles, using -NoProfile is advisable.
You don't need shell=True, which, due to calling via cmd.exe, only slows your command down.
To put it all together:
import subprocess
p = subprocess.run(
["powershell.exe",
"-NoProfile",
"-ExecutionPolicy", "Bypass",
"-File", "C:/Users/xxxxx/Documents/betterNetstatOut.ps1"],
capture_output=True, text=True
)
print('--- stdout --')
print(p.stdout)
print('--- stderr --')
print(p.stderr)
import subprocess
p = subprocess.run(["powershell.exe", "powershell -ExecutionPolicy Bypass -File", "C://Users//xxxxx//Documents//betterNetstatOut.ps1"], shell=True, capture_output=True, text=True)
print(p.stdout)
I am writing a script which is executing CMD commands on Windows. I use it to parse commands to diferent application. Those commands return some values or errors. How do I force Python/CMD to store whatever command returns (no matter if it's returned value or error) in a variable and force it NOT to print it to console. I tried subprocess and os.system() and all of those I tried allows to store value but when command returns an error, it still is being printed to the console and not stored in a variable.
That is a property of the shell / of cmd and how you call the process. By default there's one input stream (stdin) and two output streams (stdout and stderr) - the latter being the default stream for all errors.
You can direct either or both to one another or to stdin or a file by calling the script appropriately. See https://support.microsoft.com/en-us/help/110930/redirecting-error-messages-from-command-prompt-stderr-stdout
For example
myscript 1> output.msg 2>&1
will direct everything into output.msg, including errors. Now combine that output redirection with writing to a variable; that is explained in this answer.
When executing a command in a shell there is 2 different outputs handlers, stdout and stderr. Usually stdout is used to print normal output and stderr to print errors and warnings.
You can use subprocess.Popen.communicate() to read both stderr and stdout.
import subprocess
p = subprocess.Popen(
"dir",
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
shell=True
)
(stdout, stderr) = p.communicate()
print(stdout.decode('utf-8')) # Standard output
print(stderr.decode('utf-8')) # Standard error
I am trying to run the following bash script in Python and store the readlist output. The readlist that I want to be stored as a python list, is a list of all files in the current directory ending in *concat_001.fastq.
I know it may be easier to do this in python (i.e.
import os
readlist = [f for f in os.listdir(os.getcwd()) if f.endswith("concat_001.fastq")]
readlist = sorted(readlist)
However, this is problematic, as I need Python to sort the list in EXACTLY the same was as bash, and I was finding that bash and Python sort certain things in different orders (eg Python and bash deal with capitalised and uncapitalised things differently - but when I tried
readlist = np.asarray(sorted(flist, key=str.lower))
I still found that two files starting with ML_ and M_ were sorted in different order with bash and Python. Hence trying to run my exact bash script through Python, then to use the sorted list generated with bash in my subsequent Python code.
input_suffix="concat_001.fastq"
ender=`echo $input_suffix | sed "s/concat_001.fastq/\*concat_001.fastq/g" `
readlist="$(echo $ender)"
I have tried
proc = subprocess.call(command1, shell=True, stdout=subprocess.PIPE)
proc = subprocess.call(command2, shell=True, stdout=subprocess.PIPE)
proc = subprocess.Popen(command3, shell=True, stdout=subprocess.PIPE)
But I just get: subprocess.Popen object at 0x7f31cfcd9190
Also - I don't understand the difference between subprocess.call and subprocess.Popen. I have tried both.
Thanks,
Ruth
So your question is a little confusing and does not exactly explain what you want. However, I'll try to give some suggestions to help you update it, or in my effort, answer it.
I will assume the following: your python script is passing to the command line 'input_suffix' and that you want your python program to receive the contents of 'readlist' when the external script finishes.
To make our lives simpler, and allow things to be more complicated, I would make the following bash script to contain your commands:
script.sh
#!/bin/bash
input_suffix=$1
ender=`echo $input_suffix | sed "s/concat_001.fastq/\*concat_001.fastq/g"`
readlist="$(echo $ender)"
echo $readlist
You would execute this as script.sh "concat_001.fastq", where $1 takes in the first argument passed on the command line.
To use python to execute external scripts, as you quite rightly found, you can use subprocess (or as noted by another response, os.system - although subprocess is recommended).
The docs tell you that subprocess.call:
"Wait for command to complete, then return the returncode attribute."
and that
"For more advanced use cases when these do not meet your needs, use the underlying Popen interface."
Given you want to pipe the output from the bash script to your python script, let's use Popen as suggested by the docs. As I posted the other stackoverflow answer, it could look like the following:
import subprocess
from subprocess import Popen, PIPE
# Execute out script and pipe the output to stdout
process = subprocess.Popen(['script.sh', 'concat_001.fastq'],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
# Obtain the standard out, and standard error
stdout, stderr = process.communicate()
and then:
>>> print stdout
*concat_001.fastq
I've got a script parent.py trying to to read stdout from a subprocess sub.py in Python.
The parent parent.py:
#!/usr/bin/python
import subprocess
p = subprocess.Popen("sub.py", stdout=subprocess.PIPE)
print p.stdout.read(1)
And the subprocess, sub.py:
#!/usr/bin/python
print raw_input( "hello world!" )
I would expect running parent.py to print the 'h' from "hello world!". Actually, it hangs. I can only get my expected behaviour by adding -u to sub.py's she-bang line.
This confuses me because the -u switch makes no difference when sub.py is run directly from a shell; the shell is somehow privy to the un-flushed output stream, unlike parent.py.
My goal is to run a C program as the subprocess, so I won't be able to control whether or not it flushes stdout. How is it that a shell has better access to a process's stdout than Python running the same thing from subprocess.Popen? Am I going to be able to read such a stdout stream from a C program that doesn't flush its buffers?
EDIT:
Here is an updated example based on korylprince's comment...
## capitalize.sh ##
#!/bin/sh
while [ 1 ]; do
read s
echo $s | tr '[:lower:]' '[:upper:]'
done
########################################
## parent.py ##
#!/usr/bin/python
from subprocess import Popen, PIPE
# cmd = [ 'capitalize.sh' ] # This would work
cmd = [ 'script', '-q', '-f', '-c', 'capitalize.sh', '/dev/null']
p = Popen(cmd, stdin=PIPE)
p.stdin.write("some string\n")
p.wait()
When running through script, I get steady printing of newlines (and if this were a Python, subprocess, it'd raise an EOFerror).
An alternative is
p = subprocess.Popen(["python", "-u", "sub.py"], stdout=subprocess.PIPE)
or the suggestions here.
My experience is that yes, you will be able to read from most C programs without any extra effort.
The Python interpreter takes extra steps to buffer its output which is why it needs the -u switch to disable output buffering. Your typical C program won't do this.
I haven't run into any program (C or otherwise) other than the Python interpreter that I expected to work and didn't within a subshell.
The reason the shell can read output immediately, regardless of "-u" is because the program you're launching from the shell has its output connected to a TTY. When the stdout is connected to a TTY, it is unbuffered (because it is up to the TTY to buffer). When you launch the python subprocess from within python, you're connecting stdout to a pipe, which means you're at the mercy of the subprocess to flush its output when it feels like it.
If you're looking to do complicated interactions with a subprocess, look into this tutorial.
I tried something like this, but with no effect:
command = "cmd.exe"
proc = subprocess.Popen(command, stdin = subprocess.PIPE, stdout = subprocess.PIPE)
proc.stdin.write("dir c:\\")
how about simply:
import os
os.system('dir c:\\')
You probably want to try something like this:
command = "cmd.exe /C dir C:\\"
I don't think you can pipe into cmd.exe... If you are coming from a unix background, well, cmd.exe has some ugly warts!
EDIT: According to Sven Marnach, you can pipe to cmd.exe. I tried following in a python shell:
>>> import subprocess
>>> proc = subprocess.Popen('cmd.exe', stdin = subprocess.PIPE, stdout = subprocess.PIPE)
>>> stdout, stderr = proc.communicate('dir c:\\')
>>> stdout
'Microsoft Windows [Version 6.1.7600]\r\nCopyright (c) 2009 Microsoft Corporatio
n. All rights reserved.\r\n\r\nC:\\Python25>More? '
As you can see, you still have a bit of work to do (only the first line is returned), but you might be able to get this to work...
Try:
import os
os.popen("Your command here")
Using ' and " at the same time works great for me (Windows 10, python 3)
import os
os.system('"some cmd command here"')
for example to open my web browser I can use this:
os.system(r'"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe"')
(Edit)
for an easier way to open your browser I can use this:
import webbrowser
webbrowser.open('website or leave it alone if you only want to open the
browser')
Try adding a call to proc.stdin.flush() after writing to the pipe and see if things start behaving more as you expect. Explicitly flushing the pipe means you don't need to worry about exactly how the buffering is set up.
Also, don't forget to include a "\n" at the end of your command or your child shell will sit there at the prompt waiting for completion of the command entry.
I wrote about using Popen to manipulate an external shell instance in more detail at: Running three commands in the same process with Python
As was the case in that question, this trick can be valuable if you need to maintain shell state across multiple out-of-process invocations on a Windows machine.
Taking some inspiration from Daren Thomas's answer (and edit), try this:
proc = subprocess.Popen('dir C:\\', shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
out, err = proc.communicate()
out will now contain the text output.
They key nugget here is that the subprocess module already provides you shell integration with shell=True, so you don't need to call cmd.exe directly.
As a reminder, if you're in Python 3, this is going to be bytes, so you may want to do out.decode() to convert to a string.
Why do you want to call cmd.exe ? cmd.exe is a command line (shell). If you want to change directory, use os.chdir("C:\\"). Try not to call external commands if Python can provide it. In fact, most operating system commands are provide through the os module (and sys). I suggest you take a look at os module documentation to see the various methods available.
It's very simple. You need just two lines of code with just using the built-in function and also it takes the input and runs forever until you stop it. Also that 'cmd' in quotes, leave it and don't change it. Here is the code:
import os
os.system('cmd')
Now just run this code and see the whole windows command prompt in your python project!
Here's a way to just execute a command line command and get its output using the subprocess module:
import subprocess
# You can put the parts of your command in the list below or just use a string directly.
command_to_execute = ["echo", "Test"]
run = subprocess.run(command_to_execute, capture_output=True)
print(run.stdout) # the output "Test"
print(run.stderr) # the error part of the output
Just don't forget the capture_output=True argument and you're fine. Also, you will get the output as a binary string (b"something" in Python), but you can easily convert it using run.stdout.decode().
In Python, you can use CMD commands using these lines :
import os
os.system("YOUR_COMMAND_HERE")
Just replace YOUR_COMMAND_HERE with the command you like.
From Python you can do directly using below code
import subprocess
proc = subprocess.check_output('C:\Windows\System32\cmd.exe /k %windir%\System32\\reg.exe ADD HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System /v EnableLUA /t REG_DWORD /d 0 /f' ,stderr=subprocess.STDOUT,shell=True)
print(str(proc))
in first parameter just executed User Account setting you may customize with yours.