I would like to be able to allow a user to view the output of a long-running GCI script as it is generated rather than after the script is complete. However even when I explicitly flush STDOUT the server seems to wait for the script to complete before sending the response to the client. This is on a Linux server running Apache 2.2.9.
Example python CGI:
#!/usr/bin/python
import time
import sys
print "Content-type: text/plain"
print
for i in range(1, 10):
print i
sys.stdout.flush()
time.sleep(1)
print "Done."
Similar example in perl:
#!/usr/bin/perl
print "Content-type: text/plain\n\n";
for ($i = 1; $i <= 10 ; $i++) {
print "$i\n";
sleep(1);
}
print "Done.";
This link says as of Apache 1.3 CGI output should be unbuffered (but this might apply only to Apache 1.x): http://httpd.apache.org/docs/1.3/misc/FAQ-F.html#nph-scripts
Any ideas?
Randal Schwartz's article Watching long processes through CGI explains a different (and IMHO, better) way of watching a long running process.
Flushing STDOUT can help. For example, the following Perl program should work as intended:
#!/usr/bin/perl
use strict;
use warnings;
local $| = 1;
print "Content-type: text/plain\n\n";
for ( my $i = 1 ; $i <= 10 ; $i++ ) {
print "$i\n";
sleep(1);
}
print "Done.";
You must put your push script into a special directory wich contain a special .htaccess
with this environnment specs:
Options +ExecCGI
AddHandler cgi-script .cgi .sh .pl .py
SetEnvIfNoCase Content-Type \
"^multipart/form-data;" "MODSEC_NOPOSTBUFFERING=Do not buffer file uploads"
SetEnv no-gzip dont-vary
According to CGI::Push,
Apache web server from version 1.3b2
on does not need server push scripts
installed as NPH scripts: the -nph
parameter to do_push() may be set to a
false value to disable the extra
headers needed by an NPH script.
You just have to find do_push equivalent in python.
Edit: Take a look at CherryPy: Streaming the response body.
When you set the config entry "response.stream" to True (and use "yield") CherryPy manages the conversation between the HTTP server and your code like this:
(source: cherrypy.org)
Related
I'm a software tester, trying to verify that the log on a remote QNX (a BSD variant) machine will contain the correct entries after specific actions are taken. I am able to list the contents of the directory in which the log resides, and use that information in the command to read (really want to use tail -n XX <file>) the file. So far, I always get a "(No such file or directory)" when trying to read the file.
We are using Froglogic Squish for automated testing, because the Windows UI (that interacts with the server piece on QNX) is built using Qt extensions for standard Windows elements. Squish uses Python 2.7, so I am using Python 2.7.
I am using paramiko for the SSH connection to the QNX server. This has worked great for sending commands to the simulator piece that also runs on the QNX server.
So, here's the code. Some descriptive names have been changed to avoid upsetting my employer.
import sys
import time
import select
sys.path.append(r"C:\Python27\Lib\site-packages")
sys.path.append(r"C:\Python27\Lib\site-packages\pip\_vendor")
import paramiko
# Import SSH configuration variables
ssh_host = 'vvv.xxx.yyy.zzz'
thelog_dir = "/logs/the/"
ssh_user = 'un'
ssh_pw = 'pw'
def execute_Command(fullCmd):
outptLines = []
#
# Try to connect to the host.
# Retry a few times if it fails.
#
i = 1
while True:
try:
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(ssh_host, 22, ssh_user, ssh_pw)
break
except paramiko.AuthenticationException:
log ("Authentication failed when connecting to %s" % ssh_host)
return 1
except:
log ("Could not SSH to %s, waiting for it to start" % ssh_host)
i += 1
time.sleep(2)
# If we could not connect within time limit
if i == 30:
log ("Could not connect to %s. Giving up" % ssh_host)
return 1
# Send the command (non-blocking?)
stdin, stdout, stderr = ssh.exec_command(fullCmd, get_pty=True)
for line in iter(stdout.readline, ""):
outptLines.append(line)
#
# Disconnect from the host
#
ssh.close()
return outptLines
def get_Latest_Log():
fullCmd = "ls -1 %s | grep the_2" %thelog_dir
files = execute_Command(fullCmd)
theFile = files[-1]
return theFile
def main():
numLines = 20
theLog = get_Latest_Log()
print("\n\nThe latest log is %s\n\n" %theLog)
fullCmd = "cd /logs/the; tail -n 20 /logs/the/%s" %theLog
#fullCmd = "tail -n 20 /logs/the/%s" %theLog
print fullCmd
logLines = execute_Command(fullCmd)
for line in logLines:
print line
if __name__ == "__main__":
# execute only if run as a script
main()
I have tried to read the file using both tail and cat. I have also tried to get and open the file using Paramiko's SFTP client.
In all cases, the response of trying to read the file fails -- despite the fact that listing the contents of the directory works fine. (?!) And BTW, the log file is supposed to be readable by 'world'. Permissions are -rw-rw-r--.
The output I get is:
"C:\Users\xsat086\Documents\paramikoTest>python SSH_THE_MsgChk.py
The latest log is the_20210628_115455_205.log
cd /logs/the; tail -n 20 /logs/the/the_20210628_115455_205.log
(No such file or directory)the/the_20210628_115455_205.log"
The file name is correct. If I copy and paste the tail command into an interactive SSH session with the QNX server, it works fine.
Is it something to do with the 'non-interactive' nature of this method of sending commands? I read that some implementations of SSH are built upon a command that offers a very limited environment. I don't see how that would impact this tail command.
Or am I doing something stupid in this code?
I cannot really explain completely, why you get the results you get.
But in general a corrupted output is a result of enabling and not handling terminal emulation. You enable the terminal emulation using get_pty=True. Remove it. You should not use the terminal emulation, when automating command execution.
Related question:
Is there a simple way to get rid of junk values that come when you SSH using Python's Paramiko library and fetch output from CLI of a remote machine?
When running bash commands from Python to a webpage, succesful output to the webpage occurs when no output options are set. However, i'd like to save the output to an xml file and display the output to the webpage. I get an error when attempting to do this:
Code:
#!/usr/bin/python
import subprocess
import os
# Import modules for CGI handling
import cgi, cgitb; cgitb.enable()
import time
# Create instance of FieldStorage
form = cgi.FieldStorage()
# Get data from fields
niktoValue = form.getvalue('niktoInput')
# Get data from fields
print "Content-type:text/html\r\n\r\n"
print "<html>"
print "<head>"
print "</head>"
print "<body>"
print "<h2>Nikto Scan Information for %s </h2>" % (niktoValue)
def bash(command):
return subprocess.check_output(['bash', '-c', command])
def niktoScan():
res = bash("nikto -h %s -output xml" % niktoValue).splitlines()
print(res)
niktoScan()
print "</body>"
print "</html>"
Error 13 is normally "Permission denied".
I suspect your CGI script is running as the _www user rather than as your regular user when you are logged in at your desktop. You can test by running id in the bash script under the http server or looking in your http server config file - maybe /etc/apache2/httpd.conf for the part that looks like this:
User/Group: The name (or #number) of the user/group to run httpd as.
It is usually good practice to create a dedicated user and group for
running httpd, as with most system services.
User _www
Group _www
I am unfamiliar with the nikto program, so you will need to find out from its author how to allow the www user to run it - maybe by adding that user to a Unix group, maybe via sudo, maybe something else.
After reading the uWSGI's documentation on reloading, my understanding was that, for an app that uses lazy-apps, writing w to uWSGI's master FIFO should trigger a restart of all workers (and hence activate changes in the Python code).
However, that doesn't seem to work for me. I need to restart the systemd service (systemctl restart myservice) for code changes to take effect. Am I misunderstanding the documentation, or is there an issue with my setup?
My myservice.service file looks like this:
...
ExecStart=/usr/lib/myservice/virtualenv/bin/uwsgi --ini /etc/myservice/uwsgi.ini
ExecReload=/bin/echo 'w' > /run/myservice/masterfifo
ExecStop=/bin/kill -INT $MAINPID
...
In particular, systemctl reload myservice should write w to the master FIFO. I can see from the logs in systemctl status myservice that the reload was executed, but the responses to HTTP requests tell me that the old code is still active.
My /etc/myservice/uwsgi.ini like this:
[uwsgi]
processes = 16
procname-master = myservice
master-fifo = /run/myservice/masterfifo
touch-chain-reload
listen = 128
thunder-lock
reload-on-as = 4096
limit-as = 8192
max-requests = 2000
; termination options
vacuum
die-on-term
; application
chdir = /usr/lib/myservice
virtualenv = /usr/lib/myservice/virtualenv
module = myservice.uwsgi
callable = app
master
need-app
enable-threads
lazy = True
lazy-apps = True
; logging
logto = /var/log/myservice/uwsgi.log
log-maxsize = 5242880
logdate = [%%Y/%%m/%%d %%H:%%M:%%S]
disable-logging
; stats server
stats-server = :8201
memory-report
; unix socket config (nginx->uwsgi)
socket = /run/myservice/myservice.sock
chown-socket = api
chmod-socket = 660
I'm running version 2.0.19.1 of uWSGI.
All I know about uWSGI is that it exists, but I noticed a mistake here:
ExecReload=/bin/echo 'w' > /run/myservice/masterfifo
The man page explains the problem:
This syntax is inspired by shell syntax, but only the meta-characters
and expansions described in the following paragraphs are understood,
and the expansion of variables is different. Specifically, redirection
using "<", "<<", ">", and ">>", pipes using "|", running programs in
the background using "&", and other elements of shell syntax are not
supported.
In other words, no redirection is taking place and echo will simply receive 3 arguments to print: char w, char > and the string /run/myservice/masterfifo.
Try this instead:
ExecReload=/bin/sh -c '/bin/echo w > /run/myservice/masterfifo'
I have a simple C program which works the following way:
Ask for input
Print it
Ask another input
Print again
Now iam using python to call this program.
import subprocess
sobj = subprocess.Popen("./cprog", stdin = subprocess.PIPE, stdout = subprocess.PIPE)
sobj.stdin.write("2 3\n")
sobj.stdin.close()
sobj.stdout.read()
This works fine. Similarly with communicate its working fine.
But when I try to do something like this it won't work
sobj = subprocess.Popen("./cprog", stdin = subprocess.PIPE, stdout = subprocess.PIPE)
sobj.stdout.readline()
sobj.stdin.write("2 3\n")
sobj.stdin.close()
sobj.stdout.read()
Here are the few things:
1. I saw pexpect but I think we should give what program asks in advance.
2. Can I reopen closed subprocess pipe ?
Iam using the above script as CGI and I don't know why but subprocess.call won't work in that. Can anyone explain why?
EDIT:
Iam doing a web based project where users write code in either C, C++ or JAVA and execute them on browser. So first I thought of using PHP for it but I couldn't find a way to call programs and run them interactively. Then I saw python subprocess module. Everything was working fine in interpreter when I was using subprocess.call. But the same python program when saved it as .cgi and opened it in browser it didn't work. Then I started looking at subprocess.popen. But with this I need to give all the inputs in beginning and then run the code. What I want to do is run an interactive session in browser.
EDIT 2:
So what I want is user runs program in browser and enters input in textbox provided whenever needed and that input is redirected to stdin of subprocess and output based on it.
EDIT 3: cprog.c
#include <stdio.h>
int main() {
int x;
printf("Enter value of x: \n");
scanf("%d", &x);
printf("Value of x: %d\n", x);
return 0;
}
I'm assuming your C application displays a prompt and expects the user to enter their input on the same line, and that in your readline() call above you're trying to get the prompt.
If this is the case, readline() will block forever because it's waiting for a newline character and never seeing it. If you convert this call to a simple read(X) (where X is a number of bytes to read in one go) then you'll probably have better luck, although you should cope with partial input (i.e. loop around collecting input until you've seen the whole prompt). The only other issue you might see is if the C application isn't flushing the output before prompting the user, but I'd expect you to see that problem in the interactive session as well if that were the case.
When running under the context of a webserver like Apache then it's generally a bad idea to use things like subprocess as they involve forking additional processes and that's often quite a tricky thing to manage. This is because the fork process duplicates much of the state of the parent and sometimes this can cause issues. I'm not saying it won't work, I'm just saying you can make some subtle problems for yourself if you're not careful, and it wouldn't surprise me if that's why you're having trouble using subprocess.
To give any more helpful advice, though, you'd need to describe exactly the error you see when you call subprocess. For example, there's quite likely an exception being thrown which will probably be in your webserver logs - reproducing that here would be a good start.
When I run C program directly through terminal its working fine. But when I run the same program with 2nd code i provided above nothing prints.
The reason you don't see any output is that C stdio uses block-buffering when the program is run in non-interactive mode. See my answer that demonstrate several solutions: pty, stdbuf, pexpect. If you can change the C code then you could also fflush the output explicitly or make it unbuffered.
If you can provide all input at once and the output is bounded then you could use .communicate():
from subprocess import Popen, PIPE
p = Popen(["./cprog"], stdin=PIPE, stdout=PIPE, stderr=PIPE,
universal_newlines=True)
out, err = p.communicate("2\n")
So what I want is user runs program in browser and enters input in textbox provided whenever needed and that input is redirected to stdin of subprocess and output based on it.
Based on ws-cli example:
#!/usr/bin/python
"""WebSocket CLI interface.
Install: pip install twisted txws
Run: twistd -ny wscli.py
Visit http://localhost:8080/
"""
import sys
from twisted.application import strports # pip install twisted
from twisted.application import service
from twisted.internet import protocol
from twisted.python import log
from twisted.web.resource import Resource
from twisted.web.server import Site
from twisted.web.static import File
from txws import WebSocketFactory # pip install txws
class Protocol(protocol.Protocol):
def connectionMade(self):
from twisted.internet import reactor
log.msg("launch a new process on each new connection")
self.pp = ProcessProtocol()
self.pp.factory = self
reactor.spawnProcess(self.pp, command, command_args)
def dataReceived(self, data):
log.msg("redirect received data to process' stdin: %r" % data)
self.pp.transport.write(data)
def connectionLost(self, reason):
self.pp.transport.loseConnection()
def _send(self, data):
self.transport.write(data) # send back
class ProcessProtocol(protocol.ProcessProtocol):
def connectionMade(self):
log.msg("connectionMade")
def outReceived(self, data):
log.msg("send stdout back %r" % data)
self._sendback(data)
def errReceived(self, data):
log.msg("send stderr back %r" % data)
self._sendback(data)
def processExited(self, reason):
log.msg("processExited")
self._sendback('program exited')
def processEnded(self, reason):
log.msg("processEnded")
def _sendback(self, data):
self.factory._send(data)
command = './cprog'
command_args = [command]
application = service.Application("ws-cli")
echofactory = protocol.Factory()
echofactory.protocol = Protocol
strports.service("tcp:8076:interface=127.0.0.1",
WebSocketFactory(echofactory)).setServiceParent(application)
resource = Resource()
resource.putChild('', File('index.html'))
strports.service("tcp:8080:interface=127.0.0.1",
Site(resource)).setServiceParent(application)
where index.html:
<!doctype html>
<title>Send input to subprocess using websocket and echo the response</title>
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.7.0/jquery.min.js">
</script>
<script>
// send keys to websocket and echo the response
$(document).ready(function() {
// create websocket
if (! ("WebSocket" in window)) WebSocket = MozWebSocket; // firefox
var socket = new WebSocket("ws://localhost:8076");
// open the socket
socket.onopen = function(event) {
// show server response
socket.onmessage = function(e) {
$("#output").text(e.data);
}
// sent input
$("#entry").keyup(function (e) {
socket.send($("#entry").attr("value")+"\n");
});
}
});
</script>
<pre id=output>Here you should see the output from the command</pre>
<input type=text id=entry value="123">
And cprog.c:
#include <stdio.h>
int main() {
int x = -1;
setbuf(stdout, NULL); // make stdout unbuffered
while (1) {
printf("Enter value of x: \n");
if (scanf("%d", &x) != 1)
return 1;
printf("Value of x: %d\n", x);
}
return 0;
}
Summary: Python cgi script runs as expected when called from a simple python debug server, but fails with 500 Premature end of script headers error when run from ~/public_html/cgi-bin/
Problems
My CGI script works fine when run through a simple python webserver, and I see the right output when navigating to nameofmyhost.com:8080/...
However, when running the same script from my public_html/cgi-bin directory it gives me a 500 premature end of script headers error. What can I do to fix this?
The permissions on the file seem ok:
drwxrwxrwx cgi-bin
-rwxr-xr-x cgi-bin/generate_list.py
This is simple_httpd.py, the simple python webserver
from http.server import HTTPServer, CGIHTTPRequestHandler
port = 8080
httpd = HTTPServer(('', port), CGIHTTPRequestHandler)
print("Starting simple_httpd on port: " + str(httpd.server_port))
httpd.serve_forever()
CGI script generate_list.py:
#! /usr/local/bin/python3
import athletemodel
import yate
import glob
data_files = glob.glob("data/*.txt")
athletes = athletemodel.put_to_store(data_files)
print(yate.start_response())
print(yate.include_header("Coach Kelly's List of Athletes"))
print(yate.start_form("generate_timing_data.py"))
print(yate.para("Select an athlete from the list to work with:"))
for each_athlete in athletes:
print(yate.radio_button("which_athlete", athletes[each_athlete].name))
print (yate.end_form("Select"))
print(yate.include_footer({"Home": "/index.html"}))
I'm guessing I need to maybe explicitly state my directory somewhere, maybe?
PS: I am going through the Head First Python book by Oreilly
Debugging steps
Check that the server can locate my files
Server finds simple html file in public_home [OK]
Check that the CGI script can execute without error.
Content-type: text/html
<html>
<head>
<title>Coach Kelly's List of Athletes</title>
[...]
CGI script ran from command line - outputs as expected [OK]
Check that the server can execute a simple CGI script in the same location
Try a simple CGI script to see if the server is able to execute any CGI scripts at all:
#!/usr/local/bin/python3
print("Content-Type; text/html")
print("")
print("<html><body><h1>hello</h1></body></html>")
Server fails to execute simple CGI script, giving the same error [FAIL]
Fixes
Fix 1: Change the data path to be absolute instead of relative:
- data_files = glob.glob("data/*.txt")
+ data_files = glob.glob("/home/delliott/public_html/webapp/data/*.txt")
The problem is that the base of your school webserver is not the same as the base of the simple_httpd.py server. This means that you will have to provide absolute paths to your data directory instead of relative paths.
Change the following lines:
- data_files = glob.glob("data/*.txt")
+ data_files = glob.glob("/home/<username>/public_html/data/*.txt")
This should now behave as expected (if I understand your problem correctly.)
[Edit]: A quick way to check if your cgi-bin scripts work is to run them from the command line. Log into your school server and try the cgi script ion its own:
$ cd /home/<username>/public_html/
$ python3 my_cgi_script.py
This should print out the html that you expect, or a stack trace.