Develop Reference

Python http.server CGI

Need to run python scrips with CGI options from a local python server
At the moment on my Apache I use CGI to get all the get and post requests anyone does to my python scripts to tell them to do things.
For example if I do a get request to 127.0.0.1:8080?filename=yomomma
My python script should print 'yomomma'
#!/usr/bin/python3
import cgi, os
form = cgi.FieldStorage()
fileitem = form['filename']
print(fileitem)
Heres the server im running in python (I have no idea what im doing apparently)
from http.server import *
from urllib import parse
import os
import cgi
class GetHandler(CGIHTTPRequestHandler):
def do_GET(self):
form = cgi.FieldStorage()
self.send_response(200)
self.send_header('Content-Type', 'text/html; charset=utf-8')
self.end_headers()
self.wfile.write('<meta http-equiv="refresh" content=1; URL=http://127.0.0.1:8080" /><pre>'.encode('utf-8'))
self.wfile.write(str(os.popen('python my_site.py').read()).encode('utf-8'))
if __name__ == '__main__':
from http.server import HTTPServer
server = HTTPServer(('localhost', 8080), GetHandler)
print('Starting server, use <Ctrl-C> to stop')
server.serve_forever()
Id like to be able to point this at any python file and get that python file to read the CGI parameters

It's unclear what you're asking to do here.
If you just want to run CGI scripts on HTTPServer, that's very simple. CGIHTTPRequestHandler is not meant to be subclassed, which is what you have done, and you don't need to rewrite the do_X functons. It simply returns the output of a CGI script like any other server, if it's under a cgi_directories folder. Read this.
So in the main function, you would have:
server = HTTPServer(('localhost', 8080), CGIHTTPRequestHandler)
Then just call a CGI script normally:
127.0.0.1:8080/cgi-bin/myscript.cgi?filename=yomomma
If you want to utilise http.server to handle requests, you need to use BaseHTTPRequestHandler, not CGIHTTPRequestHandler. Getting form data is then slightly different but easy. Read the section of this under "HTTP POST"

Mock S3 server stalls if using HTTP/1.1

I'm writing a test where a simple mock S3 is loaded up in the test environment using http.server.HTTPServer/http.server.BaseHTTPRequestHandler, to test multipart download behaviour involving Boto's S3Transfer.
It works fine, unless I specify that the server uses HTTP/1.1. In this case, it would download 2 8mb parts of a 100mb file, and then just hang. I would like for the mock server to use HTTP/1.1 since that's what the real S3 uses (I believe).
A simplified version of the test is below, that can be run by...
pip3 install boto3
python3 test.py
# test.py
import http.server
import re
import threading
import boto3
from botocore import (
UNSIGNED,
)
from botocore.client import (
Config,
)
length = 100 * 2**20
class MockS3(http.server.BaseHTTPRequestHandler):
# If the below line is commented, the download completes
protocol_version = 'HTTP/1.1'
def do_GET(self):
range_header = self.headers['Range']
match = re.search(r'^bytes=(\d+)-(\d*)', range_header)
start_inclusive_str, end_inclusive_str = match.group(1), match.group(2)
start = int(start_inclusive_str)
end = int(end_inclusive_str) + 1 if end_inclusive_str else length
bytes_to_send = end - start
self.send_response(206)
self.send_header('Content-Length', str(bytes_to_send))
self.end_headers()
self.wfile.write(bytearray(bytes_to_send))
def do_HEAD(self):
self.send_response(200)
self.send_header('Content-Length', length)
self.end_headers()
server_address = ('localhost', 5678)
server = http.server.HTTPServer(server_address, MockS3)
thread = threading.Thread(target=server.serve_forever)
thread.daemon = True
thread.start()
class Writable():
def write(self, data):
pass
s3_client = boto3.client('s3',
endpoint_url='http://localhost:5678',
config=Config(signature_version=UNSIGNED),
)
s3_client.download_fileobj(
Bucket='some',
Key='key',
Fileobj=Writable(),
)
Note that Writable is deliberately not seekable: in my real code, I'm using a non-seekable file-like object.
Yes, moto can be used to to make a mock S3, and I do so for other tests, but for this particular test I would like "real" server. There are custom file objects involved, and want to ensure that S3Transfer, and other code that isn't relevant to this question, behaves together as I expect.
How can I setup a mock S3 server that uses HTTP/1.1 and that S3Transfer can download from?

There is a bug in your threading logic. What you're currently doing is serving on a separate thread, but what you really want to do is concurrently handle requests on multiple threads.
This can be achieved by creating a very dumb HTTP server which just mixes in a threading capabilities:
class ThreadingServer(ThreadingMixIn, HTTPServer):
pass
and serving from this server instead of the base HTTPServer.
As for why this works with HTTP/1.0, the connection was closed after a single request was serviced.

Importing from external servers not files

Is there a Python extension that would allow me to import remote functions and objects with the same syntax that I import now from files? I'm thinking of something like:
from com.coolmaps.drawing.v21 import draw_map
Then when later in my code i call draw_map, that automatically would be translated to mean a remote method call on that given server.
Wouldn't that be cool?

Check modules from that question for remote method call: What is the current
choice for doing RPC in python?
Here is example from SOAPpy:
A simple “Hello World” http SOAP server:
import SOAPpy
def hello():
return "Hello World"
server = SOAPpy.SOAPServer(("localhost", 8080))
server.registerFunction(hello)
server.serve_forever()
And the corresponding client:
import SOAPpy
server = SOAPpy.SOAPProxy("http://localhost:8080/")
print server.hello()
But I don't know if any of them allow to do it with import/from syntax.

Python CGIHTTPServer Default Directories

I've got the following minimal code for a CGI-handling HTTP server, derived from several examples on the inner-tubes:
#!/usr/bin/env python
import BaseHTTPServer
import CGIHTTPServer
import cgitb;
cgitb.enable() # Error reporting
server = BaseHTTPServer.HTTPServer
handler = CGIHTTPServer.CGIHTTPRequestHandler
server_address = ("", 8000)
handler.cgi_directories = [""]
httpd = server(server_address, handler)
httpd.serve_forever()
Yet, when I execute the script and try to run a test script in the same directory via CGI using http://localhost:8000/test.py, I see the text of the script rather than the results of the execution.
Permissions are all set correctly, and the test script itself is not the problem (as I can run it fine using python -m CGIHTTPServer, when the script resides in cgi-bin). I suspect the problem has something to do with the default CGI directories.
How can I get the script to execute?

My suspicions were correct. The examples from which this code is derived showed the wrong way to set the default directory to be the same directory in which the server script resides. To set the default directory in this way, use:
handler.cgi_directories = ["/"]
Caution: This opens up potentially huge security holes if you're not behind any kind of a firewall. This is only an instructive example. Use only with extreme care.

The solution doesn't seem to work (at least for me) if the .cgi_directories requires multiple layers of subdirectories ( ['/db/cgi-bin'] for instance). Subclassing the server and changing the is_cgi def seemed to work. Here's what I added/substituted in your script:
from CGIHTTPServer import _url_collapse_path
class MyCGIHTTPServer(CGIHTTPServer.CGIHTTPRequestHandler):
def is_cgi(self):
collapsed_path = _url_collapse_path(self.path)
for path in self.cgi_directories:
if path in collapsed_path:
dir_sep_index = collapsed_path.rfind(path) + len(path)
head, tail = collapsed_path[:dir_sep_index], collapsed_path[dir_sep_index + 1:]
self.cgi_info = head, tail
return True
return False
server = BaseHTTPServer.HTTPServer
handler = MyCGIHTTPServer

Here is how you would make every .py file on the server a cgi file (you probably don't want that for production/a public server ;):
import BaseHTTPServer
import CGIHTTPServer
import cgitb; cgitb.enable()
server = BaseHTTPServer.HTTPServer
# Treat everything as a cgi file, i.e.
# `handler.cgi_directories = ["*"]` but that is not defined, so we need
class Handler(CGIHTTPServer.CGIHTTPRequestHandler):
def is_cgi(self):
self.cgi_info = '', self.path[1:]
return True
httpd = server(("", 9006), Handler)
httpd.serve_forever()

Using Pyro4 to get the system information

I try to using the Pyro4 on autotesting, but now I confused for some ability for Pyro4.
Does there existed some method to get the system information by the Pyro4 object.
In my ideas, I expose a pyro object that can get the system information, and the remote machine can using this object to show the system information. But on my code, the remote machine always show the server information. I guess that I misunderstand or misuse the Pyro4
Sorry for this stupid question, i'm the newbie for Pyro4. The follow is my sample code.
The server exposes the pyro object
# Server.py
#! /usr/bin/env python
import Pyro4
import os
class Ex(object):
def run(self):
print os.uname()
if __name__ == "__main__":
daemon = Pyro4.Daemon()
ns = Pyro4.locateNS()
uri = daemon.register(Ex())
ns.register("ex", uri)
print uri
daemon.requestLoop()
and the client using the Pyro object
# Remote
#! /usr/bin/env python
import Pyro4
if __name__ == "__main__":
uri = raw_input("input the uri> ")
fn = Pyro4.Proxy(uri)
fn.run()
p.s
I know i can get the os information on the client side, but I want to use the Pyro object to get the information instead client does this itself.

I got the answer!
Just using the Pyro4.utils.flame. This module can provides the ability to using the module or function on remote machine.
Following is the sample code:
The remote machine
python -Wignore -m Pyro4.utils.flameserver --host <yourIP> --port 9999
The host
#! /usr/bin/env python
import Pyro4.utils.flame
if __name__ == "__main__":
flame = Pyro4.utils.flame.connect("192.168.16.29:64024")
os = flame.module("os")
print os.listdir('.')