Develop Reference

What me need fix in my POST webserver on python?

I have a script to get json and run it on my ubuntu server with a white ip
I accept json, but not completely, and after this acceptance, the script closes the connection and does not work
I think the problem is that I receive packets incorrectly, but why it does not accept the second packet and why it closes in an infinite loop is not clear to me due to my little experience
import os
import socket
from pathlib import Path
from dotenv import load_dotenv
import json
#Init .env
env_path = Path('.') / '.env'
load_dotenv(dotenv_path=env_path)
#create webserver socket
def start_my_server():
socket_server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
socket_server.bind(('ip', port))
socket_server.listen(32768)
print('Working...')
global data
global HDRS
while True:
client_socket, address = socket_server.accept()
data = client_socket.recv(32768).decode('utf-8')
# content = 'Well done'.encode('utf-8')
HDRS = 'HTTP/1.1 200 OK\r\nContent-Type: text/html; charset=utf-8\r\n\r\n'
# content = load_page_from_get_request(data)
client_socket.send(HDRS.encode('utf-8'))
# a = client_socket.send(HDRS.encode('utf-8'))
# print(a, '+'*20)
client_socket.shutdown(socket.SHUT_WR)
load_page_from_get_request(32768)
# print('end')
# socket_server.close()
def load_page_from_get_request(request_data):
HDRS = 'HTTP/1.1 200 OK\r\nContent-Type: text/html; charset=utf-8\r\n\r\n'
HDRS_404 = 'HTTP/1.1 404 OK\r\nContent-Type: text/html; charset=utf-8\r\n\r\n'
try:
with open('data.json', 'w') as output_file:
json.dump(data, output_file)
return HDRS.encode('utf-8')
except EOFError:
return HDRS_404.encode('utf-8')
# try
if __name__ == '__main__':
start_my_server()

The end version of my script
import os
import socket
from pathlib import Path
from dotenv import load_dotenv
import json
import datetime
#Init .env
env_path = Path('.') / '.env'
load_dotenv(dotenv_path=env_path)
ip = os.environ['ip']
port = os.environ['port']
HDRS = 'HTTP/1.1 200 OK\r\nContent-Type: text/html; charset=utf-8\r\n\r\n'
HDRS_404 = 'HTTP/1.1 404 OK\r\nContent-Type: text/html; charset=utf-8\r\n\r\n'
#create webserver socket
def start_my_server():
socket_server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
socket_server.bind((ip, port))
socket_server.listen(32768)
print('Working...')
global data
while True:
client_socket, address = socket_server.accept()
print('loop', address)
data = client_socket.recv(0).decode('utf-8')
# content = load_page_from_get_request(data)
load_page_from_get_request(data)
client_socket.send(HDRS.encode('utf-8'))
client_socket.shutdown(socket.SHUT_WR)
# print('end')
# socket_server.close()
def load_page_from_get_request(request_data):
# HDRS = 'HTTP/1.1 200 OK\r\nContent-Type: text/html; charset=utf-8\r\n\r\n'
# HDRS_404 = 'HTTP/1.1 404 OK\r\nContent-Type: text/html; charset=utf-8\r\n\r\n'
with open(f'data.json{datetime.time}', 'wr') as output_file:
json.dump(data, output_file)
return HDRS.encode('utf-8')
# except EOFError:
# return HDRS_404.encode('utf-8')
# try
print('loop script')
if __name__ == '__main__':
start_my_server()

Retrieving an image from HTTP via python

I am kinda new to Python, but I really want to know, what does that mean.
In my book, I have a code:
import socket
import time
HOST = 'data.pr4e.org'
PORT = 80
mysock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysock.connect((HOST, PORT))
mysock.sendall(b'GET http://data.pr4e.org/cover3.jpg HTTP/1.0\r\n\r\n')
count = 0
picture = b""
while True:
data = mysock.recv(5120)
if len(data) < 1: break
#time.sleep(0.25)
count = count + len(data)
print(len(data), count)
picture = picture + data
mysock.close()
# Look for the end of the header (2 CRLF)
pos = picture.find(b"\r\n\r\n")
print('Header length', pos)
print(picture[:pos].decode())
# Skip past the header and save the picture data
picture = picture[pos+4:]
fhand = open("stuff.jpg", "wb")
fhand.write(picture)
fhand.close()
And the response is:
$ python urljpeg.py
5120 5120
5120 10240
4240 14480
5120 19600
...
5120 214000
3200 217200
5120 222320
5120 227440
3167 230607
Header length 393
HTTP/1.1 200 OK
Date: Wed, 11 Apr 2018 18:54:09 GMT
Server: Apache/2.4.7 (Ubuntu)
Last-Modified: Mon, 15 May 2017 12:27:40 GMT
ETag: "38342-54f8f2e5b6277"
Accept-Ranges: bytes
Content-Length: 230210
Vary: Accept-Encoding
Cache-Control: max-age=0, no-cache, no-store, must-revalidate
Pragma: no-cache
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Connection: close
Content-Type: image/jpeg
I would like to know:
- What do these number in the beginning of the response mean?
- Could anyone recommend a source to read about retrieving image for dummies?

They are the result of
print(len(data), count)
Respectively, the length of the data and the total of data it has received so far.

Since data transfers through a computer network divides the data in packets, you don't receive the whole data at once. Therefore what that code is doing is receiving "chunks" of data and what's printed is the size of the "chunk" and the total amount of data received (by the line print(len(data), count)).
Once all the data has been received, you can extract the picture data. Since you are working with HTTP, the data you receive will be an HTTP response. An HTTP response has two parts a header, which is a series of key-value pairs that give information about the response itself and is what's being printed at the end of your output; and the content which is the data and in your case contains the picture.
In HTTP header and content are separated by two line endings \r\n\r\n. Therefore picture.find(b"\r\n\r\n") looks the position where the header ends and the content (the image) begins.
If all you need to do is retrieve images using HTTP, I would suggest taking a look into the requests package.

python extract few lines from the output

I've created a python class to execute a curl command which in turn generates a token. The code works as expected.The script generates the expected output.Now, need few lines from the below mentioned output and not the entire output.
How do I extract the following lines from the output.
lines starting after
Set-Cookie: SESSION= till =;
. How do I extract these lines.
import requests
import json
from akamai.edgegrid import EdgeGridAuth, EdgeRc
from tabulate import tabulate
import os
import re
import base64
class generate_token(object):
"""
This library is to generate token for the test application
"""
def __init__(self,kwargs):
self.username = 'user'
self.password = 'abcdef'
self.contractName = kwargs.get("Names")
self.contractType = kwargs.get("Type")
self.accountId = kwargs.get("Id")
self.env = kwargs.get("env")
def requestSession(self):
if self.env == "test":
ping = os.popen('ping -c 1 hostname.com').read()
ip_addr = re.findall(r"\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b",ping)[0]
else:
ping = os.popen('ping -c 1 different_hostname.com').read()
ip_addr = re.findall(r"\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b",ping)[0]
url = "https://%s/EdgeAuth/asyncUserLogin" %(ip_addr)
command = "curl -is -v -k -d username=%s -d password=%s --request POST '%s' " %(self.username,self.password,url)
self.session = os.popen(command).read()
if __name__ == "__main__":
initData = {
'Names': 'names',
'Type': 'type',
'Id': 'ID',
'env' : 'test'
}
c = generate_token(kwargs=initData)
result = c.requestSession()
Entire output :
* TCP_NODELAY set
* WARNING: disabling hostname validation also disables SNI.
> POST /EdgeAuth/asyncUserLogin HTTP/1.1
> Host: 143.111.112.202
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Length: 33
> Content-Type: application/x-www-form-urlencoded
>
} [33 bytes data]
* upload completely sent off: 33 out of 33 bytes
< HTTP/1.1 302 Moved Temporarily
< Date: Thu, 26 Jul 2018 15:54:07 GMT
< Server: Apache-Coyote/1.1
< Location: https://192.168.93.201/homeng/view/main
< Content-Length: 0
< Set-Cookie: SESSION=1-7FOmRFhEcci+8fAfh+DvLg==-1bk61eksykUHbWc9kx3LzL0KdkaxyDodxsEsu6siMR/1cpR7yLaD7HtTjXEr+XDowzpYcug4bfKIMjk2/T0aqJiHg5BrbRbnBeZnTcAUIpdyKGvtnmEQkSADoaxvgfYGrxqQctrVPdqJtPziqvIbjO1X8xh0Xtj/ZNvJXaSFs//w9ISNZ5hvIDBygUo+0EuYT8PSTYVPrcBCLaJUkC8ACg==-jdN937gosTsMx8fdaJqP5sA7wKPRgqv1RxRkX65SWeE=;
Path=/; Secure
< Via: 1.1 %{HTTP_HOST}
< X-Content-Type-Options: nosniff
< Content-Type: text/plain
Required output :
17FOmRFhEcci+8fAfh+DvLg==-1bk61eksykUHbWc9kx3LzL0KdkaxyDodxsEsu6siMR/1cpR7yLaD7HtTjXEr+XDowzpYcug4bfKIMjk2/T0aqJiHg5BrbRbnBeZnTcAUIpdyKGvtnmEQkSADoaxvgfYGrxqQctrVPdqJtPziqvIbjO1X8xh0Xtj/ZNvJXaSFs//w9ISNZ5hvIDBygUo+0EuYT8PSTYVPrcBCLaJUkC8ACg==-jdN937gosTsMx8fdaJqP5sA7wKPRgqv1RxRkX65SWeE=;

Don't use os.popen and friends to execute external programs - use the subprocess module for that:
ping = subprocess.check_output(['ping', '-c', '1', 'hostname.com'])
In the same way, don't execute a subprocess just to get a cookie from a url - you're already importing the excellent requests library in your code, why not use it!
r = requests.post(url, data={'username': self.username, 'password': self.password})
print (r.cookies['SESSION'])

Python MJPEG stream don't work outside VLC

I'm creating a streaming mjpeg from some IP cameras,the streaming with VLC works perfectly, just do not work inside a web page and other enterprise devices where other MJPEG streams taken from the internet work perfectly
here is my code:
class Streamer(Thread):
def __init__(self, *args, **kwargs):
self.container = kwargs.pop("container", None)
# self.q = kwargs.pop("queue", None)
# self.stream = kwargs.pop("stream", None)
super(Streamer, self).__init__(*args, **kwargs)
def run(self):
try:
print("dichiarazione handler")
server = ThreadedHTTPServer(('192.168.1.182', 8080), HandlerArguments(self.container))
print('started httpserver...')
server.serve_forever()
except KeyboardInterrupt:
print('^C received, shutting down server')
server.socket.close()
def HandlerArguments(init_args):
class MyHandler(BaseHTTPRequestHandler):
def __init__(self, *args, **kwargs):
self.container = init_args
super(MyHandler, self).__init__(*args, **kwargs)
def do_GET(self):
print("GET")
if self.path.endswith('.mjpeg'):
self.container.stream = True
self.send_response(200)
self.send_header('Pragma:', 'no-cache')
self.send_header('Cache-Control:', 'no-cache')
self.send_header('Content-type:', 'multipart/x-mixed-replace;boundary=--myboundary')
self.end_headers()
try:
while self.container.stream == True:
# print(self.queue.qsize())
print("ciclo")
img = self.container.finalq.get()
ret, img_jpg = cv2.imencode(".jpeg", img, (cv2.IMWRITE_JPEG_QUALITY, 90))
self.wfile.write('--myboundary'.encode('utf-8'))
self.send_header('Content-type:','image/jpeg')
# self.send_header('Content-length:', str(len(img_jpg)))
self.end_headers()
img_str = img_jpg.tostring()
self.wfile.write(img_str)
# self.wfile.write('--myboundary\r\n'.encode('utf-8'))
# self.send_response(200)
# time.sleep(0.05)
except:
self.container.stream = False
while not self.container.finalq.empty():
self.container.finalq.get()
elif self.path.endswith('.jpg'):
self.send_response(200)
self.send_header('Pragma:', 'no-cache')
self.send_header('Cache-Control:', 'no-cache')
self.send_header('Content-type:','image/jpeg')
self.end_headers()
img = self.container.finalq.get()
ret, img_jpg = cv2.imencode(".jpeg", img, (cv2.IMWRITE_JPEG_QUALITY, 90))
# self.send_header('Content-length:', str(len(img_jpg)))
img_str = img_jpg.tostring()
self.wfile.write(img_str)
return MyHandler
class ThreadedHTTPServer(ThreadingMixIn, HTTPServer):
#class ThreadedHTTPServer(HTTPServer):
"""Handle requests in a separate thread."""
I think the problem is in the request header, in particular this is a stream taken from the internet compared with my stream
curl -I http://85.90.40.19/mjpg/video.mjpg
HTTP/1.0 200 OK
Cache-Control: no-cache
Pragma: no-cache
Expires: Thu, 01 Dec 1994 16:00:00 GMT
Connection: close
Content-Type: multipart/x-mixed-replace; boundary=--myboundary
///////
curl -I http://192.168.1.182:8080/stream.mjpeg
HTTP/1.0 501 Unsupported method ('HEAD')
Server: BaseHTTP/0.6 Python/3.4.0
Date: Mon, 20 Jul 2015 10:00:54 GMT
Content-Type: text/html;charset=utf-8
Connection: close
Content-Length: 474
Especially in my request i see:
Content-Type: text/html;charset=utf-8
while in my code i run
self.send_header('Content-type:','image/jpeg')
Thanks in advance guys :)

curl -I sends a HTTP HEAD request.
In your code you're handling only the HTTP GET requests:
def do_GET(self):
See the error message: HTTP/1.0 501 Unsupported method ('HEAD')
Handle HEAD requests too:
def do_HEAD(self):
and output the expected headers since the clients may issue a HEAD before a GET.

Making HTTP HEAD request with urllib2 from Python 2

I'm trying to do a HEAD request of a page using Python 2.
I am trying
import misc_urllib2
.....
opender = urllib2.build_opener([misc_urllib2.MyHTTPRedirectHandler(), misc_urllib2.HeadRequest()])
with misc_urllib2.py containing
class HeadRequest(urllib2.Request):
def get_method(self):
return "HEAD"
class MyHTTPRedirectHandler(urllib2.HTTPRedirectHandler):
def __init__ (self):
self.redirects = []
def http_error_301(self, req, fp, code, msg, headers):
result = urllib2.HTTPRedirectHandler.http_error_301(
self, req, fp, code, msg, headers)
result.redirect_code = code
return result
http_error_302 = http_error_303 = http_error_307 = http_error_301
But I am getting
TypeError: __init__() takes at least 2 arguments (1 given)
If I just do
opender = urllib2.build_opener(misc_urllib2.MyHTTPRedirectHandler())
then it works fine

This works just fine:
import urllib2
request = urllib2.Request('http://localhost:8080')
request.get_method = lambda : 'HEAD'
response = urllib2.urlopen(request)
print response.info()
Tested with quick and dirty HTTPd hacked in python:
Server: BaseHTTP/0.3 Python/2.6.6
Date: Sun, 12 Dec 2010 11:52:33 GMT
Content-type: text/html
X-REQUEST_METHOD: HEAD
I've added a custom header field X-REQUEST_METHOD to show it works :)
Here is HTTPd log:
Sun Dec 12 12:52:28 2010 Server Starts - localhost:8080
localhost.localdomain - - [12/Dec/2010 12:52:33] "HEAD / HTTP/1.1" 200 -
Edit: there is also httplib2
import httplib2
h = httplib2.Http()
resp = h.request("http://www.google.com", 'HEAD')

Try httplib
>>> import httplib
>>> conn = httplib.HTTPConnection("www.google.com")
>>> conn.request("HEAD", "/index.html")
>>> res = conn.getresponse()
>>> print res.status, res.reason
200 OK
>>> print res.getheaders()
[('content-length', '0'), ('expires', '-1'), ('server', 'gws'), ('cache-control', 'private, max-age=0'), ('date', 'Sat, 20 Sep 2008 06:43:36 GMT'), ('content-type', 'text/html; charset=ISO-8859-1')]
See How do you send a HEAD HTTP request in Python 2?

The problem lies with your class HeadRequest, which inherits from urllib2.Request. According to doc, urllib2.Request.__init__ signature is
__init__(self, url, data=None, headers={}, origin_req_host=None, unverifiable=False)
so you must pass an url argument to it. In your second try, you just do not use HeadRequest, this is why it works.

you shoud not add HeadRequest to build_opener or add_handler it should be called like this
opener = urllib2.build_opener(MyHTTPRedirectHandler)
response = opener.open(HeadRequest(url))
print response.getheaders()