Develop Reference

How can I call a python function with Vercel Python Serverless Functions

I'm trying to create a Next.js api endpoint in python, its full path is api/search/[slug].py. It makes calls to a python TikTok api wrapper. The following code works but I would like to make the tag value be obtained dynamically from the slug.
from http.server import BaseHTTPRequestHandler
from TikTokApi import TikTokApi
import json
#search
api = TikTokApi()
videos=list(api.hashtag(name='tag').videos(count=10))
userDict={}
for v in videos:
userDict[v.author.username]=v.as_dict['authorStats']['followerCount']
userDict = dict(sorted(userDict.items(),key=lambda item: item[1], reverse=True))
json_object = json.dumps(userDict, indent = 4)
#end search
class handler(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.send_header('Content-type', 'text/plain')
self.end_headers()
self.wfile.write(json_object.encode())
return
If I extract the function and call it in the BaseHTTPRequestHandler handler I get the error:
127.0.0.1 - - [03/Oct/2022 15:02:36] "GET /api/search/slug?slug=slug HTTP/1.1" 200 -
----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 59466)
Traceback (most recent call last):
File "/opt/homebrew/Cellar/python#3.10/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/socketserver.py", line 316, in _handle_request_noblock
self.process_request(request, client_address)
File "/opt/homebrew/Cellar/python#3.10/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/socketserver.py", line 347, in process_request
self.finish_request(request, client_address)
File "/opt/homebrew/Cellar/python#3.10/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/socketserver.py", line 360, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/opt/homebrew/Cellar/python#3.10/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/socketserver.py", line 747, in __init__
self.handle()
File "/opt/homebrew/Cellar/python#3.10/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/server.py", line 432, in handle
self.handle_one_request()
File "/opt/homebrew/Cellar/python#3.10/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/server.py", line 420, in handle_one_request
method()
File "/private/var/folders/5r/_0bfsh314nxcvsxckx0nk8mm0000gn/T/zeit-fun-2f2e7e68416d5/./api/search/[slug].py", line 25, in do_GET
json_object = search()
File "/private/var/folders/5r/_0bfsh314nxcvsxckx0nk8mm0000gn/T/zeit-fun-2f2e7e68416d5/./api/search/[slug].py", line 6, in search
api = TikTokApi()
File "/opt/homebrew/lib/python3.10/site-packages/TikTokApi/tiktok.py", line 159, in __init__
self._initialize(
File "/opt/homebrew/lib/python3.10/site-packages/TikTokApi/tiktok.py", line 205, in _initialize
self._browser = asyncio.get_event_loop().run_until_complete(
File "/opt/homebrew/Cellar/python#3.10/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/events.py", line 656, in get_event_loop
raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'Dummy-1'.
----------------------------------------

Python wsgi server http streaming m4a file

I'm trying to stream m4a file over http using a simple wsgi server in python.
My code is like this:
mp3file = urllib2.urlopen(media_url)
data = mp3file.read()
headers = [('Content-type', 'audio/mp4'), ('Accept-Ranges', 'bytes')]
write = start_response('200 OK', headers)
write(data)
The problem is that if I set 'Content-Length' header like below, the client (Chrome browser for example) doesn't play the media. And if I don't set this header, it does play the media, but it doesn't know the length of the media so I can't seek/fast forward over the media file.
headers = [('Content-type', 'audio/mp4'), ('Content-Length', mp3file.headers['Content-Length']), ('Accept-Ranges', 'bytes')]
When setting the header I get this on console:
Traceback (most recent call last):
File "C:\Python27\lib\wsgiref\handlers.py", line 86, in run
self.finish_response()
File "C:\Python27\lib\wsgiref\handlers.py", line 128, in finish_response
self.write(data)
File "C:\Python27\lib\wsgiref\handlers.py", line 217, in write
self._write(data)
File "C:\Python27\lib\socket.py", line 324, in write
self.flush()
File "C:\Python27\lib\socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 10053] An established connection was aborted by the software in your host machine
127.0.0.1 - - [28/Sep/2015 16:37:20] "GET /?yb=QfCqNbzJi3k HTTP/1.1" 500 59
Traceback (most recent call last):
File "C:\Python27\lib\SocketServer.py", line 295, in _handle_request_noblock
self.process_request(request, client_address)
File "C:\Python27\lib\SocketServer.py", line 321, in process_request
self.finish_request(request, client_address)
File "C:\Python27\lib\SocketServer.py", line 334, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Python27\lib\SocketServer.py", line 653, in __init__
self.finish()
File "C:\Python27\lib\SocketServer.py", line 712, in finish
self.wfile.close()
File "C:\Python27\lib\socket.py", line 279, in close
self.flush()
File "C:\Python27\lib\socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 10053] An established connection was aborted by the software in your host machine

urlfetch DeadlineExceededError

I had a create a REST API using bottle framework, which receives calls from GAE. Once this REST API is invoked, it did some calculations and sent outputs as a zip file to AMAZON S3 server and return the link to GAE. Everything works fine expect the timeout issue. I tried to adjust the deadline of urlfetch to 60 seconds, which did not solve the problem. I appreciate any suggestions.
GAE side:
response = urlfetch.fetch(url=url, payload=data, method=urlfetch.POST, headers=http_headers, deadline=60)
Broser error info.:
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1102, in __call__
return handler.dispatch()
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "D:\Dropbox\ubertool_src\przm5/przm5_output.py", line 22, in post
przm5_obj = przm5_rest_model.przm5(args)
File "D:\Dropbox\ubertool_src\przm5\przm5_rest_model.py", line 351, in __init__
self.convertSoil1, self.convert1to3, self.convert2to3)
File "D:\Dropbox\ubertool_src\przm5\przm5_rest_model.py", line 135, in get_jid
response = urlfetch.fetch(url=url, payload=data, method=urlfetch.POST, headers=http_headers)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\api\urlfetch.py", line 270, in fetch
return rpc.get_result()
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\api\apiproxy_stub_map.py", line 612, in get_result
return self.__get_result_hook(self)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\api\urlfetch.py", line 410, in _get_fetch_result
'Deadline exceeded while waiting for HTTP response from URL: ' + url)
DeadlineExceededError: Deadline exceeded while waiting for HTTP response from URL: http://localhost:7777/my_model
REST server:
#route('/my_model', method='POST')
#auth_basic(check)
def my_model():
#run the model
run_my model()
#zip output files
zout=zipfile.ZipFile("test.zip","w")
for name in os.listdir(src1):
zout.write(name)
zout.close()
##upload file to S3
conn = S3Connection(key, secretkey)
bucket = Bucket(conn, 'przm5')
k=Key(bucket)
name1='PRZM5_'+name_temp+'.zip'
k.key=name1
###All the above steps are fine####
k.set_contents_from_filename('test.zip')
link='https://s3.amazonaws.com/'+my_file_path
return {'ff': ff}
run(host='localhost', port=7777, debug=True)
Errors from the REST server:
127.0.0.1 - - [07/Jan/2014 16:16:36] "POST /my_model HTTP/1.1" 200 1663
Traceback (most recent call last):
File "C:\Python27\Lib\wsgiref\handlers.py", line 86, in run
self.finish_response()
File "C:\Python27\Lib\wsgiref\handlers.py", line 128, in finish_response
self.write(data)
File "C:\Python27\Lib\wsgiref\handlers.py", line 212, in write
self.send_headers()
File "C:\Python27\Lib\wsgiref\handlers.py", line 270, in send_headers
self.send_preamble()
File "C:\Python27\Lib\wsgiref\handlers.py", line 194, in send_preamble
'Date: %s\r\n' % format_date_time(time.time())
File "C:\Python27\Lib\socket.py", line 324, in write
self.flush()
File "C:\Python27\Lib\socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 10053] An established connection was aborted by the software in your host machine
127.0.0.1 - - [07/Jan/2014 16:16:36] "POST /my_model HTTP/1.1" 500 59
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 50953)
Traceback (most recent call last):
File "C:\Python27\Lib\SocketServer.py", line 295, in _handle_request_noblock
self.process_request(request, client_address)
File "C:\Python27\Lib\SocketServer.py", line 321, in process_request
self.finish_request(request, client_address)
File "C:\Python27\Lib\SocketServer.py", line 334, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Python27\Lib\SocketServer.py", line 651, in __init__
self.finish()
File "C:\Python27\Lib\SocketServer.py", line 710, in finish
self.wfile.close()
File "C:\Python27\Lib\socket.py", line 279, in close
self.flush()
File "C:\Python27\Lib\socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 10053] An established connection was aborted by the software in your host machine

The deadline is a maximum value, once reached it'll fail. And it's failing with
Deadline exceeded while waiting for HTTP response
So you should try to catch that exception and try again.
If the entire operation can't be done in under 60 seconds then there is nothing else to be done, it's a hard limit in GAE that HTTP requests can't exceed 60 seconds.

Combining tornado and zmq ioloops: Connection reset by peer exception

I faced with one annoying (while not critical) problem when I tried to combine Tornado and pyzmq ioloops as described in the pyzmq official documentation.
I have a process running tornado (T) server which accepts REST API requests from clients (C) and proxies them though ZMQ transport to another process (Z) that does real work.
C <-> T <-> Z
If C closes the connection before Z replies to T, Z (tornado) outputs long bunch of exception traces (see at the bottom). Imagine the following example:
import tornado.ioloop
from tornado.web import Application, RequestHandler, asynchronous
from zmq.eventloop import ioloop
import time
def time_consuming_task():
time.sleep(5)
class TestHandler(RequestHandler):
def get(self, arg):
print "Test arg", arg
time_consuming_task()
print "Ok, time to reply"
self.write("Reply")
if __name__ == "__main__":
app = tornado.web.Application(
[
(r"/test/([0-9]+)", TestHandler)
])
ioloop.install()
app.listen(8080)
tornado.ioloop.IOLoop.instance().start()
This example doesn't actually talk to any ZMQ peer, it just attaches pyzmq ioloop to tornado's ioloop. Though, it's enough to illustrate the problem.
From console one run server:
% python example.py
From console two run client and interrupt it before server replies (i.e. during 5 secs):
% curl -is http://localhost:8080/test/1
^C
The output of server is:
Test arg 1
Ok, time to reply
WARNING:root:Read error on 24: [Errno 54] Connection reset by peer
ERROR:root:Uncaught exception GET /test/1 (::1)
HTTPRequest(protocol='http', host='localhost:8080', method='GET', uri='/test/1', version='HTTP/1.1', remote_ip='::1', body='', headers={'Host': 'localhost:8080', 'Accept': '*/*', 'User-Agent': 'curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5'})
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/tornado/web.py", line 1023, in _execute
self.finish()
File "/Library/Python/2.7/site-packages/tornado/web.py", line 701, in finish
self.request.finish()
File "/Library/Python/2.7/site-packages/tornado/httpserver.py", line 433, in finish
self.connection.finish()
File "/Library/Python/2.7/site-packages/tornado/httpserver.py", line 187, in finish
self._finish_request()
File "/Library/Python/2.7/site-packages/tornado/httpserver.py", line 223, in _finish_request
self.stream.read_until(b("\r\n\r\n"), self._header_callback)
File "/Library/Python/2.7/site-packages/tornado/iostream.py", line 153, in read_until
self._try_inline_read()
File "/Library/Python/2.7/site-packages/tornado/iostream.py", line 386, in _try_inline_read
if self._read_to_buffer() == 0:
File "/Library/Python/2.7/site-packages/tornado/iostream.py", line 421, in _read_to_buffer
chunk = self._read_from_socket()
File "/Library/Python/2.7/site-packages/tornado/iostream.py", line 402, in _read_from_socket
chunk = self.socket.recv(self.read_chunk_size)
error: [Errno 54] Connection reset by peer
ERROR:root:Cannot send error response after headers written
ERROR:root:Uncaught exception, closing connection.
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/tornado/iostream.py", line 304, in wrapper
callback(*args)
File "/Library/Python/2.7/site-packages/tornado/httpserver.py", line 262, in _on_headers
self.request_callback(self._request)
File "/Library/Python/2.7/site-packages/tornado/web.py", line 1412, in __call__
handler._execute(transforms, *args, **kwargs)
File "/Library/Python/2.7/site-packages/tornado/web.py", line 1025, in _execute
self._handle_request_exception(e)
File "/Library/Python/2.7/site-packages/tornado/web.py", line 1065, in _handle_request_exception
self.send_error(500, exc_info=sys.exc_info())
File "/Library/Python/2.7/site-packages/tornado/web.py", line 720, in send_error
self.finish()
File "/Library/Python/2.7/site-packages/tornado/web.py", line 700, in finish
self.flush(include_footers=True)
File "/Library/Python/2.7/site-packages/tornado/web.py", line 660, in flush
self.request.write(headers + chunk, callback=callback)
File "/Library/Python/2.7/site-packages/tornado/httpserver.py", line 429, in write
self.connection.write(chunk, callback=callback)
File "/Library/Python/2.7/site-packages/tornado/httpserver.py", line 177, in write
assert self._request, "Request closed"
AssertionError: Request closed
ERROR:root:Exception in callback
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/pyzmq-2.2.0-py2.7-macosx-10.7-intel.egg/zmq/eventloop/ioloop.py", line 434, in _run_callback
callback()
File "/Library/Python/2.7/site-packages/tornado/iostream.py", line 304, in wrapper
callback(*args)
File "/Library/Python/2.7/site-packages/tornado/httpserver.py", line 262, in _on_headers
self.request_callback(self._request)
File "/Library/Python/2.7/site-packages/tornado/web.py", line 1412, in __call__
handler._execute(transforms, *args, **kwargs)
File "/Library/Python/2.7/site-packages/tornado/web.py", line 1025, in _execute
self._handle_request_exception(e)
File "/Library/Python/2.7/site-packages/tornado/web.py", line 1065, in _handle_request_exception
self.send_error(500, exc_info=sys.exc_info())
File "/Library/Python/2.7/site-packages/tornado/web.py", line 720, in send_error
self.finish()
File "/Library/Python/2.7/site-packages/tornado/web.py", line 700, in finish
self.flush(include_footers=True)
File "/Library/Python/2.7/site-packages/tornado/web.py", line 660, in flush
self.request.write(headers + chunk, callback=callback)
File "/Library/Python/2.7/site-packages/tornado/httpserver.py", line 429, in write
self.connection.write(chunk, callback=callback)
File "/Library/Python/2.7/site-packages/tornado/httpserver.py", line 177, in write
assert self._request, "Request closed"
AssertionError: Request closed
NOTE: It seems it's pyzmq related problem because disappears after excluding pyzmq ioloop.
Server doesn't die, it can be used by other clients, so the problem is not critical. Though, it's very annoying to find these huge confusing traces in logfiles.
So, are there any well-known methods to solve this problem?
Thanks.

It's not ZMQ problem. Request can be closed by other than timeout reasons. ZMQ only problem here is that they are raising AssertionError, which is common, instead of more specific exception.
If you are sure, that you don't want to have these exceptions in log files - do something like this:
try:
time_consuming_task()
except AssertionError as e:
if e.message == 'Request closed':
logging.info('Bad, annoying client, came to us again!')
else:
raise e

Sending mutlitple requests of httplib, raising traceback exceptions

Here is the code:
conn = httplib.HTTPConnection("127.0.0.1:8000")
conn.request("POST", "/api/job/", some_params, headers)
conn.close()
no problem with sending request to server
but if i use loop for example:
for i in range(n):
conn = httplib.HTTPConnection("127.0.0.1:8000")
conn.request("POST", "/api/job/", some_params, headers)
conn.close()
it rises an exception, but it is interesting that request is successfull:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/django/core/servers/basehttp.py", line 284, in run
self.finish_response()
File "/usr/lib/python2.7/site-packages/django/core/servers/basehttp.py", line 324, in finish_response
self.write(data)
File "/usr/lib/python2.7/site-packages/django/core/servers/basehttp.py", line 403, in write
self.send_headers()
File "/usr/lib/python2.7/site-packages/django/core/servers/basehttp.py", line 467, in send_headers
self.send_preamble()
File "/usr/lib/python2.7/site-packages/django/core/servers/basehttp.py", line 385, in send_preamble
'Date: %s\r\n' % http_date()
File "/usr/lib/python2.7/socket.py", line 324, in write
self.flush()
File "/usr/lib/python2.7/socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 60438)
Traceback (most recent call last):
File "/usr/lib/python2.7/SocketServer.py", line 284, in _handle_request_noblock
self.process_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 310, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 323, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.7/site-packages/django/core/servers/basehttp.py", line 570, in __init__
BaseHTTPRequestHandler.__init__(self, *args, **kwargs)
File "/usr/lib/python2.7/SocketServer.py", line 641, in __init__
self.finish()
File "/usr/lib/python2.7/SocketServer.py", line 694, in finish
self.wfile.flush()
File "/usr/lib/python2.7/socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
----------------------------------------
any suggestions ???

Looks to me like your buffer is getting filled. Your buffer will fill with any network requests you make, then is cleared when the server acknowledges receipt of the data. Not sure if there isn't a better way to do this, but you could try giving the server some time to acknowledge receipt by doing a short time.pause within your loop.