Develop Reference

smtplib times out connecting to AWS SES despite receiving a 220 service ready

I am attempting to send an email with AWS SES using a Dockerized Python program. When I try to connect to SES by making an SMTP instance, the program hangs and times out.
To reproduce:
Start the Python 3.6.8 REPL
Import smtplib
>>> import smtplib
Try to connect to SES, and observe how the statement hangs.
>>> conn = smtplib.SMTP(host='email-smtp.us-west-2.amazonaws.com', port=25)
# terminal hangs for 10 seconds
Try using the connection, which fails.
>>> conn.noop()
(451, b'4.4.2 Timeout waiting for data from client.')
>>> conn.noop()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python3.6/smtplib.py", line 514, in noop
return self.docmd("noop")
File "/usr/lib64/python3.6/smtplib.py", line 421, in docmd
return self.getreply()
File "/usr/lib64/python3.6/smtplib.py", line 394, in getreply
raise SMTPServerDisconnected("Connection unexpectedly closed")
smtplib.SMTPServerDisconnected: Connection unexpectedly closed
Unsurprisingly, step (3) sends a DNS query for the SES endpoint, and then connects; the SES endpoint responds with 220 service ready nearly instantaneously. No traffic is exchanged until 10 seconds later, when SES closes the connection with 451 timeout waiting for client. The statement completes, and then runs the rest of my program which, of course, doesn't work.
For context, the rest of my script is as follows:
smtp_host = 'email-smtp.us-west-2.amazonaws.com'
smtp_port = 25
smtp_proto = 'tls'
with smtplib.SMTP(host=smtp_host, port=smtp_port) as connection:
try:
ctx = ssl.SSLContext(ssl.PROTOCOL_SSLv23 if smtp_proto == 'ssl' else ssl.PROTOCOL_TLS)
connection.starttls(context=ctx)
connection.ehlo()
connection.login(smtp_user, smtp_pass)
connection.send_message(from_addr=sender, to_addrs=recipients, msg=message)
except smtplib.SMTPHeloError as e:
print(f"SMTP HELO Failed: {e}")
except smtplib.SMTPAuthenticationError as e:
print(f"SMTP AUTH Failed: {e}")
except smtplib.SMTPException as e:
print(f"Failed to send email: {e}")
I've attempted to connect on ports 25, 587, 2587, and 465. I've done the same with the SMTP_SSL object instead, as well as changing the context in the starttls call to various SSL / TLS versions (the SSL versions result in this error - but this isn't relevant since I can't get to this portion of the script, anyway).
I've tested my connection to SES according to this article. I've also tried parts of this and this SO post (as well as a myriad of others that are lost in my browser history).
To actually send emails, I need to connect a second time. Connecting, then waiting for a timeout, then connecting again, seems wrong. What is the proper way to do this?

Google Cloud Speech to Text Audio Timeout Error when used with Twilio "Stream" verb and Websocket

I'm currently trying to make a system that can transcribe a phone call in real time and then display the conversation in my command line. To do this, im using a Twilio phone number which sends out a http request when called. Then using Flask, Ngrok and Websockets to compile my server code, make my local port public and to transfer the data, the TwiML verb "Stream" is used to stream the audio data to the Google Cloud Speech-Text API. I have so far used Twilio's python demo on GitHub (https://github.com/twilio/media-streams/tree/master/python/realtime-transcriptions).
My server code:
from flask import Flask, render_template
from flask_sockets import Sockets
from SpeechClientBridge import SpeechClientBridge
from google.cloud.speech_v1 import enums
from google.cloud.speech_v1 import types
import json
import base64
import os
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "./<KEY>.json"
HTTP_SERVER_PORT = 8080
config = types.RecognitionConfig(
encoding=enums.RecognitionConfig.AudioEncoding.MULAW,
sample_rate_hertz=8000,
language_code='en-US')
streaming_config = types.StreamingRecognitionConfig(
config=config,
interim_results=True)
app = Flask(__name__)
sockets = Sockets(app)
#app.route('/home')
def home():
return render_template("index.html")
#app.route('/twiml', methods=['POST'])
def return_twiml():
print("POST TwiML")
return render_template('streams.xml')
def on_transcription_response(response):
if not response.results:
return
result = response.results[0]
if not result.alternatives:
return
transcription = result.alternatives[0].transcript
print("Transcription: " + transcription)
#sockets.route('/')
def transcript(ws):
print("WS connection opened")
bridge = SpeechClientBridge(
streaming_config,
on_transcription_response
)
while not ws.closed:
message = ws.receive()
if message is None:
bridge.terminate()
break
data = json.loads(message)
if data["event"] in ("connected", "start"):
print(f"Media WS: Received event '{data['event']}': {message}")
continue
if data["event"] == "media":
media = data["media"]
chunk = base64.b64decode(media["payload"])
bridge.add_request(chunk)
if data["event"] == "stop":
print(f"Media WS: Received event 'stop': {message}")
print("Stopping...")
break
bridge.terminate()
print("WS connection closed")
if __name__ == '__main__':
from gevent import pywsgi
from geventwebsocket.handler import WebSocketHandler
server = pywsgi.WSGIServer(('', HTTP_SERVER_PORT), app, handler_class=WebSocketHandler)
print("Server listening on: http://localhost:" + str(HTTP_SERVER_PORT))
server.serve_forever()
streams.xml:
<?xml version="1.0" encoding="UTF-8"?>
<Response>
<Say> Thanks for calling!</Say>
<Start>
<Stream url="wss://<ngrok-URL/.ngrok.io/"/>
</Start>
<Pause length="40"/>
</Response>
Twilio WebHook:
http://<ngrok-URL>.ngrok.io/twiml
Im am getting the following error when I run the server code and then call the Twilio number:
C:\Users\Max\Python\Twilio>python server.py
Server listening on: http://localhost:8080
POST TwiML
WS connection opened
Media WS: Received event 'connected': {"event":"connected","protocol":"Call","version":"0.2.0"}
Media WS: Received event 'start': {"event":"start","sequenceNumber":"1","start":{"accountSid":"AC8abc5aa74496a227d3eb489","streamSid":"MZe6245f23e2385aa2ea7b397","callSid":"CA5864313b4992607d3fe46","tracks":["inbound"],"mediaFormat":{"encoding":"audio/x-mulaw","sampleRate":8000,"channels":1}},"streamSid":"MZe6245f2397c1285aa2ea7b397"}
Exception in thread Thread-4:
Traceback (most recent call last):
File "C:\Users\Max\AppData\Local\Programs\Python\Python37\lib\site-packages\google\api_core\grpc_helpers.py", line 96, in next
return six.next(self._wrapped)
File "C:\Users\Max\AppData\Local\Programs\Python\Python37\lib\site-packages\grpc\_channel.py", line 416, in __next__
return self._next()
File "C:\Users\Max\AppData\Local\Programs\Python\Python37\lib\site-packages\grpc\_channel.py", line 689, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.OUT_OF_RANGE
details = "Audio Timeout Error: Long duration elapsed without audio. Audio should be sent close to real time."
debug_error_string = "{"created":"#1591738676.565000000","description":"Error received from peer ipv6:[2a00:1450:4009:807::200a]:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Audio Timeout Error: Long duration elapsed without audio. Audio should be sent close to real time.","grpc_status":11}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\Max\AppData\Local\Programs\Python\Python37\lib\threading.py", line 917, in _bootstrap_inner
self.run()
File "C:\Users\Max\AppData\Local\Programs\Python\Python37\lib\threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\Max\Python\Twilio\SpeechClientBridge.py", line 37, in process_responses_loop
for response in responses:
File "C:\Users\Max\AppData\Local\Programs\Python\Python37\lib\site-packages\google\api_core\grpc_helpers.py", line 99, in next
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.OutOfRange: 400 Audio Timeout Error: Long duration elapsed without audio. Audio should be sent close to real time.
Media WS: Received event 'stop': {"event":"stop","sequenceNumber":"752","streamSid":"MZe6245f2397c125aa2ea7b397","stop":{"accountSid":"AC8abc5aa74496a60227d3eb489","callSid":"CA5842bc6431314d502607d3fe46"}}
Stopping...
WS connection closed
I cant work out why im getting the audio timeout error? Is it a firewall issue with Twilio and Google? An encoding issue?
Any help would be greatly appreciated.
System:
Windows 10
Python 3.7.1
ngrok 2.3.35
Flask 1.1.2

As your streams.xml returned socket url "wss://<ngrok-URL/.ngrok.io/", please make sure it matches with your routing (e.g. #sockets.route('/'))
If your socket starting with '/', then your should rewrite the streams.xml, see below as an example.
<?xml version="1.0" encoding="UTF-8"?>
<Response>
<Say> Thanks for calling!</Say>
<Start>
<Stream url="wss://YOUR_NGROK_ID.ngrok.io/"/>
</Start>
<Pause length="40"/>
</Response>

I ran some tests on this to try to establish what was happening. I put a timer over the
bridge = SpeechClientBridge(
streaming_config,
on_transcription_response)
section of code and found that it was taking ~10.9s to initialize. I believe the google API has a timeout of 10s. I tried running this on my google cloud instance which has more oomph than my laptop and it works perfectly well. Either this, or there are some different versions of libraries/code etc installed on the GCP instance, which I need to check.

This is related to gevent (used by flask_sockets) and grpc (used by google cloud speech) conflict described in this issue https://github.com/grpc/grpc/issues/4629
the solution is to add the following code
import grpc.experimental.gevent as grpc_gevent
grpc_gevent.init_gevent()

pydbus reconnection after target is rebooted

I'm facing the following problem.
I'm using pydbus python package to test API by connecting to a target linux machine which is in the same network by using the environment variable DBUS_SYSTEM_BUS_ADDRESS
Under normal circumstances this works well, I'm able to call any valid API over the proxy object created.
Eg:
import os
import pydbus
os.environ['DBUS_SYSTEM_BUS_ADDRESS'] = \
"tcp:host=192.168.1.100,port=55556,family=ipv4"
bus = pydbus.SystemBus()
proxy = bus.get("busname", "object_path")
return_value = proxy.method-name(args)
As part of my testing, my target linux machine gets rebooted in between and when I re-use the proxy object, I get the following error
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/harman/.local/lib/python2.7/site-packages/pydbus/proxy.py", line 47, in get
0, timeout_to_glib(timeout), None)
GLib.Error: g-io-error-quark: The connection is closed (18)
I realize that the previous connection no longer exists, so I tried re-initialising the connection with the following
bus = pydbus.SystemBus()
proxy = bus.get("busname", "object_path")
I get the same error on the bus.get() statement.
Is there a solution/possible workaround to this problem?

can't login with Paramiko ssh client

local-host --->Aterm server (security server ) -----> target-machine(
I am trying to write a code in Python using Paramiko to first SSH from local-host to the target-machine. From the target-machine, I want to capture some outputs and store them locally either as a variable or as a file (havent got to that point yet). I found an example from stackoverflow where they talk about using nested SSH with paramiko, and I follow it but I get stuck here:
i need just reaching the target-machine
My code:
import paramiko
import sys
import subprocess
hostname = '10.10.10.1'
port = 22
username = 'mohamed.hosseny'
password ='Pass#1'
client = paramiko.Transport((hostname, port))
client.connect(username=username, password=password)
client.close()
but i found the below error message :
Traceback (most recent call last):
File "C:/Users/mohamed.hosseny/Desktop/Paramiko.py", line 13, in <module>
client = paramiko.Transport((hostname, port))
File "C:\Python27\lib\site-packages\paramiko\transport.py", line 332, in
__init__
'Unable to connect to {}: {}'.format(hostname, reason))
SSHException: Unable to connect to 10.10.10.1: [Errno 10060] A connection
attempt failed because the connected party did not properly respond after
a period of time, or established connection failed because connected host
has failed to respond

paramiko.Transport is a lower level API. Don't use it unless you have a good reason. Instead, you can use paramiko.SSHClient.

Bokeh Server on IIS Setup

I'm trying to figure out how to deploy a Bokeh slider chart over an IIS server.
I recently finished up a Flask application, so I figured I'd try the route where you embed through flask:
https://github.com/bokeh/bokeh/tree/master/examples/howto/server_embed
It's nice and easy when I launch the script locally.. but I can't seem to set it up properly over IIS. I believe the complexity is stemming from the fact that the wfastcgi.py module I'm using to deploy over IIS can't easily multi-thread without some sort of hack-like work around.
So, my second attempt was to wrap the flask app in tornado as below OPTION B
(without much success, but still think this is my best lead here)
Run Flask as threaded on IIS 7
My third attempt was to try and run Bokeh server standalone on a specific port. I figured I'd be able to run the server via standalone_embed.py using wfastcgi.py on say port 8888 & while using port 5000 for the server callbacks. However, the Server function:
from bokeh.server.server import Server
still launches it locally on the host machine
server = Server({'/': bokeh_app}, io_loop=io_loop, port=5000)
server.start()
So this actually works if I go to http://localhost:5000/ on the host,
but fails if I go to http://%my_host_ip%:5000/ from a remote machine.
I even tried manually setting the host but get an "invalid host" error:
server = Server({'/': bokeh_app}, io_loop=io_loop, host='%my_host_ip_address_here%:5000')
server.start()
ERR:
Error occurred while reading WSGI handler: Traceback (most recent call last): File "C:\Python34\lib\site-packages\bokeh\server\server.py", line 45, in _create_hosts_whitelist int(parts[1]) ValueError: invalid literal for int() with base 10: '' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\WebsitesFlask\bokehTest\wfastcgi.py", line 711, in main env, handler = read_wsgi_handler(response.physical_path) File "C:\WebsitesFlask\bokehTest\wfastcgi.py", line 568, in read_wsgi_handler return env, get_wsgi_handler(handler_name) File "C:\WebsitesFlask\bokehTest\wfastcgi.py", line 537, in get_wsgi_handler handler = import(module_name, fromlist=[name_list[0][0]]) File ".\app.py", line 41, in server = Server({'/': bokeh_app}, io_loop=io_loop, host='%my_host_ip_address_here%:5000') File "C:\Python34\lib\site-packages\bokeh\server\server.py", line 123, in init tornado_kwargs['hosts'] = _create_hosts_whitelist(kwargs.get('host'), self._port) File "C:\Python34\lib\site-packages\bokeh\server\server.py", line 47, in _create_hosts_whitelist raise ValueError("Invalid port in host value: %s" % host) ValueError: Invalid port in host value: : StdOut: StdErr:

First off, the --host parameter should no longer be needed in the next 0.12.5 release. It's probably been the most confusing stumbling block for people trying to deploy a Bokeh server app in a "production" environment. You can follow the discussion on this issue on GitHub for more details.
Looking at the actual implementation in Bokeh that generates the error you are seeing, it is just this:
parts = host.split(':')
if len(parts) == 1:
if parts[0] == "":
raise ValueError("Empty host value")
hosts.append(host+":80")
elif len(parts) == 2:
try:
int(parts[1])
except ValueError:
raise ValueError("Invalid port in host value: %s" % host)
The exception you are reporting that states that int(parts[1]) is failing:
Traceback (most recent call last):
File "C:\Python34\lib\site-packages\bokeh\server\server.py", line 45,
in _create_hosts_whitelist int(parts[1])
ValueError: invalid literal for int() with base 10:
So, there is something amiss with the string you are passing for hosts that's causing the part after the colon to not be able to be converted to in int But without seeing the actual string, it's impossible to say much more. Maybe there is some encoding issue that needs to be handled differently or better. If you can provide a concrete string example that reproduces the problem I can take a closer look.