Develop Reference

SMTP_HELO returns timeout when running email address validation

Using library py3-validate-email-1.0.5 (more here) to check if email address is valid, including SMTP check, I wasn't able to make it through check_smtp step, because I get following error:
Python script
from validate_email import validate_email
from validate_email import validate_email_or_fail
from csv import DictReader
# iterate over each line by column name
with open('email-list.csv', 'r') as read_obj:
csv_dict_reader = DictReader(read_obj, delimiter=';')
for row in csv_dict_reader:
i = 1
while i < 21:
header_name = 'Email'+str(i)
if validate_email_or_fail(
email_address=row[header_name],
check_format=True,
check_blacklist=True,
check_dns=True,
dns_timeout=10,
check_smtp=True,
smtp_timeout=5,
smtp_helo_host='emailsrv.domain.com',
smtp_from_address='email#domain.com',
smtp_skip_tls=False,
smtp_tls_context=None,
smtp_debug=False):
print('Email ' + row[header_name] + ' is valid.')
else:
print('Email ' + row[header_name] + ' is invalid.')
i += 1
Error:
Traceback (most recent call last):
File "//./main.py", line 13, in <module>
if validate_email_or_fail(
File "/usr/local/lib/python3.9/site-packages/validate_email/validate_email.py", line 59, in validate_email_or_fail
return smtp_check(
File "/usr/local/lib/python3.9/site-packages/validate_email/smtp_check.py", line 229, in smtp_check
return smtp_checker.check(hosts=mx_records)
File "/usr/local/lib/python3.9/site-packages/validate_email/smtp_check.py", line 197, in check
raise SMTPTemporaryError(error_messages=self.__temporary_errors)
validate_email.exceptions.SMTPTemporaryError: Temporary error in email address verification:
mx.server.com: 451 timed out (in reply to 'connect')
I figured there is problem with my DNS settings (probably), so I dockerized the script and run it on AWS EC2, where I have used elastic IP, attached it to the EC2 instance where docker container is running, I also setup reverse DNS for domain emailsrv.domain.com with this elastic IP. Tried to run the script, no change.
Then I added MX record pointing to the emailsrv.domain.com, but still no change. The DNS records are setup properly, because I have checked it with multiple DNS tools available.
Since the library doesn't require to actually use my email address login details, I wonder what can be the problem? Just to be sure, the email address used in the script doesn't exist, since I don't have smtp server setup on that instance, obviously.
Any ideas?

Reason behind this was closed port on AWS EC2 instance. Opening the port in security group is not enough, you must send a request to AWS so they remove the restriction on port 25.
When they did that, works flawlessly.

Python Kafka client cannot connect to remote Kafka server

I have an Ubuntu VM on cloud, where I downloaded Kafka version 2.8.1 from the Official Kafka site and followed the instructions in Kafka's official quickstart guide.
I am using a python client to consume one of the topics that I created as part of the quickstart guide. When I run it on the VM, everything runs fine, however, when I run the same program on my local system, I get the below error
Unable connect to node with id 0: [Errno 8] nodename nor servname provided, or not known
Traceback (most recent call last):
...
...
File "/Path/python3.9/site-packages/aiokafka/client.py", line 547, in check_version
raise KafkaConnectionError(
kafka.errors.KafkaConnectionError: KafkaConnectionError: No connection to node with id 0
The python program I am using:
import asyncio
import aiokafka
async def consume(self):
consumer = aiokafka.AIOKafkaConsumer(
"quickstart-events", bootstrap_servers="IP:9092"
)
try:
await consumer.start()
async for msg in self.consumer:
print(
"consumed: ",
msg.topic,
msg.partition,
msg.offset,
msg.key,
msg.value,
msg.timestamp,
)
finally:
await consumer.stop()
asyncio.run(consume())
I have ensured that the necessary ports (9022) on Ubuntu is open -
I checked that I could telnet into port 9022 from my local system.
I am not sure what could be the reason that I am unable to access Kafka over internet. Am I missing something obvious?

Change the following attribute in config/server.properties to bootstrap server address you are using in your code.
advertised.listeners = PLAINTEXT://IP or FQDN:9092

SFTP hang and authentication failure with paramiko

I'm trying to connect via SFTP with paramiko and Python 2.7, to eventually get a file from remote server and put to my server. (Note it uses a non-standard port too)
But when I try to connect - it takes a really long time and then I get an authentication error. Have you have this issue have suggestions for fixing?
I don't have a key, it just uses a username/password. I can connect with a graphical SSH program without issues, so the credentials seem correct.
Here is code:
hostname = 'remotehostname.com'
username= 'AB1239'
password= ‘password’
port = 10022
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect(hostname=hostname, username=username, password=password,port=port)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/site-packages/paramiko/client.py", line 380, in connect
look_for_keys, gss_auth, gss_kex, gss_deleg_creds, gss_host)
File "/usr/lib/python2.7/site-packages/paramiko/client.py", line 597, in _auth
raise saved_exception
paramiko.ssh_exception.AuthenticationException: Authentication failed.
I am using:
>>> print paramiko.__version__
1.16.1
and python Python 2.7.5 (on linux)
http://docs.paramiko.org/en/2.4/api/transport.html# (these are for version 2.4, but note I'm using earlier version)
http://docs.paramiko.org/en/2.4/api/client.html
I also looked at this : Why does Paramiko hang if you use it while loading a module?
but still having connection issues.

I'm not quite sure if that answers your question 100 percent, but I thought better than no answer at all, especially since I'm pretty sure it can solve your problem, even though I haven't tested it in this release (no default port).
# coding: utf-8
from fabric.api import env, execute # hosts
from fabric.network import ssh
from fabric.operations import get, run, sudo # put
ssh.util.log_to_file("paramiko.log", 10)
env.host_string = 'remotehostname.com'
env.port = 10022
env.user = 'AB1239'
env.password = 'password'
# Attention: "rm -r" won't ask for password anymore!
env.warn_only = True
def blah():
sudo('ls')
get('/tmp/lolo*.xml', '/tmp', use_sudo=True) # from remote to local
# sudo('rm -r /tmp/what/no/')
execute(blah)

Executing sparql query from python to virtuoso server in linux?

I am having problem for running the following program (sparql_test.py). I am running it from Linux machine. I am installing Virtuoso server in the same Linux machine. In the Linux server, I don't have sudo permission nor browser access. But, I can execute SPARQL query from isql prompt (SQL>) successfully.
Program: sparql_test.py
from SPARQLWrapper import SPARQLWrapper, JSON
sparql = SPARQLWrapper("http://localhost:8890/sparql")
sparql.setQuery("select ?s where { ?s a <http://ehrofip.com/data/Admissions>.} limit 10")
sparql.setReturnFormat(JSON)
result = sparql.query().convert()
for res in result["results"]["bindings"]:
print(res)
I got the following error:
[suresh#deodar complex2vec]$ python sparql_test.py
Traceback (most recent call last):
File "sparql1.py", line 14, in "<module>"
result = sparql.query().convert()
File "/home/suresh/.local/lib/python2.7/site-packages/SPARQLWrapper/Wrapper.py", line 687, in query
return QueryResult(self._query())
File "/home/suresh/.local/lib/python2.7/site-packages/SPARQLWrapper/Wrapper.py", line 667, in _query
raise e
urllib2.HTTPError: HTTP Error 502: Bad Gateway
However, the above program run smoothly in my own laptop. What might be the problem? Is this issue of connection?
Thank you
Best,
Suresh

I do not believe this error is raised by Virtuoso. I believe it is raised by SPARQLWrapper.
It looks like there's something between the outside world (which includes the Linux machine itself) and the Virtuoso listener on port 8890. The "Bad Gateway" suggests there may be two things -- a reverse proxy, and a firewall.
Port 8890 (set as [HttpServer]:Listen in the INI file) must be open to communications, direct or proxied, for SPARQL access to work.
iSQL talks to port 1111 (set as [Parameters]:Listen in the INI file), which apparently doesn't have a similar block/proxy.

Cannot acquire connection to Neo4j database

I am trying to connect to my Neo4j graph database server from a new machine. I can successfully connect from an older machine but do not wish to use the older one anymore.
I have reduced the problem to a simple script that returns an exception:
from neo4j.v1 import GraphDatabase, basic_auth
auth = basic_auth("username","password")
session = GraphDatabase.driver("bolt://remote.server:7687",auth=auth).session()
statement = """MATCH (a:Protein)
WHERE a.name={name}
RETURN a.Accession"""
tx = session.begin_transaction()
record = tx.run(statement,{'name':"ARCH_HUMAN"}).single()
print record['a.Accession']
session.close()
And the error message is:
File "Test.py", line 10, in <module>
tx = session.begin_transaction()
File "/home/username/anaconda2/lib/python2.7/site-packages/neo4j/v1/api.py", line 432, in begin_transaction
self._connect()
File "/home/username/anaconda2/lib/python2.7/site-packages/neo4j/v1/api.py", line 269, in _connect
self._connection = self._acquirer(access_mode)
File "/home/username/anaconda2/lib/python2.7/site-packages/neo4j/v1/direct.py", line 52, in acquire
raise ServiceUnavailable("Cannot acquire connection to {!r}".format(self.address))
neo4j.exceptions.ServiceUnavailable: Cannot acquire connection to Address(host='remote.server', port=7687)
Port 7687 is open (confirmed via netstat -tulpn and iptables -L), and neo4j is configured to listen to 0.0.0.0:7687. In addition, .neo4j/known_hosts contains an entry for host 0.0.0.0
What's strange is that I get a different error message (neo4j.exceptions.AuthError) if I break the authentication by using an incorrect password. So the connection is being made to check the password, but still I cannot connect with the correct auth.
What's going on?

I too had the same issue and turns out the driver was the issue.
I did some experiments and found out that the last driver that it works for is neo4j-driver==v1.1.0 but the next version neo4j-driver==v1.2.0 it stops working for some reason.

Try uncomment dbms.connectors.default_listen_address=0.0.0.0 And check this
# Bolt connector
dbms.connector.bolt.enabled=true
dbms.connector.bolt.tls_level=OPTIONAL
dbms.connector.bolt.listen_address=:7687
# HTTP Connector. There must be exactly one HTTP connector.
dbms.connector.http.enabled=true
dbms.connector.http.listen_address=:7474
# HTTPS Connector. There can be zero or one HTTPS connectors.
dbms.connector.https.enabled=true
dbms.connector.https.listen_address=:7473