I'm running a script to check if pymongo exceptions are successfully caught, but so far the only errors I get are Pycharm IDE errors. I turned the mongo daemon off to test this out. The relevant portion of the script is as follows, with IDE errors following after it:
import pymongo
from pymongo import MongoClient
from pymongo import errors
import os
from os.path import basename
def make_collection_name(path):
filename = os.path.splitext(basename(path))[0]
collection_name = filename
if collection_name in db.collection_names():
db[collection_name].drop()
return collection_name
if __name__ == '__main__':
try:
client = MongoClient()
except pymongo.errors.ConnectionFailure as e:
print("Could not connect to MongoDB: %s") % e
except pymongo.errors.ServerSelectionTimeoutError as e:
print("Could not connect to MongoDB: %s") % e
filepath = **hidden filepath**
db = client.TESTDB
collection_name = make_collection_name(filepath)
Instead of having the exceptions handled, I rather get the following errors from the IDE:
Traceback (most recent call last):
File "**hidden path**", line 278, in <module>
collection_name = make_collection_name(filepath)
File "**hidden path**", line 192, in make_collection_name
if collection_name in db.collection_names():
File "C:\Python34\lib\site-packages\pymongo\database.py", line 530, in collection_names
ReadPreference.PRIMARY) as (sock_info, slave_okay):
File "C:\Python34\lib\contextlib.py", line 59, in __enter__
return next(self.gen)
File "C:\Python34\lib\site-packages\pymongo\mongo_client.py", line 859, in _socket_for_reads
with self._get_socket(read_preference) as sock_info:
File "C:\Python34\lib\contextlib.py", line 59, in __enter__
return next(self.gen)
File "C:\Python34\lib\site-packages\pymongo\mongo_client.py", line 823, in _get_socket
server = self._get_topology().select_server(selector)
File "C:\Python34\lib\site-packages\pymongo\topology.py", line 214, in select_server
address))
File "C:\Python34\lib\site-packages\pymongo\topology.py", line 189, in select_servers
self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [WinError 10061] No connection could be made because the target machine actively refused it
Process finished with exit code 1
Beginning in PyMongo 3 (not Python 3, PyMongo 3!), the MongoClient constructor no longer blocks trying to connect to the MongoDB server. Instead, the first actual operation you do will wait until the connection completes, and then throw an exception if connection fails.
http://api.mongodb.com/python/current/migrate-to-pymongo3.html#mongoclient-connects-asynchronously
As you can see from your stack trace, the exception is thrown from db.collection_names(), not from MongoClient(). So, wrap your call to make_collection_name in try / except, not the MongoClient call.
Related
A MySQL server is initialized in Flask (with connexion) on service startup.
service.app.datastore = DatastoreMySQL(service.config)
class DatastoreMySQL(Datastore):
def __init__(self, config):
...
self.connection_pool = pooling.MySQLConnectionPool(
database=self.database,
host=self.hostname,
username=self.username,
password=self.password,
pool_name="pool_name",
pool_size=self.pool_size,
autocommit=True
)
def exec_query(self, query, params=None):
try:
connection = self.connection_pool.get_connection()
connection.ping(reconnect=True)
with closing(connection.cursor(dictionary=True, buffered=True)) as cursor:
if params:
cursor.execute(query, params)
else:
cursor.execute(query)
finally:
connection.close()
The view functions use the database by passing the DB reference from current_app.
def new():
do_something_in_db(current_app.datastore, request.get_json())
def do_something_in_db(db, data):
db.create_new_item(data)
...
However, a background process (run with APScheduler) must also run do_something_in_db(), but when passed a datastore reference an mysql.connector.errors.OperationalError error is thrown.
My understanding is that this error comes from two sources:
The server timed out and closed the connection. However, in this service the exec_query() function obtains a connection and executes right away, so there should be no reason that it times out. The monitor is also initialized at service startup with a datastore reference, but I am not sure how that can time out given that a new connection is created each time exec_query() is called.
The server dropped an incorrect or too large packet. However, there are no packets here - the process is run by a local background scheduler.
The error in full:
Job "Monitor.monitor_running_queries" raised an exception
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/mysql/connector/connection_cext.py", line 509, in cmd_query
raw_as_string=raw_as_string)
_mysql_connector.MySQLInterfaceError: MySQL server has gone away
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/k8s-service/lib/datastore/datastore_mysql.py", line 88, in exec_query
cursor.execute(query, params)
File "/usr/local/lib/python3.6/site-packages/mysql/connector/cursor_cext.py", line 276, in execute
raw_as_string=self._raw_as_string)
File "/usr/local/lib/python3.6/site-packages/mysql/connector/connection_cext.py", line 512, in cmd_query
sqlstate=exc.sqlstate)
mysql.connector.errors.DatabaseError: 2006 (HY000): MySQL server has gone away
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/apscheduler/executors/base.py", line 125, in run_job
retval = job.func(*job.args, **job.kwargs)
File "/opt/k8s-service/lib/background.py", line 60, in monitor_running_queries
self.handle_process_state(query.id, datastore, hive)
File "/opt/k8s-service/lib/background.py", line 66, in handle_process_state
query = datastore.get_item(query_id)
File "/opt/k8s-service/lib/datastore/datastore.py", line 48, in get_item
return_results=True)
File "/opt/k8s-service/lib/datastore/datastore.py", line 97, in exec_query
connection.close()
File "/usr/local/lib/python3.6/site-packages/mysql/connector/pooling.py", line 131, in close
cnx.reset_session()
File "/usr/local/lib/python3.6/site-packages/mysql/connector/connection_cext.py", line 768, in reset_session
raise errors.OperationalError("MySQL Connection not available.")
mysql.connector.errors.OperationalError: MySQL Connection not available.
I'm writing a small app that writes JSON files to a Mongo collection and then retrieves them based on a given input. I have my Mongo settings stored as a Pydantic model as follows in settings.py:
from typing import Optional
from pydantic import BaseSettings
__all__ = ["mongo_settings"]
class MongoSettings(BaseSettings):
host: str = "mongodb://127.0.0.1"
port: int = 27017
minPoolSize: int = 10
maxPoolSize: int = 10
username: str = None
password: str = None
authSource: Optional[str] = ""
tls: bool = True
tlsAllowInvalidCertificates: bool = True
mongo_settings = MongoSettings()
these are used in a Database class with a method that returns a MongoClient:
from pymongo import MongoClient
from .settings import mongo_settings
class Database:
client: MongoClient = None
def get_database() -> MongoClient:
client = MongoClient(**mongo_settings.dict())
return client
I've written an (extremely) basic test to check that my app can connect to a locally running instance of Mongo:
from pymongo import MongoClient
from database.db import Database
def test_database_connection():
mongo: MongoClient = Database.get_database()
mongo.server_info()
However, this fails with the following errors:
Traceback (most recent call last):
File "/code/project-env/lib/python3.8/site-packages/pymongo/pool.py", line 909, in _configured_socket
sock = ssl_context.wrap_socket(sock, server_hostname=host)
File "/home/.pyenv/versions/3.8.2/lib/python3.8/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/home/.pyenv/versions/3.8.2/lib/python3.8/ssl.py", line 1040, in _create
self.do_handshake()
File "/home/.pyenv/versions/3.8.2/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:1108)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/code/project-env/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1749, in _process_periodic_tasks
self._topology.update_pool()
File "/code/project-env/lib/python3.8/site-packages/pymongo/topology.py", line 438, in update_pool
server._pool.remove_stale_sockets(pool_id)
File "/code/project-env/lib/python3.8/site-packages/pymongo/pool.py", line 1060, in remove_stale_sockets
sock_info = self.connect()
File "/code/project-env/lib/python3.8/site-packages/pymongo/pool.py", line 1089, in connect
sock = _configured_socket(self.address, self.opts)
File "/code/project-env/lib/python3.8/site-packages/pymongo/pool.py", line 922, in _configured_socket
_raise_connection_failure(address, exc, "SSL handshake failed: ")
File "/code/project-env/lib/python3.8/site-packages/pymongo/pool.py", line 290, in _raise_connection_failure
raise AutoReconnect(msg)
pymongo.errors.AutoReconnect: SSL handshake failed: 127.0.0.1:27017: EOF occurred in violation of protocol (_ssl.c:1108)
I've tried playing around with the /etc/mongod.conf file to see if that will solve the problem, with the net block currently looking like this:
net:
port: 27017
ssl:
mode: requireSSL
CAFile: "~/.ssl/ca.pem"
PEMKeyFile: "~/.ssl/server.pem"
PEMKeyPassword: 'XXXXXXXXX'
allowConnectionsWithoutCertificates: true
but this hasn't yielded any results. I feel like I'm missing something obvious here which is causing it to not work, does anybody have any insight as to what I'm doing wrong?
Turns out there was nothing wrong with the code itself - the tilde shortcut in the mongod.conf file meant the certificates weren't being read properly. Swapping that out for an absolute path seems to have solved the problem 👍
I receive following output:
Traceback (most recent call last):
File "/home/ec2-user/env/lib64/python3.7/site-packages/redis/connection.py", line 1192, in get_connection
raise ConnectionError('Connection has data')
redis.exceptions.ConnectionError: Connection has data
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ec2-user/env/lib64/python3.7/site-packages/eventlet/hubs/hub.py", line 457, in fire_timers
timer()
File "/home/ec2-user/env/lib64/python3.7/site-packages/eventlet/hubs/timer.py", line 58, in __call__
cb(*args, **kw)
File "/home/ec2-user/env/lib64/python3.7/site-packages/eventlet/greenthread.py", line 214, in main
result = function(*args, **kwargs)
File "crawler.py", line 53, in fetch_listing
url = dequeue_url()
File "/home/ec2-user/WebCrawler/helpers.py", line 109, in dequeue_url
return redis.spop("listing_url_queue")
File "/home/ec2-user/env/lib64/python3.7/site-packages/redis/client.py", line 2255, in spop
return self.execute_command('SPOP', name, *args)
File "/home/ec2-user/env/lib64/python3.7/site-packages/redis/client.py", line 875, in execute_command
conn = self.connection or pool.get_connection(command_name, **options)
File "/home/ec2-user/env/lib64/python3.7/site-packages/redis/connection.py", line 1197, in get_connection
raise ConnectionError('Connection not ready')
redis.exceptions.ConnectionError: Connection not ready
I couldn't find any issue related to this particular error. I emptied/flushed all redis databases, so there should be no data there. I assume it has something to do with eventlet and patching. But even when I put following code right at the beginning of the file, the error appears.
import eventlet
eventlet.monkey_path()
What does this error mean?
Finally, I came up with the answer to my problem.
When connecting to redis with python, I specified the database with the number 0.
redis = redis.Redis(host=example.com, port=6379, db=0)
After changing the dabase to number 1 it worked.
redis = redis.Redis(host=example.com, port=6379, db=1)
Another way is to set protected_mode to no in etc\redis\redis.conf. Recommended when running redis locally.
I'm using Python 3.4 the latest version of MongoDB and already installed pymongo using
pip install pymongo
When I run this .py file
from pymongo import MongoClient
if __name__ == "__main__":
con = MongoClient()
db = con.test_database
people = db.people
people.insert({'name':'Mike', 'food':'cheese'})
people.insert({'name':'John', 'food':'ham', 'location':'UK'})
people.insert({'name':'Michelle', 'food':'cheese'})
peeps = people.find()
print("INSERT & FIND TEST")
for person in peeps:
print(person)
I got this error:
Traceback (most recent call last):
File "C:/Users/Alberto/Documents/Python/Kivy/Code/testmongo.py", line 10, in <module>
people.insert({'name':'Mike', 'food':'cheese'})
File "C:\Python34\lib\site-packages\pymongo\collection.py", line 2197, in insert
with self._socket_for_writes() as sock_info:
File "C:\Python34\lib\contextlib.py", line 59, in __enter__
return next(self.gen)
File "C:\Python34\lib\site-packages\pymongo\mongo_client.py", line 712, in _get_socket
server = self._get_topology().select_server(selector)
File "C:\Python34\lib\site-packages\pymongo\topology.py", line 142, in select_server
address))
File "C:\Python34\lib\site-packages\pymongo\topology.py", line 118, in select_servers
self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [WinError 10061] No connection could be made because the target machine actively refused it
Anyone know how to fix this?
if you are using Local Mongodb
myclient = pymongo.MongoClient('mongodb://<yourUsername>:<yourPassword>#<IPAddress>:27017/<yourDB>?retryWrites=true&w=majority')
If you are using Atlas Mongodb
myclient = pymongo.MongoClient('mongodb+srv://<yourUsername>:<yourPassword>#<YourCluster>.mongodb.net/<yourDB>?retryWrites=true&w=majority')
Just try to replace the connection with the following line.
con = MongoClient('localhost', 27017)
It works fine for me.
I am trying to connect to a MySQL database using python but I am getting a strange error. It is compounded by the fact that I can use the same connection values from the mysql console command and it connects with no problems.
Here is the exact code I am using:
import pymysql
from checks import AgentCheck
class DelayedJobCheck(AgentCheck):
def check(self, instance):
self.log.info("testing connection")
self.log.info(instance)
connection = pymysql.connect(**instance)
cur = cnx.cursor(buffered=True)
cur.execute("SHOW STATUS LIKE 'Ssl_cipher'")
print(cur.fetchone())
cur.close()
cnx.close()
self.gauge('hello.world', 1)
This is the error that I am getting:
Traceback (most recent call last):
File "/opt/datadog-agent/agent/checks/__init__.py", line 661, in run
self.check(copy.deepcopy(instance))
File "/opt/datadog-agent/agent/checks.d/delayed_job.py", line 10, in check
connection = pymysql.connect(**instance)
File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/pymysql/__init__.py", line 88, in Connect
return Connection(*args, **kwargs)
File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/pymysql/connections.py", line 644, in __init__
self._connect()
File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/pymysql/connections.py", line 869, in _connect
raise exc
OperationalError: (2003, u"Can't connect to MySQL server on '192.168.199.86' ([SSL: SSL_NEGATIVE_LENGTH] dh key too small (_ssl.c:590))")
I am running this code on a Ubuntu box and I though initially that it might be because the SSL CA is a self generated cert. So I followed the steps here But, it did not make any difference. Also I have verified that the process that is running this code has full access to the cert files
Any ideas what else might be causing this?
As the err info said dh key is too small, a larger one might help. Replace the default dh512.pem file with dh4096.pem
sudo wget "https://git.openssl.org/gitweb/?p=openssl.git;a=blob_plain;f=apps/dh4096.pem" -O dh4096.pem
Ref: http://www.alexrhino.net/jekyll/update/2015/07/14/dh-params-test-fail.html