Develop Reference

mysql.connector to AWS RDS database timing out

I have an RDS database that a program I created using python and Mysql connect to, in order to keep track of usage of the program. Anytime the program is used, it adds 1 to a counter on the RDS database. Just this week the program has started throwing an error connecting to the RDS SQL database after about an hour of use. Previous to this, I could leave the software running for days without ever timing out. Closing the software and re-opening it, to re-establish the connection allows me to connect for approx another hour or so before it times out again.
I am connecting using the following parameters:
awsConn = mysql.connector.connect(host='myDatabase.randomStringofChars.us-east-1.rds.amazonaws.com', database='myDatabase', port=3306, user='username', password='password')
Did something recently change with AWS/RDS, do I just need to pass a different parameter into the connection string, or do I just need to add somewhere into my program to attempt to re-establish the connection every so often?
Thanks

server starts to closed the connection unexpectedly

I have a project with 10+ parsers and at the end have this code:
`
cursor = conn.cursor()
my_file = open(r'csv\file.csv')
sql_statement = """
CREATE TEMP TABLE temp
(
LIKE vhcl
)
ON COMMIT DROP;
COPY temp FROM STDIN WITH
CSV
HEADER
DELIMITER AS ',';
INSERT INTO vhcl
SELECT *
FROM temp
ON CONFLICT (id) DO UPDATE SET name= EXCLUDED.name"""
cursor.copy_expert(sql=sql_statement, file=my_file)
conn.commit()
cursor.close()
`
Everything worked fine until a couple of weeks ago I started to get these errors:
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
I noticed, that if parsers works (for example) less, than 10 minutes, I won't get those errors
I tried to make a separate function, that adds data to the DB after the parser ends working.
It still gives me that error. The strange thing is that I ran my parsers on my home pc, and it works fine, also, if I add data manually with the same function, but in a different file, it also works fine.
I asked about banned IP for db, but it's okay. So I have no idea why I have this error.
PostgreSQL log

Finally, I found a solution. I still don't know what the problem was. It isn't a connection issue, cause some parsers with the same IP and same network connections work normally. And I'm was still able to add data with the same script, but in a separate project file.
My solution is to add 'keepalives' settings in connection:
conn = psycopg2.connect(
host=hostname,
dbname=database,
user=username,
password=password,
port=port_id,
keepalives=1,
keepalives_idle=30,
keepalives_interval=10,
keepalives_count=5)

You have a network problem.
Both the server and the client complain that the other side unexpectedly hung up on them. So it was some misconfigured network component in the middle that cut the line. You have two options:
fix the network
lower tcp_keepalives_idle on the PostgreSQL client or server, so that the operating system sends "keepalive packets" frequently and the network doesn't consider the connection idle
You might want to read this for more details.

Django can't close persistent Mysql connection

Problem:
Our Django application includes zipping large folders, which takes too long (up to 48 hours) so the Django connection with the database gets timed out and throws: "MySQL server has gone away error".
Description:
We have a Django version==3.2.1 application whose CONN_MAX_AGE value is set to 1500(seconds). The default wait_timeout in Mysql(MariaDB) is 8 hours.
ExportStatus table has the following attributes:
package_size
zip_time
Our application works this way:
Zip the folders
set the attribute 'package_size' of ExportStatus table after zipping and save in the database.
set the attribute 'zip_time' of ExportStatus table after zipping and save in the database.
Notice the setting of the columns' values in database. These require django connection with database, which gets timedout after long zipping process. Thus throws the MySQL server gone away error.
What we have tried so far:
from django.db import close_old_connections`
close_old_connections()
This solution doesn't work.
Just after zipping, if the time taken is more than 25 minutes, we close all the connections and ensure new connection as:
from django.db import connections
for connection in connections.all():
try:
# hack to check if the connection still persists with the database.
with connection.cursor() as c:
c.execute("DESCRIBE auth_group;")
c.fetchall()
except:
connection.close()
connection.ensure_connection()
Upon printing the value of the length of connections.all(), it is 2. What we don't understand is how Django persists those old connections and retrieves connections from the connection pool. When we close connections from connections.all(), aren't we closing all the connections in the thread pool?
We first set the package_size and then set the zip_time. The problem with this solution is that occasionally (not always), it throws the same error when setting the zip_time attribute. Sometimes, this solution does seem to work. There is no problem in setting the package_size but throws an error occasionally when setting the 'zip_time' attribute. So our question is if we already reset connections after zipping, why does this still take a stale connection from the connection pool and throws the MySQL server gone away error? Do we have any way to close all the old persistent connections and recreate new ones?

get process name for mongodb open connections

I need to know process/file name of the open MongoDB connections.
for example, assume there are files called F1,F2 ..., Fn using Connection pool to get mongodb connection. each running in parallel in different process.
Is there any way to get file name which having open connection to mongodb.
Because, I am on mission to reduce number of open mongodb connections.
when I did below query,
db.serverStatus().connections
It giving me current consumed connections count, available count. But I need filenames which opened connection to optimize.
stack: python,django,some server running in apache, mongodb, pymongo

I figured out myself how to know more about connections information.
db.currentOp(true).inprog
Above command will give all current connections information in array. you can see information such as client ip,whether its active or not,connection id,operation type and everything.

You can get the connection details in the most rudimentary form using a quick shell command such as
ps -eAf | grep mongo
If you use this command on the host running your mongod process. Essentially you can make a note of all active pid's and take corrective actions

MySQL, should I stay connected or connect when needed?

I have been logging temperatures at home to a MySQL database (read 10 sensors in total every 5 minutes), and have been using Python, but I am wondering something...
Currently when I first run my program, I run the normal connect to MySQL, which is only run once.
db = MySQLdb.connect(mysql_server, mysql_username, mysql_passwd, mysql_db)
cursor = db.cursor()
Then I collect the data and publish it to the database successfully. The script then sleeps for 5 minutes, then starts again and collects and publishes the data again and so on. However, I only connect once, and I don't ever disconnect; it just keeps going in a loop. I only disconnect if I terminate the program.
Is this the best practice? That is, keeping the connection open all the time to the MySQL server, or should I disconnect after I have done a insert/commit?
The reason I ask: every now and then, I have to restart the script because maybe my MySQL server has gone offline or some other issue. Should I:
Keep doing what I am doing and just handle any MySQL database disconnections with a reconnect,
Put it in the crontab to collect data every five minutes and have no loop and no sleep, or
Something else?

MySQL servers are configured to handle a fixed limited number of connections. It's not a good practice to tie up a connection that you are not using constantly. So typically you should close the connection as soon as you are done with it, and reconnect only when you need it again. MySQLdb's connections are context mangagers, so you could use the with-statement syntax to make closing the connection automatic.
connection = MySQLdb.connect(
host=config.HOST, user=config.USER,
passwd=config.PASS, db=config.MYDB, )
with connection as cursor:
print(cursor)
# the connection is closed for you automatically
# when Python leaves the `with-suite`.
For robustness, you might want to use try..except to handle the case when (even on the first run) connect fails to make a connection.
Having said that, I would just put it in a crontab entry and dispense with sleeping.