I'm using Flask-SqlAlchemy with a MySQL database. And recently I started getting this error:
sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (1040, 'Too many connections')
After some digging, it seems that I'm not using the connection pooling.
Based on my research SQLALCHEMY_POOL_SIZE is now deprecated in v2.4 and shouldn't be used anymore.
SQLAlchemy==1.3.7
Flask-SQLAlchemy==2.4.1
So what is the correct way of doing it?
Config:
SQLALCHEMY_DATABASE_URI = 'mysql://root:PASSWORD#localhost/main_db'
SQLALCHEMY_BINDS = {
'radius': 'mysql://root:PASSWORD#localhost/radius_db',
'cache': 'mysql://root:PASSWORD#localhost/cache_db',
}
Code:
def make_app():
app = Flask(__name__, template_folder="../templates")
app.config.from_object(config)
db.init_app(app)
app = my_fabric.make_app()
According to SqlAlchemy, I'm supposed to do this via create_engine
engine = create_engine("mysql+pymysql://user:pw#host/db", pool_size=20, max_overflow=0))
But Flask-SqlAlchemy is supposed to abstract this out. So I don't know how this should be configured.
UPDATE:
I'm running uWSGI with two processes.
I have now increased the max_connections to 500. It's hard to say if I have high traffic, but my database statistics after 16 hours showing this:
mysql> show status like '%onn%';
+-------------------------------------------------------+---------------------+
| Variable_name | Value |
+-------------------------------------------------------+---------------------+
| Aborted_connects | 5 |
| Connection_errors_accept | 0 |
| Connection_errors_internal | 0 |
| Connection_errors_max_connections | 0 |
| Connection_errors_peer_address | 0 |
| Connection_errors_select | 0 |
| Connection_errors_tcpwrap | 0 |
| Connections | 3897 |
| Locked_connects | 0 |
| Max_used_connections | 167 |
| Max_used_connections_time | 2019-11-29 00:11:51 |
| Mysqlx_connection_accept_errors | 0 |
| Mysqlx_connection_errors | 0 |
| Mysqlx_connections_accepted | 0 |
| Mysqlx_connections_closed | 0 |
| Mysqlx_connections_rejected | 0 |
| Performance_schema_session_connect_attrs_longest_seen | 117 |
| Performance_schema_session_connect_attrs_lost | 0 |
| Ssl_client_connects | 0 |
| Ssl_connect_renegotiates | 0 |
| Ssl_finished_connects | 0 |
| Threads_connected | 97 |
+-------------------------------------------------------+---------------------+
AND
mysql> SHOW STATUS WHERE variable_name LIKE "Threads_%" OR variable_name = "Connections";
+-------------------+-------+
| Variable_name | Value |
+-------------------+-------+
| Connections | 3896 |
| Threads_cached | 8 |
| Threads_connected | 97 |
| Threads_created | 365 |
| Threads_running | 2 |
+-------------------+-------+
AND
mysql> SHOW VARIABLES LIKE 'max_connections';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| max_connections | 500 |
+-----------------+-------+
1 row in set (0.01 sec)
You can use sessions instead of connections
my_session = Session(engine)
results = my_session.execute(query)
my_session.close()
and when creating the engine you can set
pool_recycle=60.
(or a little higher). https://docs.sqlalchemy.org/en/13/core/pooling.html#pool-setting-recycle
Not saying this will solve your issues entirely, but I've rarely encountered problems using this setup.
Related
I am working on automating a process that uses our ancient HRIS system that unfortunately doesn't have API Access.
I am fairly new to Python, so I have been taking this task bit by bit. I've managed to connect to the app and input my username and password to sign in. However, I am stuck on selecting a menu item. I've tried everything that I know to do and have Googled until I've gone cross-eyed.
Dialog - 'City of Conway LIVE Springbrook V7' (L0, T0, R1032, B1039)
['Dialog', 'City of Conway LIVE Springbrook V7Dialog', 'City of Conway LIVE Springbrook V7', 'Dialog0', 'Dialog1']
child_window(title="City of Conway LIVE Springbrook V7", auto_id="MainMenu", control_type="Window")
|
| Pane - '' (L231, T87, R1024, B118)
| ['Pane', 'Pane0', 'Pane1']
| child_window(auto_id="_panelExWorkArea", control_type="Pane")
| |
| | Pane - 'Desktop' (L231, T90, R1021, B115)
| | ['DesktopPane', 'Desktop', 'Pane2']
| | child_window(title="Desktop", auto_id="_ssiGroupHeaderWorkArea", control_type="Pane")
|
| Pane - '' (L228, T87, R231, B1005)
| ['Pane3']
| child_window(auto_id="_ssiExpandableSplitter1", control_type="Pane")
|
| Pane - '' (L8, T87, R228, B1005)
| ['Pane4']
| child_window(auto_id="_panelTaskArea", control_type="Pane")
| |
| | Pane - '' (L11, T90, R228, B1002)
| | ['Pane5']
| | child_window(auto_id="328582", control_type="Pane")
| | |
| | | TreeView - '' (L11, T115, R228, B1002)
| | | ['TreeView', 'TreeView0', 'TreeView1']
| | | child_window(auto_id="1775914", control_type="Tree")
| | | |
| | | | Pane - '' (L28, T269, R194, B609)
| | | | ['Pane6']
| | | | child_window(auto_id="726924", control_type="Pane")
| | | | |
| | | | | Pane - '' (L28, T269, R194, B609)
| | | | | ['Pane7']
| | | | | child_window(auto_id="1317216", control_type="Pane")
| | | | | |
| | | | | | TreeView - '' (L28, T269, R194, B609)
| | | | | | ['TreeView2']
| | | | | | child_window(auto_id="2101028", control_type="Tree")
| | | | | | |
| | | | | | | Custom - 'Maintenance' (L0, T0, R0, B0)
| | | | | | | ['Custom', 'Maintenance', 'MaintenanceCustom', 'Custom0', 'Custom1', 'Maintenance0', 'Maintenance1', 'MaintenanceCustom0', 'MaintenanceCustom1']
| | | | | | | child_window(title="Maintenance", control_type="Custom")
I'm using a few tools to inspect the GUI, and this one specifically allows me to do the desired task by selecting "do it". It allows me to expand and collapse the section, so surely I've got to be missing something somewhere?
enter image description here
enter image description here
Here is my code:
from pywinauto import Application
app=Application(backend="uia").connect(path=r"C:\Users\skywalker\AppData\Local\Apps\2.0\C38DNYDP.PZ6\07BV1NGN.8G6\spri..ons1_b443b3e57637483a_0007.000f_52ec298e739bfebb", timeout = 30)
maintenance = app.CityofConwayLIVESpringbrookV7.WindowsForms10.Window.8.app.0.a0f91b_r8_ad1, 263022
maintenance.click()
I would also like to mention that I CAN get it to work with Click_Input, but I would like to avoid that if at all possible.
I have this table in Postgres
ID| | IP | Remote-as | IRR-Record |
1 | | 192.168.1.1 |100 | |
2 | | 192.168.2.1 |200 | |
3 | | 192.168.3.1 |300 | |
4 | | 192.168.4.1 |400 | |
I want to add for each ip address the IIR-Record.
The IIR-record is inside a variable.
c = conn.cursor()
query = 'select * From "peers"'
c.execute(query)
for row in c:
c.execute('''INSERT INTO "peers" ("IRR-
Record") VALUES(variable)
conn.commit()
This code doesn't work because i gat the IIR-Record at the end of my table.
ID| | IP | Remote-as | IRR-Record |
1 | | 192.168.1.1 |100 | |
2 | | 192.168.2.1 |200 | |
3 | | 192.168.3.1 |300 | |
4 | | 192.168.4.1 |400 |
|Variable
|Variable
|Varibale
any Idea!!!!
You need to use an UPDATE query instead of INSERT
UPDATE peers
SET IRR-Record = <<MYVALUE>>
WHERE ID = <<MYID>>
I think it's something like this:
for row in c:
c.execute('UPDATE peers SET "IRR-Record"=%s WHERE ID=%s', (record_var, id_var))
Edit: Use UPDATE sql statement as per #Devasta's answer.
In Ruby console, it is possible to display SQL query results in a very human-friendly way (ActiveRecord + Hirb):
>> Tag.all :limit=>3, :order=>"id DESC"
+-----+-------------------------+-------------+-------------------+-----------+-----------+----------+
| id | created_at | description | name | namespace | predicate | value |
+-----+-------------------------+-------------+-------------------+-----------+-----------+----------+
| 907 | 2009-03-06 21:10:41 UTC | | gem:tags=yaml | gem | tags | yaml |
| 906 | 2009-03-06 08:47:04 UTC | | gem:tags=nomonkey | gem | tags | nomonkey |
| 905 | 2009-03-04 00:30:10 UTC | | article:tags=ruby | article | tags | ruby |
+-----+-------------------------+-------------+-------------------+-----------+-----------+----------+
3 rows in set
Is there a module that will allow me to do display SQLAlchemy result sets in a similar way in IPython?
According to the mysql documentation which supports only up to 3 byte utf-8 unicode encoding.
My question is, how can I replace characters that require 4 byte utf-8 encoding in my database? And how do I decode those characters in order to display exactly what the user wrote?
Part of the integration test:
description = u'baaam á ✓ ✌ ❤'
print description
test_convention = Blog.objects.create(title="test title",
description=description,
login=self.user,
tag=self.tag)
Error:
Creating test database for alias 'default'...
baaam á ✓ ✌ ❤
E..
======================================================================
ERROR: test_post_blog (blogs.tests.PostTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/admin/Developer/project/pro/blogs/tests.py", line 64, in test_post_blog
tag=self.tag)
File "build/bdist.macosx-10.9-intel/egg/MySQLdb/cursors.py", line 201, in execute
self.errorhandler(self, exc, value)
File "build/bdist.macosx-10.9-intel/egg/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
DatabaseError: (1366, "Incorrect string value: '\\xE2\\x9C\\x93 \\xE2\\x9C...' for column 'description' at row 1")
----------------------------------------------------------------------
Ran 3 tests in 1.383s
FAILED (errors=1)
Destroying test database for alias 'default'...
Table's configuration:
+----------------------------------+--------+---------+-------------------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+----------+----------------+---------+
| Name | Engine | Version | Collation | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Checksum | Create_options | Comment |
+----------------------------------+--------+---------+-------------------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+----------+----------------+---------+
| blogs_blog | InnoDB | 10 | utf8_general_ci | Compact | 25 | 1966 | 49152 | 0 | 32768 | 0 | 35 | 2014-02-09 00:57:59 | NULL | NULL | NULL | | |
+----------------------------------+--------+---------+-------------------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+----------+----------------+---------+
Update: I already changed the table and column configurations from utf-8 to utf8mb4 and still getting the same error, any ideas?
+----------------------------------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+--------------------+----------+----------------+---------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |
+----------------------------------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+--------------------+----------+----------------+---------+
| blogs_blog | InnoDB | 10 | Compact | 5 | 3276 | 16384 | 0 | 32768 | 0 | 36 | 2014-02-17 22:24:18 | NULL | NULL | utf8mb4_general_ci | NULL | | |
+----------------------------------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+--------------------+----------+----------------+---------+
and:
+---------------+--------------+--------------------+------+-----+---------+----------------+---------------------------------+---------+
| Field | Type | Collation | Null | Key | Default | Extra | Privileges | Comment |
+---------------+--------------+--------------------+------+-----+---------+----------------+---------------------------------+---------+
| id | int(11) | NULL | NO | PRI | NULL | auto_increment | select,insert,update,references | |
| title | varchar(500) | latin1_swedish_ci | NO | | NULL | | select,insert,update,references | |
| description | longtext | utf8mb4_general_ci | YES | | NULL | | select,insert,update,references | |
| creation_date | datetime | NULL | NO | | NULL | | select,insert,update,references | |
| login_id | int(11) | NULL | NO | MUL | NULL | | select,insert,update,references | |
| tag_id | int(11) | NULL | NO | MUL | NULL | | select,insert,update,references | |
+---------------+--------------+--------------------+------+-----+---------+----------------+---------------------------------+---------+
It is supported, but not asutf8. Add the following to the [mysqld] section of my.cnf:
character-set-server=utf8mb4
collation-server=utf8mb4_unicode_ci
When creating a database, use:
CREATE DATABASE xxxxx DEFAULT CHARACTER SET utf8mb4 DEFAULT COLLATE utf8mb4_unicode_ci;
At the end of a CREATE TABLE command, add:
ENGINE=InnoDB ROW_FORMAT=COMPRESSED DEFAULT CHARSET=utf8mb4;
In Python, how do I turn RDF/SKOS taxonomy data into a dictionary that represents the concept hierarchy only?
The dictionary must have this format:
{ 'term1': [ 'term2', 'term3'], 'term3': [{'term4' : ['term5', 'term6']}, 'term6']}
I tried using RDFLib with JSON plugins, but did not get the result I want.
I'm not much of a Python user, and I haven't worked with RDFLib, but I just pulled the SKOS and vocabulary from the SKOS vocabularies page. I wasn't sure what concepts (RDFS or OWL classes) were in the vocabulary, nor what their hierarchy was, so I ran this a SPARQL query using Jena's ARQ to select classes and their subclasses. I didn't get any results. (There were classes defined of course, but none had subclasses.) Then I decided to use both the SKOS and SKOS-XL vocabularies, and to ask for properties and subproperties as well as classes and subclasses. This is the SPARQL query I used:
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
SELECT ?property ?subproperty ?class ?subclass WHERE {
{ ?subclass rdfs:subClassOf ?class }
UNION
{ ?subproperty rdfs:subPropertyOf ?property }
}
ORDER BY ?class ?property
The results I got were
-------------------------------------------------------------------------------------------------------------------
| property | subproperty | class | subclass |
===================================================================================================================
| rdfs:label | skos:altLabel | | |
| rdfs:label | skos:hiddenLabel | | |
| rdfs:label | skos:prefLabel | | |
| skos:broader | skos:broadMatch | | |
| skos:broaderTransitive | skos:broader | | |
| skos:closeMatch | skos:exactMatch | | |
| skos:inScheme | skos:topConceptOf | | |
| skos:mappingRelation | skos:broadMatch | | |
| skos:mappingRelation | skos:closeMatch | | |
| skos:mappingRelation | skos:narrowMatch | | |
| skos:mappingRelation | skos:relatedMatch | | |
| skos:narrower | skos:narrowMatch | | |
| skos:narrowerTransitive | skos:narrower | | |
| skos:note | skos:changeNote | | |
| skos:note | skos:definition | | |
| skos:note | skos:editorialNote | | |
| skos:note | skos:example | | |
| skos:note | skos:historyNote | | |
| skos:note | skos:scopeNote | | |
| skos:related | skos:relatedMatch | | |
| skos:semanticRelation | skos:broaderTransitive | | |
| skos:semanticRelation | skos:mappingRelation | | |
| skos:semanticRelation | skos:narrowerTransitive | | |
| skos:semanticRelation | skos:related | | |
| | | _:b0 | <http://www.w3.org/2008/05/skos-xl#Label> |
| | | skos:Collection | skos:OrderedCollection |
-------------------------------------------------------------------------------------------------------------------
It looks like there's not much concept hierarchy in SKOS at all. Could that explain why you didn't get the results you wanted before?