I am developing a 3D Scanner using a Raspberry Pi and a Canon SLR. To control the reflex and the turntable, I use a simple web interface that I developed.
But I have to make the link between a "Start" or "Pause" buttons between the HTML interface and the Python control program. So I create a database to store the settings for the SLR (ISO, Shutterspeed ...).
For example, is it possible to have a box in the database with a state true or false for the pause button and then read it continuously on the python program?
I made a little test code like this below :
import mysql.connector
import time
mydb = mysql.connector.connect(
host="localhost",
user="scanner",
passwd="valentin",
database="Scanner3D"
)
print(mydb)
mycursor = mydb.cursor()
mycursor.execute("SELECT a FROM test")
myresult = mycursor.fetchall()
print(myresult)
myresult = str(myresult)
print(myresult)
x = myresult[2:4]
print(x)
while x == "10":
print("Hello World")
time.sleep(1)
mycursor.execute("SELECT a FROM test")
myresult = mycursor.fetchall()
myresult = str(myresult)
x = myresult[2:4]
print(x)
Problem is when I change the value of A in the database, the program never displays the new value and never leaves the loop, what is wrong in here?
Consider to use real-time communication like websocket.
The simplest way is: When scanner scans some value and saves it to DB, send via websocket command to client, to pull new data from server. Or you can send new data right in websocket message.
Very nice tutorial is here: https://realpython.com/python-sockets/
Related
i was trying to follow this example from stackoverflow using pg_notify with python. I am not getting this to work. Nothing happens and python does not receive the notification.
Python 3.85 Using Psycopg2. PostgreSQL 13
python & postgresql: reliably check for updates in a specific table
First. I created a python function that listened to the postgreSQL. Then i went to pgadmin and
executed
select pg_notify('process', 'update');
My python function is below
def dblisten(connection):
cur = connection.cursor()
print("inside")
cur.execute("LISTEN process")
while True:
print("listening")
select.select([connection],[],[])
connection.poll()
events = []
while connection.notifies:
notify = connection.notifies.pop().payload
print ("Got NOTIFY:", datetime.datetime.now(), notify.pid, notify.channel, notify.payload)
if __name__ == '__main__': # If it's executed like a script (not imported)
connection = psycopg2.connect(host='host', user='user',
password='password')
connection.set_isolation_level(psycopg2.extensions.ISOLATION_LEVEL_AUTOCOMMIT)
dblisten(connection)
I have a program using Python + python mysql connector + Mysql which is installed in a network and uses the same database. How do I refresh a connection without restarting the program on other Machines?
The Program is installed in more than one Machine and connects to the same Mysql database which is located on a Server. The program is working properly, but,
when a new data is entered or modified "SQL INSERT, UPDATE.... Statments" from one machine is not reflected in all other Machines until the program is restarted, which means that a new connection is necessary to show the new database data.
So, I would like to know how to refresh the connection without restarting the program on other machines.
Here is the sample Connection code:
import mysql.connector as mc
conn = mc.connect(host='host', user='user', passwd='passw', db='db_name', port=port_number)
cur = conn.cursor()
How to refresh This Connection while the program is running?
closing and reopening connection may also solve your problem,
see following example in loop
import mysql.connector as mc
conn = mc.connect(host='host', user='user', passwd='passw', db='db_name', port=port_number)
while True:
# check if youre connected, if not, connect again
if (conn.is_connected() == False):
conn = mc.connect(host='host', user='user', passwd='passw', db='db_name', port=port_number)
cur = conn.cursor()
sql = "INSERT INTO db_name.myTable (name) VALUES (%(val)s);"
val = {'val':"Slim Shady"}
cur.execute(sql,val)
conn.commit()
conn.close()
After inserting or updating, you should do a commit to make sure all data it's writed to the DB. Then it will be visible for everyone:
conn.commit()
I have a script running which I want to process data when it's added to the database.
import mysql.connector
import time
wait_time = 2
mydb = mysql.connector.connect(
host="localhost",
user="xxx",
passwd="yyy",
database="my_database"
)
mycursor = mydb.cursor()
while True:
sql = "SELECT * FROM data WHERE processed = 0"
mycursor.execute(sql)
records = mycursor.fetchall()
for i, r in enumerate(records):
print(r)
time.sleep(wait_time)
However, if insert a row via different connection, this connection doesn't show it.
I.e. if I connect to my database via a third party app, and insert a row to
However if I restart the above script, it appears.
Any ideas?
Use a message queue (e.g. RabbitMQ). Get the third party App to use it. Message queue implementation has better APIs for processing information asynchronously. Even if you just use the message queue for storing the primary key of the database content.
Alternately enable binary logging and use a replication protocol library to process events.
I just faced the same error. and the easiest way to solve it is... defining mydb and mycursor inside the loop.
import mysql.connector
import time
wait_time = 2
while True:
mydb = mysql.connector.connect(
host="localhost",
user="xxx",
passwd="yyy",
database="my_database"
)
mycursor = mydb.cursor()
sql = "SELECT * FROM data WHERE processed = 0"
mycursor.execute(sql)
records = mycursor.fetchall()
for i, r in enumerate(records):
print(r)
time.sleep(wait_time)
I am currently using AWS Lambda (Python 3.6) to talk to a MySQL database. I also have Slack commands triggering the queries to the database. On occasion, I have noticed that I can change things directly through MySQL Workbench and then trigger a query through Slack which returns old values. I currently connect to MySQL outside of the python handler like this:
BOT_TOKEN = os.environ["BOT_TOKEN"]
ASSET_TABLE = os.environ["ASSET_TABLE"]
REGION_NAME = os.getenv('REGION_NAME', 'us-east-2')
DB_NAME = os.environ["DB_NAME"]
DB_PASSWORD = os.environ["DB_PASSWORD"]
DB_DATABASE = os.environ["DB_DATABASE"]
RDS_HOST = os.environ["RDS_HOST"]
port = os.environ["port"]
try:
conn = pymysql.connect(RDS_HOST, user=DB_NAME, passwd=DB_PASSWORD, db=DB_DATABASE, connect_timeout=5, cursorclass=pymysql.cursors.DictCursor)
cursor = conn.cursor()
except:
sys.exit()
The MySQL connection is done outside of any definition at the very top of my program. When Slack sends a command, I call another definition that then queries MySQL. This works okay sometimes, but other times can send my old data that has not updated. The whole layout is like this:
imports
SQL connections
SQL query definitions
handler definition
I tried moving the MySQL connection portion inside of the handler, but then the SQL query definitions do not recognize my cursor (out of scope, I guess).
So my question is, how do I handle this MySQL connection? Is it best to keep the MySQL connection outside of any definitions? Should I open and close the connection each time? Why is my data stale? Will Lambda ALWAYS run the entire routine or can it try to split the load between servers (I swear I read somewhere that I cannot rely on Lambda to always read my entire routine; sometimes it just reads the handler)?
I'm pretty new to all this, so any suggestions are much appreciated. Thanks!
Rest of the code if it helps:
################################################################################################################################################################################################################
# Slack Lambda handler.
################################################################################################################################################################################################################
################################################################################################################################################################################################################
# IMPORTS
###############
import sys
import os
import pymysql
import urllib
import math
################################################################################################################################################################################################################
################################################################################################################################################################################################################
# Grab data from AWS environment.
###############
BOT_TOKEN = os.environ["BOT_TOKEN"]
ASSET_TABLE = os.environ["ASSET_TABLE"]
REGION_NAME = os.getenv('REGION_NAME', 'us-east-2')
DB_NAME = os.environ["DB_NAME"]
DB_PASSWORD = os.environ["DB_PASSWORD"]
DB_DATABASE = os.environ["DB_DATABASE"]
RDS_HOST = os.environ["RDS_HOST"]
port = os.environ["port"]
################################################################################################################################################################################################################
################################################################################################################################################################################################################
# Attempt SQL connection.
###############
try:
conn = pymysql.connect(RDS_HOST, user=DB_NAME, passwd=DB_PASSWORD, db=DB_DATABASE, connect_timeout=5, cursorclass=pymysql.cursors.DictCursor)
cursor = conn.cursor()
except:
sys.exit()
################################################################################################################################################################################################################
# Define the URL of the targeted Slack API resource.
SLACK_URL = "https://slack.com/api/chat.postMessage"
################################################################################################################################################################################################################
# Function Definitions.
###############
def get_userExistance(user):
statement = f"SELECT 1 FROM slackDB.users WHERE userID LIKE '%{user}%' LIMIT 1"
cursor.execute(statement, args=None)
userExists = cursor.fetchone()
return userExists
def set_User(user):
statement = f"INSERT INTO `slackDB`.`users` (`userID`) VALUES ('{user}');"
cursor.execute(statement, args=None)
conn.commit()
return
################################################################################################################################################################################################################
################################################################################################################################################################################################################
# Slack command interactions.
###############
def lambda_handler(data, context):
# Slack challenge answer.
if "challenge" in data:
return data["challenge"]
# Grab the Slack channel data.
slack_event = data['event']
slack_userID = slack_event['user']
slack_text = slack_event['text']
channel_id = slack_event['channel']
slack_reply = ""
# Check sql connection.
try:
conn = pymysql.connect(RDS_HOST, user=DB_NAME, passwd=DB_PASSWORD, db=DB_DATABASE, connect_timeout=5, cursorclass=pymysql.cursors.DictCursor)
cursor = conn.cursor()
except pymysql.OperationalError:
connected = 0
else:
connected = 1
# Ignore bot messages.
if "bot_id" in slack_event:
slack_reply = ""
else:
# Start data sift.
if slack_text.startswith("!addme"):
if get_userExistance(slack_userID):
slack_reply = f"User {slack_userID} already exists"
else:
slack_reply = f"Adding user {slack_userID}"
set_user(slack_userID)
# We need to send back three pieces of information:
data = urllib.parse.urlencode(
(
("token", BOT_TOKEN),
("channel", channel_id),
("text", slack_reply)
)
)
data = data.encode("ascii")
# Construct the HTTP request that will be sent to the Slack API.
request = urllib.request.Request(
SLACK_URL,
data=data,
method="POST"
)
# Add a header mentioning that the text is URL-encoded.
request.add_header(
"Content-Type",
"application/x-www-form-urlencoded"
)
# Fire off the request!
urllib.request.urlopen(request).read()
# Everything went fine.
return "200 OK"
################################################################################################################################################################################################################
All of the code outside the lambda handler is only run once per container. All code inside the handler is run every time the lambda is invoked.
A lambda container lasts for between 10 and 30 minutes depending on usage. A new lambda invocation may or may not run on an already running container.
It's possible you are invoking a lambda in a container that is over 5 minutes old where your connection has timed out.
I'm trying to run a Python script which leaves a connection open permanently, and responds to changes made outside of the script.
So for example:
Data script: Accepts form posts and commits form data to the database
Worker script: Monitors the database for new form posts and takes action accordingly
The relevant code in the worker script is:
import pymysql
conn = pymysql.connect(host='127.0.0.1', port=3306, user='dbuser', passwd='dbpass', db='my_db')
def processForms(Formdat):
c = conn.cursor(pymysql.cursors.DictCursor)
myform.sendEmail(c)
conn.commit()
c.close()
def doForms():
while True:
... get data and store in 'myforms' ...
futures = [executor.submit(processForms, myform) for myform in myforms]
time.sleep(30)
doForms()
Now I don't understand why this is not picking up new forms... If I create a new connection in each iteration of doForms(), the new forms are picked up, but I don't want to be creating and destroying connections all the time.
For example, this modification works:
conn = None
def doForms():
while True:
global conn
conn = pymysql.connect(host='127.0.0.1', port=3306, user='root', passwd='', db='mw_py')
... get data and store in 'myforms' ...
futures = [executor.submit(processForms, myform) for myform in myforms]
conn.close()
time.sleep(30)
Is there a way for me to use the open connection and have it poll the latest data?
Open 1 connection in the beginning of your script. It is not a cheap operation to connect.
Remember ID of last row fetched.
On every iteration select rows with ID greater than last seen.