Want a way to automate the transaction logshipping process for SQL server through Python script. Also the secondary DB should get created in "stand-by" mode once we do the log shipping configuration.
Related
I am trying to connect to a SQL Server instance using SQLAlchemy through Python, however I require the SQL connection to come from a specific Windows AD account that isn't the one I run VSCode with. Is there any way to modify the connection string below to explicitly feed SQL Server a Windows login that isn't the same login I am using the run VSCode?
(I am able to connect with this connection string if I "Run as a different User" in VSCode, however the AD accounts with SQL access does not have shared drive access and therefore cannot access shared files, so this won't scale long-term)
import urllib
from sqlalchemy import create_engine
params = urllib.parse.quote_plus('DRIVER={SQL Server};SERVER={server};DATABASE={database};Trusted_Connection=Yes')
engine = create_engine(f'mssql+pyodbc:///?odbc_connect={params}')
The task, when I update a column in the database table, "report" this to the Python application. Working with SQL Server is new to me, maybe I'm missing something.
The database has a trigger on the table I need. I tried to add Exec xp_cmdshell there, but because of this, the application that makes changes to the database hangs. In the task manager, you can see that the Python application is running, but it does not open.
After several attempts, the Python process exits and the main application reports the error "The creator of the error did not provide a reason." I ask for your help.
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER TRIGGER [dbo].[OnEmployeePhotoUpdate]
ON [dbo].[EmployeePhoto]
AFTER UPDATE
AS
BEGIN
SET NOCOUNT ON;
UPDATE EmployeePhoto
SET ModificationDateTime = GETDATE()
WHERE _id IN (SELECT _id FROM inserted)
EXEC xp_cmdshell 'C:\Users\IT\AppData\Local\Programs\Python\Python39\pythonw.exe D:\untitled\1.pyw'
END
I am trying to identify the number of connections to a postgres database. This is in context of the connection limit on heroku-postgres for dev and hobby plans, which is limited to 20. I have a python django application using the database. I want to understand what constitute a connection. Will each instance of an user using the application count as one connection ? Or The connection from the application to the database is counted as one.
To figure this out I tried the following.
Opened multiple instances of the application from different clients (3 separate machines).
Connected to the database using an online Adminer tool(https://adminer.cs50.net/)
Connected to the database using pgAdmin installed in my local system.
Created and ran dataclips (query reports) on the database from heroku.
Ran the following query from adminer and pgadmin to observe the number of records:
select * from pg_stat_activity where datname ='db_name';
Initial it seemed there was a new record for each for the instance of the application I opened and 1 record for the adminer instance. After some time the query from adminer was showing 6 records (2 connections for adminer, 2 for the pgadmin and 2 for the web-app).
Unfortunately I am still not sure if each instance of users using my web application would be counted as a connection or will all connections to the database from the web app be counted as one ?
Thanks in advance.
Best Regards!
Using PostgreSQL parameters to log connections and disconnections (with right log_line_prefix parameter to have client information) should help:
log_connections (boolean)
Causes each attempted connection to the server to be logged, as well as successful completion of client authentication. Only
superusers can change this parameter at session start, and it cannot
be changed at all within a session. The default is off.
log_disconnections (boolean)
Causes session terminations to be logged. The log output provides information similar to log_connections, plus the duration of the
session. Only superusers can change this parameter at session start,
and it cannot be changed at all within a session. The default is off.
How do I keep my connection to a SQL server database alive when running scripts from a Python pyodbc program?
The reason I ask is because I want to automate a task at my job that uses a Temp DB to store information, and then uses Excel to refresh its data based on those temp databases. However, when I run the query through pyodbc in my python script, the databases fall off automatically, and I'm assuming that's because it kills the connection after it's done running.
Is there a way to keep the connection open in Python so that I can still refresh my Excel spreadsheets?
Thanks
Working with a simple HiveQL query that looks like this:
SELECT event_type FROM {{table}} where dt=20140103 limit 10;
The {{table}} part is just interpolated via the runner code im using via Jinja2. I'm running my query using the -e flag on the hive command line using subprocess.Popen from python.
For some reason, this setup is attempting to write into the regular /user directory in HDFS? Sudoing the command has no effect. The error produced is as follows:
Job Submission failed with exception:
org.apache.hadoop.security.AccessControlException(Permission denied:user=username, access=WRITE, inode="/user":hdfs:hadoop:drwxrwxr-x\n\tat org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:234)
Why would hive attempt to write to /users? Additionally, why would a select statement like this need an output location at all?
Hive is a SQL frontend to MapReduce and so needs to compile and stage Java code for execution. It's not trying to put output there but rather the program it will execute. Depending on your version of Hadoop this is controlled by the variables:
mapreduce.jobtracker.staging.root.dir
And on YARN / Hadoop 2:
yarn.app.mapreduce.am.staging-dir
These are set in mapred-site.xml.
Your runner needs to be authenticated to the cluster and have a writable directory it can use.