Develop Reference

Dealing with special characters in file paths - Python script and Athena Query

I have a script that I use to parse out S3 info using Athena and S3 inventory. Currently when it finds a specific file type, the script will use that path to generate a SQL query that is then passed to Athena. Here's an example:
def getS3TagReportData(session, tag, table):
whereStatement = ""
for v in tag.values:
first = True
for p in v.paths:
# print(p) #iterate through paths
if first:
whereStatement = f"WHERE dt = (SELECT MAX(dt) FROM \"{awsAthenaDatabase}\".\"{table}\") AND key LIKE '{p}/%'"
first = False
else:
whereStatement += f" OR key LIKE '{p}%'"
This works fine until I hit a path where someone has put in an '
For example: bucketname\data\int'l hotel files\data.jpg
Then the query that gets dynamically generated is:
WHERE dt = (SELECT MAX(dt) FROM \"{awsAthenaDatabase}\".\"{table}\") AND key LIKE 'bucketname\data\int'l hotel files\data.jpg'
As you can see, the Where statement is going to fail because there is that extra ' in there. How can I programmatically work around these errors and report them?
I appreciate the help!

Changing output of speedtest.py and speedtest-cli to include IP address in output .csv file

I added a line in the python code “speedtest.py” that I found at pimylifeup.com. I hoped it would allow me to track the internet provider and IP address along with all the other speed information his code provides. But when I execute it, the code only grabs the next word after the find all call. I would also like it to return the IP address that appears after the provider. I have attached the code below. Can you help me modify it to return what I am looking for.
Here is an example what is returned by speedtest-cli
$ speedtest-cli
Retrieving speedtest.net configuration...
Testing from Biglobe (111.111.111.111)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by GLBB Japan (Naha) [51.24 km]: 118.566 ms
Testing download speed................................................................................
Download: 4.00 Mbit/s
Testing upload speed......................................................................................................
Upload: 13.19 Mbit/s
$
And this is an example of what it is being returned by speediest.py to my .csv file
Date,Time,Ping,Download (Mbit/s),Upload(Mbit/s),myip
05/30/20,12:47,76.391,12.28,19.43,Biglobe
This is what I want it to return.
Date,Time,Ping,Download (Mbit/s),Upload (Mbit/s),myip
05/30/20,12:31,75.158,14.29,19.54,Biglobe 111.111.111.111
Or may be,
05/30/20,12:31,75.158,14.29,19.54,Biglobe,111.111.111.111
Here is the code that I am using. And thank you for any help you can provide.
import os
import re
import subprocess
import time
response = subprocess.Popen(‘/usr/local/bin/speedtest-cli’, shell=True, stdout=subprocess.PIPE).stdout.read().decode(‘utf-8’)
ping = re.findall(‘km]:\s(.*?)\s’, response, re.MULTILINE)
download = re.findall(‘Download:\s(.*?)\s’, response, re.MULTILINE)
upload = re.findall(‘Upload:\s(.*?)\s’, response, re.MULTILINE)
myip = re.findall(‘from\s(.*?)\s’, response, re.MULTILINE)
ping = ping[0].replace(‘,’, ‘.’)
download = download[0].replace(‘,’, ‘.’)
upload = upload[0].replace(‘,’, ‘.’)
myip = myip[0]
try:
f = open(‘/home/pi/speedtest/speedtestz.csv’, ‘a+’)
if os.stat(‘/home/pi/speedtest/speedtestz.csv’).st_size == 0:
f.write(‘Date,Time,Ping,Download (Mbit/s),Upload (Mbit/s),myip\r\n’)
except:
pass
f.write(‘{},{},{},{},{},{}\r\n’.format(time.strftime(‘%m/%d/%y’), time.strftime(‘%H:%M’), ping, download, upload, myip))

Let me know if this works for you, it should do everything you're looking for
#!/usr/local/env python
import os
import csv
import time
import subprocess
from decimal import *
file_path = '/home/pi/speedtest/speedtestz.csv'
def format_speed(bits_string):
""" changes string bit/s to megabits/s and rounds to two decimal places """
return (Decimal(bits_string) / 1000000).quantize(Decimal('.01'), rounding=ROUND_UP)
def write_csv(row):
""" writes a header row if one does not exist and test result row """
# straight from csv man page
# see: https://docs.python.org/3/library/csv.html
with open(file_path, 'a+', newline='') as csvfile:
writer = csv.writer(csvfile, delimiter=',', quotechar='"')
if os.stat(file_path).st_size == 0:
writer.writerow(['Date','Time','Ping','Download (Mbit/s)','Upload (Mbit/s)','myip'])
writer.writerow(row)
response = subprocess.run(['/usr/local/bin/speedtest-cli', '--csv'], capture_output=True, encoding='utf-8')
# if speedtest-cli exited with no errors / ran successfully
if response.returncode == 0:
# from the csv man page
# "And while the module doesn’t directly support parsing strings, it can easily be done"
# this will remove quotes and spaces vs doing a string split on ','
# csv.reader returns an iterator, so we turn that into a list
cols = list(csv.reader([response.stdout]))[0]
# turns 13.45 ping to 13
ping = Decimal(cols[5]).quantize(Decimal('1.'))
# speedtest-cli --csv returns speed in bits/s, convert to bytes
download = format_speed(cols[6])
upload = format_speed(cols[7])
ip = cols[9]
date = time.strftime('%m/%d/%y')
time = time.strftime('%H:%M')
write_csv([date,time,ping,download,upload,ip])
else:
print('speedtest-cli returned error: %s' % response.stderr)

$/usr/local/bin/speedtest-cli --csv-header > speedtestz.csv
$/usr/local/bin/speedtest-cli --csv >> speedtestz.csv
output:
Server ID,Sponsor,Server Name,Timestamp,Distance,Ping,Download,Upload,Share,IP Address
Does that not get you what you're looking for? Run the first command once to create the csv with header row. Then subsequent runs are done with the append '>>` operator, and that'll add a test result row each time you run it
Doing all of those regexs will bite you if they or a library that they depend on decides to change their debugging output format
Plenty of ways to do it though. Hope this helps

Reading Excel File with openpyxl while populating form using Selenium takes too long

I'm filling out a web form which has input fields, dropdown menus, autocomplete fields and action buttons.
I'm pulling the data from an excel sheet using openpyxl. Initially it used to take between 3-4 seconds to populate these fields. After adding read_only=True to my readData function, it improved a bit but not as expected.
Does anyone has any suggestions on how I would be able to reduce the time it takes to populate each field? Any help is really appreciated. I'm leaving both the readData function as well as the populate_form which I use to fill out a text field as an example.
Cheers.
Method to read each cell:
workbook = openpyxl.load_workbook(file, read_only=True)
def readData(file, sheetName, row_num, column_num):
sheet = workbook.get_sheet_by_name(sheetName)
return sheet.cell(row=row_num, column=column_num).value
Method to populate input field:
def fill_out_form(driver, path, input_sel, row_num, column_num):
try:
wait_for_element(driver, "//input[#id='" + input_sel + "']", 5)
xls = readData(path, "Callcenter", row_num, column_num)
input_el = driver.find_element_by_xpath("//input[#id='" + input_sel + "']")
input_el.click()
if column_num == 9 or column_num == 40 or column_num == 67 or column_num == 121:
xls = datetime.strftime(xls,'%d/%m/%Y')
input_el.send_keys(xls)
input_el.send_keys(Keys.TAB)
loading_el = WebDriverWait(driver, 4).until(EC.presence_of_element_located((By.XPATH, "//*[#class='sk-attr js-sk-attr sk-attr--labeled sk-attr--mandatory sk-attr--infonnized sk-attr--error sk-textbox clearfix']")))
WebDriverWait(driver, 4).until(wait_not_spinning(loading_el))
except TimeoutException:
print("Loading took too much time!-Try again")

Unless your spreadsheet is huge I'm fairly certain the wait_for_element and WebDriverWait calls are taking the most time.
As was already suggested, try caching the spreadsheet(s) data using an efficient structure such as:
dict[file][sheet] = list[row][column]
Since it seems you only have one file you can load the data using:
def load_data(filename):
data = {}
workbook = openpyxl.load_workbook(filename, data_only=True, read_only=True, keep_vba=False)
for sheet_name in workbook.sheetnames:
data[sheet_name] = []
sheet = workbook[sheet_name]
for rows in sheet.iter_rows():
row_elements = []
for cell in rows:
try:
value = cell.value
except IndexError:
value = cell.internal_value
row_elements.append(value)
data[sheet_name].append(row_elements)
return data
In order to use it, you would call load_data(filename) once (when your application starts) and access the loaded data later on using xls_data instead of readData:
#application start
xls_data = load_data(filename)
....
# sheet_name->str, row_num->int, col_num->int
xls = xls_data[sheet_name][row_num][col_num]
The above will throw KeyError if the sheet name is invalid or IndexError for an invalid row,column combination.

Try implementing the readData method using the 'xlrd' library.
It does not provide rich API like openpyxl but I'm sure it'll run faster.

When you fill in a web form, in the end the data will be sent to a server with a POST request.
What I would recommend is to use e.g. wireshark to capture that POST request.
Analyse that request to see what exactly is sent to the server. Then you can create such a POST request using the requests module.
That means you don't have to deal with selenium at all.
And as the others have mentioned, read the excel file only once.

Looping SQL query in python

I am writing a python script which queries the database for a URL string. Below is my snippet.
db.execute('select sitevideobaseurl,videositestring '
'from site, video '
'where siteID =1 and site.SiteID=video.VideoSiteID limit 1')
result = db.fetchall()
filename = '/home/Site_info'
output = open(filename, "w")
for row in result:
videosite= row[0:2]
link = videosite[0].format(videosite[1])
full_link = link.replace("http://","https://")
print full_link
output.write("%s\n"%str(full_link))
output.close()
The query basically gives a URL link.It gives me baseURL from a table and the video site string from another table.
output: https://www.youtube.com/watch?v=uqcSJR_7fOc
SiteID is the primary key which is int and not in sequence.
I wish to loop this sql query to pick a new siteId for every execution so that i have unique site URL everytime and write all the results to a file.
desired output: https://www.youtube.com/watch?v=uqcSJR_7fOc
https://www.dailymotion.com/video/hdfchsldf0f
There are about 1178 records.
Thanks for your time and help in advance.

I'm not sure if I completely understand what you're trying to do. I think your goal is to get a list of all links to videos. You get a link to a video by joining the sitevideobaseurl from site and videositestring from video.
From my experience it's much easier to let the database do the heavy lifting, it's build for that. It should be more efficient to join the tables, return all the results and then looping trough them instead of making subsequent queries to the database for each row.
The code should look something like this: (Be careful, I didn't test this)
query = """
select s.sitevideobaseurl,
v.videositestring
from video as v
join site as s
on s.siteID = v.VideoSiteID
"""
db.execute(query)
result = db.fetchall()
filename = '/home/Site_info'
output = open(filename, "w")
for row in result:
link = "%s%s" % (row[0],row[1])
full_link = link.replace("http://","https://")
print full_link
output.write("%s\n" % str(full_link))
output.close()
If you have other reasons for wanting to fetch these ony by one an idea might be to fetch a list of all SiteIDs and store them in a list. Afterwards you start a loop for each item in that list and insert the id into the query via a parameterized query.

How to check if a SQLite3 database exists in Python?

I am trying to create a function in Python 2.7.3 to open a SQLite database.
This is my code at the moment:
import sqlite3 as lite
import sys
db = r'someDb.sqlite'
def opendb(db):
try:
conn = lite.connect(db)
except sqlite3.Error:
print "Error open db.\n"
return False
cur = conn.cursor()
return [conn, cur]
I have tried the code above and I have observed that the sqlite3 library opens the database declared if exists, or creates a new database if this one doesn't exist.
Is there a way to check if the database exists with sqlite3 methods or I have to use file operation like os.path.isfile(path)?

In Python 2, you'll have to explicitly test for the existence using os.path.isfile:
if os.path.isfile(db):
There is no way to force the sqlite3.connect function to not create the file for you.
For those that are using Python 3.4 or newer, you can use the newer URI path feature to set a different mode when opening a database. The sqlite3.connect() function by default will open databases in rwc, that is Read, Write & Create mode, so connecting to a non-existing database will cause it to be created.
Using a URI, you can specify a different mode instead; if you set it to rw, so Read & Write mode, an exception is raised when trying to connect to a non-existing database. You can set different modes when you set the uri=True flag when connecting and pass in a file: URI, and add a mode=rw query parameter to the path:
from urllib.request import pathname2url
try:
dburi = 'file:{}?mode=rw'.format(pathname2url(db))
conn = lite.connect(dburi, uri=True)
except sqlite3.OperationalError:
# handle missing database case
See the SQLite URI Recognized Query Parameters documentation for more details on what parameters are accepted.

os.path.isfile() is just telling you if a file exists, not if it exists AND is a SQLite3 database! Knowing http://www.sqlite.org/fileformat.html, you could do this :
def isSQLite3(filename):
from os.path import isfile, getsize
if not isfile(filename):
return False
if getsize(filename) < 100: # SQLite database file header is 100 bytes
return False
with open(filename, 'rb') as fd:
header = fd.read(100)
return header[:16] == 'SQLite format 3\x00'
and subsequently use it like :
for file in files:
if isSQLite3(file):
print "'%s' is a SQLite3 database file" % file
else:
print "'%s' is not a SQLite3 database file" % file

Yes, there is a way to do what you want with Python 3.4+.
Use the sqlite3.connect() function to connect, but pass it a URI instead of a file path, and add mode=rw to its query string.
Here is a complete working code example:
import sqlite3
con = sqlite3.connect('file:aaa.db?mode=rw', uri=True)
This will open an existing database from a file named aaa.db in the current folder, but will raise an error in case that file can not be opened or does not exist:
Traceback (most recent call last):
File "aaa.py", line 2, in <module>
con = sqlite3.connect('file:aaa.db?mode=rw', uri=True)
sqlite3.OperationalError: unable to open database file
Python sqlite.connect() docs state that:
If uri is true, database is interpreted as a URI. This allows you to specify options. For example, to open a database in read-only mode you can use:
db = sqlite3.connect('file:path/to/database?mode=ro', uri=True)
More information about this feature, including a list of recognized options, can be found in the SQLite URI documentation.
Here's an excerpt of all the relevant URI option information collected from http://www.sqlite.org/c3ref/open.html:
mode: The mode parameter may be set to either "ro", "rw", "rwc", or "memory". Attempting to set it to any other value is an error. If "ro" is specified, then the database is opened for read-only access, just as if the SQLITE_OPEN_READONLY flag had been set in the third argument to sqlite3_open_v2(). If the mode option is set to "rw", then the database is opened for read-write (but not create) access, as if SQLITE_OPEN_READWRITE (but not SQLITE_OPEN_CREATE) had been set. Value "rwc" is equivalent to setting both SQLITE_OPEN_READWRITE and SQLITE_OPEN_CREATE. If the mode option is set to "memory" then a pure in-memory database that never reads or writes from disk is used. It is an error to specify a value for the mode parameter that is less restrictive than that specified by the flags passed in the third parameter to sqlite3_open_v2().
The sqlite3_open_v2() interface works like sqlite3_open() except that it accepts two additional parameters for additional control over the new database connection. The flags parameter to sqlite3_open_v2() can take one of the following three values, optionally combined with the SQLITE_OPEN_NOMUTEX, SQLITE_OPEN_FULLMUTEX, SQLITE_OPEN_SHAREDCACHE, SQLITE_OPEN_PRIVATECACHE, and/or SQLITE_OPEN_URI flags:
SQLITE_OPEN_READONLY
The database is opened in read-only mode. If the database does not already exist, an error is returned.
SQLITE_OPEN_READWRITE
The database is opened for reading and writing if possible, or reading only if the file is write protected by the operating system. In either case the database must already exist, otherwise an error is returned.
SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE
The database is opened for reading and writing, and is created if it does not already exist. This is the behavior that is always used for sqlite3_open() and sqlite3_open16().
For convenience, here's also a Python 3.4+ function for converting a regular path to an URI usable by sqlite.connect():
import pathlib
import urllib.parse
def _path_to_uri(path):
path = pathlib.Path(path)
if path.is_absolute():
return path.as_uri()
return 'file:' + urllib.parse.quote(path.as_posix(), safe=':/')

This is a fork (using Python 3) based on Tom Horen's answer, which presents a solution more complete and reliable that the elected answer.
The elected answer, does not evaluate any content, header, etc., in order to determine whether the file actually contains any data related to a SQLite3 database or not.
I tried to present something more pragmatic here:
#!/usr/bin/python3
import os
import sys
if os.path.isfile('test.sqlite3'):
if os.path.getsize('test.sqlite3') > 100:
with open('test.sqlite3','r', encoding = "ISO-8859-1") as f:
header = f.read(100)
if header.startswith('SQLite format 3'):
print("SQLite3 database has been detected.")

Building on a couple of other answers above. Here is a clean solution that works in Python 3.7.7:
def isSqlite3Db(db):
if not os.path.isfile(db): return False
sz = os.path.getsize(db)
# file is empty, give benefit of the doubt that its sqlite
# New sqlite3 files created in recent libraries are empty!
if sz == 0: return True
# SQLite database file header is 100 bytes
if sz < 100: return False
# Validate file header
with open(db, 'rb') as fd: header = fd.read(100)
return (header[:16] == b'SQLite format 3\x00')
Usage:
if isSqlite3Db('<path_to_db>'):
# ... <path_to_db> is a Sqlite 3 DB
Notes:
The answers checking file size is > 100 does not work as a new sqlite3 db created in recent python creates an file with length of 0.
Other examples reading file header returned bytes in Python 3.7.7 and not string so comparison would fail.
Examples that use sqlite3.connect(dburl, uri=True) did not work for me in Python 3.7.7 as it gave false positives.

I am using a function like the following at the beginning of my script so that I can try and figure out why a sqlite3 db script might not be working. Like the comments say, it uses 3 phases, checks if a path exist, checks if the path is a file, checks if that file's header is a sqlite3 header.
def checkdbFileforErrors():
#check if path exists
try:
with open('/path/to/your.db'): pass
except IOError:
return 1
#check if path if a file
if not isfile('/path/to/your.db'):
return 2
#check if first 100 bytes of path identifies itself as sqlite3 in header
f = open('/path/to/your.db', "rx")
ima = f.read(16).encode('hex')
f.close()
#see http://www.sqlite.org/fileformat.html#database_header magic header string
if ima != "53514c69746520666f726d6174203300":
return 3
return 0