Develop Reference

How to add current date in filename in python script

I'm trying to unload data from snowflakes to GCS, for that I'm using snowflakepython connector and python script. In the below python script in the file name 'LH_TBL_FIRST20200908' if the script runs today then the name will be same, if the script runs tomorrow then the file name should be 'LH_TBL_FIRST20200909' similarly if it runs day after then 'LH_TBL_FIRST202009010'.
Also please tell me if the code has any mistakes in it. Code is below
import snowflake.connector
# Gets the version
ctx = snowflake.connector.connect(
user='*****',
password='*******',
account='********',
warehouse='*******',
database='********',
schema='********'
)
cs = ctx.cursor()
sql = "copy into #unload_gcs/LH_TBL_FIRST20200908.csv.gz
from ( select * from TEST_BASE.LH_TBL_FIRST )
file_format =
( type=csv compression='gzip'
FIELD_DELIMITER = ','
field_optionally_enclosed_by='"'
NULL_IF=()
EMPTY_FIELD_AS_NULL = FALSE
)
single = fals
e max_file_size=5300000000
header = false;"
cur.execute(sql)
cur.close()
conn.close()

You can use f-strings to fill in (part of) your filename. Python has the datetime module to handle dates and times.
from datetime import datetime
date = datetime.now().strftime('%Y%m%d')
myFileName = f'LH_TBL_FIRST{date}.csv.gz'
print(myFileName)
>>> LH_TBL_FIRST20200908.csv.gz
As for errors in your code:
you declare your cursor as ctx.cursor() and further along you just use cur.execute(...) and cur.close(...). These won't work. Run your code to find the errors and fix them.
Edit suggested by #Lysergic:
If your python version is too old, you could use str.format().
myFileName = 'LH_TBL_FIRST{0}.csv.gz'.format(date)

from datetime import datetime
class FileNameWithDateTime(object):
def __init__(self, fileNameAppender, fileExtension="txt"):
self.fileNameAppender = fileNameAppender
self.fileExtension = fileExtension
def appendCurrentDateTimeInFileName(self,filePath):
currentTime = self.fileNameAppender
print(currentTime.strftime("%Y%m%d"))
filePath+=currentTime.strftime("%Y%m%d")
filePath+="."+self.fileExtension
try:
with open(filePath, "a") as fwrite1:
fwrite1.write(filePath)
except OSError as oserr:
print("Error while writing ",oserr)

I take the following approach
#defining what time/date related values your variable will contain
date_id = (datetime.today()).strftime('%Y%m%d')
Write the output file.
#Creating the filename
with open(date_id + "_" + "LH_TBL.csv.gz" 'w') as gzip:
output: YYYY/MM/DD _ filename
20200908_filename

Moving Files by creation/modification date then moving with Python

I am new to programming, even more so with Python. So please excuse any ignorance on my part. I am trying to write a script for myself that will move files that have been modified in the last 24 hours. So far I have came up with this:
import datetime
import os
import shutil
src = "C:\Users\Student\Desktop\FolderA"
dst = "C:\Users\Student\Desktop\FolderB"
now = dt.datetime.now()
before = now - dt.timedelta(hours=24)
def mins_since_mod(fname):
return (os.path.getmtime(fname))
for fname in os.listdir(src):
if mins_since_mod > before:
src_fname = os.path.join(src,fname)
os.path.join(dst,fname)
shutil.move(src_fname, dst)
I know i'm close to the solution, but I can't seem to figure out how to get this to work. I looked around here on the community and was not able to find a solution to my problem. Thank you for any leads or suggestions.

There are a few things to change. First, you can't compare the datetime in before to the Unix timestamp that getmtime() returns. It's easier to just use that directly. Also, you actually need to pass the (full) filename to mins_since_mod() for it to do anything.
Here's something that should work, changing the name of mins_since_mod() to reflect what it does better:
import time
import os
import shutil
SECONDS_IN_DAY = 24 * 60 * 60
src = "C:\Users\Student\Desktop\FolderA"
dst = "C:\Users\Student\Desktop\FolderB"
now = time.time()
before = now - SECONDS_IN_DAY
def last_mod_time(fname):
return os.path.getmtime(fname)
for fname in os.listdir(src):
src_fname = os.path.join(src, fname)
if last_mod_time(src_fname) > before:
dst_fname = os.path.join(dst, fname)
shutil.move(src_fname, dst_fname)

Hey mate I have actually just done something like this myself. I found that there will be a few issues will the time comparison as well as some issues in comparing and moving folders.
Try this:
import os
import shutil
import datetime
def filter_by_date(src_folder, archive_date):
os.chdir(src_folder)
delay_time = 24 * 60 * 60
archive_period = archive_date - delay_time
return [
name for name in os.listdir(u'.')
if os.path.isdir(name)
and datetime.datetime.fromtimestamp(os.path.getmtime(name)) < archive_period
]
if __name__ == '__main__':
folders = filter_by_date("C:/Users/Student/Desktop/FolderA", time.time())
for files in folders:
print files
try:
shutil.copytree(files, os.path.join("C:/Users/Student/Desktop/New", files))
except OSError as e:
print('\nDirectory not copied. Error: %s' % e)
except shutil.Error as e:
try:
files = files.encode('UTF-8')
dst_path = os.path.join('C:/Users/Student/Desktop/FolderB/', files)
shutil.copytree(files, dst_path)
finally:
print('\nDirectory not copied. Error: %s' % e)
print "\Completed"
This is going to ensure any file name (including Chinese, Russian and Japanese will be copied) and any folder (directory or sub-directory) is copied. It will also keep all file attributes.

Print new line not working in Python

The \n just doesn't seem to work for me when I use it with print. I am using Python 2.7.8. I don't get whats wrong, I think \n with a print should print a new line very straight forwardly.
import sys
import os
import subprocess
from collections import OrderedDict
import xmlrpclib
import hawkey
op_name = sys.argv[1]
pkg_name = sys.argv[2]
# Hawkey Configurations
sack = hawkey.Sack()
path = "/home/thejdeep/test_repo/repodata/%s"
repo = hawkey.Repo("test")
repo.repomd_fn = path % "repomd.xml"
repo.primary_fn = path % "b6f6911f7d9fb63f001388f1ecd0766cec060c1d04c703c6a74969eadc24ec97-primary.xml.gz"
repo.filelists_fn = path % "df5897ed6d3f87f2be4432543edf2f58996e5c9e6a7acee054f9dbfe513df4da-filelists.xml.gz"
sack.load_repo(repo,load_filelists=True)
# Main Function
if __name__ == "__main__":
print "Querying the repository\n"
print "-----------------------\n"
print "Found packages :\n"
print "--------------\n"
q = hawkey.Query(sack)
q = q.filter(name=pkg_name,latest_per_arch=True)[0]
if q:
for pkg in q:
print str(pkg)
else:
print "No packages with name "+pkg_name+" found. Exiting"
sys.exit()
print "--------------------"
print "Performing Dependency Check"
Output is something like this. Basically its printing in the same line :
Querying the repository ----------------------- Found packages : --------------

Using print method automatically add \n at the end of the line so its not necessary put \n at the end of each line.

Python scp copy file with spaces in filename

I'm trying to copy files in local network with scp.
It's working well with filenames without spaces, but it crash with.
I've tried to replace " " with "\ " as this exemple, but it don't work.
Here is my code:
def connection(locals):
a = (int(re.search(br'(\d+)%$', locals['child'].after).group(1)))
print a
perc = (Decimal(a)/100)
print (type(perc)), perc
while gtk.events_pending():
gtk.main_iteration()
FileCopy.pbar.set_text("Copy of the file in the Pi... " + str(a) + "%")
while gtk.events_pending():
gtk.main_iteration()
FileCopy.pbar.set_fraction(perc)
file_pc = "/home/guillaume/folder/a very large name of file with space .smthg"
file_pi = "pi#192.168.X.X:/home/pi/folder/a very large name of file with space .smthg"
if " " in file_pc:
file_pc = fichier_pc.replace(" ", '\\\ ') # tried '\\ ' or '\ '
file_pi = fichier_pi.replace(" ", '\\\ ') # but no way
else:
pass
command = "scp %s %s" % tuple(map(pipes.quote, [file_pc, file_pi]))
pexpect.run(command, events={r'\d+%': connection}) # this command is using to get the %
How can I fix this problem ?
Thanks

Use subprocess module and/or shlex.split():
import subprocess
subprocess.call(['scp', file_pc, file_pi])
and you don't need to worry about escaping or quoting anything

You may keep local file file_pc as is (pipes.quote will escape the spaces). The remote file should be changed:
import pipes
file_pi = 'pi#192.168.X.X:/home/pi/folder/file with space.smth'
host, colon, path = file_pi.partition(':')
assert colon
file_pi = host + colon + pipes.quote(path)
i.e., user#host:/path/with space should be changed to user#host:'/path/with space'

You might want to look into fabric, a Python library that streamlines the use of SSH.
from fabric.state import env
from fabric.operations import get
env.user = 'username'
env.key_filename = '/path/to/ssh-key'
get('/remote_path/*', 'local_path/')

Compare RPM Packages using Python

I'm trying to compare a csv file containing required Linux packages with the current installed packages. The comparison should output any packages not installed or newer than the current installed packages.
The problem is that I'm unable to loop through the list of installed packages and show all hits, for instance packages with the same name and version, but different architecture should be shown twice(for instance compat-libstdc++-33), but I only getting the first hit with the script below.
#!/usr/bin/python
import rpm
import csv
import sys
import os
'''
Script to check installed rpms against a csv file containing the package name and version similar to the list below:
atk,1.12.2
libart_lgpl,2.3
info,4.9
libsepol,1.15.2
libusb,0.1.12
libfontenc,1.4.2
'''
if len(sys.argv) !=2:
print ''
print 'Usage: ', sys.argv[0], '/path/to/csv_input_file'
print ''
sys.exit(1)
if not os.path.isfile(sys.argv[1]):
print ''
print sys.argv[1], 'not found!'
print ''
sys.exit(1)
else:
input_csv = sys.argv[1]
pkgRequired = csv.reader(open(input_csv),delimiter=',')
pkgInstalledName = []
pkgInstalledVersion = []
pkgInstalledArch = []
ts = rpm.TransactionSet()
mi = ts.dbMatch()
for h in mi:
pkgInstalledName.append((h['name']))
pkgInstalledVersion.append((h['version']))
pkgInstalledArch.append((h['arch']))
for row in pkgRequired:
pkgRequiredName = row[0]
pkgRequiredVersion = row[1]
#pkgRequiredArch = row[2]
if pkgRequiredName in pkgInstalledName:
if pkgInstalledVersion[pkgInstalledName.index(pkgRequiredName)] >= pkgRequiredVersion:
pass
else:
print '\nInstalled: ',pkgInstalledName[pkgInstalledName.index(pkgRequiredName)], pkgInstalledVersion[pkgInstalledName.index(pkgRequiredName)], pkgInstalledArch[pkgInstalledName.index(pkgRequiredName)], ' \nRequired: ', ' ', pkgRequiredName,pkgRequiredVersion

Assuming that there's no problem with the way that you're reading the list of installed packages (I'm not familiar with the rpm module), then your only problem is with using the index() function. This function return the first occurrence of an item with the specified value - and it isn't what you want.
A correct implementation (which is also much more efficient) would be:
installedPackages = {} #create a hash table, mapping package names to LISTS of installed package versions and architectures
for h in mi:
l = installedPackages.get(h['name'], list()) #return either the existing list, or a new one if this is the first time that the name appears.
l.append( (h['version'], h['arch']) )
...
if requiredPackageName in installedPackages:
for ver, arch in installedPackages[requiredPackageName]: print ...

This is what I ended up doing to get this working. The script currently is not checking for the architecture of required packages, but at least it shows the arch installed. The script works (as far as I know) but can be improved as its my first at python :)
#!/usr/bin/python
import rpm
import csv
import sys
import os
'''
Script to check installed rpms against a csv file containing the package name and version similar to the list below:
atk,1.12.2
libart_lgpl,2.3
info,4.9
libsepol,1.15.2
libusb,0.1.12
libfontenc,1.4.2
'''
#silverbullet - 20120301
if len(sys.argv) !=2:
print ''
print 'Usage: ', sys.argv[0], '/path/to/csv_input_file'
print ''
sys.exit(1)
if not os.path.isfile(sys.argv[1]):
print ''
print sys.argv[1], 'not found!'
print ''
sys.exit(1)
else:
input_csv = sys.argv[1]
pkgRequired = csv.reader(open(input_csv),delimiter=',')
pkgInstalledName = []
pkgInstalledVersion = []
pkgInstalledArch = []
ts = rpm.TransactionSet()
mi = ts.dbMatch()
for h in mi:
pkgInstalledName.append((h['name']))
pkgInstalledVersion.append((h['version']))
pkgInstalledArch.append((h['arch']))
for row in pkgRequired:
try:
pkgRequiredName = row[0]
pkgRequiredVersion = row[1]
#pkgRequiredArch = row[2] - This is not implemented yet, ie, script will ignore architecture in csv input file
except:
print "Unexpected Error. Check if input is csv format with no blank lines. "#, sys.exc_info()[1]
break
else:
for pos, pkg in enumerate(pkgInstalledName):
if pkg == pkgRequiredName:
if pkgInstalledVersion[pos] >= pkgRequiredVersion:
pass
else:
print '\nInstalled:', pkgInstalledName[pos], pkgInstalledVersion[pos], pkgInstalledArch[pos], '\nRequired: ', pkg, pkgRequiredVersion