Develop Reference

How can I ensure a dataframe has completed being written via pandas.to_csv()?

I have been creating a little script that queries a database and returns the result. I have then been using Pandas.to_csv() to write it out to a CSV tempfile before I upload that CSV result to a cloud location. The trouble I am running into is ensuring that the pandas.to_csv() function has completed writing the CSV tempfile before I upload it to the cloud location. The only way I have consistently ensured that that date makes it to the temp file before the upload is by keeping the
print(temp.tell())
line of code in the example below. If I comment it out, no data gets uploaded.
Example code below:
def write_to_temporary_csv_file(df, file_name, token, folder_id):
with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False) as temp:
print("DataFrame: ", df)
df.to_csv(temp, index=False, encoding='utf-8')
print("temp.tell() size: ", temp.tell())
print("File size: ", str(round((os.stat(temp.name).st_size/1024), 2)), "kb")
new_file_path = tempfile.gettempdir() + '/' + customer_name + '_' + file_name + '_' + current_date + '.csv'
## Check if newly created renamed temp file already exist, if it does remove it to create it
remove_temporary_file(new_file_path)
os.link(temp.name, new_file_path)
upload_response = upload_file(token, folder_id, new_file_path)
## Remove both the temp file and the newly created renamed temp file
remove_temporary_file(temp.name)
remove_temporary_file(new_file_path)
Image 1 (with temp.tell() included:
Image 2 (with temp.tell() commented out:

I think it might be caused by the fact that you keep your file opened (as long as you are inside the with block). That might cause the content not being flushed to disk.
def write_to_temporary_csv_file(df, file_name, token, folder_id):
with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False) as temp:
print("DataFrame: ", df)
df.to_csv(temp, index=False, encoding='utf-8')
# at this point we can close the file by exiting the with block
print("temp.tell() size: ", temp.tell())
print("File size: ", str(round((os.stat(temp.name).st_size/1024), 2)), "kb")
new_file_path = tempfile.gettempdir() + '/' + customer_name + '_' + file_name + '_' + current_date + '.csv'
## Check if newly created renamed temp file already exist, if it does remove it to create it
remove_temporary_file(new_file_path)
os.link(temp.name, new_file_path)
upload_response = upload_file(token, folder_id, new_file_path)
## Remove both the temp file and the newly created renamed temp file
remove_temporary_file(temp.name)
remove_temporary_file(new_file_path)

saving a filename with an extension to an existing filename

I would like to save a file by adding an extension to the existing filename, by extension I do not mean change .csv to .HTML.
What I mean is if I have an existing file file1.csv
I would like to save the other file as file1_processed.csv.
I tried doing this
data = pd.read_csv("file1.csv")
df = x
df.to_csv(os.path.basename(data) + '_' + 'processed' + '.csv')
however, it gives an error
TypeError: expected str, bytes or os.PathLike object, not DataFrame

filename = "file1"
data = pd.read_csv(filename + ".csv")
# ...
df.to_csv(filename + '_processed.csv')

data has contents of file file1.csv, not the file name. You need to assign the file name to a variable if the name of the file is not same always.
file_name = "file1.csv"
data = pd.read_csv(file_name )
df = x
name, extension = os.path.splitext(os.path.basename(file_name))
df.to_csv(name + '_' + 'processed' + extension)

Python - Mac - renamed file is created at the same level with the folder that was containing it

I am trying to rename a file in a specific path, but it moves the file at the same level with the folder that was containing it before being renamed.
import os
path_to_file = "/Users/Me/file.txt" #NOTE that in my code, file name will be randomly generated
path_before_file, file_name = os.path.split(path_to_file) # I need file_name from here, which is the file name without the rest of the path: "file.txt"
renamed_file = os.path.splitext(file_name)[0] + '_renamed' + os.path.splitext(file_name)[1] #This is how renamed file I want to look "file_renamed.txt"
renamed_file_path = os.path.join(path_to_file, renamed_file) #"/Users/Me/file_renamed.txt"
os.rename(path_to_file, renamed_file_path)
After renaming, "/Users/Me/file.txt" becomes "/Users/file.txt"

renamed_file_path = os.path.join(path_to_file, renamed_file) #"/Users/Me/file_renamed.txt"
Above line has a bug. Above line should be as below
renamed_file_path = os.path.join(path_before_file, renamed_file) #"/Users/Me/file_renamed.txt"

I figured out what was the problem. path_before_file was something like "Users/Me"(without slash at the end). If I do:
renamed_file_path = os.path.join(path_before_file + '/' + renamed_file)
or
renamed_file_path = path_before_file + '/' + renamed_file
it works. But AFAIK, you wouldn't need this when working with os.path.join()
This is the code that works:
import os
path_to_file = "/Users/Me/file.txt"
path_before_file, file_name = os.path.split(path_to_file)
renamed_file = os.path.splitext(file_name)[0] + '_renamed' + os.path.splitext(file_name)[1]
renamed_file_path = path_before_file + '/' + renamed_file
os.rename(path_to_file, renamed_file_path)

Copy file from one folder to another and then rename using current date/time and new file ext using PowerShell or Python script

I do not have any experience with PowerShell scripts so I need some help to create a simple shell script. The shell script needs to do the following:
The script will look for a file name with .upl file extension in the directory M:\ProgramData\Aspect Software\Unified IP\Advanced List Management\tenant1\xfer\CRM_Upload.upl.
The shell script needs to copy the .upl file and place it into a folder in the same directory, the destination folder is called "Uploads" (xfer\uploads).
The shell script also needs to append the current date/time (hhmmssmmddyyyy) onto the end of the file name after it has been copied to the Uploads folder with underscore (CRM_Upload_hhmmssmmddyyyy).
Finally the shell script needs to change the file extension to .txt instead of the original .upl (CRM_Upload_hhmmssmmddyyyy.txt).
This as far as I got:
This is another stab I took at this using PowerShell script...I would prefer if it copied the file and renamed it instead of moving it. Can someone help me with this below?
Function RenameMoveFile($locationPath, $fileName, $extension1, $extension2, $archiveFolder)
{
$date = Get-Date -uFormat ""%y%m%d-%H%M%S""
$old = $locationPath + $fileName + $extension1
$new = $locationPath + $fileName + "_" + $date + $extension2
$archiveFolder = $locationPath + $archiveFolder + "\"
Rename-Item $old $new
Move-Item $new $archiveFolder
}
## This is the only part that we'd edit
RenameMoveFile -locationPath "M:\ProgramData\Aspect Software\Unified IP\Advanced List Management\tenant1\xfer" -fileName "CRM_DIALER_UPLOAD" - extension1 ".upl" -extension2 ".txt" -archiveFolder "Uploads"
*** Here is a Python script I was trying to work on but its still not right..
import shutil
import datetime
Variable Definitions
from System import Environment
data_dir_base = Environment.GetEnvironmentVariable('LMSRVSHARED')
xfer_dir = data_dir_base + "tenant1\\xfer\\"
source_file = xfer_dir + "CRM_DIALER_UPLOAD.txt"
now = datetime.datetime.now()
date = now.strftime("%Y%m%d_%H%M%S")
new_folder = xfer_dir + "Uploads\\"
dest = new_folder + '_' + str(date) + '.txt'
Main Script
shutil.copy(source_file, dest)
***With a little bit of hard work and trial and error I got the script working..
import shutil
import datetime
### Variable Definitions
from System import Environment
data_dir_base = Environment.GetEnvironmentVariable('LMSRVSHARED')
xfer_dir = data_dir_base + "tenant1\\xfer\\"
source_file = xfer_dir + "CRM_DIALER_UPLOAD.txt"
now = datetime.datetime.now()
date = now.strftime("%Y%m%d_%H%M%S")
new_folder = xfer_dir + "Uploads\\"
dest = new_folder + 'CRM_DIALER_UPLOAD' + '_' + str(date) + '.txt'
### Main Script
shutil.copy(source_file, dest)

For shell scripting in windows you can use batch (.bat), but i prefer (.vbs)
Here is what you want with VbScript :
Dim fso, folder, objFolder, file, objFile
Dim strDir, strFileExtension, strNewDir, strNewExtension, strDate
' Parameters
strDir = "M:\ProgramData\Aspect Software\Unified IP\Advanced List Management\tenant1\xfer\"
strFileExtension = ".upl"
strNewDir = "M:\ProgramData\Aspect Software\Unified IP\Advanced List Management\tenant1\xfer\uploads\"
strNewExtension = ".txt"
Set fso = CreateObject("Scripting.FileSystemObject")
Set folder = fso.GetFolder(strDir)
' create new dir
If Not fso.FolderExists(strNewDir) Then
Set objFolder = fso.CreateFolder(strNewDir)
End If
' search for file, get date, then copy and rename
For each file In folder.Files
If (InStr(file.name, ".") > 0) Then
If (LCase(Mid(file.name, InStrRev(file.name, "."))) = strFileExtension) Then
' get only the file name
Set objFile = fso.GetFile(file.name)
' get date
strDate = Right("0" & DatePart("h",now), 2) _
& Right("0" & DatePart("n",now), 2) _
& Right("0" & DatePart("s",now), 2) _
& Right("0" & DatePart("m",now), 2) _
& Right("0" & DatePart("d",now), 2) _
& DatePart("yyyy",now) _
' copy file
CreateObject("Scripting.FileSystemObject").CopyFile file.name, strNewDir & fso.GetBaseName(objFile) & "_" & strDate &
strNewExtension
End If
End If
Next

import shutil
import datetime
### Variable Definitions
from System import Environment
data_dir_base = Environment.GetEnvironmentVariable('LMSRVSHARED')
xfer_dir = data_dir_base + "tenant1\\xfer\\"
source_file = xfer_dir + "CRM_DIALER_UPLOAD.txt"
now = datetime.datetime.now()
date = now.strftime("%Y%m%d_%H%M%S")
new_folder = xfer_dir + "Uploads\\"
dest = new_folder + 'CRM_DIALER_UPLOAD' + '_' + str(date) + '.txt'
### Main Script
shutil.copy(source_file, dest)

How to get file metadata from external drive in Python?

I wrote a Python script that collects file metadata (filename, creation date, creation time, last modified data, last modified time) from a file directory. However, when the directory is a path that is located in an external hard drive the script doesn't work. I can't figure out why.
Here is the code:
import os
from os.path import basename
import datetime
import time
def getSize(filename):
st = os.stat(filename)
print st
return st.st_size
#get last modified date
def getMTime(filename):
fileModTime = os.path.getmtime(filename)
return fileModTime
#get creation date
def getCTime(filename):
fileModTime = os.path.getctime(filename)
return fileModTime
#get data from directory
MyDirectory = "H:\0_tempfiles\150115_Portfolio\Work\Work\BarBackUp"
MyExtension = ".jpg"
#write to file
WorkingDirectory = "C:\\Users\Admin\Downloads\demo\\"
MyTxtFile = WorkingDirectory + "fileData6.txt"
delim = ";"
with open(MyTxtFile, 'wb') as f:
f.write(delim.join(["FILENAME", "FILESIZE", "mDATE","mTIME",
"cDATE","cTIME"]) + "\n")
for root, dirs, files in os.walk(MyDirectory):
for file in files:
if file.endswith(MyExtension):
#get File Name
a = (os.path.join(root, file))
#print a
filename = a
MyFileName = basename(a)
#get File Size
MyFileSize = getSize(filename) / 1000
print MyFileName + " >>> file size: " + str(MyFileSize) + "Kb"
#get modification time V2
modTimeV2 = getMTime(filename)
modTimeV2 = time.strftime("%Y/%d/%m;%I:%M:%S %p", \
time.localtime(modTimeV2))
print "time modified: " + str(modTimeV2)
#get creation time
creTime = getCTime(filename)
creTime = time.strftime("%Y/%d/%m;%I:%M:%S %p", \
time.localtime(creTime))
print "time created: " + str(creTime)
#--------
#write data to file
entry = delim.join([str(MyFileName), str(MyFileSize), \
str(modTimeV2), str(creTime)]) + "\n"
f.write(entry)
print "<<<<<<everything went fine>>>>>>"

Your code works fine for me. Your "MyDirectory" variable has escape characters in it. Try adding an r in front of the quotations:
MyDirectory = r"H:\0_tempfiles\150115_Portfolio\Work\Work\BarBackUp"
or
MyDirectory = "H:/0_tempfiles/150115_Portfolio/Work/Work/BarBackUp"
or
MyDirectory = "H:\\0_tempfiles\\150115_Portfolio\\Work\\Work\\BarBackUp"