Develop Reference

Use command prompt from python

I write a python scripts that after execute some db queries, save the result of that queries on different csv files.
Now, it's mandatory to rename this file with the production's timestamps and so every hour i got new file with new name.
The script run with a task scheduler every hour and after save my csv files I need to run automatically the command prompt and execute some command that includes my csv files name in the path....
Is it possible to run the cmd and paste him the path of csv file like a variable? in python I save the file in this way:
date_time_str_csv1 = now.strftime("%Y%m%d_%H%M_csv1")
I don't know how to write automatically the different file name when i call the cmd

If I understand your question correctly, one solution would be to simply execute the command-line command directly from the Python script.
You can use the subprocess module from Python (as also explained here: How do I execute a program or call a system command?).
This could look like this for example:
csv_file_name = date_time_str_csv1 +".csv"
subprocess.run(["cat", csv_file_name)

You can run a system cmd from within Python using os.system:
import os
os.system('command filename.csv')
Since the argument to os.system is a string, you can build it with your created filename above.

you can try using the subprocess library, and get a list of the files in the folder in an array. This example is using the linux shell:
import subprocess
str = subprocess.check_output('ls', shell=True)
arr = str.decode('utf-8').split('\n')
print(arr)
After this you can iterate to find the newest file and use that one as the variable.

Obtaining metadata "Where from" of a file on Mac

I am trying to obtain the "Where from" extended file attribute which is located on the "get info" context-menu of a file in MacOS.
Example
When right-clicking on the file and displaying the info it shows the this metadata.
The highlighted part in the image below shows the information I want to obtain (the link of the website where the file was downloaded from).
I want to use this Mac-specific function using Python.
I thought of using OS tools but couldn't figure out any.

TL;DR: Get the extended attribute like MacOS's "Where from" by e.g. pip-install pyxattr and use xattr.getxattr("file.pdf", "com.apple.metadata:kMDItemWhereFroms").
Extended Attributes on files
These extended file attributes like your "Where From" in MacOS (since 10.4) store metadata not interpreted by the filesystem. They exist for different operating systems.
using the command-line
You can also query them on the command-line with tools like:
exiftool:
exiftool -MDItemWhereFroms -MDItemTitle -MDItemAuthors -MDItemDownloadedDate /path/to/file
xattr (apparently MacOS also uses a Python-script)
xattr -p -l -x /path/to/file
On MacOS many attributes are displayed in property-list format, thus use -x option to obtain hexadecimal output.
using Python
Ture Pålsson pointed out the missing link keywords. Such common and appropriate terms are helpful to search Python Package Index (PyPi):
Search PyPi by keywords: extend file attributes, meta data:
xattr
pyxattr
osxmetadata, requires Python 3.7+, MacOS only
For example to list and get attributes use (adapted from pyxattr's official docs)
import xattr
xattr.listxattr("file.pdf")
# ['user.mime_type', 'com.apple.metadata:kMDItemWhereFroms']
xattr.getxattr("file.pdf", "user.mime_type")
# 'text/plain'
xattr.getxattr("file.pdf", "com.apple.metadata:kMDItemWhereFroms")
# ['https://example.com/downloads/file.pdf']
However you will have to convert the MacOS specific metadata which is stored in plist format, e.g. using plistlib.
File metadata on MacOS
Mac OS X 10.4 (Tiger) introduced Spotlight a system for extracting (or harvesting), storing, indexing, and querying metadata. It provides an integrated system-wide service for searching and indexing.
This metadata is stored as extended file attributes having keys prefixed with com.apple.metadata:. The "Where from" attribute for example has the key com.apple.metadata:kMDItemWhereFroms.
using Python
Use osxmetadata to use similar functionality like in MacOS's md* utils:
from osxmetadata import OSXMetaData
filename = 'file.pdf'
meta = OSXMetaData(filename)
# get and print "Where from" list, downloaded date, title
print(meta.wherefroms, meta.downloadeddate, meta.title)
See also
MacIssues (2014): How to look up file metadata in OS X
OSXDaily (2018): How to View & Remove Extended Attributes from a File on Mac OS
Ask Different: filesystem - What all file metadata is available in macOS?
Query Spotlight for a range of dates via PyObjC
Mac OS X : add a custom meta data field to any file

macOS stores metadata such as the "Where from" attribute under the key com.apple.metadata:kMDItemWhereFroms.
import xattr
value = xattr.getxattr("sublime_text_build_4121_mac.zip",'com.apple.metadata:kMDItemWhereFroms').decode("ISO-8859-1")
print(value)
'bplist00¢\x01\x02_\x10#https://download.sublimetext.com/sublime_text_build_4121_mac.zip_\x10\x1chttps://www.sublimetext.com/\x08\x0bN\x00\x00\x00\x00\x00\x00\x01\x01\x00\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00m'
I had faced a similar problem long ago. We did not use Python to solve it.

On linux , using a bash script how do I rename an Excel file to include the row count at the end of the existing filename

First post so be gentle please.
I have a bash script running on a Linux server which does a daily sftp download of an Excel file. The file is moved to a Windows share.
An additional requirement has arisen in that i'd like to add the number of rows to the filename which is also timestamped so different each day. Ideally at the end before the xlsx extension.
After doing some research it would seem I may be able to do it all in the same script if I use Python and one of the Excel modules. I'm a complete noob in Python but i have done some experimenting and have some working code using the Pandas module.
Here's what i have working in a test spreadsheet with a worksheet named mysheet and counting a column named code.
>>> excel_file = pd.ExcelFile('B:\PythonTest.xlsx')
>>> df = excel_file.parse('mysheet')
>>> df[['code']].count()
code 10
dtype: int64
>>> mycount = df[['code']].count()
>>> print(mycount)
code 10
dtype: int64
>>>
I have 2 questions please.
First how do I pass todays filename into the python script to then do the count on and how do i return this to bash. Also how do i just return the count value e.g 10 in the above example. i dont want column name or dtype passed back.
Thanks in advance.

Assuming we put your python into a separate script file, something like:
# count_script.py
import sys
import pandas as pd
excel_file = pd.ExcelFile(sys.argv[1])
df = excel_file.parse('mysheet')
print(df[['code']].count().at(0))
We could then easily call that script from within the bash script that invoked it in the first place (the one that downloads the file).
TODAYS_FILE="PythonTest.xlsx"
# ...
# Download the file
# ...
# Pass the file into your python script (manipulate the file name to include
# the correct path first, if necessary).
# By printing the output in the python script, the bash subshell (invoking a
# command inside the $(...) will slurp up the output and store it in the COUNT variable.
COUNT=$(python count_script.py "${TODAYS_FILE}")
# this performs a find/replace on $TODAYS_FILE, replacing the ending ".xlsx" with an
# underscore, then the count obtained via pandas, then tacks on a ".xlsx" again at the end.
NEW_FILENAME="${TODAYS_FILE/\.xlsx/_$COUNT}.xlsx"
# Then rename it
mv "${TODAYS_FILE}" "${NEW_FILENAME}"

You can pass command-line arguments to python programs, by invoking them as such:
python3 script.py argument1 argument2 ... argumentn
They can then be accessed within the script using sys.argv. You must import sys before using it. sys.argv[0] is the name of the python script, and the rest are the additional command-line arguments.
Alternatively you may pass it in stdin, which can be read in Python using normal standard input functions like input(). To pass input in stdin, in bash do this:
echo $data_to_pass | python3 script.py
To give output you can write to stdout using print(). Then redirect output in bash, to say, a file:
echo $data_to_pass | python3 script.py > output.txt
To get the count value within Python, you simply need to add .at(0) at the end to get the first value; that is:
df[["code"]].count().at(0)
You can then print() it to send it to bash.

Is there a better way to utilize redshift, python, and powershell to automate my report?

I'm automating my current reports and need help getting up and running.
I have a PS script that first fixes a csv file and then calls a python script. Within the python script I am importing modules that 'cannot be found' when the PS script runs.
I have tried importing the modules separately(but within the powershell script) to no avail. I have been trying to run from PyCharms using a powershell plugin. Even though I have tried running it in Powershell IDE and it produces the same error.
This is where it chokes. If I need to delve further please let me know and I will remove the sensitive information in the code and post it.
#POWERSHELL CODE
#start excel
$Excel = New-Object -Com Excel.Application
#make it visible (just to check what is happening)
$Excel.Visible = $true
#open file
$FilePath = 'file path here'
$Workbook = $Excel.Workbooks.Open($FilePath)
$Worksheets = $Workbooks.worksheets
#change column format type for DateTime issue in logs
$workbook.ActiveSheet.Columns.Item("X").NumberFormat = "yyyy-mm-dd hh:mm:ss"
$Output.ExitCode
#save doc and close excel
$ext=".csv"
$path="file here"
$workbook.SaveAs($path)
$workbook.Close
$Excel.DisplayAlerts = 'False'
$Excel.Quit()
$Output.ExitCode
python "file here" #calling python script here
$Output.ExitCode
#PYTHON CODE
import MySQLConnection2 as mc
import RedshiftConnection2 as rc
from Data_download2 import dict_to_csv
from os.path import join, basename
import datetime
import csv
import pandas as pd
import numpy as np
import sys
The python code generates several csv files that are needed for this report. Once I get past this choke I will need to call another python script and then alter the csv files.
Error Message > 'Import-Module : The specified module 'SQLalchemy' was not loaded because no valid module file was found in any module
directory.'

You cannot use Import-Module on python scripts and python modules.

Generating Model code with pwiz Error

I am trying to generate some model values using pwiz. The database in question is a Sqlite database, which actually contains some Django tables in addition to some regular tables generated using a python script. However when I try the following in a Linux terminal
python -m pwiz -e sqlite -t mapping_table db.sqlite3
I get the following error:
/python2.7/site-packages/peewee.py", line 3001, in get_indexes
for _, name, is_unique in cursor.fetchall():
ValueError: too many values to unpack
The table I am trying to retrieve is one generated using another python script. It only has a couple of columns and rows in it. Not sure how to proceed here exactly.

python -m pwiz -e sqlite db.sqlite3 > db_map.py
pwiz reads the database and creates a file with the database mapping.
db.sqlite3 is the name of your database (put your database filename)
db_map.py is the filename of the output file (name it as you like but keep the .py extension)