Strange Caching Happening When Running Python Script From Bat - python

I have a Python script that tags MP3's and renames them. It works when assigning sys args via script parameters in Pycharm.
However, when I call the Python file from a bat, parsing in the same two parameters, I get problems with the file rename.
When printing os.listdir debugging the script, I see a bunch of old test MP3's that are no longer in the dir.
I've checked I'm referencing the correct dir. But it seems like somehow the list of files is cached and being passed over (since I have a validation check - abstracted function in the second part of the if statement blow).
Could it be possible Windows is caching something, or perhaps the .pyc is?
Here is part of the Python in question...
os.chdir(directory)
for files in os.listdir("."):
if files.endswith(".mp3") and not the_file_name_is_valid(files):
log.info("Starting work on: " + files)...
When run from the below bat file, os.listdir(".") produces a list of files that are not current for the working directory. Yet when the script is run from Pycharm it works perfectly.
Here is the bat
echo off
set SET_PREP="set_prep.py"
set DIR=%cd%
set IMAGE="cover_artwork.jpg"
c:\python27\python.exe %SET_PREP% %DIR% %IMAGE%
Pause
Why does running my script with command parameters behave differently to using Script Parameters in Pycharm?

Related

Is it possible to access the launching shortcut directory from a Python executalbe?

I currently have a Python scrip that runs through all Excel files in the current directory and generates a PDF report.
It works fine now but I don't want the users to be anywhere near frozen Python scripts. I created an MSI with cxFreeze which puts the EXE and scripts in the Program Files directory.
What I would like to be able to do is create a shortcut to this executable and pass the directory the shortcut was run from to the Python program so that can be set as the working directory. This would allow the user to move the shortcut to any folder of Excel files and generate a report there.
Does Windows send the location of a opened shortcut to the executable and is there a way to access it from Python?
When you launch a shortcut, Windows changes the working directory to the directory specified in the shortcut, in the Start in field. At this point, Windows has no memory of where the shortcut was stored.
You could change the Start in field to point to the directory that the shortcut is in. But you'd have to do that for every single shortcut, and never make a mistake.
The better approach is to use a script, rather than a shortcut. Place your actual Python script (which we'll call doit.py for sake of example) somewhere in your PYTHONPATH. Then create a single-line Python script that imports it:
import doit
Save it (but don't name it doit.py) and copy it to each directory from which you want to be able to invoke the main script. In doit.py you can use os.getcwd() to find out what directory you're being invoked from.
You could also do it with a batch file. This is a little more flexible in that you can specify the exact name of the script and which Python interpreter should be used, and don't need to store the script in a directory in PYTHONPATH. Also, you don't need to worry about the file's name clashing with the name of a Python module. Simply put this line in a file:
C:\path\to\your\python.exe C:\path\to\your\script.py
Save it as (e.g.) doit.bat and copy it into the directories from which you want to invoke it. As before, your Python script can call os.getcwd() to get the directory. Or you can write it so your Python script accepts it as the first argument, and write your batch file like:
C:\path\to\your\python.exe C:\path\to\your\script.py %cd%
Another thing you can do with the batch file approach is add a pause command to the end so that the user is asked to press a key after the script runs, giving them the opportunity to read any output generated by the script. You could even make this conditional so that it only happens if an error occurs (which requires returning a proper exit code from the script). I'll leave that as an exercise. :-)
Is there a problem with modifying the script to take the directory to process as a command line argument?
You could then configure the different shortcuts to pass in the appropriate directory.
Type the following into a batch file (i.e. script.bat):
python \absolute\path\to\your\script.py %~dp0
pause
Then add these imports at the top of your python file script.py (if not already included):
import os
import sys
And add this to the bottom of the python file (or combine it with a similar statement):
if __name__ == "__main__":
# set current working directory:
if len(sys.argv) > 1:
os.chdir(sys.argv[1])
main()
replace main() with whatever function you want to call or code you want to run.
The following is how I came to my answer:
I tried using kindall's answer and had the following issues:
The first suggestion of storing the script somewhere in PYTHONPATH could not be applied to my situation because my script will be used on a server and needs to be independent of the client computer's python environment (besides having the required pip installations).
I tried calling my python script from a Windows Batch File which could be moved to a different location. Instead of the batch file's location being used as the current working directory, it was C:\Windows.
I tried passing %cd% as an argument to my python script, then setting that to be my CWD. This still resulted in a CWD of C:\Windows.
After reviewing the comments, I tried Eryk Sun's suggestion of instead passing %~dp0 as an argument to the python script. This resulted in the CWD being correctly set to the batch file's location.
I hope this helps others facing similar difficulties.

Python os.chdir() not changing directory

So, I am following a simple tutorial "Python Tutorial: Automate Parsing and Renaming of Multiple Files" and am already encountering a problem where os.chdir() is not working. I am on a Windows 10 system, running python 3.6, and I have tried using both my regular terminal (which has cygwin installed) and bash on ubuntu on Windows.
Here is the code:
import os
print(os.getcwd())
os.chdir('c:/Users/Michelle Kaiser/Desktop/Lab_Progs/PI3Kalpha')
print(os.getcwd())
Here is the reg terminal:
C:\Users\Michelle Kaiser\Desktop\Lab_Progs>python rename.py
C:\Users\Michelle Kaiser\Desktop\Lab_Progs
C:\Users\Michelle Kaiser\Desktop\Lab_Progs>`
The path that it is returning corresponds to the folder my program is located in. I have moved the program 3 times to verify this. Also, it's obviously returning a path only once, so it's probably not responding to the 2 print statements.
Here is the bash terminal:
mkaiser#ZIPPY:/mnt/c/Users/Michelle Kaiser/Desktop/Lab_Progs$ python rename.py
/mnt/c/Users/Michelle Kaiser/Desktop/Lab_Progs
mkaiser#ZIPPY:/mnt/c/Users/Michelle Kaiser/Desktop/Lab_Progs$
I also tried running the code with os.path.exists(), which did not change the output on either terminal. I have definitely double checked that I am saving my program file from one test to the next. Thanks.
I've been trying to change a file that has no whitespace.
It seems like this person has a similar problem:
Python reading whitespace-separated file lines as separate lines

Python utility fails to successfully run a non-Python script that uses relative paths

My Python3 utility has a function that doesn't work (unless it's placed within selected directories, where it can then run the non-python pdflatex scripts successfully). I want to run the utility from a set location on any of the template.tex files I have, stored in various other locations.
The Python utility prompts the user to select a pdflatex template file from an absolute path using a tkinter.filedialog GUI, then runs the user's selected pdflatexscript using, for example: os.system("pdflatex /afullpath/a/b/c/mytemplate.tex")
Python's os.system runs pdflatex, which then runs its mytemplate.tex script. mytemplate.tex has numerous inputs written with relative paths like ./d/another.tex.
So, the Python utility works fine as long as it's in the exact same path as /afullpath/a/b/c/mytemplate.tex that the user selects. Otherwise pdflatex can't finds its own input files. pdflatex delivers an error message like: ! LaTeX Error: File ./d/another.tex not found because the execution path is relative to the Python script and not the pdflatex script.
[pdflatex needs to use relative paths because the folders with its .tex files get moved around, as needed.]
I found the following similar case on Stack Overflow, but I don't think the answers are geared towards this situation: Relative Paths In Python -- Stack Overflow
By referring to other files with relative paths like ./d/another.tex, your mytemplate.tex file is assuming (and requiring) that pdflatex is only run on it from the same directory that mytemplate.tex is located in. You thus need to satisfy this requirement by changing to the directory containing mytemplate.tex before calling os.system:
input_file = '/afullpath/a/b/c/mytemplate.tex'
olddir = os.getcwd()
os.chdir(os.path.dirname(input_file))
os.system('pdflatex ' + input_file)
os.chdir(olddir)
Even better is to use subprocess.call, as it handles the change of directory for you and isn't vulnerable to shell quoting issues:
subprocess.call(['pdflatex', input_file], cwd=os.path.dirname(input_file))
Use subprocess.run instead of os.system and pass in the cwd parameter as the directory of the latex script.
See the subprocess.run docs here, and look at the cwd parameter for subprocess.Popen.
Example:
subprocess.run(["pdflatex", "/afullpath/a/b/c/mytemplate.tex"], cwd="/afullpath/a/b/c/")

Why does Task Manager not run some lines of code in script?

Python novice here.
I have a Python script that performs some geodatabase management (reconcile/post versions, compress, etc). I have the following line of code in my script:
createLog = open(str(datetime.date.today()) + ".txt", "w")
along each step of the script I add to the text file with the following statements:
createLog.write("Database connections blocked.\n")
When I run the script in my IDE (PyCharm) I get the desired result: A text file with each step written to the .txt file. When I run it in Task Scheduler no .txt file is created and therefore no log. Everything else runs as far as I can tell. I'm able to track edits made to the data.
I have experienced things like this before with task scheduler but have never been able to resolve the problem in the past.
Any ideas?
I think this is a working directory problem. Python's open function opens a file in the current working directory, NOT in the same folder as the script. This is a common misconception! (Which confused me for ages when learning Python...)
So what is a working directory? Well to quote my good friend Wikipedia:
In computing, the working directory of a process is a directory of a hierarchical file system, if any,[1] dynamically associated with each process. When the process refers to a file using a simple file name or relative path (as opposed to a file designated by a full path from a root directory), the reference is interpreted relative to the current working directory of the process. So for example a process with working directory /rabbit-shoes that asks to create the file foo.txt will end up creating the file /rabbit-shoes/foo.txt.
(Source: https://en.wikipedia.org/wiki/Working_directory)
So how is this working directory selected?
Well it is selected by the parent process of that processes! When you run a program from a shell like bash, the shell (the parent process) helpfully sets the working directory of the program you are running (the child process) to the directory you are currently in. (That is, the directory you cd'd to.)
Since your IDE is smart and helpful, it is starting your Python script process and setting the working directory to the same place the script itself is located. The task scheduler is less helpful... I have absolutely no idea what it is setting the working directory to. However if you search your system, I am sure you will find the log file lying about somewhere!

Python os.system() call runs in incorrect directory

My coworker is having trouble with a Python install. When running the code below, from 'C:\my\folder\', 'C:\' is returned instead of the current working directory. When I or anyone else run the script on our systems, we get 'C:\my\folder\'.
We're assuming that some global setting must be causing the issue, so I've had the person uninstall Python, delete the local Python2.7 folder, clean the registry and reinstall the Python, but it's still not working.
NOTE: We have a large number of legacy scripts, so revising all of them to use subprocess is impractical. :(
Any ideas?
Environment: Windows XP, Python 2.7
import os
#
# This test script demonstrates issue on the users computer when python invokes
# a subshell via the standard os.system() call.
#
print "This is what python thinks the current working directory is..."
print os.getcwd()
print
print
print "but when i execute a command *from* python, this is what i get for the current working directory"
os.system('echo %cd%')
raw_input()
you could also try something like this
os.chdir("C:\\to\\my\\folder")
print os.system("echo %CD%")
raw_input()
also to get the current working directory i use a different approach
cur_dir = os.path.abspath(".")
os.getcwd() isn't guarenteed to get the location of your script when it is called. Your coworker might be calling the script a different way or his computer (for some reason) handles the current working directory differently.
To get the actual script location you should use the following:
import os
os.path.dirname(os.path.realpath(__file__))
As an example I wrote getcwd and the above line in the same script and ran it from C:\.
Results:
C:\>python C:\Users\pies\Desktop\test.py
C:\Users\pies\Desktop
C:\
It depends on what your real purpose for this script is, whether you actually need the current working directory, or just the current scripts directory. As a little caveat this call will return a different directory if you call a script from script which then uses this call.
os.system("cd dir;command params")

Categories

Resources