I'm working with HPC and am trying to run a python script that reads and modify every tif files in a folder and its subfolders. Right now I'm running a for loop:
files = glob.glob("./Examples/**/*.tif", recursive = True)
for file in files:
But it doesn't modify every file in the subfolder, only the final one. I run the exact same script locally and it worked just fine. So I'm thinking it's something to do with SLURM sitting on top of the code and maybe controlling how and when outputs get written. So could there be a way to write a python/slurm script that looks at the files in the directory and calls a slurm script, passing it a file name? Or any other solutions are also welcomed.
Thanks.
Related
SOLVED
chkdsk /f and rebooting fixed the problem.
Problem. Python script freezes when doing multiple writings in files located inside 'Desktop' folder. This happens at around 10-15's file. Doesn't matter what kind of file, but amount of files, that script have modified (around 10-15).
If they are moved outside 'Desktop' folder (C:\Users\User\Desktop) or python script is running by CMD with admin rights, then there is no freeze.
EDIT This behaviour has nothing to do with specific file. Because I've experimented with any random files I found. Also I tried to delete file at which script froze with prior file. The result was the same — freeze after exact amount of files.
Also when I deleted first 9 files. Then script succesfully modified files where previosly it froze. But again when around 10-15 files passed script freeze.
Here is an example of code where script freezes:
root = ...
for path, subdirs, files in os.walk(root):
for name in files:
file = (os.path.join(path, name))
print("1")
with open(file, 'r+b') as f: #Here script freezes
print("2") #No matter what the file is
f.seek(1) #but iterations of loop are limited to 10-15
f.write(b'\x0a')
Python 3.10, Windows 10(updated recently), Windows Defender deactivated.
Replacing with() by open()+close() doesnt help at all.
I tried script on .txt, .css, .bin files — everywhere the same freeze occurs, but at different amount of files that script went through (~10-20 cycles in the above code example before freeze)
Q. Why does this happen without Exceptions, Errors or notifications by System? What causes a script to freeze? And why the freeze happens after 10-20 files were successfully modified?
I have a Python script that tags MP3's and renames them. It works when assigning sys args via script parameters in Pycharm.
However, when I call the Python file from a bat, parsing in the same two parameters, I get problems with the file rename.
When printing os.listdir debugging the script, I see a bunch of old test MP3's that are no longer in the dir.
I've checked I'm referencing the correct dir. But it seems like somehow the list of files is cached and being passed over (since I have a validation check - abstracted function in the second part of the if statement blow).
Could it be possible Windows is caching something, or perhaps the .pyc is?
Here is part of the Python in question...
os.chdir(directory)
for files in os.listdir("."):
if files.endswith(".mp3") and not the_file_name_is_valid(files):
log.info("Starting work on: " + files)...
When run from the below bat file, os.listdir(".") produces a list of files that are not current for the working directory. Yet when the script is run from Pycharm it works perfectly.
Here is the bat
echo off
set SET_PREP="set_prep.py"
set DIR=%cd%
set IMAGE="cover_artwork.jpg"
c:\python27\python.exe %SET_PREP% %DIR% %IMAGE%
Pause
Why does running my script with command parameters behave differently to using Script Parameters in Pycharm?
The program is separate. Once it runs, I get the .bov files in another directory I call output.
Then I type, module load allpy
then, python
and then any bit of python code I want. But I don't know how to open these .bov files.
Python novice here.
I have a Python script that performs some geodatabase management (reconcile/post versions, compress, etc). I have the following line of code in my script:
createLog = open(str(datetime.date.today()) + ".txt", "w")
along each step of the script I add to the text file with the following statements:
createLog.write("Database connections blocked.\n")
When I run the script in my IDE (PyCharm) I get the desired result: A text file with each step written to the .txt file. When I run it in Task Scheduler no .txt file is created and therefore no log. Everything else runs as far as I can tell. I'm able to track edits made to the data.
I have experienced things like this before with task scheduler but have never been able to resolve the problem in the past.
Any ideas?
I think this is a working directory problem. Python's open function opens a file in the current working directory, NOT in the same folder as the script. This is a common misconception! (Which confused me for ages when learning Python...)
So what is a working directory? Well to quote my good friend Wikipedia:
In computing, the working directory of a process is a directory of a hierarchical file system, if any,[1] dynamically associated with each process. When the process refers to a file using a simple file name or relative path (as opposed to a file designated by a full path from a root directory), the reference is interpreted relative to the current working directory of the process. So for example a process with working directory /rabbit-shoes that asks to create the file foo.txt will end up creating the file /rabbit-shoes/foo.txt.
(Source: https://en.wikipedia.org/wiki/Working_directory)
So how is this working directory selected?
Well it is selected by the parent process of that processes! When you run a program from a shell like bash, the shell (the parent process) helpfully sets the working directory of the program you are running (the child process) to the directory you are currently in. (That is, the directory you cd'd to.)
Since your IDE is smart and helpful, it is starting your Python script process and setting the working directory to the same place the script itself is located. The task scheduler is less helpful... I have absolutely no idea what it is setting the working directory to. However if you search your system, I am sure you will find the log file lying about somewhere!
I am currently writing a very simple script on python which requires that I download a .dat file posted online. I then use loadtxt to plot the data using matplotlib. However, in order to read the file, I had to change directory (os.chdir) to 'downloads' (where the file had been saved). This works fine for me, but I will need to send the script to somebody else, in which case it seems as though the directory would again need to be something else in order to find the file... Where might I save the file so that no matter to whom I send it, the script will run properly?
User-specific path to the dat file can be acquired using:
os.chdir(raw_input('Which is the path to the download folder?'))