Develop Reference

Finding which files are being read from during a session (python code)

I have a large system written in python. when I run it, it reads all sorts of data from many different files on my filesystem. There are thousands lines of code, and hundreds of files, most of them are not actually being used. I want to see which files are actually being accessed by the system (ubuntu), and hopefully, where in the code they are being opened. Filenames are decided dynamically using variables etc., so the actual filenames cannot be determined just by looking at the code.
I have access to the code, of course, and can change it.
I try to figure how to do this efficiently, with minimal changes in the code:
is there a Linux way to determine which files are accessed, and at what times? this might be useful, although it won't tell me where in the code this happens
is there a simple way to make an "open file" command also log the file name, time, etc... of the open file? hopefully without having to go into the code and change every open command, there are many of them, and some are not being used at runtime.
Thanks

You can trace file accesses without modifying your code, using strace.
Either you start your program with strace, like this
strace -f -e trace=file your_program.py
Otherwise you attach strace to a running program like this
strace -f -e trace=file -p <PID>

For 1 - You can use
ls -la /proc/<PID>/fd`
Replacing <PID> with your process id.
Note that it will give you all the open file descriptors, some of them are stdin stdout stderr, and often other things, such as open websockets (which use a file descriptor), however filtering it for files should be easy.
For 2- See the great solution proposed here -
Override python open function when used with the 'as' keyword to print anything
e.g. overriding the open function with your own, which could include the additional logging.

One possible method is to "overload" the open function. This will have many effects that depend on the code, so I would do that very carefully if needed, but basically here's an example:
>>> _open = open
>>> def open(filename):
... print(filename)
... return _open(filename)
...
>>> open('somefile.txt')
somefile.txt
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in open
FileNotFoundError: [Errno 2] No such file or directory: 'somefile.txt'
As you can see my new open function will return the original open (renamed as _open) but will first print out the argument (the filename). This can be done with more sophistication to log the filename if needed, but the most important thing is that this needs to run before any use of open in your code

Remove file in python with absolute path: error "no such file or directory" but file exists

I'm trying to remove a file in Python 3 on Linux (RHEL) the following way:
os.remove(or.getcwd() + '/file.txt')
(sorry not allowed to publish the real paths).
and it gives me the usual error
No such file or directory: '/path/to/file/file.txt'
(I've respected slash or antislash in the path)
What is strange is that when I just ls the file (by copy pasting, so the very same path) the file does exist.
I've read this post but i'm not on Windows and slash direction seems correct.
Any idea ?
EDIT: as suggested by #DominicPrice os.system('ls') is showing the file while os.listdir() does not show it (but shows other files in the same directory)
EDIT 2: So my issue was due a a bad usage of os.popen. I used this method to copy file but did not wait for the subprocess to be terminated. So my understanding is that the file was not copied yet when I tried to delete it.

The problem is that, as you have explained in the comments, you are creating the file using os.popen("cp ..."). This works asynchronously, so it may not have had time to complete by the time you call os.remove(). You can force python to wait for it to finish by calling the close method:
proc = os.popen("cp myfile myotherfile")
proc.close() # wait for process to finish
os.remove("myotherfile") # we're all good
I would highly recommend staying away from using os.popen in favour of the subprocess library, which has a run function which is way safer to use.
For the specific functions of copying a file, an even better (and cross platform) solution is to use the shutil library:
import shutil
shutil.copyfile("myfile", "myotherfile")

you should use os.path.dirname(__file__).
this is an inbuilt function of os module in python.
you can read more here.
https://www.geeksforgeeks.org/find-path-to-the-given-file-using-python/

Python Path.rglob failing on network error when encountering folder without permission

I am new to Python and have been using this site as a reference...thanks for everything, I have learned a ton. First question:
I am running a basic recursive file search with Path.rglob(). I am running into a error when it encounters a folder that it does not have permission to access. I am running Python 3.7 on Windows and connecting to a windows share on a network drive.
Here's my code:
scan_folder = pathlib.Path("//192.168.1.242/Media")
nfo_files = list(scan_folder.rglob("*.nfo"))
It works perfectly until I encounter a folder that I do not have permission to access, then it errors out with:
Traceback (most recent call last):
File "D:/Working/media_tools/media_tools/movies_nfo_cataloger.py", line 337, in <module>
nfo_files = list(scan_folder.rglob("*.nfo"))
File "C:\Users\ulrick65\Anaconda3\lib\pathlib.py", line 1094, in rglob
for p in selector.select_from(self):
File "C:\Users\ulrick65\Anaconda3\lib\pathlib.py", line 544, in _select_from
for p in successor_select(starting_point, is_dir, exists, scandir):
File "C:\Users\ulrick65\Anaconda3\lib\pathlib.py", line 507, in _select_from
entries = list(scandir(parent_path))
OSError: [WinError 59] An unexpected network error occurred: '\\\\192.168.1.242\\Media\\#recycle'
Process finished with exit code 1
I searched and found the following Issue for Pathlib that appears to have been fixed, however the error is different in my case as it points to "Unexpected network error" instead of permissions.
https://bugs.python.org/issue24120
I verified that this is indeed a permissions error, I do not have access to that Recycle folder as the user I am logged in as. I edited the permissions for that folder and gave myself access and the code runs fine after that.
I know I could use oswalk as it ignores these...but I figured given the bug fix I linked to above, so should path.glob however it doesn't. Also, using path.rglob() is pretty slick, one line of code and is fast (not that oswalk wouldn't be just as fast).
Any help is appreciated.

Python script from batch file won't run in task scheduler

Hi folks so I got the following problem,
I have the following code in a batch file:
..\python-2.7.10.amd64\python.exe ./bin/bla.py ./conf/config.conf > ./logs/output.txt
This works like a charme by double clicking the batch. Next my plan was to automate the call of this batch by adding it to the task scheduler in windows. So I changed all the relative paths to absolute paths:
D:\path\to\python-2.7.10.amd64\python.exe D:\path\to\bin\bla.py D:\path\to\conf\config.conf > D:\path\to\logs\output.txt
This also still works by double clicking the batch file.
So my next step was adding the batch to the task scheduler but when I run it from there I get this error message:
Traceback (most recent call last): File "D:\path\to\bin\bla.py", line 159, in logging.config.fileConfig(logFile) File "D:\path\to\python-2.7.10.amd64\lib\logging\confi eConfig formatters = _create_formatters(cp) File "D:\path\to\python-2.7.10.amd64\lib\logging\confi reate_formatters flist = cp.get("formatters", "keys") File "D:\path\to\python-2.7.10.amd64\lib\ConfigParser. raise NoSectionError(section) ConfigParser.NoSectionError: No section: 'formatters'
So for some reason the python script can't find the conf file by the absolute path I think but I don't understand why. I also tried it with the relative paths in the task scheduler it obviously also doesn't work.
Does anyone of you have a clue why it works straight from the batch but not from the task scheduler ?

Thank you guys for your help. It was indeed "just" the working directory I had to set to the location of the bat file

Python: NameError - what should I do with this?

I am working on a large-scale software system that is written in Python right now.
The thing is, I am not sure how to make sure if each individual .py file in the system is correct. The only way for me to run the software is to run the main.py file, which uses all the other .py files.
So either everything works, or one thing doesn't (causing everything to not work).
I keep getting a NameError even when importing the correct file. I think this may have to do with the fact that the class associated with that name in the NameError may have errors in it. Any suggestions? NameError is giving me this:
File "<string>", line 1, in <module>
NameError: name 'RGBox' is not defined
It's not a very helpful error message, and I'm not sure why it's giving "string" and 'module' instead of actual values.....
[EDIT]- I am working through ssh into a remote unix machine

This is a straight-forward error message which indicates that the execution flow has not yet encountered class/module/variable RGBox prior to it being called.
RGBox is either being called out of sequence or has been mispelt.
Perform a commandline search through the app files for the name 'RGBox' or its regex equivalents. for example with grep you can do a case-insensitive search:
$ grep -lsri 'rgbox' ./my_project_folder
which will output any file which contains the patterns 'RGBox', 'rgBox', etc.
If you are unfamiliar with the code and its structure, then you may as well insert strategic logging (or print) statements at significant locations in the code to understand its flow and execution logic.