Develop Reference

Problem on adressing nested relative directories in server

I want to run some image processing code on a Linux server.
The code is located in
/q/w/e/r/t
The images are located in
/abc/d/e/f/g # the images are in the g folder
This is the code I'm using:
path = "/abc/d/e/f"
new_path = os.path.join(path, 'g', '001.png')
img1= cv2.imread(new_path)
However, this is the error I'm getting:
[ WARN:0#"a number"] global /io/opencv/modules/imgcodecs/src/loadsave.cpp (239) findDecoder imread_('/abc/d/e/f/g/001.png'): can't open/read file: check file path/integrity
How can I solve the problem?

Probably it's because of permissions, but to reveal what is happening exactly best is to run the program with strace: https://man7.org/linux/man-pages/man1/strace.1.html which will show you the exact problem while it's trying to access to the file.

You can try the following:
Check whether you have permission to read and write the file. You can simply go to directory that contains all subdirectories and give read and write access to all subdirectories to all users.
Check whether you can access the directory stored in variable path.
Check the spelling of your pathname
Use try and catch to open the files - this way you see whether the problem is only one file or all of them which narrows the source of bug down.
Check the start of your path. Is that correct and does your program know the path?
If you can, download an image and have a look at it. Is it corrupted?
The OpenCV error sounds like certain library paths are not set. Either OpenCV is not properly installed or the path to the library not set. Might be worth to reinstall and set the paths properly.

Check that...
the path is indeed correct, that is, /abc/d/e/f/g/001.png exists
the process executing the python script has the necessary permissions to read this file
the file isn't corrupt (copy it to your local computer and open it in a browser/image viewer) (After reading the opencv source that's probably not it)

You can also "walk" the filesystem so you get all files below a certain directory:
def get_files(path=os.getcwd()):
files = []
if not path.endswith(os.sep):
path += os.sep
for root, dirs, files in os.walk(path)
for file in files:
filepath = os.path.join(root, file)
files.append(filepath)
return files
files = get_files("/abc/d/e/f")
Also, check your file and path. On the linux system's shell (I assume it is Bash):
$ ls -l <file> # check if it exists and whether the permissions are correct
$ file <file> # check the filetype (should hint at being an image)

From dataframe to Excel in Desktop

I exported dataframe from Python to Excel. I have no problem, but I have a question if someone can help me. Can I change the path of the file without having to go through personal folders like "crisf" in my case, to save it to the desktop? I am looking for a generic route that can be applied to all computers
xlwriter= pd.ExcelWriter(r'C:\Users\crisf\Desktop\OHH + GIT + M3.xls')

I doubt that all computers will have user called "crisf". Instead use this and get universal path (path contains user’s home directory):
import os
***some code
xlwriter= pd.ExcelWriter(os.path.expanduser("~/Desktop") + path of folder on desktop + file name)
I run print(os.path.expanduser("~\Desktop")) command on my PC and got:
C:\Users\korne\Desktop

h5py.File(path) doesn't recognize folder path

I am in my project folder call "project". I have two neural network h5 file, one in "project/my_folder/my_model_1.h5", I also copy it to folder "project/my_model_2.h5". So I open my Jupyter Notebook which is working at "project" folder.
import h5py
f = h5py.File("my_model_2.h5") # has NO Issue
but
f = h5py.File("my_folder/my_model_1.h5") # OSError
It says OSError: Unable to open file (unable to open file: name = 'my_folder/my_model_1.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
Interestingly, I only have this issue when I do the same thing on my Mac, but I don't encounter any issue in Linux machine.
Please let me know if you know how to fix this. Thank you in advance.

So it looks like some hidden invalid character incidentally got copied when I simply copy and paste the file path from Mac folder system. Take a look at the code in the screen.
The Line 92 is the path name I directly copy and paste from Mac folder.
The Line 93 is the path I literally type with every single letter, then there is no error and .h5 file is loaded properly. It's a kinda of similar issue that has been spotted by someone at this link: Invalid character in identifier
I simply copy the error code to Pycharm, and the unwelcome character got busted.
So solution, for Mac user, be careful of of just simply copying the text from folder system, if something obviously weird, try type every letter into the text editor.

Specifying the absolute path using the os worked in windows
file_name = os.path.dirname(__file__) +'\\my_folder\\my_model_1.h5'
f = h5py.File(file_name)
dont forget to import os though

How do I access a file for reading/writing in a different (non-current) directory?

I am working on the listener portion of a backdoor program (for an ETHICAL hacking course) and I would like to be able to read files from any part of my linux system and not just from within the directory where my listener python script is located - however, this has not proven to be as simple as specifying a typical absolute path such as "~/Desktop/test.txt"
So far my code is able to read files and upload them to the virtual machine where my reverse backdoor script is actively running. But this is only when I read and upload files that are in the same directory as my listener script (aptly named listener.py). Code shown below.
def read_file(self, path):
with open(path, "rb") as file:
return base64.b64encode(file.read())
As I've mentioned previously, the above function only works if I try to open and read a file that is in the same directory as the script that the above code belongs to, meaning that path in the above content is a simple file name such as "picture.jpg"
I would like to be able to read a file from any part of my filesystem while maintaining the same functionality.
For example, I would love to be able to specify "~/Desktop/another_picture.jpg" as the path so that the contents of "another_picture.jpg" from my "~/Desktop" directory are base64 encoded for further processing and eventual upload.
Any and all help is much appreciated.
Edit 1:
My script where all the code is contained, "listener.py", is located in /root/PycharmProjects/virus_related/reverse_backdoor/. within this directory is a file that for simplicity's sake we can call "picture.jpg" The same file, "picture.jpg" is also located on my desktop, absolute path = "/root/Desktop/picture.jpg"
When I try read_file("picture.jpg"), there are no problems, the file is read.
When I try read_file("/root/Desktop/picture.jpg"), the file is not read and my terminal becomes stuck.
Edit 2:
I forgot to note that I am using the latest version of Kali Linux and Pycharm.
I have run "realpath picture.jpg" and it has yielded the path "/root/Desktop/picture.jpg"
Upon running read_file("/root/Desktop/picture.jpg"), I encounter the same problem where my terminal becomes stuck.
[FINAL EDIT aka Problem solved]:
Based on the answer suggesting trying to read a file like "../file", I realized that the code was fully functional because read_file("../file") worked without any flaws, indicating that my python script had no trouble locating the given path. Once the file was read, it was uploaded to the machine running my backdoor where, curiously, it uploaded the file to my target machine but in the parent directory of the script. It was then that I realized that problem lied in the handling of paths in the backdoor script rather than my listener.py
Credit is also due to the commentator who pointed out that "~" does not count as a valid path element. Once I reached the conclusion mentioned just above, I attempted read_file("~/Desktop/picture.jpg") which failed. But with a quick modification, read_file("/root/Desktop/picture.jpg") was successfully executed and the file was uploaded in the same directory as my backdoor script on my target machine once I implemented some quick-fix code.
My apologies for not being so specific; efforts to aid were certainly confounded by the unmentioned complexity of my situation and I would like to personally thank everyone who chipped in.
This was my first whole-hearted attempt to reach out to the stackoverflow community for help and I have not been disappointed. Cheers!

A solution I found is putting "../" before the filename if the path is right outside of the dictionary.
test.py (in some dictionary right inside dictionary "Desktop" (i.e. /Desktop/test):
with open("../test.txt", "r") as test:
print(test.readlines())
test.txt (in dictionary "/Desktop")
Hi!
Hello!
Result:
["Hi!", "Hello!"]
This is likely the simplest solution. I found this solution because I always use "cd ../" on the terminal.

This not only allows you to modify the current file, but all other files in the same directory as the one you are reading/writing to.
path = os.path.dirname(os.path.abspath(__file__))
dir_ = os.listdir(path)
for filename in dir_:
f = open(dir_ + '/' + filename)
content = f.read()
print filename, len(content)
try:
im = Image.open(filename)
im.show()
except IOError:
print('The following file is not an image type:', filename)

converting/mapping linux reference path without altering the file?

Currently on a project that my client needs the reference file path to
remain in linux format. For example
A.ma , referencing objects from --> //linux/project/scene/B.ma
B.ma , referencing objects from --> //linux/project/scene/C.ma
Most of our Maya license here however are on Windows. I can run a
Python script that convert all the paths windows paths and save the
file. For example
Z:\project\scene\B.ma
However I'm trying to figure out a way to do this without converting
or altering the original file.... I'll try to explain what I'm trying to do.
Run the script to open the file.
The script checks for the linux formatted reference path, and all
child path down the hierarchy.
Maps all paths to their appropriate windows formatted paths.
Giving the animators the ability to "save" files normally without running a separate save script.
Is this possible to achieve this with Python script? Or will I need a
fully-compiled plug in to get this to work?
Any suggestion is greatly appreciated.
edit: Thank you for your input.
A little more clarification. The projects were set up for us by a remote company and part of the requirement is that we have to keep the path as is. They come as absolute path and we have no choice in that matter.
We match the mount //linux/ on our Fedora workstations. That same drive is mapped to Z:\ on our windows workstations. We only have 2 Maya license for Linux tho which is why I'm trying to do this.

Here is a solution. First step is to create a dict that keeps track of linux/windows references (don't forget to import the re module for regexp):
>>> def windows_path(path):
return path.replace('//linux', 'Z:').replace('/', '\\')
>>> reg = re.compile('(\w+\.ma) , referencing objects from --> (.*)')
>>> d = {}
>>> for line in open('D:\\temp\\Toto.txt'):
match = reg.match(line)
if match:
file_name = match.groups()[0]
linux_path = match.groups()[1]
d[file_name] = (linux_path, windows_path(linux_path))
>>> d
{'B.ma': ('//linux/project/scene/C.ma', 'Z:\\project\\scene\\C.ma'),
'A.ma': ('//linux/project/scene/B.ma', 'Z:\\project\\scene\\B.ma')}
Then you just need to loop on this dict to ask for file save:
>>> for file_name in d.keys():
s = raw_input('do you want to save file %s ? ' % file_name)
if s.lower() in ('y', 'yes'):
# TODO: save your file thanks to d[file][0] for linux path,
# d[file][1] for windows path
print '-> file %s was saved' % file_name
else:
print '-> file %s was not saved' % file_name
do you want to save file B.ma ? n
-> file B.ma was not saved
do you want to save file A.ma ? yes
-> file A.ma was saved

Many Windows applications will interpret paths with two leading "/"s as UNC paths. I don't know if Maya is one of those, but try it out. If Maya can understand paths like "//servername/share/foo", then all you need to do is set up a SMB server named "linux", and the paths will work as they are. I would guess that this is actually what your client does, since the path "//linux" would not make sense in a Linux-only environment.

You can use environment variables to do this. Maya will expand environment vars present in a file path, you could use Maya.env to set them up properly for each platform.

What you are looking for is the dirmap mel command. It is completely non-intrusive to your files as you just define a mapping from your linux paths to windows and/or vice versa. Maya will internally apply the mapping to resolve the paths, without changing them when saving the file.
To setup dirmap, you need to run a MEL script which issues the respective commands on maya startup. UserSetup.mel could be one place to put it.
For more details, see the official documentation - this particular link points to maya 2012, the command is available in Maya 7.0 and earlier as well though:
http://download.autodesk.com/global/docs/maya2012/en_us/Commands/dirmap.html