Develop Reference

mkdir not setting permissions correctly

I am trying to make a directory that has the "set group ID on execution" set
in Python
In the shell:
$ mkdir --mode 2775 mail
$ ls -ld mail
drwxrwsr-x 1 john john 0 May 3 08:34 mail
^
In Python I am using pathlib.Path.mkdir, and it seems there some bits for mode are not taken into account,
even when the umask is cleared:
from pathlib import Path
import os
os.umask(0)
md = Path('mail')
md.mkdir(mode=0o2775)
drwxrwxr-x 1 john john 0 May 3 08:35 mail
^
Is there some cut-off at nine bits that is not documented

Ultimately, on non-Windows systems, Path.mkdir calls os.mkdir which in turn invokes the C system call mkdirat. That call's behavior is defined in terms of mkdir, which, on Linux, documents the mode with:
The argument mode specifies the permissions to use. It is modified by the process's umask in the usual way: the permissions of the created directory are (mode & ~umask & 0777)
Note that the permissions explicitly mask off all bits above 0777; this is just how mkdir system calls behave on some OSes. You'll just have to chmod it after the fact.

pathlib's mkdir essentially just calls os.mkdir(). The documentation there includes the section:
On some systems, mode is ignored. Where it is used, the current umask value is first masked out. If bits other than the last 9 (i.e. the last 3 digits of the octal representation of the mode) are set, their meaning is platform-dependent. On some platforms, they are ignored and you should call chmod() explicitly to set them.
So only the last 9 bits are guaranteed to be used.
You'll have to do (no need to set umask):
md.mkdir()
md.chmod(0o2775)
It would be better if pathlib would not silently ignore extra bits, but actually threw a ValueError.

Does path.mkdir()'s updates the file permissions if path already exists and mode is passed?

I have a volume attached in kubernetes with path /var/www/aaa/tmp.
That volume was created using path.mkdir() and currently have 755 permissions.
It was created with code path.mkdir(parents=True, exist_ok=True) initially.
I'm trying to update its permissions without deleting the existing path.
I'm using path.mkdir(parents=True, exist_ok=True, mode=0o777). I'm still facing issues related to permissions and getting 502 Bad gateway for the flask app that is creating the above directories.
Does the path.mkdir(parents=True, exist_ok=True, mode=0o777) updates the path permissions if it already exists and have 755 permissions? Or will it ignore it completely as we've mentioned exists_ok=True ? I don't see the permissions getting updated for the path.
Should I be deleting the path completely and re-running the path.mkdir..... with mode=0o777 which creates new directories and set permissions?
Edit 1:
I've tried using os.chmod() on the path. But it's throwing PermissionError.
Here's the code snippet.
path.mkdir(parents=True, exist_ok=True)
os.chmod(path, mode=0o777)
Error:
File "./app/init.py", line 79, in create_prediction_app
create_directories(app) File "./app/init.py", line 36, in create_directories
os.chmod(path, mode=0o777) PermissionError: [Errno 1] Operation not permitted: '/var/www/aaa/tmp' unable to load app 0
(mountpoint='') (callable not found or import error) * no app
loaded. GAME OVER *

If the path is already existing, you should use the os.chmod(path, mode) instead of deleting/re-creating.
For example:
import os
os.chmod("/var/www/aaa/tmp", 0o777)
Furthermore, the chmod can get the permission from stat module.
stat.S_ISUID − Set user ID on execution.
stat.S_ISGID − Set group ID on execution.
stat.S_ENFMT − Record locking enforced.
stat.S_ISVTX − Save text image after execution.
stat.S_IREAD − Read by owner.
stat.S_IWRITE − Write by owner.
stat.S_IEXEC − Execute by owner.
stat.S_IRWXU − Read, write, and execute by owner.
stat.S_IRUSR − Read by owner.
stat.S_IWUSR − Write by owner.
stat.S_IXUSR − Execute by owner.
stat.S_IRWXG − Read, write, and execute by group.
stat.S_IRGRP − Read by group.
stat.S_IWGRP − Write by group.
stat.S_IXGRP − Execute by group.
stat.S_IRWXO − Read, write, and execute by others.
stat.S_IROTH − Read by others.
stat.S_IWOTH − Write by others.
stat.S_IXOTH − Execute by others.
For example:
import os
import stat
# Set a file write by others.
os.chmod("/var/www/aaa/tmp", stat.S_IWOTH)
You can set more permission with the bitwise operator.
For example:
import os
import stat
os.chmod(
'/var/www/aaa/tmp',
stat.S_IRUSR |
stat.S_IROTH |
stat.S_IRGRP
)
Complete testing:
>>> touch test_perm.sh
>>> ll test_perm.sh
-rw-rw-r-- test_perm.sh
>>> python -c "import os; os.chmod('test_perm.sh', 0755)"
>>> ll test_perm.sh
-rwxr-xr-x test_perm.sh
EDIT:
If you get PermissionError: [Errno 1] Operation not permitted:... exception when you want to change the permission with os.chmod, you should try the following code part to solve it. It's important to run the script with admin rights (with sudo in Linux environment).
Code:
from getpwnam import pwd
from getgrnam import grp
import os
uid = getpwnam("USERNAME")[2]
gid = grp.getgrnam("GROUPNAME")[2]
os.chown("/var/www/aaa/tmp", uid, gid)
Based on the official chown documentation:
os.chown(path, uid, gid, *, dir_fd=None, follow_symlinks=True)
Change the owner and group id of path to the numeric uid and gid. To leave one of the ids unchanged, set it to -1.
This function can support specifying a file descriptor, paths relative to directory descriptors and not following symlinks.
See shutil.chown() for a higher-level function that accepts names in addition to numeric ids.
Availability: Unix.
New in version 3.3: Added support for specifying path as an open file descriptor, and the dir_fd and follow_symlinks arguments.
Changed in version 3.6: Supports a path-like object.
Link to documentation: https://docs.python.org/3/library/os.html#os.chown

milanbalazs' answer is very nice, good explanation, but will only chmod the last directory in the path. In my case this was not good enough, because I needed to ensure every subfolder all the way down the last one was also chmodded.
I couldn't find any simple way to do this with existing bash tools, the only one close to what I needed with was chmod -R but I also didn't want to interfere without anything other than specifically each folder specified in the path.
To clarify, in the example, we have the path /var/www/aaa/tmp. I might want to start from /var/www and ensure that both /var/www/aaa and /var/www/aaa/tmp have their permissions set to whatever I want.
So I wrote the following script to solve this use case;
import os
import argparse
def base8(val):
return int(val, base=8)
def path(val):
assert os.path.exists(val)
return val
def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument('start_dir', type=path, help='starting point for chmod to check for subdirectories')
parser.add_argument('end_dir', help='last directory to be chmodded, relative to start_dir')
parser.add_argument('permission', type=base8, help='permission to set on each directory (must be base 8 format e.g. "0o750")')
args = parser.parse_args()
return args
def main():
args = parse_args()
directories = os.path.split(args.end_dir)
current_dir = args.start_dir
for directory in directories:
current_dir = os.path.join(current_dir, directory)
os.chmod(current_dir, args.permission)
if __name__ == '__main__':
main()
Hopefully someone finds this useful one day :)

How do you do a simple "chmod +x" from within python?

I want to create a file from within a python script that is executable.
import os
import stat
os.chmod('somefile', stat.S_IEXEC)
it appears os.chmod doesn't 'add' permissions the way unix chmod does. With the last line commented out, the file has the filemode -rw-r--r--, with it not commented out, the file mode is ---x------. How can I just add the u+x flag while keeping the rest of the modes intact?

Use os.stat() to get the current permissions, use | to OR the bits together, and use os.chmod() to set the updated permissions.
Example:
import os
import stat
st = os.stat('somefile')
os.chmod('somefile', st.st_mode | stat.S_IEXEC)

For tools that generate executable files (e.g. scripts), the following code might be helpful:
def make_executable(path):
mode = os.stat(path).st_mode
mode |= (mode & 0o444) >> 2 # copy R bits to X
os.chmod(path, mode)
This makes it (more or less) respect the umask that was in effect when the file was created: Executable is only set for those that can read.
Usage:
path = 'foo.sh'
with open(path, 'w') as f: # umask in effect when file is created
f.write('#!/bin/sh\n')
f.write('echo "hello world"\n')
make_executable(path)

If you're using Python 3.4+, you can use the standard library's convenient pathlib.
Its Path class has built-in chmod and stat methods.
from pathlib import Path
import stat
f = Path("/path/to/file.txt")
f.chmod(f.stat().st_mode | stat.S_IEXEC)

If you know the permissions you want then the following example may be the way to keep it simple.
Python 2:
os.chmod("/somedir/somefile", 0775)
Python 3:
os.chmod("/somedir/somefile", 0o775)
Compatible with either (octal conversion):
os.chmod("/somedir/somefile", 509)
reference permissions examples

Respect umask like chmod +x
man chmod says that if augo is not given as in:
chmod +x mypath
then a is used but with umask:
A combination of the letters ugoa controls which users' access to the file will be changed: the user who owns it (u), other users in the file's group (g), other users not in the file's group (o), or all users (a). If none of these are given, the effect is as if (a) were given, but bits that are set in the umask are not affected.
The goal of this is so that you don't accidentally give too many permissions. umask determines the default permissions of a new file, e.g. with a umask of 0077, touch newfile.txt produces permissions rw for the current user because the 77 would exclude group and other (x is not given by default by touch anyways though). And chmod +x would similarly only add +x for user, ignoring group and other due to the 0011 part of the mask: you would need chmod o+x, chmod g+x, chmod go+x or chmod a+x to force them to be set.
Here is a version that simulates that behavior exactly:
#!/usr/bin/env python3
import os
import stat
def get_umask():
umask = os.umask(0)
os.umask(umask)
return umask
def chmod_plus_x(path):
os.chmod(
path,
os.stat(path).st_mode |
(
(
stat.S_IXUSR |
stat.S_IXGRP |
stat.S_IXOTH
)
& ~get_umask()
)
)
chmod_plus_x('.gitignore')
See also: How can I get the default file permissions in Python?
Tested in Ubuntu 16.04, Python 3.5.2.

You can also do this
>>> import os
>>> st = os.stat("hello.txt")
Current listing of file
$ ls -l hello.txt
-rw-r--r-- 1 morrison staff 17 Jan 13 2014 hello.txt
Now do this.
>>> os.chmod("hello.txt", st.st_mode | 0o111)
and you will see this in the terminal.
ls -l hello.txt
-rwxr-xr-x 1 morrison staff 17 Jan 13 2014 hello.txt
You can bitwise or with 0o111 to make all executable, 0o222 to make all writable, and 0o444 to make all readable.

we can directly call chmod +x command in python using os.system()
import os
os.system("chmod +x somefile")

This turns on all executable bits for a file:
os.chmod("path", os.stat("path").st_mode | 0o111)

In python3:
import os
os.chmod("somefile", 0o664)
Remember to add the 0o prefix since permissions are set as an octal integer, and Python automatically treats any integer with a leading zero as octal. Otherwise, you are passing os.chmod("somefile", 1230) indeed, which is octal of 664.

How can I get the default file permissions in Python?

I am writing a Python script in which I write output to a temporary file and then move that file to its final destination once it is finished and closed. When the script finishes, I want the output file to have the same permissions as if it had been created normally through open(filename,"w"). As it is, the file will have the restrictive set of permissions used by the tempfile module for temp files.
Is there a way for me to figure out what the "default" file permissions for the output file would be if I created it in place, so that I can apply them to the temp file before moving it?

For the record, I had a similar issue, here is the code I have used:
import os
from tempfile import NamedTemporaryFile
def UmaskNamedTemporaryFile(*args, **kargs):
fdesc = NamedTemporaryFile(*args, **kargs)
# we need to set umask to get its current value. As noted
# by Florian Brucker (comment), this is a potential security
# issue, as it affects all the threads. Considering that it is
# less a problem to create a file with permissions 000 than 666,
# we use 666 as the umask temporary value.
umask = os.umask(0o666)
os.umask(umask)
os.chmod(fdesc.name, 0o666 & ~umask)
return fdesc

There is a function umask in the os module. You cannot get the current umask per se, you have to set it and the function returns the previous setting.
The umask is inherited from the parent process. It describes, which bits are not to be set when creating a file or directory.

This way is slow but safe and will work on any system that has a 'umask' shell command:
def current_umask() -> int:
"""Makes a best attempt to determine the current umask value of the calling process in a safe way.
Unfortunately, os.umask() is not threadsafe and poses a security risk, since there is no way to read
the current umask without temporarily setting it to a new value, then restoring it, which will affect
permissions on files created by other threads in this process during the time the umask is changed.
On recent linux kernels (>= 4.1), the current umask can be read from /proc/self/status.
On older systems, the safest way is to spawn a shell and execute the 'umask' command. The shell will
inherit the current process's umask, and will use the unsafe call, but it does so in a separate,
single-threaded process, which makes it safe.
Returns:
int: The current process's umask value
"""
mask: Optional[int] = None
try:
with open('/proc/self/status') as fd:
for line in fd:
if line.startswith('Umask:'):
mask = int(line[6:].strip(), 8)
break
except FileNotFoundError:
pass
except ValueError:
pass
if mask is None:
import subprocess
mask = int(subprocess.check_output('umask', shell=True).decode('utf-8').strip(), 8)
return mask

import os
def current_umask() -> int:
tmp = os.umask(0o022)
os.umask(tmp)
return tmp
This function is implemented in some python packages, e.g. pip and setuptools.

Write file with specific permissions in Python

I'm trying to create a file that is only user-readable and -writable (0600).
Is the only way to do so by using os.open() as follows?
import os
fd = os.open('/path/to/file', os.O_WRONLY, 0o600)
myFileObject = os.fdopen(fd)
myFileObject.write(...)
myFileObject.close()
Ideally, I'd like to be able to use the with keyword so I can close the object automatically. Is there a better way to do what I'm doing above?

What's the problem? file.close() will close the file even though it was open with os.open().
with os.fdopen(os.open('/path/to/file', os.O_WRONLY | os.O_CREAT, 0o600), 'w') as handle:
handle.write(...)

This answer addresses multiple concerns with the answer by vartec, especially the umask concern.
import os
import stat
# Define file params
fname = '/tmp/myfile'
flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL # Refer to "man 2 open".
mode = stat.S_IRUSR | stat.S_IWUSR # This is 0o600.
umask = 0o777 ^ mode # Prevents always downgrading umask to 0.
# For security, remove file with potentially elevated mode
try:
os.remove(fname)
except OSError:
pass
# Open file descriptor
umask_original = os.umask(umask)
try:
fdesc = os.open(fname, flags, mode)
finally:
os.umask(umask_original)
# Open file handle and write to file
with os.fdopen(fdesc, 'w') as fout:
fout.write('something\n')
If the desired mode is 0600, it can more clearly be specified as the octal number 0o600. Even better, just use the stat module.
Even though the old file is first deleted, a race condition is still possible. Including os.O_EXCL with os.O_CREAT in the flags will prevent the file from being created if it exists due to a race condition. This is a necessary secondary security measure to prevent opening a file that may already exist with a potentially elevated mode. In Python 3, FileExistsError with [Errno 17] is raised if the file exists.
Failing to first set the umask to 0 or to 0o777 ^ mode can lead to an incorrect mode (permission) being set by os.open. This is because the default umask is usually not 0, and it will be applied to the specified mode. For example, if my original umask is 2 i.e. 0o002, and my specified mode is 0o222, if I fail to first set the umask, the resulting file can instead have a mode of 0o220, which is not what I wanted. Per man 2 open, the mode of the created file is mode & ~umask.
The umask is restored to its original value as soon as possible. This getting and setting is not thread safe, and a threading.Lock must be used in a multithreaded application.
For more info about umask, refer to this thread.

update
Folks, while I thank you for the upvotes here, I myself have to argue against my originally proposed solution below. The reason is doing things this way, there will be an amount of time, however small, where the file does exist, and does not have the proper permissions in place - this leave open wide ways of attack, and even buggy behavior.
Of course creating the file with the correct permissions in the first place is the way to go - against the correctness of that, using Python's with is just some candy.
So please, take this answer as an example of "what not to do";
original post
You can use os.chmod instead:
>>> import os
>>> name = "eek.txt"
>>> with open(name, "wt") as myfile:
... os.chmod(name, 0o600)
... myfile.write("eeek")
...
>>> os.system("ls -lh " + name)
-rw------- 1 gwidion gwidion 4 2011-04-11 13:47 eek.txt
0
>>>
(Note that the way to use octals in Python is by being explicit - by prefixing it with "0o" like in "0o600". In Python 2.x it would work writing just 0600 - but that is both misleading and deprecated.)
However, if your security is critical, you probably should resort to creating it with os.open, as you do and use os.fdopen to retrieve a Python File object from the file descriptor returned by os.open.

The question is about setting the permissions to be sure the file will not be world-readable (only read/write for the current user).
Unfortunately, on its own, the code:
fd = os.open('/path/to/file', os.O_WRONLY, 0o600)
does not guarantee that permissions will be denied to the world. It does try to set r/w for the current user (provided that umask allows it), that's it!
On two very different test systems, this code creates a file with -rw-r--r-- with my default umask, and -rw-rw-rw- with umask(0) which is definitely not what is desired (and poses a serious security risk).
If you want to make sure that the file has no bits set for group and world, you have to umask these bits first (remember - umask is denial of permissions):
os.umask(0o177)
Besides, to be 100% sure that the file doesn't already exist with different permissions, you have to chmod/delete it first (delete is safer, since you may not have write permissions in the target directory - and if you have security concerns, you don't want to write some file where you're not allowed to!), otherwise you may have a security issue if a hacker created the file before you with world-wide r/w permissions in anticipation of your move. In that case, os.open will open the file without setting its permissions at all and you're left with a world r/w secret file...
So you need:
import os
if os.path.isfile(file):
os.remove(file)
original_umask = os.umask(0o177) # 0o777 ^ 0o600
try:
handle = os.fdopen(os.open(file, os.O_WRONLY | os.O_CREAT, 0o600), 'w')
finally:
os.umask(original_umask)
This is the safe way to ensure the creation of a -rw------- file regardless of your environment and configuration. And of course you can catch and deal with the IOErrors as needed. If you don't have write permissions in the target directory, you shouldn't be able to create the file, and if it already existed the delete will fail.

I would like to suggest a modification of A-B-B's excellent answer that separates the concerns a bit more clearly. The main advantage would be that you can handle exceptions that occur during opening the file descriptor separately from other problems during actual writing to the file.
The outer try ... finally block takes care of handling the permission and umask issues while opening the file descriptor. The inner with block deals with possible exceptions while working with the Python file object (as this was the OP's wish):
try:
oldumask = os.umask(0)
fdesc = os.open(outfname, os.O_WRONLY | os.O_CREAT, 0o600)
with os.fdopen(fdesc, "w") as outf:
# ...write to outf, closes on success or on exceptions automatically...
except IOError, ... :
# ...handle possible os.open() errors here...
finally:
os.umask(oldumask)
If you want to append to the file instead of writing, then the file descriptor should be opened like this:
fdesc = os.open(outfname, os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o600)
and the file object like this:
with os.fdopen(fdesc, "a") as outf:
Of course all other usual combinations are possible.

I'd do differently.
from contextlib import contextmanager
#contextmanager
def umask_helper(desired_umask):
""" A little helper to safely set and restore umask(2). """
try:
prev_umask = os.umask(desired_umask)
yield
finally:
os.umask(prev_umask)
# ---------------------------------- […] ---------------------------------- #
[…]
with umask_helper(0o077):
os.mkdir(os.path.dirname(MY_FILE))
with open(MY_FILE, 'wt') as f:
[…]
File-manipulating code tends to be already try-except-heavy; making it even worse with os.umask's finally isn't going to bring your eyes any more joy. Meanwhile, rolling your own context manager is that easy, and results in somewhat neater indentation nesting.