I am trying to make a directory that has the "set group ID on execution" set
in Python
In the shell:
$ mkdir --mode 2775 mail
$ ls -ld mail
drwxrwsr-x 1 john john 0 May 3 08:34 mail
^
In Python I am using pathlib.Path.mkdir, and it seems there some bits for mode are not taken into account,
even when the umask is cleared:
from pathlib import Path
import os
os.umask(0)
md = Path('mail')
md.mkdir(mode=0o2775)
drwxrwxr-x 1 john john 0 May 3 08:35 mail
^
Is there some cut-off at nine bits that is not documented
Ultimately, on non-Windows systems, Path.mkdir calls os.mkdir which in turn invokes the C system call mkdirat. That call's behavior is defined in terms of mkdir, which, on Linux, documents the mode with:
The argument mode specifies the permissions to use. It is modified by the process's umask in the usual way: the permissions of the created directory are (mode & ~umask & 0777)
Note that the permissions explicitly mask off all bits above 0777; this is just how mkdir system calls behave on some OSes. You'll just have to chmod it after the fact.
pathlib's mkdir essentially just calls os.mkdir(). The documentation there includes the section:
On some systems, mode is ignored. Where it is used, the current umask value is first masked out. If bits other than the last 9 (i.e. the last 3 digits of the octal representation of the mode) are set, their meaning is platform-dependent. On some platforms, they are ignored and you should call chmod() explicitly to set them.
So only the last 9 bits are guaranteed to be used.
You'll have to do (no need to set umask):
md.mkdir()
md.chmod(0o2775)
It would be better if pathlib would not silently ignore extra bits, but actually threw a ValueError.
Related
I have a volume attached in kubernetes with path /var/www/aaa/tmp.
That volume was created using path.mkdir() and currently have 755 permissions.
It was created with code path.mkdir(parents=True, exist_ok=True) initially.
I'm trying to update its permissions without deleting the existing path.
I'm using path.mkdir(parents=True, exist_ok=True, mode=0o777). I'm still facing issues related to permissions and getting 502 Bad gateway for the flask app that is creating the above directories.
Does the path.mkdir(parents=True, exist_ok=True, mode=0o777) updates the path permissions if it already exists and have 755 permissions? Or will it ignore it completely as we've mentioned exists_ok=True ? I don't see the permissions getting updated for the path.
Should I be deleting the path completely and re-running the path.mkdir..... with mode=0o777 which creates new directories and set permissions?
Edit 1:
I've tried using os.chmod() on the path. But it's throwing PermissionError.
Here's the code snippet.
path.mkdir(parents=True, exist_ok=True)
os.chmod(path, mode=0o777)
Error:
File "./app/init.py", line 79, in create_prediction_app
create_directories(app) File "./app/init.py", line 36, in create_directories
os.chmod(path, mode=0o777) PermissionError: [Errno 1] Operation not permitted: '/var/www/aaa/tmp' unable to load app 0
(mountpoint='') (callable not found or import error) * no app
loaded. GAME OVER *
If the path is already existing, you should use the os.chmod(path, mode) instead of deleting/re-creating.
For example:
import os
os.chmod("/var/www/aaa/tmp", 0o777)
Furthermore, the chmod can get the permission from stat module.
stat.S_ISUID − Set user ID on execution.
stat.S_ISGID − Set group ID on execution.
stat.S_ENFMT − Record locking enforced.
stat.S_ISVTX − Save text image after execution.
stat.S_IREAD − Read by owner.
stat.S_IWRITE − Write by owner.
stat.S_IEXEC − Execute by owner.
stat.S_IRWXU − Read, write, and execute by owner.
stat.S_IRUSR − Read by owner.
stat.S_IWUSR − Write by owner.
stat.S_IXUSR − Execute by owner.
stat.S_IRWXG − Read, write, and execute by group.
stat.S_IRGRP − Read by group.
stat.S_IWGRP − Write by group.
stat.S_IXGRP − Execute by group.
stat.S_IRWXO − Read, write, and execute by others.
stat.S_IROTH − Read by others.
stat.S_IWOTH − Write by others.
stat.S_IXOTH − Execute by others.
For example:
import os
import stat
# Set a file write by others.
os.chmod("/var/www/aaa/tmp", stat.S_IWOTH)
You can set more permission with the bitwise operator.
For example:
import os
import stat
os.chmod(
'/var/www/aaa/tmp',
stat.S_IRUSR |
stat.S_IROTH |
stat.S_IRGRP
)
Complete testing:
>>> touch test_perm.sh
>>> ll test_perm.sh
-rw-rw-r-- test_perm.sh
>>> python -c "import os; os.chmod('test_perm.sh', 0755)"
>>> ll test_perm.sh
-rwxr-xr-x test_perm.sh
EDIT:
If you get PermissionError: [Errno 1] Operation not permitted:... exception when you want to change the permission with os.chmod, you should try the following code part to solve it. It's important to run the script with admin rights (with sudo in Linux environment).
Code:
from getpwnam import pwd
from getgrnam import grp
import os
uid = getpwnam("USERNAME")[2]
gid = grp.getgrnam("GROUPNAME")[2]
os.chown("/var/www/aaa/tmp", uid, gid)
Based on the official chown documentation:
os.chown(path, uid, gid, *, dir_fd=None, follow_symlinks=True)
Change the owner and group id of path to the numeric uid and gid. To leave one of the ids unchanged, set it to -1.
This function can support specifying a file descriptor, paths relative to directory descriptors and not following symlinks.
See shutil.chown() for a higher-level function that accepts names in addition to numeric ids.
Availability: Unix.
New in version 3.3: Added support for specifying path as an open file descriptor, and the dir_fd and follow_symlinks arguments.
Changed in version 3.6: Supports a path-like object.
Link to documentation: https://docs.python.org/3/library/os.html#os.chown
milanbalazs' answer is very nice, good explanation, but will only chmod the last directory in the path. In my case this was not good enough, because I needed to ensure every subfolder all the way down the last one was also chmodded.
I couldn't find any simple way to do this with existing bash tools, the only one close to what I needed with was chmod -R but I also didn't want to interfere without anything other than specifically each folder specified in the path.
To clarify, in the example, we have the path /var/www/aaa/tmp. I might want to start from /var/www and ensure that both /var/www/aaa and /var/www/aaa/tmp have their permissions set to whatever I want.
So I wrote the following script to solve this use case;
import os
import argparse
def base8(val):
return int(val, base=8)
def path(val):
assert os.path.exists(val)
return val
def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument('start_dir', type=path, help='starting point for chmod to check for subdirectories')
parser.add_argument('end_dir', help='last directory to be chmodded, relative to start_dir')
parser.add_argument('permission', type=base8, help='permission to set on each directory (must be base 8 format e.g. "0o750")')
args = parser.parse_args()
return args
def main():
args = parse_args()
directories = os.path.split(args.end_dir)
current_dir = args.start_dir
for directory in directories:
current_dir = os.path.join(current_dir, directory)
os.chmod(current_dir, args.permission)
if __name__ == '__main__':
main()
Hopefully someone finds this useful one day :)
I'm working with shelve in python 2.7.6 for caching calculations, and I've ran into the issue described HERE for shelve files produced by me and implemented the suggested solution in a function that merges a file other into the file target:
... # target and other are strings
# Loads the gdbm module, as suggested
mod = __import__("gdbm")
# Open target for modifications
tar = shelve.Shelf(mod.open(target, 'c', 0666)) # THROWS EXCEPTION
# Open other for reading
oth = shelve.Shelf(mod.open(other, 'r'))
...
The two files are owned by me, are recorded in a local filesystem and have posix permissions set to 0666, or, equivalently, -rw-rw-rw- in my Linux Mint box, so that the obvious checks have been performed:
$ ls -l
-rw-rw-rw- 1 myusr mygrp 11694080 Sep 17 21:24 cache
-rw-rw-rw- 1 myusr mygrp 12189696 Sep 17 21:23 cache.0
Here, cache is the target, and cache.0 is the other file. The current working directory is owned by me and has permissions 0775, and I can create files with touch, cp, etc., at will with no problems, and I've even set my umask to 0000, so new files are created with 0666, or, equivalently, -rw-rw-rw- permissions.
I've even matched the actual file permissions with the permissions in the gdbm.open() call, according to its documentation; however, to no avail.
Update: By running the python code with sudo, i.e., with superuser privileges, the error occurs on the same line; however with a different message: gdbm error: Bad magic number! This is very strange as the very point of using a (seemingly) lower-level module (gdbm as opposed to shelve) was precisely bypassing database type detection.
Update #2: Running python whichdb.py on the files returns dbhash; however, changing the module to the dbhash on the loading code still gives the following errors:
bsddb.db.DBAccessError: (13, 'Permission denied')
when running as user, but
bsddb.db.DBInvalidArgError: (22, 'Invalid argument -- BDB0210 ././merge-cache.py: metadata page checksum error')
when running with sudo; merge-cache.py is my code's name.
This new error is discussed here, in connection with python version, but (i) my python version is different than the one in that post, and (ii) files are created and later read with the same version of python.
This answer indicates shelve gets "wasted" with large sets, but the problem I'm reporting occurs with smaller databases as well.
Question: How can I open these shelve files using python-2.7.6? (upgrading python is not an option).
I want to create a file from within a python script that is executable.
import os
import stat
os.chmod('somefile', stat.S_IEXEC)
it appears os.chmod doesn't 'add' permissions the way unix chmod does. With the last line commented out, the file has the filemode -rw-r--r--, with it not commented out, the file mode is ---x------. How can I just add the u+x flag while keeping the rest of the modes intact?
Use os.stat() to get the current permissions, use | to OR the bits together, and use os.chmod() to set the updated permissions.
Example:
import os
import stat
st = os.stat('somefile')
os.chmod('somefile', st.st_mode | stat.S_IEXEC)
For tools that generate executable files (e.g. scripts), the following code might be helpful:
def make_executable(path):
mode = os.stat(path).st_mode
mode |= (mode & 0o444) >> 2 # copy R bits to X
os.chmod(path, mode)
This makes it (more or less) respect the umask that was in effect when the file was created: Executable is only set for those that can read.
Usage:
path = 'foo.sh'
with open(path, 'w') as f: # umask in effect when file is created
f.write('#!/bin/sh\n')
f.write('echo "hello world"\n')
make_executable(path)
If you're using Python 3.4+, you can use the standard library's convenient pathlib.
Its Path class has built-in chmod and stat methods.
from pathlib import Path
import stat
f = Path("/path/to/file.txt")
f.chmod(f.stat().st_mode | stat.S_IEXEC)
If you know the permissions you want then the following example may be the way to keep it simple.
Python 2:
os.chmod("/somedir/somefile", 0775)
Python 3:
os.chmod("/somedir/somefile", 0o775)
Compatible with either (octal conversion):
os.chmod("/somedir/somefile", 509)
reference permissions examples
Respect umask like chmod +x
man chmod says that if augo is not given as in:
chmod +x mypath
then a is used but with umask:
A combination of the letters ugoa controls which users' access to the file will be changed: the user who owns it (u), other users in the file's group (g), other users not in the file's group (o), or all users (a). If none of these are given, the effect is as if (a) were given, but bits that are set in the umask are not affected.
The goal of this is so that you don't accidentally give too many permissions. umask determines the default permissions of a new file, e.g. with a umask of 0077, touch newfile.txt produces permissions rw for the current user because the 77 would exclude group and other (x is not given by default by touch anyways though). And chmod +x would similarly only add +x for user, ignoring group and other due to the 0011 part of the mask: you would need chmod o+x, chmod g+x, chmod go+x or chmod a+x to force them to be set.
Here is a version that simulates that behavior exactly:
#!/usr/bin/env python3
import os
import stat
def get_umask():
umask = os.umask(0)
os.umask(umask)
return umask
def chmod_plus_x(path):
os.chmod(
path,
os.stat(path).st_mode |
(
(
stat.S_IXUSR |
stat.S_IXGRP |
stat.S_IXOTH
)
& ~get_umask()
)
)
chmod_plus_x('.gitignore')
See also: How can I get the default file permissions in Python?
Tested in Ubuntu 16.04, Python 3.5.2.
You can also do this
>>> import os
>>> st = os.stat("hello.txt")
Current listing of file
$ ls -l hello.txt
-rw-r--r-- 1 morrison staff 17 Jan 13 2014 hello.txt
Now do this.
>>> os.chmod("hello.txt", st.st_mode | 0o111)
and you will see this in the terminal.
ls -l hello.txt
-rwxr-xr-x 1 morrison staff 17 Jan 13 2014 hello.txt
You can bitwise or with 0o111 to make all executable, 0o222 to make all writable, and 0o444 to make all readable.
we can directly call chmod +x command in python using os.system()
import os
os.system("chmod +x somefile")
This turns on all executable bits for a file:
os.chmod("path", os.stat("path").st_mode | 0o111)
In python3:
import os
os.chmod("somefile", 0o664)
Remember to add the 0o prefix since permissions are set as an octal integer, and Python automatically treats any integer with a leading zero as octal. Otherwise, you are passing os.chmod("somefile", 1230) indeed, which is octal of 664.
In Python (tried this in 2.7 and below) it looks like a file created using tempfile.NamedTemporaryFile doesn't seem to obey the umask directive:
import os, tempfile
os.umask(022)
f1 = open ("goodfile", "w")
f2 = tempfile.NamedTemporaryFile(dir='.')
f2.name
Out[33]: '/Users/foo/tmp4zK9Fe'
ls -l
-rw------- 1 foo foo 0 May 10 13:29 /Users/foo/tmp4zK9Fe
-rw-r--r-- 1 foo foo 0 May 10 13:28 /Users/foo/goodfile
Any idea why NamedTemporaryFile won't pick up the umask? Is there any way to do this during file creation?
I can always workaround this with os.chmod(), but I was hoping for something that did the right thing during file creation.
This is a security feature. The NamedTemporaryFile is always created with mode 0600, hardcoded at tempfile.py, line 235, because it is private to your process until you open it up with chmod. There is no constructor argument to change this behavior.
In case it might help someone, I wanted to do more or less the same thing, here is the code I have used:
import os
from tempfile import NamedTemporaryFile
def UmaskNamedTemporaryFile(*args, **kargs):
fdesc = NamedTemporaryFile(*args, **kargs)
# we need to set umask to get its current value. As noted
# by Florian Brucker (comment), this is a potential security
# issue, as it affects all the threads. Considering that it is
# less a problem to create a file with permissions 000 than 666,
# we use 666 as the umask temporary value.
umask = os.umask(0o666)
os.umask(umask)
os.chmod(fdesc.name, 0o666 & ~umask)
return fdesc
When including the line
*.py diff=python
in a local .gitattributes file, git diff produces nice labels for the different diff hunks of Python files (with the name of the function where the changes are, etc.).
Is is possible to ask git to use this diff mode for all Python files across all git projects? I tried to set a global ~/.gitattributes, but it is not used by local git repositories. Is there a more convenient method than initializing each new git project with a ln -s ~/.gitattributes?
Quoting from gitattributes(5):
Attributes that should affect all repositories for a single user should be placed in a file specified by the core.attributesfile configuration option (see git-config(1)). Its default value is $XDG_CONFIG_HOME/git/attributes. If $XDG_CONFIG_HOME is either not set or empty, $HOME/.config/git/attributes is used instead. Attributes for all users on a system should be placed in the $(prefix)/etc/gitattributes file.
TL;DR: echo '*.py diff=python' >> "${XDG_CONFIG_HOME:-$HOME/.config}"/git/attributes
Update, 7 years later
Ok, it's not necessary to configure diff=python for *.py files — it's the default since long ago.
But the general point remains: anything you can set up in local (per-repository) .gitattributes, you can also make global (per-machine).
There're many good examples in man 5 gitattributes itself, so please go RTFM.
Let's do just one custom setup: --word-diff for all Markdown files (kudos to #RayLuo for suggesting this in comments).
First, we add an external diff driver:
git config --global diff.stackoverflow-word-diff.command ~/.local/bin/stackoverflow-word-diff
The API is such that we must make a standalone wrapper executable.
cat > ~/.local/bin/stackoverflow-word-diff << 'EOF'
#!/bin/bash -eu
#-- uncomment for debug:
#echo >&2 "$(basename $0) args: $#"; set -x
FILENAME="$1"
OLDFILE="$2"
OLDHASH="$3"
OLDMODE="$4"
NEWFILE="$5"
NEWHASH="$6"
NEWMODE="$7"
git diff --no-ext-diff --word-diff "$OLDFILE" "$NEWFILE" || exit 0
#-- from https://stackoverflow.com/a/18948381/531179
#-- see `man 1 git` /EXTERNAL_DIFF, or https://www.git-scm.com/docs/git
EOF
chmod +x ~/.local/bin/stackoverflow-word-diff
Finally, we tie that in to *.md, *.markdown via global gitattributes:
mkdir -vp "${XDG_CONFIG_HOME:-$HOME/.config}"/git
{ echo '*.md diff=stackoverflow-word-diff'; \
echo '*.markdown diff=stackoverflow-word-diff; \
} \
>> "${XDG_CONFIG_HOME:-$HOME/.config}"/git/attributes
And that's all folks! Test it.
To tell git to use ~/.gitattributes you need to put this in ~/.gitconfig:
[core]
attributesfile = ~/.gitattributes
No, git only looks for attributes locally: .gitattributes and .git/info/attributes