This is a strange one and strace is not giving me any useful information. I am using pyinotify 0.9.6 to watch a directory recursively so I can commit changes about it to a MySQL database. The problem is when I have pyinotify running (daemonized or not), i can remove files in the sub-directories but not the sub-directories themselves. rmdir exits with status 0 and everything looks right on the system level, but the sub directory is still there just chilling. I may just be being stupid as I am new to the framework, but here is an example of how I am initializing the watch:
wm = pyinotify.WatchManager()
notifier = pyinotify.Notifier(wm)
wm.add_watch('/path/to/dir', pyinotify.IN_CLOSE_WRITE, rec=True, auto_add=True,\
proc_fun=CommitFunction() # I can supply this if you think it's relevant
notifier.loop(daemonize=True, callback=None,\
stdout='/path/to/log.log, stderr='/path/to/log.log')
So in this example if there was a file '/path/to/dir/file.txt' I could run a rm on 'file.txt' and it gets deleted, but if there was a sub-directory '/path/to/dir/subdir' running rm -r on 'subdir' exits cleanly but the directory doesn't actually get deleted.
Furthermore my logs aren't getting written, but I am pretty sure that's my fault somewhere.
EDIT:
Here is an example CommitFunction:
class CommitFunction(pyinotify.ProcessEvent):
def process_default(self, event):
dir = event.path
file = event.name
print "%s is the new file" % os.path.join(dir, file)
EDIT2: Actually my logs probably aren't getting written because I don't call a log or print function anywhere during the commit. I just write straight to my MySQL database
EDIT3: Ok here goes. Hopefully I am not delving too deep. Here is what I try on the command line:
bash$ cd /var/www
bash$ mkdir subdir
bash$ rmdir subdir
bash$ ls
subdir
Here is the actual commit function:
class CommitFunction(pyinotify.ProcessEvent):
def process_default(self, event):
dir = event.path
file = event.name
dbaccess.commit(dir, file) # dir corresponds to a project name
I can keep going deeper if you want, but dbaccess has all my functions for committing and querying the database (nothing really touching the fs) and it further draws from a 'models' file that defines my tables. It's a flask/uwsgi app if that helps at all
Using this code:
import pyinotify
import os
class CommitFunction(pyinotify.ProcessEvent):
def process_default(self, event):
dir = event.path
file = event.name
print "%s is the new file" % os.path.join(dir, file)
wm = pyinotify.WatchManager()
notifier = pyinotify.Notifier(wm)
wm.add_watch('/tmp/testdir', pyinotify.IN_CLOSE_WRITE, rec=True,
auto_add=True, proc_fun=CommitFunction())
notifier.loop(daemonize=False, callback=None)
I can create and delete subdirectories in the target directory
without error:
bash-4.3$ pwd
/tmp/testdir
bash-4.3$ mkdir subdir
bash-4.3$ rm -r subdir
bash-4.3$ ls
bash-4.3$
In response to the above session, the code logged the following on
stdout:
/tmp/testdir/subdir is the new file
/tmp/testdir/subdir/ is the new file
Can you confirm whether the above exhibits the behavior you have described in your environment? If not, can you update your question with a complete example that does demonstrate the problem?
OK so the problem I was having is that IN_CLOSE_WRITE was also getting called on removal of a sub-directory, which in turn was firing a function I had to make sure a directory for the "project" existed when uploaded through the web portal.
I am still confused, but this particular question is solved
Related
I'm trying to keep passwords that are usually written in a py file separated from the script and make it so that those passwords are only accessible by root and python whenever a script needs it. I got the idea reading this:
https://stackoverflow.com/a/158248/3892678
To do this, I'm trying to hide passwords to be used in a_script in another passwords py file. passwords can only be read, written, and executed (-rwxrwx---)by root:root. As another user tomato, I want to run a_script, which imports the password from passwords to be used in the file. To make it so that this user can run the file as root, I've elevated the file's setuid and setgid withchmod 6777 a_script.py so that the file has -rwsrwsrwx permissions. Now, as user tomato, I run python a_script.py, but I get back ImportError: No module named passwords. I thought that setting the uid and groupid as s would run the file as root, which should have permissions to read passwords. What am I doing wrong?
Here's a_script.py
import os
print "uid: %s" % os.getuid()
print "euid: %s" % os.getgid()
print "gid: %s" % os.geteuid()
print "egid: %s" % os.getegid()
from passwords import MYPASS
print MYPASS
All the print statements before I get the ImportError are 1001 which is tomato. Thanks for your help. Might there be a better way to "hide" passwords in another file so that only root and programs that need it are the only ones that have access to it?
how bout
os.popen("echo ROOT_PASSWORD | sudo -s -p '' cat /path/to/secure/file.txt").read()
(note its probably better to use the subprocess module ... but it requires more typing)
I've been trying to trouble this for days now, and would appreciate some help --
Basically, I wrote the following Python script
import os, sys
# =__=__=__=__=__=__=__ START MAIN =__=__=__=__=__=__=__
if __name__ == '__main__':
# initialize variables
all_files = []
# directory to download data siphon files to
dDir = '/path/to/download/directory/'
# my S3 bucket
s3bucket = "com.mybucket/"
foldername = "test"
# get a list of available feeds
feeds = <huge JSON object with URLs to feeds>
for item in range(feeds['count']):
# ...check if the directory exists, and if not, create the directory...
if not os.path.exists(folderName):
os.makedirs(folderName)
... ... ...
# Loop through all the splits
for s in dsSplits:
... ... ...
location = requestFeedLocation(name, timestamp)
... ... ...
downloadFeed(location[0], folderName, nameNotGZ)
# THIS IS WHERE I AM HAVING PROBLEMS!!!!!!!!!!!
cmd = 's3cmd sync 'dDir+folderName+'/ s3://'+s3bucket+'/'
os.system(cmd)
Everything in my code works...when I run this straight from the command line, everything runs as expected...however, when I have it executed via cron -- the following DOES NOT execute (everything else does)
# THIS IS WHERE I AM HAVING PROBLEMS!!!!!!!!!!!
cmd = 's3cmd sync 'dDir+folderName+'/ s3://'+s3bucket+'/'
os.system(cmd)
To answer a few questions, I am running the cron as root, s3cmd is configured for the root user, OS is Ubuntu 12.04, python version is 2.7, all of the necessary directories have Read / Write permissions...
What am I missing?
First check variable 'foldername' in command you have used N in variable.
In command syntax you need to add plus (+) sign in prefix like +dDir+folderName+
So I hope command would be like below..
cmd = 's3cmd sync '+dDir+foldername+'/ s3://'+s3bucket+'/'
os.system(cmd)
Here I found some more s3cmd sync help: http://tecadmin.net/s3cmd-file-sync-with-s3bucket/
OpenShift has these default dir's:
# $_ENV['OPENSHIFT_INTERNAL_IP'] - IP Address assigned to the application
# $_ENV['OPENSHIFT_GEAR_NAME'] - Application name
# $_ENV['OPENSHIFT_GEAR_DIR'] - Application dir
# $_ENV['OPENSHIFT_DATA_DIR'] - For persistent storage (between pushes)
# $_ENV['OPENSHIFT_TMP_DIR'] - Temp storage (unmodified files deleted after 10 days)
How do reference them in a python script?
Example script "created a log file in log directory and log in data directory?
from time import strftime
now= strftime("%Y-%m-%d %H:%M:%S")
fn = "${OPENSHIFT_LOG_DIR}/test.log"
fn2 = "${OPENSHIFT_DATA_DIR}/test.log"
#fn = "test.txt"
input = "appended text " + now + " \n"
with open(fn, "ab") as f:
f.write(input)
with open(fn2, "ab") as f:
f.write(input)
Can these script be used with cron?
EDIT the BASH File:
#! /bin/bash
#date >> ${OPENSHIFT_LOG_DIR}/new.log
source $OPENSHIFT_HOMEDIR/python-2.6/virtenv/bin/activate
python file.py
date >> ${OPENSHIFT_DATA_DIR}/new2data.log
import os
os.getenv("OPENSHIFT_INTERNAL_IP")
should work.
So with your example, modify to:-
import os
OPENSHIFT_LOG_DIR = os.getenv("OPENSHIFT_LOG_DIR")
fn = os.path.join(OPENSHIFT_LOG_DIR, "test.log")
And, yes, you can call this python script with a cron by referencing your bash script if you want... Like this for example:-
#!/bin/bash
date >> ${OPENSHIFT_LOG_DIR}/status.log
chmod +x status
cd ${OPENSHIFT_REPO_DIR}/wsgi/crawler
nohup python file.py 2>&1 &
Those variables OPENSHIFT_* are provided as environment variables on OpenShift -- so the $_ENV["OPENSHIFT_LOG_DIR"] is an example to get the value inside a php script.
In python, the equivalent would just be os.getenv("OPENSHIFT_LOG_DIR").
Made edits to Calvin's post above and submitted 'em.
Re: the question of where file.py exists -- use os.getenv("OPENSHIFT_REPO_DIR") as the base directory where all your code would be located on the gear where you app is running.
So if your file is located in .openshift/misc/file.py -- then just use:
os.path.join(os.getenv("OPENSHIFT_REPO_DIR"), ".openshift", "misc", "file.py")
to get the full path.
Or in bash, the equivalent would be:
$OPENSHIFT_REPO_DIR/.openshift/misc/file.py
HTH
I'm trying to add to our existing pre-commit SVN hook so that it will check for and block an increase in file size for files in specific directory/s.
I've written a python script to compare two file sizes, which takes two files as arguments and uses sys.exit(0) or (1) to return the result, this part seems to work fine.
My problem is in calling the python script from the batch file, how to reference the newly committed and previous versions of each file? The existing code is new to me and a mess of %REPOS%, %TXN%s etc and I'm not sure how to go about using them. Is there a simple, standard way of doing this?
It also already contains code to loop through the changed files using svnlook changed, so that part shouldn't be an issue.
Thanks very much
If comparing file sizes is all you need to do, look no further than the svnlook filesize command. The default invocation - svnlook filesize repo path - will give you the size of the HEAD revision of path. To get the size of the path in the incoming commit use svnlook filesize repo path -t argv[2].
Still, here is an example of listing all revisions of a versioned path (except the incoming one, since this is pre-commit hook).
#!/usr/bin/env python
from sys import argv, stderr, exit
from subprocess import check_output
repo = argv[1]
transaction = argv[2]
def path_history(path, limit=5):
path = '/%s' % path
cmd = ('svnlook', 'history', '-l', str(limit), repo, path)
out = check_output(cmd).splitlines()[2:]
for rev, _path in (i.split() for i in out):
if _path == path:
yield rev
def commit_changes():
cmd = ('svnlook', 'changed', repo, '-t', transaction)
out = check_output(cmd).splitlines()
for line in out:
yield line.split()
def filesize(path, rev=None, trans=None):
cmd = ['svnlook', 'filesize', repo, path]
if rev: cmd.extend(('-r', str(rev)))
elif trans: cmd.extend(('-t', str(trans)))
out = check_output(cmd)
return out.rstrip()
def filesize_catwc(path, rev=None, trans=None):
'''A `svnlook filesize` substitute for older versions of svn.
Uses `svnlook cat ... | wc -c` and should be very inefficient
for large files.'''
arg = '-r %s' % rev if rev else '-t %s' % trans
cmd = 'svnlook cat %s %s %s | wc -c' % (arg, repo, path)
out = check_output(cmd, shell=True)
return out.rstrip()
for status, path in commit_changes():
if status in ('A', 'M', 'U'):
# get the last 5 revisions of the added/modified path
revisions = list(path_history(path))
headrev = revisions[0]
oldsize = filesize(path, rev=headrev)
newsize = filesize(path, trans=transaction)
It is probably easier to write a whole pre-commit script in python. According to the subversion handbook, there are three inputs to pre-commit;
Two command line arguments
repository path
commit transaction name
lock-token info on standard input
If you want to know which files have changed, I suggest you use the subprocess.check_output() function to call svnlook changed. For the files which contents have changed, you should call svnlook filesize, to get the size of the file as it is in the last revision in the repository. The size of the equivalent file in the working directory you'd have to query with os.stat(), as shown in the function getsizes().
import subprocess
import sys
import os
repo = sys.argv[1]
commit_name = sys.argv[2]
def getsizes(rname, rfile):
'''Get the size of the file in rfile from the repository rname.
Derive the filename in the working directory from rfile, and use
os.stat to get the filesize. Return the two sizes.
'''
localname = rfile[10:].strip() # 'U trunk/foo/bar.txt' -> 'foo/bar.txt'
reposize = subprocess.check_output(['svnlook', 'filesize', rname, rfile])
reposize = int(reposize)
statinfo = os.stat(localname)
return (reposize, statinfo.st_size)
lines = subprocess.check_output(['svnlook', 'changed', repo]).splitlines()
for line in lines:
if line.startswith('U ') or line.startswith('UU'):
# file contents have changed
reposize, wdsize = getsizes(repo, line)
# do something with the sizes here...
elif line.startswith('_U'):
# properties have changed
pass
How would I go about getting a privilege elevation dialog to pop up in my Python app? I want the UAC dialog on Windows and the password authentication dialog on Mac.
Basically, I need root privileges for part of my application and I need to get those privileges through the GUI. I'm using wxPython. Any ideas?
On Windows you cannot get the UAC dialog without starting a new process, and you cannot even start that process with CreateProcess.
The UAC dialog can be brought about by running another application that has the appropriate manifest file - see Running compiled python (py2exe) as administrator in Vista for an example of how to do this with py2exe.
You can also programatically use the runas verb with the win32 api ShellExecute http://msdn.microsoft.com/en-us/library/bb762153(v=vs.85).aspx - you can call this by using ctypes http://python.net/crew/theller/ctypes/ which is part of the standard library on python 2.5+ iirc.
Sorry don't know about Mac. If you give more detail on what you want to accomplish on Windows I might be able to provide more specific help.
I know the post is a little old, but I wrote the following as a solution to my problem (running a python script as root on both Linux and OS X).
I wrote the following bash-script to execute bash/python scripts with administrator privileges (works on Linux and OS X systems):
#!/bin/bash
if [ -z "$1" ]; then
echo "Specify executable"
exit 1
fi
EXE=$1
available(){
which $1 >/dev/null 2>&1
}
platform=`uname`
if [ "$platform" == "Darwin" ]; then
MESSAGE="Please run $1 as root with sudo or install osascript (should be installed by default)"
else
MESSAGE="Please run $1 as root with sudo or install gksu / kdesudo!"
fi
if [ `whoami` != "root" ]; then
if [ "$platform" == "Darwin" ]; then
# Apple
if available osascript
then
SUDO=`which osascript`
fi
else # assume Linux
# choose either gksudo or kdesudo
# if both are avilable check whoch desktop is running
if available gksudo
then
SUDO=`which gksudo`
fi
if available kdesudo
then
SUDO=`which kdesudo`
fi
if ( available gksudo && available kdesudo )
then
if [ $XDG_CURRENT_DESKTOP = "KDE" ]; then
SUDO=`which kdesudo`;
else
SUDO=`which gksudo`
fi
fi
# prefer polkit if available
if available pkexec
then
SUDO=`which pkexec`
fi
fi
if [ -z $SUDO ]; then
if available zenity; then
zenity --info --text "$MESSAGE"
exit 0
elif available notify-send; then
notify-send "$MESSAGE"
exit 0
elif available xmessage notify-send; then
xmessage -buttons Ok:0 "$MESSAGE"
exit 0
else
echo "$MESSAGE"
fi
fi
fi
if [ "$platform" == "Darwin" ]
then
$SUDO -e "do shell script \"$*\" with administrator privileges"
else
$SUDO $#
fi
Basically, the way I set up my system is that I keep subfolders inside the bin directories (e.g. /usr/local/bin/pyscripts in /usr/local/bin), and create symbolic links to the executables. This has three benefits for me:
(1) If I have different versions, I can easily switch which one is executed by changing the symbolic link and it keeps the bin directory cleaner (e.g. /usr/local/bin/gcc-versions/4.9/, /usr/local/bin/gcc-versions/4.8/, /usr/local/bin/gcc --> gcc-versions/4.8/gcc)
(2) I can store the scripts with their extension (helpful for syntax highlighting in IDEs), but the executables do not contain them because I like it that way (e.g. svn-tools --> pyscripts/svn-tools.py)
(3) The reason I will show below:
I name the script "run-as-root-wrapper" and place it in a very common path (e.g. /usr/local/bin) so python doesn't need anything special to locate it. Then I have the following run_command.py module:
import os
import sys
from distutils.spawn import find_executable
#===========================================================================#
def wrap_to_run_as_root(exe_install_path, true_command, expand_path = True):
run_as_root_path = find_executable("run-as-root-wrapper")
if(not run_as_root_path):
return False
else:
if(os.path.exists(exe_install_path)):
os.unlink(exe_install_path)
if(expand_path):
true_command = os.path.realpath(true_command)
true_command = os.path.abspath(true_command)
true_command = os.path.normpath(true_command)
f = open(exe_install_path, 'w')
f.write("#!/bin/bash\n\n")
f.write(run_as_root_path + " " + true_command + " $#\n\n")
f.close()
os.chmod(exe_install_path, 0755)
return True
In my actual python script, I have the following function:
def install_cmd(args):
exe_install_path = os.path.join(args.prefix,
os.path.join("bin", args.name))
if(not run_command.wrap_to_run_as_root(exe_install_path, sys.argv[0])):
os.symlink(os.path.realpath(sys.argv[0]), exe_install_path)
So if I have a script called TrackingBlocker.py (actual script I use to modify the /etc/hosts file to re-route known tracking domains to 127.0.0.1), when I call "sudo /usr/local/bin/pyscripts/TrackingBlocker.py --prefix /usr/local --name ModifyTrackingBlocker install" (arguments handled via argparse module), it installs "/usr/local/bin/ModifyTrackingBlocker", which is a bash script executing
/usr/local/bin/run-as-root-wrapper /usr/local/bin/pyscripts/TrackingBlocker.py [args]
e.g.
ModifyTrackingBlocker add tracker.ads.com
executes:
/usr/local/bin/run-as-root-wrapper /usr/local/bin/pyscripts/TrackingBlocker.py add tracker.ads.com
which then displays the authentification dialog needed to get the privileges to add:
127.0.0.1 tracker.ads.com
to my hosts file (which is only writable by a superuser).
If you want to simplify/modify it to run only certain commands as root, you could simply add this to your script (with the necessary imports noted above + import subprocess):
def run_as_root(command, args, expand_path = True):
run_as_root_path = find_executable("run-as-root-wrapper")
if(not run_as_root_path):
return 1
else:
if(expand_path):
command = os.path.realpath(command)
command = os.path.abspath(command)
command = os.path.normpath(command)
cmd = []
cmd.append(run_as_root_path)
cmd.append(command)
cmd.extend(args)
return subprocess.call(' '.join(cmd), shell=True)
Using the above (in run_command module):
>>> ret = run_command.run_as_root("/usr/local/bin/pyscripts/TrackingBlocker.py", ["status", "display"])
>>> /etc/hosts is blocking approximately 16147 domains
I'm having the same problem on Mac OS X. I have a working solution, but it's not optimal. I will explain my solution here and continue looking for a better one.
At the beginning of the program I check if I'm root or not by executing
def _elevate():
"""Elevate user permissions if needed"""
if platform.system() == 'Darwin':
try:
os.setuid(0)
except OSError:
_mac_elevate()
os.setuid(0) will fail if i'm not already root and that will trigger _mac_elevate() which relaunch my program from the start as administrator with the help of osascript. osascript can be used to execute applescript and other stuff. I use it like this:
def _mac_elevate():
"""Relaunch asking for root privileges."""
print "Relaunching with root permissions"
applescript = ('do shell script "./my_program" '
'with administrator privileges')
exit_code = subprocess.call(['osascript', '-e', applescript])
sys.exit(exit_code)
The problem with this is if I use subprocess.call as above I keep the current process running and there will be two instances of my app running giving two dock icons. If I use subprocess.Popen instead and let the non-priviledged process die instantly I can't make use of the exit code, nor can I fetch the stdout/stderr streams and propagate to the terminal starting the original process.
Using osascript with with administrator privileges is actually just Apple Script wrapping a call to AuthorizationExecuteWithPrivileges().
But you can call AuthorizationExecuteWithPrivileges() directly from Python3 with ctypes.
For example, the following parent script spawn_root.py (run as a non-root user) spawns a child process root_child.py (run with root privileges).
The user will be prompted to enter their password in the OS GUI pop-up. Note that this will not work on a headless session (eg over ssh). It must be run inside the GUI (eg Terminal.app).
After entering the user's password into the MacOS challenge dialog correctly, root_child.py executes a soft shutdown of the system, which requires root permission on MacOS.
Parent (spawn_root.py)
#!/usr/bin/env python3
import sys, ctypes
import ctypes.util
from ctypes import byref
sec = ctypes.cdll.LoadLibrary(ctypes.util.find_library("Security"))
kAuthorizationFlagDefaults = 0
auth = ctypes.c_void_p()
r_auth = byref(auth)
sec.AuthorizationCreate(None,None,kAuthorizationFlagDefaults,r_auth)
exe = [sys.executable,"root_child.py"]
args = (ctypes.c_char_p * len(exe))()
for i,arg in enumerate(exe[1:]):
args[i] = arg.encode('utf8')
io = ctypes.c_void_p()
sec.AuthorizationExecuteWithPrivileges(auth,exe[0].encode('utf8'),0,args,byref(io))
Child (root_child.py)
#!/usr/bin/env python3
import os
if __name__ == "__main__":
f = open( "root_child.out", "a" )
try:
os.system( "shutdown -h now" )
f.write( "SUCCESS: I am root!\n" )
except Exception as e:
f.write( "ERROR: I am not root :'(" +str(e)+ "\n" )
f.close()
Security Note
Obviously, any time you run something as root, you need to be very careful!
AuthorizationExecuteWithPrivileges() is deprecated, but it can be used safely. But it can also be used unsafely!
It basically boils down to: do you actually know what you're running as root? If the script you're running as root is located in a Temp dir that has world-writeable permissions (as a lot of MacOS App installers have done historically), then any malicious process could gain root access.
To execute a process as root safely:
Make sure that the permissions on the process-to-be-launched are root:root 0400 (or writeable only by root)
Specify the absolute path to the process-to-be-launched, and don't allow any malicious modification of that path
Sources
https://github.com/cloudmatrix/esky/blob/master/esky/sudo/sudo_osx.py
https://github.com/BusKill/buskill-app/issues/14
https://www.jamf.com/blog/detecting-insecure-application-updates-on-macos/