Develop Reference

linux directory permission check and/or dir exist or not

I have a script which is running as a ROOT on linux collecting data from different users. Given a nfs path for each individual user 1) verify the director does not exist 2) verify permission denied
verify_model_path_not_found = '/usr/bin/su ' + userID + ' -c \'/usr/bin/ls ' + u_model_path + ' | /usr/bin/grep \"Permission denied\|No such file or directory\"\''
perm_denied_str = 'Permission denied'
no_file_dir_str = 'No such file or directory'
print("verify_model_path_not_found:", verify_model_path_not_found)
#Run as a root
try:
verify_cmd_out = subprocess.check_output(verify_model_path_not_found, shell=True)
verify_cmd_out = str(verify_cmd_out)
print("verify_cmd_out:", verify_cmd_out, "\n")
except subprocess.CalledProcessError as errorcatch:
print(datetime.datetime.now(), " Error while executing ", verify_model_path_not_found)
print("error code", errorcatch.returncode, errorcatch.output, "\n")
continue
#only add items that are not accessible (perm denied or no file found)
if ((perm_denied_str in verify_cmd_out) or (no_file_dir_str in verify_cmd_out)):
#remaining actions .... send an email to the user ...
Example output Error:
verify_model_path_not_found: /usr/bin/su xxxx -c '/usr/bin/ls /nfs/x/y/z | /usr/bin/grep "Permission denied\|No such file or directory"'
2021-08-10 17:00:31.827186 Error while executing /usr/bin/su xxxx -c '/usr/bin/ls /nfs/x/y/z | /usr/bin/grep "Permission denied\|No such file or directory"'
error code 1 b'' #I know this dir does not exist or perm denied - still getting error
given /nfs/x/y/z, if the user does not have a read access, I would like to get "Permission denied" using grep - "Permission denied" should be the value of verify_cmd_out
given /nfs/x/y/z, if the dir does not exist, I would like to get "No such file or directory" using grep - "No such file or directory" should be the value of verify_cmd_out
once perm denied or no such file is confirmed for the user, certain actions need to place.
The /usr/bin/su xxxx -c ... command is not properly working, any thought or idea how to resolve the issue?

You are examining standard output (file descriptor 1), but error messages (and progress and diagnostics in general) are posted on standard error (file descriptor 2).
Your code is quite clunky anyway. Probably try something along the lines of
import subprocess
def assert_error_stderr(cmd, expected_stderr):
"""
Accept a shell command; verify that it fails with error,
and emits the expected message on standard error.
"""
try:
result = subprocess.run(cmd, check=True, text=True, capture_output=True)
raise ValueError("%r did not raise an error" % cmd)
except CalledProcessError:
assert expected_stderr in result.stderr
def assert_silent_failure(cmd):
"""
Check that a command fails; verify that standard error is empty
"""
try:
result = subprocess.run(cmd, check=True, text=True, capture_output=True)
except CalledProcessError:
assert result.stderr == ''
raise ValueError("%r did not fail", cmd)
assert_silent_failure(['su', '-c' 'test -d ' + u_model_path])
...
but of course using Python when you fundamentally want to test the shell might not make much sense.
#!/bin/sh
! test -d "$1"
Basically never use ls in scripts and generally probably don't rely on a particular error message (it could be localized, or change between versions).
Also, in Python subprocess code, avoid shell=True whenever you can. See also Actual meaning of shell=True in subprocess

Running vulture from a python script

I'm trying to find a way to run vulture (which finds unused code in python projects) inside a python script.
vulture documentation can be found here:
https://pypi.org/project/vulture/
Does anyone know how to do it?
The only way I know to use vulture is by shell commands.
I tried to tun the shell commands from the script, using module subprocess, something like this:
process = subprocess.run(['vulture', '.'], check=True,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT,universal_newlines=True)
which I though would have the same effect as running the shell command "vulture ."
but it doesn't work.
Can anyone help?
Thanks

Vulture dev here.
The Vulture package exposes an API, called scavenge - which it uses internally for running the analysis after parsing command line arguments (here in vulture.main).
It takes in a list of Python files/directories. For each directory, Vulture analyzes all contained *.py files.
To analyze the current directory:
import vulture
v = vulture.Vulture()
v.scavenge(['.'])
If you just want to print the results to stdout, you can call:
v.report()
However, it's also possible to perform custom analysis/filters over Vulture's results. The method vulture.get_unused_code returns a list of vulture.Item objects - which hold the name, type and location of unused code.
For the sake of this answer, I'm just gonna print the name of all unused objects:
for item in v.get_unused_code():
print(item.name)
For more info, see - https://github.com/jendrikseipp/vulture

I see you want to capture the output shown at console:
Below code might help:
import tempfile
import subprocess
def run_command(args):
with tempfile.TemporaryFile() as t:
try:
out = subprocess.check_output(args,shell=True, stderr=t)
t.seek(0)
console_output = '--- Provided Command: --- ' + '\n' + args + '\n' + t.read() + out + '\n'
return_code = 0
except subprocess.CalledProcessError as e:
t.seek(0)
console_output = '--- Provided Command: --- ' + '\n' + args + '\n' + t.read() + e.output + '\n'
return_code = e.returncode
return return_code, console_output
Your expected output will be displayed in console_output
Link:
https://docs.python.org/3/library/subprocess.html

Check if a Python script is already running in Windows

What is the best/easiest why to check if a specific Python script already running in Windows?
I have a script that goes over all files in a folder and copies them to another folder (sort to Movie or TV Shows folder).
I want to make sure when the script starts that there isn't another process (of the same script) that is already running, so I wouldn't have issues with 2 scripts that are trying to move the same files.
I have tried to create a file in the start of the script and deleting it when the script finishes, but I got into problems when the script crashes and/or throws an error.
I know that I can use psutil, but then I will get the process name (python.exe) and I'm looking for a why to distinguish if the Python process is running my script or another program.

You can use psutil.Process().cmdline() to see the complete command line of a process.
Alternatively, you could lock the files you're working on. See the answer to this question how to do this on ms-windows. The thing with locks is that you have to be careful to remove them, especially when an error occurs.

For Windows o.s. you can use timestamp.txt file
timestamp = 'timestamp.txt'
...
elif windows:
try:
new_timestamp = False
if not os.path.exists(timestamp):
new_timestamp = True
try:
with open(timestamp, 'a') as f_timestamp:
f_timestamp.write(str(int_t0))
except IOError as e:
out1 = 'M. Cannot open file for writing. Error: %s - %s.' \
% (e.logfile, e.strerror) + ' -> Exit code 3'
logging.error(out1)
sys.exit(3)
if not new_timestamp and os.path.exists(timestamp):
out1 = 'N. Script ' + __file__ + ' is already running.'
print(out1)
logging.error(out1)
sys.exit(0)
except IOError as e:
out1 = 'J. Cannot open file. Error: %s - %s.' \
% (e.filepath, e.strerror) + ' -> Exit code 4'
logging.error(out1)
...
try:
f_timestamp.close()
os.remove(timestamp)
except OSError as e:
logging.error('B. Cannot delete ' + timestamp + \
' Error: %s - %s.' % (e.filename, e.strerror))

Use lockfile. It is cross-platform, uses native OS functions and much more robust than any home-brewn lock file creation schemas

I have solved it by using an empty dummy file.
At the start of the process I check if the file exists, if not I create a new file, run the process and delete it in the end (even if the process failed), if the file does exist, that means that the process is running now so I terminate the current (new) process.

Python subprocess not returning

I want to call a Python script from Jenkins and have it build my app, FTP it to the target, and run it.
I am trying to build and the subprocess command fails. I have tried this with both subprocess.call() and subprocess.popen(), with the same result.
When I evaluate shellCommand and run it from the command line, the build succeeds.
Note that I have 3 shell commands: 1) remove work directory, 2) create a fresh, empty, work directory, then 3) build. The first two commands return from subprocess, but the third hangs (although it completes when run from the command line).
What am I doing wrongly? Or, what alternatives do I have for executing that command?
# +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=
def ExcecuteShellCommandAndGetReturnCode(arguments, shellCommand):
try:
process = subprocess.call(shellCommand, shell=True, stdout=subprocess.PIPE)
#process.wait()
return process #.returncode
except KeyboardInterrupt, e: # Ctrl-C
raise e
except SystemExit, e: # sys.exit()
raise e
except Exception, e:
print 'Exception while executing shell command : ' + shellCommand
print str(e)
traceback.print_exc()
os._exit(1)
# +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=
def BuildApplciation(arguments):
# See http://gnuarmeclipse.github.io/advanced/headless-builds/
jenkinsWorkspaceDirectory = arguments.eclipseworkspace + '/jenkins'
shellCommand = 'rm -r ' + jenkinsWorkspaceDirectory
ExcecuteShellCommandAndGetReturnCode(arguments, shellCommand)
shellCommand = 'mkdir ' + jenkinsWorkspaceDirectory
if not ExcecuteShellCommandAndGetReturnCode(arguments, shellCommand) == 0:
print "Error making Jenkins work directory in Eclipse workspace : " + jenkinsWorkspaceDirectory
return False
application = 'org.eclipse.cdt.managedbuilder.core.headlessbuild'
shellCommand = 'eclipse -nosplash -application ' + application + ' -import ' + arguments.buildRoot + '/../Project/ -build myAppApp/TargetRelease -cleanBuild myAppApp/TargetRelease -data ' + jenkinsWorkspaceDirectory + ' -D DO_APPTEST'
if not ExcecuteShellCommandAndGetReturnCode(arguments, shellCommand) == 0:
print "Error in build"
return False

I Googled further and found this page, which, at 1.2 says
One way of gaining access to the output of the executed command would
be to use PIPE in the arguments stdout or stderr, but the child
process will block if it generates enough output to a pipe to fill up
the OS pipe buffer as the pipes are not being read from.
Sure enough, when I deleted the , stdout=subprocess.PIPE from the code above, it worked as expected.
As I only want the exit code from the subprocess, the above code is enough for me. Read the linked page if you want the output of the command.

Compare over directory and text file using Python

My goal is to compare two data one is from text file and one is from directory and after comparing it this is will notify or display in the console what are the data that is not found for example:
ls: /var/patchbundle/rpms/:squid-2.6.STABLE21-7.el5_10.x86_64.rpm NOT FOUND!
ls: /var/patchbundle/rpms/:tzdata-2014j-1.el5.x86_64.rpm
ls: /var/patchbundle/rpms/:tzdata-java-2014j-1.el5.x86_64.rpm
ls: /var/patchbundle/rpms/:wireshark-1.0.15-7.el5_11.x86_64.rpm
ls: /var/patchbundle/rpms/:wireshark-gnome-1.0.15-7.el5_11.x86_64.rpm
ls: /var/patchbundle/rpms/:yum-updatesd-0.9-6.el5_10.noarch.rpm NOT FOUND
It must be like that. So Here's my python code.
import package, sys, os, subprocess
path = '/var/tools/tools/newrpms.txt'
newrpms = open(path, "r")
fds = newrpms.readline()
def checkrc(rc):
if(rc != 0):
sys.exit(rc)
cmd = package.Errata()
for i in newrpms:
rc = cmd.execute("ls /var/patchbundle/rpms/ | grep %newrpms ")
if ( != 0):
cmd.logprint ("%s not found !" % i)
checkrc(rc)
sys.exit(0)
newrpms.close
Please see the shell script. This script its executing file but because I want to use another language that's why Im trying python
retval=0
for i in $(cat /var/tools/tools/newrpms.txt)
do
ls /var/patchbundle/rpms/ | grep $i
if [ $? != 0 ]
then
echo "$i NOT FOUND!"
retval=255
fi
done
exit $retval
Please see my Python code. What is wrong because it is not executing like the shell executing it.

You don't say what the content of "newrpms.txt" is; you say the script is not executing how you want - but you don't say what it is doing; I don't know what package or package.Errata are, so I'm playing guess-the-problem; but lots of things are wrong.
if ( != 0): is a syntax error. If {empty space} is not equal to zero?
cmd.execute("ls /var/patchbundle/rpms/ | grep %newrpms ") is probably not doing what you want. You can't put a variable in a string in Python like that, and if you could newrpms is the file handle not the current line. That should probably be ...grep %s" % (i,)) ?
The control flow is doing:
Look in this folder, try to find files
Call checkrc()
Only quit with an error if the last file was not found
newrpms.close isn't doing anything, it would need to be newrpms.close() to call the close method.
You're writing shell-script-in-Python. How about:
import os, sys
retval=0
for line in open('/var/tools/tools/newrpms.txt'):
rpm_path = '/var/patchbundle/rpms/' + line.strip()
if not os.path.exists(rpm_path):
print rpm_path, "NOT FOUND"
retval = 255
else:
print rpm_path
sys.exit(retval)
Edited code slightly, and an explanation:
The code is almost a direct copy of the shell script into Python. It loops over every line in the text file, and calls line.strip() to get rid of the newline character at the end. It builds rpm_path which will be something like "/var/patchbundle/rpms/:tzdata-2014j-1.el5.x86_64.rpm".
Then it uses sys.path.exists() which tests if a file exists and returns True if it does, False if it does not, and uses that test to set the error value and print the results like the shell script prints them. This replaces the "ls ... | grep " part of your code for checking if a file exists.