I have the following command.
/usr/bin/node_modules/phantomjs/bin/phantomjs RequestURL.js https://www.ubank.com.au/ > body.html which gets executed fine at the terminal, but does not get executed from python as follows.
def get_generated_html(self, url, has_headers):
"""
Method: Method to get the generated HTML content from Phantomas.
Args: Takes the url as an argument for which to get the HTML content.
hasHeaders defaulted to false for no headers.
Returns: Nothing.
"""
if not urlparse(url).scheme:
url = 'http://'+url
if has_headers == False:
command = PAGE_SOURCE_CMD % url
utils.execute_command(command).communicate()
else:
command = FEO_PAGE_SOURCE_CMD % url
print command
utils.execute_command(command).communicate()
The print statement prints out the exact command.
Here is the execute_command() method.
def execute_command(command):
"""Executes the command and returns the process."""
process = None
try:
process = subprocess.Popen(command, shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
except subprocess.CalledProcessError:
print (
'Process utility could not process %s'
' inside execute_command() method'
% url)
return process
I call the generated html as follows.
def start_parser(self, analysis_id, url, hasHeaders=False):
"""
Method: Method to start the parser.
Args: Analsyis ID and URL as an argument.
Returns: Nothing.
"""
feed = None
path = self.create_analysis_folder(analysis_id, hasHeaders)
self.get_generated_html(url, hasHeaders)
for root, dirs, files in os.walk(path):
for file in files:
if file.endswith('.html'):
feed = BeautifulSoup(open(path + '/' +file).read())
if hasHeaders:
os.chdir('..')
print "deleting"
shutil.rmtree(os.getcwd())
break
return feed
The feed returned here is not the page source as it returns from command line.
I'm not sure what get_generated_html is supposed to do, but it doesn't return anything and it only prints in one of the cases. I'm confused by the docstring, since it says the function shouldn't return anything, but it doesn't say anyting about output either. The "defaults to false" part is not correct in the current code.
Also, communicate returns a tuple, so you will only return one of the elements of the return value (probably).
Try something like:
output = utils.execute_command(command).communicate()
return output[0]
If we are doing code review, this is more elegant to me:
if not has_headers:
command = PAGE_SOURCE_CMD % url
else:
command = FEO_PAGE_SOURCE_CMD % url
output = utils.execute_command(command).communicate()
return output[0] # or print it?
Related
How can I evaluate the return of ping command used on subprocess.call(), by having a value of True or False like in the last "if" statement ?
Here is the code:
import subprocess
from openpyxl import load_workbook
from openpyxl import worksheet
_list_test = []
def ping_rng():
wb = load_workbook(filename="IP_gav.xlsx")
ws = wb["172.16.7.X"]
i = 0
for IP_ in range(2,258):
# Get correct cell value
cell_ = ws.cell(row=IP_, column=1).value
_list_test.append(cell_)
# Construct and execute "ping" command
cmd_param = "-n"
command = ['ping', cmd_param, '1', str(_list_test[i])]
response = subprocess.call(command)
print(response) # Just to check the value
# Check the "response"
if response == True:
print("Is up.")
elif response == False:
print("Is down.")
else: print("Nothing Good")
i = i + 1
ping_rng()
The code is working as it should, by iterating on a range of IP addresses (which it get from a XLSX file) and by pinging the single IP address like so:
ping 172.16.7.0 -n 1
As a further attempt, I've found different posts about verifying if subprocess.check_output() is True or False, but this didn't worked for me since it returns the string of ping command in "cmd style", and that is why I've opted for subprocess.call() which returns the state of the command.
Tests:
In the code above, I printed the value of "response" variable by using the 2 different methods:
With subprocess.call() the return is '0' in any cases (so return False in the last "if" statement)
or
With subprocess.check_output() the return is the ping command itself in "cmd style" (return "Nothing Good" in the last "if" statement)
ping will exit with 0 as an exit code if it ran successfully. Given that you seem to want to handle error codes in a generic way, you could keep things simple:
import subprocess
def ping_test(ip):
cmd = ["ping","-n","1",ip]
response = subprocess.call(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
if response == 0:
print(f"{ip} ping was successful!")
return
print(f"{ip} ping was not successful.")
ping_test("127.0.0.1")
ping_test("300.0.0.0")
prints:
#127.0.0.1 ping was successful!
#300.0.0.0 ping was not successful.
I am using the 7za.exe command line version of 7 zip for my compression/encryption and decompression/decryption needs.
For improved security, I am not executing the command in CMD with the password. Rather, I am passing in the password using subprocess and passing it in to the CMD's standard input.
Here is my execute function which is being used to run system commands with input.
def execute(self, cmd : typing.List[str], input: typing.Optional[bytes] = None, verbose=False, debug=False, normal_priority=False):
if verbose:
print(cmd)
creationflags = subprocess.CREATE_NO_WINDOW
if normal_priority:
creationflags |= subprocess.NORMAL_PRIORITY_CLASS
else:
creationflags |= subprocess.BELOW_NORMAL_PRIORITY_CLASS
if debug:
process = subprocess.Popen(cmd, shell=False, stdout=sys.stdout, stderr=sys.stderr, stdin=subprocess.PIPE,creationflags=creationflags)
else:
process = subprocess.Popen(cmd, shell=False, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
stdin=subprocess.PIPE, creationflags=creationflags)
if input:
process.stdin.write(input)
process.stdin.flush()
returncode = process.wait()
if returncode:
raise OSError(returncode)
Here is my decompression/Decryption function
def sevenZipDecompressDecryptFileFolder(self, file, destinationPath=None, password=None):
try:
if not os.path.exists(file):
self.CELogger.error(f"The file ({file}) does not exist for encryption")
return False
theFilepath = pathlib.Path(file)
theFilesuffix = theFilepath.suffix
theFilename = theFilepath.stem
if theFilesuffix != '.7z':
self.CELogger.error(f"The file ({file}) does not have the .7z extension")
return False
if destinationPath is None:
parentFolders = str(theFilepath.parent)
decryptedFilepath = os.path.join(parentFolders)
else:
if not os.path.exists(destinationPath):
try:
os.makedirs(destinationPath)
except FileExistsError:
pass
decryptedFilepath = destinationPath
sevenZipExecutablePath = self.getSevenZipExecutablePath()
if sevenZipExecutablePath is False:
self.CELogger.error("Unable to obtain the seven zip executable path")
return False
if password is None:
CMD = [sevenZipExecutablePath, 'x','-r', '-y', file,"-o"+decryptedFilepath]
executeResult = self.execute(CMD)
else:
CMD = [sevenZipExecutablePath, 'x', '-r', '-y', file, '-p', "-o"+decryptedFilepath]
stringCMD = f'"{sevenZipExecutablePath}" x -r -y "{file}" -p{password} -o"{decryptedFilepath}"'
#os.system(stringCMD)
input = (password + "\r\n").encode("ascii")
executeResult = self.execute(CMD, input)
return decryptedFilepath
except:
self.CELogger.error(f"A problem occurred while attempting to decrypt the file ({file})")
tb = traceback.format_exc()
self.CELogger.exception(tb)
return False
My compression/Encryption program uses the same approach and execute function and works without problem.
def sevenZipCompressFileFolder(self, thePath,password=None,destinationDirectory=None,compressionLevel=9,multithreadedCompression="off"):
try:
if not os.path.exists(thePath):
self.CELogger.error(f"Could not compress the file/folder at the path ({thePath}). Does not exist")
return False
pathlibPath = pathlib.Path(thePath)
thePathname = pathlibPath.stem
if destinationDirectory is None:
destinationDirectory = str(pathlibPath.parent)
else:
if not os.path.exists(destinationDirectory):
try:
os.makedirs(destinationDirectory)
except FileExistsError:
pass
destinationFilepath = os.path.join(destinationDirectory,thePathname+'.7z')
sevenZipExecutablePath = self.getSevenZipExecutablePath()
mxEntry = f"-mx={compressionLevel}"
mmtEntry = f"-mmt={multithreadedCompression}"
if password is None:
CMD = [sevenZipExecutablePath, "a", "-t7z", mxEntry, mmtEntry, "-mf=on", "-mhc=on", "-mhe=on", "-ms=on","-m0=LZMA2", "-r", "-y", "-aoa", "-ssw", destinationFilepath, thePath]
executeResult = self.execute(CMD)
else:
CMD = [sevenZipExecutablePath, "a", "-t7z",'-p', mxEntry, mmtEntry, "-mf=on", "-mhc=on", "-mhe=on", "-ms=on","-m0=LZMA2", "-r", "-y", "-aoa", "-ssw", destinationFilepath, thePath]
input = (password + "\r\n").encode("ascii")
executeResult = self.execute(CMD, input)
except:
self.CELogger.error("An error has occurred while attempting to compress file with seven zip")
tb = traceback.format_exc()
self.CELogger.exception(tb)
return False
The decompression/decryption function works without the password input. However, when supplying the password, it is unable to decrypt it (the password has been verified to be correct). I suspect this is caused by the execute function.
Furthermore, you may notice that I have the os.system call commented out. This is because I have tried using the stringCMD command with the password pre-entered and not passing into the input stream (less secure version). This is also does not work in Python and os.system. However, if I copy the stringCMD variable, and manually open a CMD window, paste, and run, it decrypts it just fine.
What am I doing wrong? I have tried many different things but can't get it to work.
Please note that the errors are isolated to python and these functions. Trivial errors such as unable to overwrite files, wrong password etc. are not a factor and such cases have already been tested.
You may also be interested to note that I am able to decompress and decrypt the file by using the standard 7 zip utility (right click on file and select extract archive, enter password etc.)
Thanks
*EDIT This is also a problem when I try to "update" the compressed archive. Similar to decompression, I need to supply the password. In this situation as well, supplying the password and command through the execute function does not work.
So, it appears there is no problem in creating an archive with the password. However, manipulating the file with the same password afterwards is the problem.
I am currently converting my unittest setup over to pytest, everything is working however on certain tests I run a command line process in order to load code into the hardware I have attached via USB. this process works fine with unittest however when using pytest or nose2 I get the response
------------------------------------------------ Captured stderr call -------------------------------------------------
Error processing Test
this happens just when my process begins to run? I get no error message am an unsure why one is not output? The command is fine as it runs on cmd and unittest is there something I am missing in order to make it work with pytest?
For reference my class I am running
class LoadCode():
def __init__(self, comport):
''' Constructor
'''
self.filename = None
self.code_comport = comport
self.code_loaded = False
self.logger = logging.getLogger(__name__)
def set_code_filename(self, new_file_name):
''' Sets the filename parameter for loading the code
'''
if (self.filename != new_file_name):
self.filename = new_file_name
self.logger.info("Setting code File to " + self.filename)
self.code_loaded = False
else:
self.logger.info("Code File Name Is Already Set !")
def write_code(self):
REBOOT_TIME = 50 #approximatly 50 seconds if enough for a reboot after loading boot and main and enough time for
SUCCESSFUL_RETURNCODE = 0 # 0 is a successful return code for subprocess
if(self.filename != None and self.code_comport != None):
#set up command line to run
command = <<COMMAND>>
self.logger.info("Running: " + command)
#run command line as subprocess (thread will wait for command line to finish)
load_code = subprocess.run(command)
#successful returncode = 0 anything else means an error has occured during subprocess
subprocess.CompletedProcess(args=[command], returncode = SUCCESSFUL_RETURNCODE)
if (load_code.returncode == SUCCESSFUL_RETURNCODE ):
self.code_loaded = True
self.logger.info(self.filename)
time.sleep(REBOOT_TIME) #Allow reboot
else:
raise AssertionError("ERROR: No code File Set/No Comport Set")
self.is_code_loaded()
def is_code_loaded(self):
'''check the bool of code to ensure it has successfully ran
'''
if self.code_loaded == False:
Print("Failed")
raise AssertionError("Code Was Not Loaded ..")
else:
print("WORKED")
subprocess.CompletedProcess(args=[command], returncode = SUCCESSFUL_RETURNCODE)
this line of code is not needed as it returns from subproccess.run(). thanks #Masklinn for pointing that out.
details: https://python.readthedocs.io/en/latest/library/subprocess.html?highlight=CompletedProcess
The path that was generated contained spaces in the middle and did not have one at the start which is why the command did not run and the error was returned! now my subprocess work perfectly fine with pytest and nose2 ! :)
I am trying to to run rsync for each folder in a folder.
__author__ = 'Alexander'
import os
import subprocess
root ='/data/shares'
arguments=["--verbose", "--recursive", "--dry-run", "--human-readable", "--remove-source-files"]
remote_host = 'TL-AS203'
for folder in os.listdir(root):
print 'Sync Team ' + folder.__str__()
path = os.path.join(root,folder, 'in')
if os.path.exists(path):
folder_arguments = list(arguments)
print (type(folder_arguments))
folder_arguments.append("--log-file=" + path +"/rsync.log")
folder_arguments.append(path)
folder_arguments.append("transfer#"+remote_host+":/data/shares/"+ folder+"/out")
print "running rsync with " + str(folder_arguments)
returncode = subprocess.call(["rsync",str(folder_arguments)])
if returncode == 0:
print "pull successfull"
else:
print "error during rsync pull"
else:
print "not a valid team folder, in not found"
If I run this I get the following output:
Sync Team IT-Systemberatung
<type 'list'>
running rsync with ['--verbose', '--recursive', '--dry-run', '--human-readable', '--remove-source-files', '--log-file=/data/shares/IT-Systemberatung/in/rsync.log', '/data/shares/IT-Systemberatung/in', 'transfer#TL-AS203:/data/shares/IT-Systemberatung/out']
rsync: change_dir "/data/shares/IT-Systemberatung/['--verbose', '--recursive', '--dry-run', '--human-readable', '--remove-source-files', '--log-file=/data/shares/IT-Systemberatung/in/rsync.log', '/data/shares/IT-Systemberatung/in', 'transfer#TL-AS203:/data/shares/IT-Systemberatung" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1040) [sender=3.0.4]
error during rsync pull
Sync Team IT-Applikationsbetrieb
not a valid team folder, in not found
transfer#INT-AS238:/data/shares/IT-Systemberatung
If i manually start rsync from bash with these arguments, everything works fine. I also tried it with shell=true but with the same result.
You need to do:
returncode = subprocess.call(["rsync"] + folder_arguments)
Calling str() on a list will return the string represention of the python list which is not what you want to pass in as an argument to rsync
You do a os.chdir(os.path.join(root,folder)), but never go back.
In order to properly resume operation on the next folder, you should either remember the last os.getpwd() and return to it, or just do os.chdir('..') at the end of one loop run.
I've been working on a Python problem for sometime now. I'm trying to use the Echoprint API to sort my music out. So i'm writing some code that does that for me.
This is how the API works :
Takes in a song name as a command line arg.
Gives the appropriate result.
But i'm writing a script that has to perform this "internally". As in, the script should take the files and perform the lookup and output the results to the terminal. (basically - NO COMMAND LINE ARGUMENTS SUPPLIED )
So is there anyway as to pass files into a function ?
I know this sounds silly but it's a problem i'm not able to solve.
If i use os.walk(), etc it returns a str object to my lookup function as a parameter. I want the audio file to be passed as a parameter.
Here's the code which takes in the song as a command line arg :
import sys
import os
import pyechonest.config as config
import pyechonest.song as song
config.CODEGEN_BINARY_OVERRIDE = os.path.abspath("/Users/******/python/minger/echoprint-codegen-master/echoprint-codegen")
config.ECHO_NEST_API_KEY='*****'
def lookup(file):
# Note that song.identify reads just the first 30 seconds of the file
fp = song.util.codegen(file)
if len(fp) and "code" in fp[0]:
# The version parameter to song/identify indicates the use of echoprint
result = song.identify(query_obj=fp, version="4.11")
print "Got result:", result
print result[0]
if len(result):
print "Artist: %s (%s)" % (result[0].artist_name, result[0].artist_id)
print "Song: %s (%s)" % (result[0].title, result[0].id)
else:
print "No match. This track may not be in the database yet."
else:
print "Couldn't decode", file
if __name__ == "__main__":
if len(sys.argv) < 2:
print >>sys.stderr, "Usage: %s <audio file>" % sys.argv[0]
sys.exit(1)
lookup(sys.argv[1])
From there, http://echonest.github.io/remix/apidocs/pyechonest.util-module.html#codegen
the method you use has signature
codegen(filename, start=0, duration=30)
so that it is the filename that has to be passed as an argument... not the file itself...
Ex use here http://nullege.com/codes/show/src#p#y#pyechonest-7.1.0#pyechonest#song.py/371/util.codegen
if filename:
if os.path.exists(filename):
query_obj = util.codegen(filename, start=codegen_start, duration=codegen_duration)
if query_obj is None:
raise Exception("The filename specified: %s could not be decoded." % filename)