Develop Reference

Generate a text file according to the successful steps. - Possible improvement of the code or other way of doing it?

I would like to keep track of the steps taken by the program in a text report file. Each step in the code returns a dataframe and there is a dependency between tasks (task n cannot be executed if task n-1 had found nothing).
My programme looks like this:
(kind of pseudo code)
import pandas as pd
step_1 = find_stuff()
if not step_1.empty:
step_2 = find_new_stuff()
if not step_2.empty:
step_3 = find_new_stuff_again()
if not step_3.empty:
report (step_1, step_2, step_3)
else:
report (step_1, step_2, step_3=pd.DataFrame())
else:
report (step_1, step_2=pd.DataFrame(), step_3=pd.DataFrame())
else:
report (step_1=pd.DataFrame(), step_2=pd.DataFrame(), step_3=pd.DataFrame())
def report (step_1, step_2, step_3) :
report_file = open("report.txt", "a")
if not step_1.empty:
report_file.write(f'Here what was found for step 1 \n : { step_1} \n')
if not step_2.empty:
report_file.write(f'Here what was found for step 2 \n : { step_2} \n')
if not step_3.empty:
report_file.write(f'Here what was found for step 3 \n : { step_3} \n')
else:
report_file.write('Nothing was found \n')
This way of doing things is very basic but does what I ask it to do. Though, I was wondering if there was a way to avoid/reduce all these "if" or an alternative way to generate this kind of report?

My answer is similar to #Nik's so that you need to iterate over some functions but I added it to a class so you can have the state together with the functions in the same scope. I also changed you file opening to use with as it is considered the safe way to handle files:
class StepFailedException(Exception): pass
class StuffFinder:
def __init__(self):
self.findings = []
def find_stuff_1(self):
stuff = None
# find stuff
if stuff.empty:
raise StepFailedException
self.findings.append(stuff)
def find_stuff_2(self):
stuff = None
# find stuff
if stuff.empty:
raise StepFailedException
self.findings.append(stuff)
def find_stuff_3(self):
stuff = None
# find stuff
if stuff.empty:
raise StepFailedException
self.findings.append(stuff)
def report(self):
if self.findings:
with open("report.txt", "a") as report_file:
for i,stuff in enumerate(self.findings):
report_file.write(f'Here what was found for step {i+1} \n : { stuff } \n')
def clear(self):
self.findings.clear()
def find_stuff(self):
self.clear()
try:
self.find_stuff_1()
self.find_stuff_2()
self.find_stuff_3()
except StepFailedException as e:
# handle exception if necessary
pass
self.report()
sf = StuffFinder() # you could add some initial values as argument to the constructor
sf.find_stuff()
# OR go step by step
sf.clear()
sf.find_stuff_1()
# sf.find_stuff_2() let's skip this one
sf.find_stuff_3()
sf.report()

In principle, I think your code is readable and gets the job done. However, if you have a lot of steps, you might want to iterate over them.
Here is an example:
import pandas as pd
def report(steps):
report_file = open("report.txt", "a")
for i, s in enumerate(steps):
report_file.write(f"Here what was found for step {i+1} \n : { s} \n")
steps = []
find_functions = [find_stuff, find_new_stuff, find_new_stuff_again]
for f in find_functions:
found_stuff = f()
if found_stuff.empty:
break
steps.append(found_stuff)
report(steps)
Also, mind that you are currently opening your report in "a" mode, so it will append results if you rerun the code.

How do I reuse the code from this function cleanly?

The following function is used within a module to query network devices and is called from multiple scripts I use. The arguments it takes are a nested dictionary (the device IP and creds etc) and a string (the command to run on the device):
def query_devices(hosts, cmd):
results = {
'success' : [],
'failed' : [],
}
for host in hosts:
device = hosts[host]
try:
swp = ConnectHandler(device_type=device['dev_type'],
ip=device['ip'],
username=device['user'],
password=device['pwd'],
secret=device['secret'])
swp.enable()
results[host] = swp.send_command(cmd)
results['success'].append(host)
swp.disconnect()
except (NetMikoTimeoutException, NetMikoAuthenticationException) as e:
results['failed'].append(host)
results[host] = e
return results
I want to reuse all of the code to update a device and the only changes would be:
The function would take the same dictionary but the cmd argument would now be a list of commands.
The following line:
results[host] = swp.send_command(cmd)
would be replaced by:
results[host] = swp.send_config_set(cmd)
I could obviously just replicate the function making those two changes and as it is in a module I reuse, I am only having to do it once but I am still basically repeating a lot of the same code.
Is there a better way to do this as I seem to come across the same issue quite often in my code.

You could just add a check on the changed line:
...
if isinstance(cmd, str):
results[host] = swp.send_command(cmd)
else:
results[host] = swp.send_config_set(cmd)
...
The rest of the function can stay the same and now you can simply call it with either a string or a list of strings...

You could use the unpacking operator to always make cmds an iterable (a tuple, actually) even if it is a single value. That way you could always call send_config_set. Here is a super simplified example to illustrate the concept.
def query_devices(hosts, *cmds):
for one_cmd in cmds:
print(one_cmd)
print('query_devices("hosts", "cmd_1")')
query_devices("hosts", "cmd_1")
print('\nquery_devices("hosts", "cmd_1", "cmd_2", "cmd_3")')
query_devices("hosts", "cmd_1", "cmd_2", "cmd_3")
print('\nquery_devices("hosts", *["cmd_1", "cmd_2", "cmd_3"])')
query_devices("hosts", *["cmd_1", "cmd_2", "cmd_3"])
Output:
query_devices("hosts", "cmd_1")
cmd_1
query_devices("hosts", "cmd_1", "cmd_2", "cmd_3")
cmd_1
cmd_2
cmd_3
query_devices("hosts", *["cmd_1", "cmd_2", "cmd_3"])
cmd_1
cmd_2
cmd_3

Checking len of azure blob storage folder causes function to not run

Weird problem I've run into. I'm currently using the following code:
generic.py
def function_in_different_pyfile(input_folder):
# do stuff here
folder_1 = f"/folder_1"
folder_1_virtualdir = CONTAINER_CLIENT.list_blobs(name_starts_with=folder_1)
folder_2 = f"/folder_2"
folder_2_virtualdir = CONTAINER_CLIENT.list_blobs(name_starts_with=folder_2)
if len([file for file in folder_1_virtualdir]) !=(len([file for file in folder_2_virtualdir]):
generic.function_in_different_pyfile(folder_1_virtualdir)
else:
print('Already done')
So what I'm trying to do is:
Check the number of files in folder_1_virtualdir and folder_2_virtualdir
If they aren't equal, run the function.
If they are, then print statement/pass.
The problem:
The generic.function() runs although doesn't do anything when you pass in the list comprehension.
The generic.function() works totally fine if you don't have a list comprehension in the code e.g:
folder_1 = f"/folder_1"
folder_1_virtualdir = CONTAINER_CLIENT.list_blobs(name_starts_with=folder_1)
folder_2 = f"/folder_2"
folder_2_virtualdir = CONTAINER_CLIENT.list_blobs(name_starts_with=folder_2)
generic.function_in_different_pyfile(folder_1_virtualdir)
Will work completely fine.
There are no error messages. It passes through the function as if the function doesn't do anything.
What I've tried:
I've tested this by modifying the function:
generic.py
def function_in_different_pyfile(input_folder):
print('Start of the function')
# do stuff here
print('End of the function')
You will see these print statements although the function doesn't process any of the files in the input_folder argument if you include the list comprehension.
This is extended to when the list comprehension is ANYWHERE in the code:
folder_1 = f"/folder_1"
folder_1_virtualdir = CONTAINER_CLIENT.list_blobs(name_starts_with=folder_1)
folder_1_contents = [file for file in folder_1_virtualdir]
folder_2 = f"/folder_2"
folder_2_virtualdir = CONTAINER_CLIENT.list_blobs(name_starts_with=folder_2)
generic.function_in_different_pyfile(folder_1_virtualdir)
# Function doesn't run.
I'm fairly new to Python although can't seem to understand why the list comprehension here completely prevents the function from running correctly.

You could try the code if the number of files in the folder is less than 5000:
folder_1 = f"/folder_1"
folder_1_virtualdir = CONTAINER_CLIENT.list_blobs(name_starts_with=folder_1)
folder_2 = f"/folder_2"
folder_2_virtualdir = CONTAINER_CLIENT.list_blobs(name_starts_with=folder_2)
folder_1_count = len(folder_1_virtualdir)
folder_2_count = len(folder_2_virtualdir)
if folder_1_count != folder_2_count :
generic.function_in_different_pyfile(folder_1_virtualdir)
else:
print('Already done')
If greater than 5000, you need to get the number iterating through your blob.
count = 0
for count, item in enumerate(blobs):
print("number", count + 1, "in the list is", item)

conflicting options string(s) on second loop

#main loop
while 1==1:
#If they click Yes on the dialog box begin recording, otherwise ask again
easygui.msgbox('This is what the last person suggested! Press ok to record.: ' + output_string, 'Title', ok_button= "OK")
N+=1
counterFile = open('counterFile','w')
counterFile.write(str(N));
counterFile.close()
camera.start_recording('video' + str(N) + '.h264')
audioRecord()
camera.stop_recording()
output_string_old = output_string;
output_string = TextEnter()
filename = ConvertMerge()
argparser.add_argument("--file")
argparser.add_argument("--title")
argparser.add_argument("--description")
argparser.add_argument("--category")
argparser.add_argument("--keywords")
argparser.add_argument("--privacyStatus")
args = argparser.parse_args(["--file", filename, "--title", str(N),"--description", output_string_old, "--category", "22", "--keywords", " ", "--privacyStatus", "public"])
initialize_upload(get_authenticated_service(args), args)
I've made this code which records footage and then uploads to youtube using the youtube api but it currently returns this error on the second loop around.
ArgumentError: argument --file: conflicting options string(s): --file
filename ='mergedVideo'+ str(N) + '.mkv' and increases each time the programme is run.
Why is this error happening on the second loop around?

You're calling parser.add_argument("--file") multiple times, when you must only call it once per argument. Just move all of the add_argument calls to right before you enter your loop.
Running this code might help understand what is going wrong:
import argparse
parser = argparse.ArgumentParser(description='test')
for i in range(2):
print i
parser.add_argument("--file")
parser.add_argument("stuff")
You'll notice that there's an error the second time through the loop, because you already added an argument called "--file".

Is there an R equivalent of the pythonic "if __name__ == "__main__": main()"?

The objective is to have two simple ways to source some code, say func.R, containing a function. Calling R CMD BATCH func.R initializes the function and evaluates is. Within a session, issuing source("func.R") simply initializes the function.
Any idea?

I think that the interactive() function might work.
This function returns TRUE when R is being used interactively and FALSE otherwise. So just use if (interactive())
i.e. the equivalent is
if (!interactive()) {
main()
}

Another option is:
#!/usr/bin/Rscript
# runs only when script is run by itself
if (sys.nframe() == 0){
# ... do main stuff
}

You could pass arguments into R, and if an argument is present run main(). More on arguments here: http://yangfeng.wordpress.com/2009/09/03/including-arguments-in-r-cmd-batch-mode/

It's a lot of work, but I finally got it (and posted at Rosetta Code).
This example exports a function called meaningOfLife. When the script is run by itself, it runs main. When imported by another R file, it does not run main.
#!/usr/bin/Rscript
meaningOfLife <- function() {
42
}
main <- function(program, args) {
cat("Main: The meaning of life is", meaningOfLife(), "\n")
}
getProgram <- function(args) {
sub("--file=", "", args[grep("--file=", args)])
}
args <- commandArgs(trailingOnly = FALSE)
program <- getProgram(args)
if (length(program) > 0 && length(grep("scriptedmain", program)) > 0) {
main(program, args)
q("no")
}

I asked a similar question, in an answer, Matthew Plourde suggested using getOption('run.main', default=TRUE) in the main script and then setting options(run.main=FALSE) before calling source(). This worked in my case.
Otherwise a simpler pattern when you have an R script creating a bunch of functions and you want to write a few lines at the end of the script to experiment with the use of a function: place these extra lines in an if(FALSE){} block.

This works fairly well for my use. If you have two files and want to source one with the other while only running a certain part of the file.
parent file: parent.R
print("Running Parent File")
`__name__` <- "__main__"
print(paste0("This file is : ", `__name__`))
`__src__` <- "__not.main__"
source("child.R")
rm(`__src__`)
child file: child.R
print("Running Child File")
`__name__` <- "__main__"
if (exists("__src__")){`__name__` <- `__src__`}
if (`__name__` == "__main__"){
print(paste0("This file is : ", `__name__`))
} else {
print(paste0("This file is : ", `__name__`))
}
Output when running Rscript parent.R
[1] "Running Parent File"
[1] "This file is : __main__"
[1] "Running Child File"
[1] "This file is : __not.main__"
Output when running Rscript child.R
[1] "Running Child File"
[1] "This file is : __main__"
A more robust method would be to write a custom source function where a list of arguments can be included.
source2 <- function(file, args = list()){
tryCatch(
expr = {
assign("__src__", "__not.main__", envir = globalenv())
assign("__src.args__", args, envir = globalenv())
source(file)
},
error = function(e){
message("File could not be sourced")
},
finally = {
rm(list = c("__src__", "__src.args__"), envir = globalenv());
assign("__name__", "__main__", envir = globalenv())
})
}
source2("child.R", args = list("list", "of", "arguments"))