Develop Reference

how to fix "SystemExit: 2 error when calling parse_args()" in jupyter notebook [duplicate]

I am trying to pass BioPython sequences to Ilya Stepanov's implementation of Ukkonen's suffix tree algorithm in iPython's notebook environment. I am stumbling on the argparse component.
I have never had to deal directly with argparse before. How can I use this without rewriting main()?
By the by, this writeup of Ukkonen's algorithm is fantastic.

An alternative to use argparse in Ipython notebooks is passing a string to:
args = parser.parse_args()
(line 303 from the git repo you referenced.)
Would be something like:
parser = argparse.ArgumentParser(
description='Searching longest common substring. '
'Uses Ukkonen\'s suffix tree algorithm and generalized suffix tree. '
'Written by Ilya Stepanov (c) 2013')
parser.add_argument(
'strings',
metavar='STRING',
nargs='*',
help='String for searching',
)
parser.add_argument(
'-f',
'--file',
help='Path for input file. First line should contain number of lines to search in'
)
and
args = parser.parse_args("AAA --file /path/to/sequences.txt".split())
Edit: It works

Using args = parser.parse_args(args=[]) would solve execution problem.
or you can declare it as class format.
class Args:
data = './data/penn'
model = 'LSTM'
emsize = 200
nhid = 200
args=Args()

I've had a similar problem before, but using optparse instead of argparse.
You don't need to change anything in the original script, just assign a new list to sys.argv like so:
if __name__ == "__main__":
from Bio import SeqIO
path = '/path/to/sequences.txt'
sequences = [str(record.seq) for record in SeqIO.parse(path, 'fasta')]
sys.argv = ['-f'] + sequences
main()

If all arguments have a default value, then adding this to the top of the notebook should be enough:
import sys
sys.argv = ['']
(otherwise, just add necessary arguments instead of the empty string)

I ended up using BioPython to extract the sequences and then editing Ilya Steanov's implementation to remove the argparse methods.
import imp
seqs = []
lcsm = imp.load_source('lcsm', '/path/to/ukkonen.py')
for record in SeqIO.parse('/path/to/sequences.txt', 'fasta'):
seqs.append(record)
lcsm.main(seqs)
For the algorithm, I had main() take one argument, his strings variable, but this sends the algorithm a list of special BioPython Sequence objects, which the re module doesn't like. So I had to extract the sequence string
suffix_tree.append_string(s)
to
suffix_tree.append_string(str(s.seq))
which seems kind of brittle, but that's all I've got for now.

I face a similar problem in invoking argsparse, the string '-f' was causing this problem. Just removing that from sys.srgv does the trick.
import sys
if __name__ == '__main__':
if '-f' in sys.argv:
sys.argv.remove('-f')
main()

Clean sys.argv
import sys; sys.argv=['']; del sys
https://github.com/spyder-ide/spyder/issues/3883#issuecomment-269131039

Here is my code which works well and I won't worry about the environment changed.
import sys
temp_argv = sys.argv
try:
sys.argv = ['']
print(sys.argv)
args = argparse.parser_args()
finally:
sys.argv = temp_argv
print(sys.argv)

Suppose you have this small code in python:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("-v", "--verbose", help="increase output verbosity",
action="store_true")
parser.add_argument("-v_1", "--verbose_1", help="increase output verbosity",
action="store_true")
args = parser.parse_args()
To write this code in Jupyter notebook write this:
import argparse
args = argparse.Namespace(verbose=False, verbose_1=False)
Note: In python, you can pass arguments on runtime but in the Jupyter notebook that will not be the case so be careful with the data types of your arguments.

If arguments passed by the iPython environment can be ignored (do not conflict with the specified arguments), then the following works like a charm:
# REPLACE args = parser.parse_args() with:
args, unknown = parser.parse_known_args()
From: https://stackoverflow.com/a/12818237/11750716

If you don't want to change any of the arguments and working mechanisms from the original argparse function you have written or copied.
To let the program work then there is a simple solution that works most of the time.
You could just install jupyter-argparser using the below command:
pip install jupyter_argparser
The codes work without any changes thanks to the maintainer of the package.

argparser override set option manual

Is it possible with argpaser in python to set the values without using command line, but using inline code. I'm working with an stitching example of opencv that uses it a lot, but i dont want that.
The image names in my situation are created by code, not by command line.
Its not only the images i like to set but several options as well so is there a common way to do this:
#in psuedo code
args = parser.parse_args()
args_setoption('images') = listToString(myimagelist)

You can use argparse without command line using parser.parse_args:
import argparse
def listToString(myimagelist):
return ' '.join(myimagelist)
myimagelist = ['/tmp/foo.png', '/tmp/bar.jpeg']
parser = argparse.ArgumentParser(description='Test')
parser.add_argument('--images', type=str, help='(string) list of images')
_a = parser.parse_args([f'--images={listToString(myimagelist)}'])
print(_a.images)
Out:
/tmp/foo.png /tmp/bar.jpeg

PySpark and argparse

How does one specify command line arguments using argparse for a PySpark script? I've been breaking my head over this one and I swear I can't find the solution anywhere else.
Here's my test script:
import argparse
from pyspark.sql import SparkSession
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--v1", "-a", type=int, default=2)
parser.add_argument("--v2", "-b", type=int, default=3)
args = vars(parser.parse_args())
spark = (SparkSession.builder
.appName("Test")
.master("local[*]")
.getOrCreate()
)
result = args['v1'] + args['v2']
return result
if __name__ == "__main__":
result = main()
print(result)
When I try running the file using spark-submit file.py --v1 5 --v2 4, I get an error as shown below:
[TerminalIPythonApp] CRITICAL | Bad config encountered during initialization:
[TerminalIPythonApp] CRITICAL | Unrecognized flag: '--v1'
However, when I don't specify the arguments (just spark-submit file.py), it does the sum correctly, using the default values 2 and 3 from the argument parser, and displays "5" as expected. So clearly it's reading the values from argparse correctly. What's going wrong with the command when I actually pass non-default values?
NOTE: Am using PySpark 2.4.4 and Python 3.6.
EDIT: Of course, I could just use sys.argv and be done with it, but argparse is so much better!

Based on the TerminalIPythonApp error message (similar to this one), Pyspark was trying to pass argparse arguments to ipython instead of python. To fix this, set the correct Spark environment as python3, not ipython.
Add/modify the lines in /path/to/pyspark/conf/spark-env.sh:
export SPARK_HOME=/home/user/spark-2.4.0-bin-hadoop2.7/
export PYSPARK_PYTHON=python3
export PYSPARK_DRIVER_PYTHON=python3
export SPARK_YARN_USER_ENV="PYSPARK_PYTHON=python3"
This ensures that Pyspark looks for the python3 executable, following which argparse arguments should be read without any issues.

Setting optional system arguments via command prompt

I am doing a project in which I want to specify one system argument on my cmd right after the script.py. My problem is that I want to specify another argument in which is optional, and the user may or may not want to give that argument. Therefore, I am struggling how to deal with the fact that the system argument might or might not be given by the user and how to read that. If everything sounds confusing, the following text might clarify:
The user types the following on the command prompt to run the program:
python script.py file.txt
I want to add an argument which may or may not be given, like:
python script.py file.txt file_added.txt
As I read these arguments on my main script, I though that this problem would solve:
If sys.argv[2] is not None:
file2 = f"\{sys.argv[2]}"
However, I still getting IndexError when doing that. So, is there a simple way to bypass such problem?

If sys.argv holds less than 2 items, you'll get an IndexError. So wrap the statement around with a try block
try:
filename = sys.argv[2]
except IndexError:
filename = None
if filename:
# ... do something

A way to accomplish this would be to check the length of sys.argv. If the length is 3 you'll know that a second argument was passed (3 because the first argument is script.py). So something along the lines:
if len(sys.argv) == 3:
file2 = f"\{sys.argv[2]}"

Here, sys.argv[2] is not None you are checking if 3rd element is None or not and that is the issue.
You are indexing outside the length of argv array and index error.
If you only have max 2 input then you could check the length of argv like if len(sys.argv) == 3 and that means you have got both the input and then you can access them via sys.argv[1] and sys.argv[2]

You can use argsparse which is a built in library in python, which makes it easy to handle command line arguments. Go to the link https://docs.python.org/3/library/argparse.html to know mor, but the basic implementation for your usecase will be like this.
import argparse
parser = argparse.ArgumentParser(description='Enter filenames')
parser.add_argument('-file', type=str,help='enter the file name',dest='filename1')
parser.add_argument('--optional','--op',type=str, dest='filename2',help='enter optional filename')
args = parser.parse_args()
file1=args.filename1
file2=args.filename2
Then in the cmd you can invoke it as
python script.py -filename="file1.txt"
or
python script.py -filename="file1.txt" --optional="file2.txt"
or
python script.py -filename="file1.txt" --op="file2.txt"

You are looking for argv[1], argv[2], and so on.
This should work:
for filename in sys.argv[1:]:
readfile(filename)

Python command line parameters

I am just starting with python so I am struggling with a quite simple example. Basically I want pass the name of an executable plus its input via the command line arguments, e.g.:
python myprogram refprogram.exe refinput.txt
That means when executing myprogram, it executes refprogram.exe and passes to it as argument refinput. I tried to do it the following way:
import sys, string, os
print sys.argv
res = os.system(sys.argv(1)) sys.argv(2)
print res
The error message that I get is:
res = os.system(sys.argv(1)) sys.argv(2)
^
SyntaxError: invalid syntax
Anyone an idea what I am doing wrong?
I am running Python 2.7

This line
res = os.system(sys.argv(1)) sys.argv(2)
Is wrong in a couple of ways.
First, sys.argv is a list, so you use square brackets to access its contents:
sys.argv[1]
sys.argv[2]
Second, you close out your parentheses on os.system too soon, and sys.argv(2) is left hanging off of the end of it. You want to move the closing parenthesis out to the very end of the line, after all of the arguments.
Third, you need to separate the arguments with commas, a simple space won't do.
Your final line should look like this:
res = os.system(sys.argv[1], sys.argv[2])

A far, far better way to do this is with the argparse library. The envoy wrapper library makes subprocess easier to work with as well.
A simple example:
import argparse
import envoy
def main(**kwargs):
for key, value in kwargs.iteritems():
print key, value
cmd = '{0} {1}'.format(kwargs['program'], ' '.join(kwargs['infiles']))
r = envoy.run(cmd)
print r.std_out
print r.std_err
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='Get a program and run it with input', version='%(prog)s 1.0')
parser.add_argument('program', type=str, help='Program name')
parser.add_argument('infiles', nargs='+', type=str, help='Input text files')
parser.add_argument('--out', type=str, default='temp.txt', help='name of output file')
args = parser.parse_args()
main(**vars(args))
This reads in the arguments, parses them, then sends them to the main method as a dictionary of keywords and values. That lets you test your main method independently from your argument code, by passing in a preconstructed dictionary.
The main method prints out the keywords and values. Then it creates a command string, and passes that to envoy to run. Finally, it prints the output from the command.
If you have pip installed, envoy can be installed with pip install envoy. The easiest way to get pip is with the pip-installer.

sys.argv is a list, and is indexed using square brackets, e.g. sys.argv[1]. You may want to check len(sys.argv) before indexing it as well.
Also, if you wanted to pass parameters to os.system(), you might want something like os.system(' '.join(sys.argv[1:])), but this won't work for arguments with spaces. You're better off using the subprocess module.

sys.argv is a list
import sys, string, os
print sys.argv
res = os.system(sys.argv[1]) sys.argv[2]
print res

If you are running Python 2.7 it is recommended to use the new subprocess module.
In this case you would write
import sys, subprocess
result = subprocess.check_output(sys.argv[1], sys.argv[2])