Develop Reference

Pyspark : is there any way to use Unix exported variables in python code

I am looking for a solution to get Unix exported variables in pyspark code when I am running it on cluster mode.
I got a solution using os.getenv but this is not working in cluster mode for me. In local mode it is working fine.
Is there any other way to replaces complete set of variables in one go. Passing n number of arguments and reading them is a bit overhead.

For a cluster mode you should use spark configuration named spark.yarn.appMasterEnv.XXX to pass variables to the driver. You can find more information in the documentation.
But, I woudn't say environment variables are the best way to deal with parameters. Did you head about optparse library? It is shipped with all python distributions (2.X and 3.X) and allows you to parse parameters in quite easy way, like:
from optparse import OptionParser
parser = OptionParser()
parser.add_option("-i", "--input-file", default="hdfs:///some/input/file.csv")
parser.add_option("-o", "--output-dir", default= "hdfs:///tmp/output")
parser.add_option("-l", "--log-level", default='INFO')
(options, args) = parser.parse_args()

why isn't getopt working on python 2.7.5 mac os x

For some reason, when I pass in an option using the long form, the getopt function isn't recognizing it properly. Any ideas? I've read the documentation here http://docs.python.org/2/library/getopt.html, and it seems it should not do what it's doing.
I'm running python 2.7.5 on mac os x for the record.
[user#macbookpro:~] python Script.py test --condition=foo --output-file abc.def
['test', '--condition=foo', '--output-file', 'abc.def']
[]
<type 'list'>
def main(argv):
try:
optlist, args = getopt.getopt(argv[1:], '', ['condition=', 'output-file=', 'testing'])
except getopt.GetoptError, msg:
logging.warning(msg)
return 1
print args
print optlist
print type(optlist)
I should be getting the following as stated in the documentation:
optlist
[('--condition', 'foo'), ('--testing', ''), ('--output-file', 'abc.def')]

The documentation doesn't say that you should be getting that. In fact, it explicitly says you shouldn't:
Note: Unlike GNU getopt(), after a non-option argument, all further arguments are considered also non-options. This is similar to the way non-GNU Unix systems work.
And if you look at the examples, the non-option arguments come after the options on the command line. If you do that, it gives you what you were hoping for:
$ python Script.py --condition=foo --output-file abc.def test
['test']
[('--condition', 'foo'), ('--output-file', 'abc.def')]
<type 'list'>
But if you do something different from the examples, you get different results from the examples. And the results you get match what the docs say you should.
But really, if you don't understand why putting test after the options is different from putting it before the options, you shouldn't be using getopt in the first place. As the docs say in a big box right at the top:
Note: The getopt module is a parser for command line options whose API is designed to be familiar to users of the C getopt() function. Users who are unfamiliar with the C getopt() function or who would like to write less code and get better help and error messages should consider using the argparse module instead.
If you really want to learn getopt, then read the POSIX definition. That's what Python is trying to emulate. It does add GNU-style -- long arguments, but that doesn't mean it includes all GNU extensions.

Using OptionParser vs sys.argv

For a script, I'm currently using OptionParser to add variables to an input. However, all of my current options are booleans, and it seems it would just be easier to parse using argv instead. For example:
$ script.py option1 option4 option6
And then do something like:
if 'option1' in argv:
do this
if 'option2' in argv:
do this
etc...
Would it be suggested to use argv over OptionParser when the optionals are all booleans?

"However, all of my current options are booleans, and it seems it
would just be easier to parse using argv instead."
There's nothing wrong with using argv, and if it's simpler to use argv, there's no reason not to.

OptionParser has been deprecated, and unless you're stuck on an older version of python, you should use the ArgParser module.
For one-off scripts, there's nothing wrong with parsing sys.argv yourself. There are some advantages to using an argument parsing module instead of writing your own.
Standardized. Do you allow options like "-test", because the standard is usually 2 underscores for multichar options (e.g. "--test"). With a module, you don't have to worry about defining standards because they're already defined.
Do you need error-catching and help messages? Because you get a lot of that for free with ArgParse.
Will someone else be maintaining your code? There's already lots of documentation and examples of ArgParse. Plus, it's somewhat self documenting, because you have to specify the type and number of arguments, which isn't always apparent from looking at a sys.argv parser.
Basically, if you ever expect your command line options to change over time, or expect that your code will have to be modified by someone else, the overhead of ArgParse isn't that bad and would probably save you time in the future.

Creating a CLI (Shell?) in Python

I am very newbie in Python but I have to implement for school a command line interpreter in Python language, but I am kinda lost in how to do that.
I have already read some tutorials and created a simple file called functions.py where i include some simple functions like this:
def delete(loc):
if os.path.exists(loc) == True:
os.remove(loc)
print "Removed"
else:
print "File not exists"
Now.. here is the thing.. in order to use this I must import it inside the python command interpreter, like...
import functions
functions.delete("file to delete")
How can I make a Shell/CLI so instead of have to write all of this I can just write like:
delete file_name
Thanks!

Or if you want a cmd shell, you could use the cmd lib. It offers python interfaces to making command lines.
http://docs.python.org/library/cmd.html

I think you should now to use simple argparse module to get command line arguments
import argparse
from functions import delete
parser = argparse.ArgumentParser()
parser.add_argument('-f', '--file')
args = parser.parse_args()
delete(args.file)
Hope this should work for you
Sultan

You might want to check my personal REPL for some inspiration. I wrote it during a tutorial series. Actual source may be found here. It probably does a few things you won't need... Still it could be a good read. :)

How to Pass variables to python script?

I know it can be achieved by command line but I need to pass at least 10 variables and command line will mean too much of programming since these variables may or may not be passed.
Actually I have build A application half in vB( for GUI ) and Half in python( for script ). I need to pass variables to python, similar, to its keywords arguments, i.e, x = val1, y = val2. Is there any way to achieve this?

If you are using Python <2.7 I would suggest optparse.
optparse is deprecated though, and in 2.7 you should use argparse
It makes passing named parameters a breeze.

you can do something fun like call it as
thepyscript.py "x = 12,y = 'hello world', z = 'jam'"
and inside your script,
parse do:
stuff = arg[1].split(',')
for item in stuff:
exec(item) #or eval(item) depending on how complex you get
#Exec can be a lot of fun :) In fact with this approach you could potentially
#send functions to your script.
#If this is more than you need, then i'd stick w/ arg/optparse

Since you're working on windows with VB, it's worth mentioning that IronPython might be one option. Since both VB and IronPython can interact through .NET, you could wrap up your script in an assembly and expose a function which you call with the required arguments.

Have you taken a look at the getopt module? It's designed to make working with command line options easier. See also the examples at Dive Into Python.
If you are working with Python 2.7 (and not lower), than you can also have a look at the argparse module which should make it even easier.

If your script is not called too often, you can use a configuration file.
The .ini style is easily readable by ConfigParser:
[Section_1]
foo1=1
foo2=2
foo3=5
...
[Section_2]
bar1=1
bar2=2
bar3=3
...
If you have a serious amount of variables, it might be the right way to go.

What do you think about creating a python script setting these variables from the gui side? When starting the python app you just start this script and you have your vars.
Execfile