How can I run a bunch of imports and path appends from the interpreter with one command/import? If I import another module that runs the commands for me the imports are not available in main namespace. Similar to running a bash script that modifies/adds commands and variables to the current session.
ex.
import os, ...
sys.path.append(...)
If I understand you correctly, you're just looking for the from … import … statement. For example:
lotsostuff.py:
import json
def foo(): pass
Now:
$ python3.3
>>> from lotsostuff import *
>>> json
<module 'json' from '/usr/local/lib/python3.3/json/__init__.py'>
>>> foo
<function lotsostuff.foo>
However, you might want to consider a different alternative. If you're just trying to control the startup of your interpreter session, you can do this:
$ PYTHONSTARTUP=lotsostuff.py
$ python3.3
>>> json
<module 'json' from '/usr/local/lib/python3.3/json/__init__.py'>
>>> foo
<function __main__.foo>
Notice the difference in the last line. You're now running lotsostuff in the __main__ namespace, rather than running in a separate namespace and grabbing all of its members.
Similarly:
$ python3.3 -i lotsostuff.py
>>> json
<module 'json' from '/usr/local/lib/python3.3/json/__init__.py'>
You'd normally use PYTHONSTARTUP if you want to do this every time in your session, -i if you want to do it just this once.
If you want to do the same thing in the middle of a session instead of at startup… well, you can't do it directly, but you can come pretty close with exec (Python 3.x) (or execfile in Python 2.x).
If you really want to do exactly what you described—importing a module, as a normal import, except merged into your namespace instead of in its own—you'll need to customize the import process. This isn't that hard with importlib; if you're not in Python 3.2 or later, you'll have a lot more work to do it with imp.
That's pretty much the difference between . ./foo instead of just ./foo in a bash script that I think you were looking for.
If you're using ipython, there are even cooler options. (And if you're not using ipython, you might want to check it out.)
Related
I am currently using the subprocess module in python for scripting purposes, and have been unable to execute the command 'module list' despite this working when I run it in shell and despite any other kind of command working fine when using the subprocess module to execute commands.
Two variations I have tried:
p = subprocess.Popen('module list', shell=True)
print(p.communicate()[0])
and
p = Popen(["module", "list"], stdout=subprocess.PIPE)
print(p.communicate()[0])
For commands such as 'echo hello world' and even longer commands with multiple arguments, either of these formats works just fine. Is the terminal I run commands from different from the shell used to execute commands using subprocess? The error I get is as follows:
/bin/bash: line 1: module: command not found
Based on what you've said in the comments, I believe you're going about using environment modules in Python the wrong way: There is actually a method in Modules itself to import module functionality into Python, as explained here:
>>> execfile('/usr/local/Modules/default/init/python.py')
>>> module('list')
No Modulefiles Currently Loaded.
>>> module('load','foo')
>>> module('list')
Currently Loaded Modulefiles:
1) foo/1.0
Of course, it's not very safe to use execfile(), so I slightly prefer the import method described here (slightly altered for Python 3 support):
import os
if 'PYTHONPATH' in os.environ:
os.environ['PYTHONPATH'] +=':'+os.environ['MODULESHOME']+"/init"
else:
os.environ['PYTHONPATH'] = os.environ['MODULESHOME']+"/init"
from python import module
The documentation of the Environment Modules software provides a recommendation on how to initialize the module command in Python (that should work on either Python 2 or 3):
import os
exec(open('/usr/share/Modules/init/python.py').read())
Once initialized, the module function is available and could be used in the following way:
module('sub-command', 'arg1', 'arg2', ...)
For example:
module('load', 'foo', 'bar')
module('list')
module('avail')
According to the python doc the -m flag should do the following:
Search sys.path for the named module and execute its contents as the
__main__ module.
When I run my script simply with the python command, everything works fine. Since I now want to import something from a higher level, I have to run the script with python -m. However the __name__ == "__main__" statement seems to return False and the following Error is produced:
/home/<name>/anaconda3/bin/python: Error while finding module specification for 'data.generate_dummies.py' (AttributeError: module 'data.generate_dummies' has no attribute '__path__')
I dont't understand what the __path__ attribute has to do with that.
The error you get occurs when python tries to look for a package/module that does not exist. As user2357112 mentions, data.generate_dummies.py is treated as a fully specified module path (that does not exist), and an attempt is made to import a submodule py (that is also non-existent).
Invoke your file without .py, if you're using the -m flag, like this:
python -m data.generate_dummies
I would like to know if it is possible to execute a python2 script from a python3 script.
I have a file written using py3 that must execute legacy code written in py2 to obtain dictionaries for processing within the initial file.
The line in py3 to call the mentioned py2 script is
exec(open('python2script.py').read())
The script runs without error until it begins processing python2script.py, at which point it crashes at the first difference with version3.
As the comments pointed out, exec() uses the current python implementation, so you can't execute python 2 code from python 3 using it.
Unless you port it, your best bet is simply to call it as a subprocess, using either os.system..:
./py3.py
#!/usr/bin/env python3
import os
print('running py2')
os.system('./py2.py')
print('done')
./py2.py
#!/usr/bin/env python2.7
print "hello from python2!"
Then (after making them both executable) run:
$ ./py3.py
Or alternatively you can use the more flexible subprocess, which allows you to pass data back and forward more easily using a serialising module such as json so that you can get your results from the python2 script in your python3 code:
./py3.py
#!/usr/bin/env python3
import json
from subprocess import PIPE, Popen
print('running py2')
py2_proc = Popen(['./py2.py'], stdout=PIPE)
# do not care about stderr
stdout, _ = py2_proc.communicate()
result = json.loads(stdout.decode())
print('value1 was %s, value2 was %s' % (result['value1'], result['value2']))
./py2.py
#!/usr/bin/env python2.7
import json
my_result = {
'value1': 1,
'value2': 3
}
print json.dumps(my_result)
Like that it may be easy to pack up the data you need and transport it over.
Note: I have used a very simple environment setup here using my system's python2.7 and python3. In the real world the most painful thing about getting this sort of thing to work properly is configuring the environment correctly. Perhaps, e.g., you are using virtual environments. Perhaps you are running as a user which doesn't have the right python2 version in their path. Perhaps you can't make the files executable and so have to specify the path to python in your subprocess / os.system call. There are many options and it is very complicated, but out of the scope of the question. You just have to read the doc pages very carefully and try a few things out!
I have a python script test.py I'm working on. I'd like to make edits and re-run it. I'm using Terminal on OSX to run the script. I can't get the script to run a second time without quitting out of terminal and starting it again.
# test.py
print "Howdy"
Terminal window:
$ python
>>> import test
Howdy
>>> import test
>>>
Question 1: How do I get the script to run again?
Question 2: Or is python designed to work like this:
# test.py
def printStuff():
print "Howdy"
Terminal:
$ python
>>> import test
>>> test.printStuff()
Howdy
>>> test.printStuff()
Howdy
>>>
1: you can use reload(moduleName) to do what you're trying to do (but see below).
2: There's a few different patterns, but I typically have a main() function inside all of my modules that have a clear "start point", or else I'll just have a bunch of library functions. So more or less what you're thinking in your example. You're not really supposed to "do stuff" on import, unless it's setting up the module. Think of a module as a library, not a script.
If you want to execute it as a script (in which case you shouldn't be using import), then there's a couple options. You can use a shebang at the top of your python script (Should I put #! (shebang) in Python scripts, and what form should it take?) and execute it directly from the command line, or you can use the __main__ module in your script as an entry point (https://docs.python.org/2/library/main.html), and then call the script with python from the command line, e.g. python myscript.
Use Python's reload() method:
https://docs.python.org/2/library/functions.html#reload
You want to use the reload method
>>> import test
Howdy
>>> reload(test)
Howdy
I would like to check if all modules imported by a script are installed before I actually run the script, because the script is quite complex and is usually running for many hours. Also, it may import different modules depending on the options passed to it, so just running it once may not check everything. So, I wouldn't like to run this script on a new system for few hours only to see it failing before completion because of a missing module.
Apparently, pyflakes and pychecker are not helpful here, correct me if I'm wrong. I can do something like this:
$ python -c "$(cat *.py|grep import|sed 's/^\s\+//g'|tr '\n' ';')"
but it's not very robust, it will break if the word 'import' appears in a string, for example.
So, how can I do this task properly?
You could use ModuleFinder from the standard lib modulefinder
Using the example from the docs
from modulefinder import ModuleFinder
finder = ModuleFinder()
finder.run_script('bacon.py')
print 'Loaded modules:'
for name, mod in finder.modules.iteritems():
print '%s: ' % name,
print ','.join(mod.globalnames.keys()[:3])
print '-'*50
print 'Modules not imported:'
print '\n'.join(finder.badmodules.iterkeys())
You could write a test.py that just contains all the possible imports for example:
import these
import are
import some
import modules
Run it and if there are any problems python will let you know