Specs: Python 2.7
I'm working on a project that has several modules, I want to activate some features from the __future__ module in all of them. I would like to import all the features I need on one module, and then import that single module to every other, and have those features be active in all of them, or something to that effect.
I tried:
[A.py]
from __future__ import division
[B.py]
import A
print(1/2)
Running B.py the division was still integer. I tried:
[A.py]
print(1/2)
[B.py]
from __future__ import division
import A
Running B.py gave the same result. With both previous examples I also tried switching 'import A' by 'from A import *' with the same results.
I searched Google for a while, and found the best description about how the __future__ module works, obviously enough, on the Python documentation. There I could only find the assurance the features would be active in the module they were imported to, without any mention of how to do it globally.
So I'd like to know if there is a way of doing this, either the way I described, or creating some sort of runtime configuration file, or through some other means.
There's no way to do this in-language; you really can't make __future__ imports global in this sense. (Well, you probably can replace the normal import statements with something complicated around imp or something. See the Future statement documentation and scroll down to "Code compiled by…" But anything like this is almost certainly a bad idea.)
The reason is that from __future__ import division isn't really a normal import. Or, rather, it's more than a normal import. You actually do get a name called division that you can inspect, but just having that value has no effect—so passing it to other modules doesn't affect those modules. On top of the normal import, Python has special magic that detects __future__ imports at the top of a module, or in the interactive interpreter, and changes the way your code is compiled. See future for the "real import" part, and Future statements for the "magic" part, if you want all the details.
And there's no configuration file that lets you do this. But there is a command-line parameter:
python -Qnew main.py
This has the same effect as doing a from __future__ import division everywhere.
You can add this to the #! lines, or alias pyfuturediv='python -Qnew' (or even alias python='python -Qnew') in your shell, or whatever, which maybe as good as a configuration file for your purposes.
But really, if you want to make sure module B gets new-style division, you probably should have the __future__ declaration in B in the first place.
Or, of course, you could just write for Python 3.0+ instead of 2.3-2.7. (Note that some of the core devs were against having command-line arguments, because "the right way to get feature X globally is to use a version of Python >= feature X's MandatoryRelease".) Or use // when you mean //.
Another possibility is to use six, a module designed to let you write code that's almost Python 3.3 and have it work properly in 2.4-2.7 (and 3.0-3.2). For example, you don't get a print function, but you do get a print_ function that works exactly the same. You don't get Unicode literals, but you get u() fake literals—which, together with a UTF-8 encoding declaration in the source, is almost good enough. And it provides a whole lot of stuff that you can't get from __future__ as well—StringIO and BytesIO, exec as a function, the next function, etc.
If the problem is that you have 1000 source files, and it's a pain to edit them all, you could use sed, or use 3to2 with just the option that fixes division, or…
Another approach would be using isort. isort has a -a command line flag to add imports to files that you specify. Simply running isort without arguments will run it recursively on all python files in the current working directory and all subdirectories.
If, like me, you have a virtual environment inside that folder, and are using git (or have an equivalent way of listing only your files) and don't want to run it on all files inside that virtual environment, you can use something like:
git ls-tree -r HEAD --name-only | grep "\.py$" | xargs isort -a -y "from __future__ import division"
Related
This may be a dumb question, but charging boldly ahead anyway.
I have a library of about a dozen Python modules I maintain for general use. Recently, after advice found here on SO, I changed all of the modules so they are imported in the import x as y style instead of from x import *. This solved several problems and made the code easier to manage.
However, there was an unintended side effect of this. Many of the modules use Python builtin modules like sys or os to do whatever, and the way the code was previously set up, if I typed import sys in module x, and used from x import * in module y, I didn't have to import sys in module y. As a result, I took this for granted quite a lot (terrible practice, I know). When I switched to import x, this caused a lot of broken functions, as you can imagine.
So here's the main issue: because Python is an interpreted language, errors about missing modules in a function won't show up until a function is actually run. And because this is just a general-use library, some of these errors could persist for months undetected, or longer.
I'm completely prepared to write a unit test for each module (if __name__ == "__main__" and all that), but I wanted to ask first: is there an automated way of checking every function in a module for import/syntax errors, or any other error that is not dependent on input? Things that a compiler would catch in C or another language. A brief Google and SO search didn't turn up anything. Any suggestions are welcome and appreciated.
Yes. PyFlakes will warn you about those most basic errors, and you should make sure it's integrated with your favorite text editor, so it tells you about missing imports or unused imports whenever you save the file.
PyFlakes will, amgonst other things, tell you about
syntax errors
undefined names
missing imports
unused imports
To just run PyFlakes on all your files in a directory, you can just do:
pyflakes /path/to/dir
One big advantage that PyFlakes has over more advanced linting tools like PyLint is that it does static analysis - which means, it doesn't need to import your code (which can be a pain if you've got some complex dependencies). It just analyses the abstract syntax tree of your Python source, and therefore catches the most basic of errors - those that usually prevent your script from having even a chance of running.
I should also mention that there is a related tool, flake8, which combines PyFlakes with PEP8 convention checks and McCabe code complexity analysis.
There's PyFlakes integrations for every editor (or IDE) I know. Here's just a couple (in no particular order):
Sublime Text
vim
emacs
TextMate
Eclipse / PyDev
Is there a way to get the behaviour of
from __future__ import unicode_literals
to apply project-wide, apart from putting this import in the top of each and every module?
I would like to just define it in one place, like the __init__.py of the package directory of project root say, and have it recursively apply to subpackages and submodules.
TL;DR - No
Command-line option -U enables unicode globally. From docs:
Turns all string literals into unicodes globally. Do not be tempted to use this option as it will probably break your world. It also produces .pyc files with a different magic number than normal.
You can't use it in practice, though, because it will mess up even the standard library. The docs advise just to enable unicode literals on a per-module basis using the future import.
I recently upgraded from python2.7 to python3 and think it may have screwed up some configurations. Now when I try to run a module, I get import errors. Let's say I have a directory structure like this:
/directory
/directory/__init__.py
/directory/run.py
/directory/app/db.py
/directory/app/views.py
/directory/app/__init__.py
with the following imports...
/directory/run.py says 'import app'
/directory/app/db.py says 'import views'
When I execute run.py, I get an error saying the module views cannot be found. However, if I go into /directory/app and execute db.py, then the import runs correctly. I've also found that if I change the /directory/app/db.py to say "from app import views" then it works correctly when executing run.py. However, this used to all work!
It seems like the import statements are not taking into account the folder it's being executed in. It seems like this wants me to base all of my imports out of the root folder, which seems incorrect and would take me time to change everything.
Any ideas as to what happened? This has been driving me crazy.
In Python3, implicit relative imports have been removed, all imports need to be either absolute, or use explicit relative imports.
This is not going to change, you need to either replace them with from app import views or from . import views.
Python 2.x and Python 3.x differ in so many ways it is usually extremely helpful to use 2to3 or another similar tool to "port" (convert) the code.
The issue you are running into likely has to do with the fact that Python 2 uses relative imports but Python 3 uses absolute imports (I may have that backwards). It is possible to change the import statement to make the import work, though for the torrent of compatibility issues that will surely follow, I highly recommend using 2to3 and then making any final adjustments manually.
Good luck!
It's considered bad Python to use imports like this:
import my_module
When you are doing a relative import and this would work:
from . import my_module
Is there a tool that can detect these non-dotted relative imports in my code and warn me so I could update them to the dotted syntax? My project has hundreds of Python modules and I would like to do this automatically. (Possibly such a tool would override __import__ and detect the bad imports as they happen when I run the program.)
Does anyone know of such tool?
[Reposted as an answer because it apparently did the trick]
2to3 will automatically convert them, because it's compulsory in Python 3.
Here's the relevant source code if you want to modify it for your purposes.
Alternatively, you could just run 2to3 with only that fixer: 2to3 -w -f import myproject/
pylint gives warnings about relative imports, along with tons of other stuff that's considered, for one reason or another, "bad Python".
Just after standard pythonmodule imports?
If I postpone it to the main function and do my specific module imports before it, it gives error (which is quite obvious). Python Style guide no where mentions the correct location for it.
It should go before the import or from statements that need it (which as you say is obvious). So for example a module could start with:
import sys
import os
import math
try:
import foo
except ImportError:
if 'foopath' in sys.path: raise
sys.path.append('foopath')
import foo
Note that I've made the append conditional (on the import failing and the specific module's path not being on sys.path yet) to avoid the risk of sys.path ending up with dozens of occurrences of string foopath, which would not be particularly helpful;-).
One reason this isn't mentioned in PEP 8 or other good Python style guides is that modifying sys.path isn't something you want to do in a real program; it makes your program less robust and portable. A better solution might be to put your package somewhere that would already be in sys.path or to define PYTHONPATH systemwide to include your package.
I often use a shell script to launch my python applications: I put my sys.path.insert (append) statement just after the "standard" python module imports in my "launch python script".
Using sys.path.insert(0, ...) gets your "imports" in priority in the path list.
I generally do it before importing anything. If you're worried that your module names might conflict with the Python stdlib names, then change your module names!
I think it's a matter of taste. But most people tend to put it behind the import sys :-)
I prefer wrapping it in an extra function:
def importmod_abs(name):
sys.path.append() ..
__import__ ...
sys.path.pop()
... this way sys.path remains clean. Of course that's only applicable to certain module structures. Anyways, I'd import everything that works without altering sys.path first.