Develop Reference

How to always add all imports to new Python files?

I am working on a Python project in PyCharm.
Every time I create a new file, I have to import all of the basic libraries again. I keep getting kicked out of my workflow for a few seconds every time I notice that I forgot to import a library.
As far as I can tell, there is no reason not to just import every library I use anywhere in every single file, or is there? I would much rather import a library I don't need than risk losing my concentration because I forgot to import something and have to waste a few seconds on it. Since the auto-sorting of libraries doesn't seem to work properly, I even have to add the import statement manually.
Is there a way to give PyCharm a large list of libraries and just import those in absolutely every file by default?
For comparison, this is possible to do in Jupyter Lab: You can have one notebook that contains all import statements, and just call "%run import_everything.ipynb" in all the other notebooks. This has saved me a lot of time, and is also much more readable than having different imports in every notebook.
(I care specifically about how to do this in PyCharm, but if there is a more generic way to do it, that information would also be appreciated)

I wrote a short script to go over all my files and combine all basic import statements. It also wraps it in comments that say "BASIC IMPORTS", so in the future I can just use PyCharms "replace string in whole project" feature to mass-replace all imports by looking for these comments with a regex.
from pathlib import Path
import re
p = Path(__file__).absolute().parent.parent
excluded_files = ['__init__.py', 'setup.py']
affected_files = [a for a in p.glob('**/*.py') if a.name not in excluded_files]
acc = {}
for a in affected_files:
b = a.read_text()
b = b[:b.index("\n\n")]
finds = re.findall("(import |from )(.*)", b)
for c in finds:
acc[c] = True
assert "intnet" not in c[1], a
lst = list(acc.keys())
lst.sort(key=lambda a: a[1])
res = '\n'.join(f"{a[0]}{a[1]}" for a in lst)
res = "# BASIC IMPORTS\n" + res + "\n# / BASIC IMPORTS"
for a in affected_files:
b = a.read_text()
b = res + b[b.index("\n\n"):]
a.write_text(b)

Error calling modules when using x = _import_("y") with dictionary values

I'm having trouble understanding an issue with calling packages after they have been imported using the __import_ function in Python. I'll note that using the usual import x as y works fine, but this is an learning exercise for me. I am importing and then checking the version for multiple packages and to learn a little more about Python, I wanted to automate this by using a dictionary.
My dictionary looks something like this:
pack = {"numpy": ["np", "1.7.1"]}
and I then use this to load and check the modules:
for keys in pack.keys():
pack[keys][0] = __import__(keys)
print("%s version: %6.6s (need at least %s)" %(keys, pack[keys][0].__version__, pack[keys][1]))
This works fine, but when I later try to call the package, it does not recognize it: x = np.linspace(0,10,30)
produces an error saying np isn't recognized, but this works: x = pack[keys][0].linspace(0,10,30)
Since this is just a way for me to learn, I'd also be interested in any solutions that change how I've approached the problem. I went with dictionaries because I've at least heard of them, and I used the _import__ function since I was forced to either use quoted characters or numeric values in my dictionary values. The quoted characters created problems for the import x as y technique.

Althought it isn't a good practice, you can create variables dynamically using the builtin locals function.
So,
for module, data in pack.items():
locals()[data[0]] = __import__(module)
nickname = locals()[data[0]]
print("%s version: %6.6s (need at least %s)" %(module, nickname.__version__, pack[module][1]))
Output:
numpy version: 1.12.1 (need at least 1.7.1)

Enable all warnings in Python, after they were disabled by an imported module

I have the following snippet:
a = np.array([1, 2, 3])
b = np.array([False, True])
print(a[b])
It triggers a VisibleDeprecationWarning as expected.
Now when I import a certain module, that warning is no longer shown:
import questionable_module
a = np.array([1, 2, 3])
b = np.array([False, True])
print(a[b])
How do I need to modify my code in order to re-enable all warnings? I neither want nor can change the questionable_module. I would prefer to do it in the code instead of command line arguments, if possible.
The questionable_module is Glumpy but I'm looking for a solution that works independently of what other modules do.

I checked and it seems for glumpy they use logging.captureWarnings to catch the warnings:
import warnings
import logging
logging.captureWarnings(True)
Source
I'm not sure if they intended to log all warnings but you can disable it with
import logging
logging.captureWarnings(False)
Other possibilities (that don't apply in this case) but could be helpful in the future:
In general it could also be that they adjusted the warnings.simplefilter you could enable it again like this:
import warnings
warnings.simplefilter("always", VisibleDeprecationWarning)
or reset it to the default with warnings.resetwarnings.
If it's actually a NumPy floating point warning you, then you need to use numpy.seterr:
import numpy as np
np.seterr(all='warn')
But it could also be that the questionable_module really replaces or patches the functions in a way that they actually never get to the point where the warning is raised. In that case you probably can't do anything.

Try this:
import warnings
import questionable_module
warnings.resetwarnings() # Reset the warnings filter. This discards the effect of all previous calls to filterwarnings(), including that of the -W command line options and calls to simplefilter().
warnings.simplefilter('default')
a = np.array([1, 2, 3])
b = np.array([False, True])
print(a[b])

Python multiprocessing pool function not defined

I need to implement a multiprocessing pool that utilizes arbitrary packages for calculations. For this, I'm using Python and joblib 0.9.0. This code is basically the structure I want.
import numpy as np
from joblib import pool
def someComputation(x):
return np.interp(x, [-1, 1], [-1, 1])
if __name__ == '__main__':
some_set_of_numbers = [-1,-0.5,0,0.5,1]
the_pool = pool.Pool(processes=2)
solutions = [the_pool.apply_async(someComputation, (x,)) for x in some_set_of_numbers]
print(solutions[0].get())
On both Windows 10 and Red Hat Enterprise Linux running Anaconda 4.3.1 Python 3.6.0 (as well as 3.5 and 3.4 with virtual envs), I get that 'np' was never passed into the someComputation() function raising the error
File "C:\Anaconda3\lib\site-packages\multiprocessing_on_dill\pool.py", line 608, in get
raise self._value
NameError: name 'np' is not defined
however, on my Mac OS X 10.11.6 running Python 3.5 and the same joblib, I get the expected output of '-1' with the exact same code. This question is essentially the same, but it dealt with pathos and not joblib. The general answer was to include the numpy import statement inside of the function
from joblib import pool
def someComputation(x):
import numpy as np
return np.interp(x, [-1, 1], [-1, 1])
if __name__ == '__main__':
some_set_of_numbers = [-1,-0.5,0,0.5,1]
the_pool = pool.Pool(processes=2)
solutions = [the_pool.apply_async(someComputation, (x,)) for x in some_set_of_numbers]
print(solutions[0].get())
This solves the issue on the Windows and Linux machines, where they now output '-1' as expected but this solution seems clunky. Is there any reason why the first bit of code would work on a Mac, but not Windows or Linux? I ultimately need to run this code on the Linux machine so is there any fix that doesn't include putting the import statement inside of the function?
Edit:
After investigating a bit further, I found an old workaround I put in years ago that looks like is causing the issue. In joblib/pool.py, I changed line 44 from
from multiprocessing.pool import Pool
to
from multiprocessing_on_dill.pool import Pool
to support pickling of arbitrary functions. For some reason, this change is what really causes the issue on Windows and Linux, but the Mac machine runs just fine. Using multiprocessing instead of multiprocessing_on_dill solves the above issue, but the code doesn't work for the majority of my cases since they can't be pickled.

I am not sure what the exact issue is, but it appears that there is some problem with transferring the global scope over to the subprocesses that run the task. You can potentially avoid name errors by binding the name np as a function parameter:
def someComputation(x, np=np):
return np.interp(x, [-1, 1], [-1, 1])
This has the advantage of not requiring a call to the import machinery every time the function is run. The name np will be bound to the function when it is first evaluated during module loading.

Differences between ipython %run and promt

I've the following simple erroneous code
from numpy import random, sqrt
points = random.randn(20,3);
points = points / sqrt(sum(points**2,1))
In ipython (with %autoreload 2) if I copy and paste it into the terminal I get a ValueError as one would expect. If I save this as a file and use %run then it runs without error (it shouldn't).
What's going on here?

I just figured it out, but I had written the question and it might be useful to someone else.
It is a difference between the numpy sum and the native sum. Changing the first line to
from numpy import random, sqrt, sum
fixes it as %run uses the native version by default (at least with my settings). The native run does not take an axis parameter, but does not throw an error either, because it is a start parameter, which is in effect just an offset to the sum. So,
>>> sum([1,2,3],10000)
10006
for the native version. And "axis out of bounds" for the numpy one.