I am working on a Python project in PyCharm.
Every time I create a new file, I have to import all of the basic libraries again. I keep getting kicked out of my workflow for a few seconds every time I notice that I forgot to import a library.
As far as I can tell, there is no reason not to just import every library I use anywhere in every single file, or is there? I would much rather import a library I don't need than risk losing my concentration because I forgot to import something and have to waste a few seconds on it. Since the auto-sorting of libraries doesn't seem to work properly, I even have to add the import statement manually.
Is there a way to give PyCharm a large list of libraries and just import those in absolutely every file by default?
For comparison, this is possible to do in Jupyter Lab: You can have one notebook that contains all import statements, and just call "%run import_everything.ipynb" in all the other notebooks. This has saved me a lot of time, and is also much more readable than having different imports in every notebook.
(I care specifically about how to do this in PyCharm, but if there is a more generic way to do it, that information would also be appreciated)
I wrote a short script to go over all my files and combine all basic import statements. It also wraps it in comments that say "BASIC IMPORTS", so in the future I can just use PyCharms "replace string in whole project" feature to mass-replace all imports by looking for these comments with a regex.
from pathlib import Path
import re
p = Path(__file__).absolute().parent.parent
excluded_files = ['__init__.py', 'setup.py']
affected_files = [a for a in p.glob('**/*.py') if a.name not in excluded_files]
acc = {}
for a in affected_files:
b = a.read_text()
b = b[:b.index("\n\n")]
finds = re.findall("(import |from )(.*)", b)
for c in finds:
acc[c] = True
assert "intnet" not in c[1], a
lst = list(acc.keys())
lst.sort(key=lambda a: a[1])
res = '\n'.join(f"{a[0]}{a[1]}" for a in lst)
res = "# BASIC IMPORTS\n" + res + "\n# / BASIC IMPORTS"
for a in affected_files:
b = a.read_text()
b = res + b[b.index("\n\n"):]
a.write_text(b)
Related
I'm working with the internet_stability library in python and I want to use it to, as the name implies test my internet stability and also speed.
The code I use works perfectly, but instead of using it only once, I would like to loop the code so that it can run throughout the day.
The problem I have is that library uses it's own exporting functions and that whenever you rerun the program it overwrites the old files.
So, I think a solution for this would be to create new folder every time it loops and put the exported files of that loop into the folders.
How would I go about doing this? The code I use is shown below.
import network_stability
net = network_stability.NetworkTest()
# Run connectivity test.
net.connection_test_interval(hours=0.1)
net.export_connection_results('connection.csv')
net.report_connection('connection.png')
# Run speed test.
net.speed_test_interval(minutes=10)
net.export_speed_results('speed.csv')
net.report_speed('speed.png')
import network_stability
import datetime
net = network_stability.NetworkTest()
# Run connectivity test.
net.connection_test_interval(hours=0.1)
net.export_connection_results(f'{datetime.datetime.now().strftime("%y_%m_%d-%H_%M_%S")}_connection.csv')
net.report_connection(f'{datetime.datetime.now().strftime("%y_%m_%d-%H_%M_%S")}_connection.png')
# Run speed test.
net.speed_test_interval(minutes=10)
net.export_speed_results(f'{datetime.datetime.now().strftime("%y_%m_%d-%H_%M_%S")}_speed.csv')
net.report_speed(f'{datetime.datetime.now().strftime("%y_%m_%d-%H_%M_%S")}_speed.png')
It will create files like this:
>>> f'{datetime.datetime.now().strftime("%y_%m_%d-%H_%M_%S")}_speed.png'
'21_12_13-11_31_22_speed.png'
Because filenames are differents, it won t overwrite them. (Unless your run twice tests a second!)
If you want all you files has the same name in a run, do this:
import network_stability
import datetime
net = network_stability.NetworkTest()
date = datetime.datetime.now().strftime("%y_%m_%d-%H_%M_%S")
# Run connectivity test.
net.connection_test_interval(hours=0.1)
net.export_connection_results(f'{date}_connection.csv')
net.report_connection(f'{date}_connection.png')
# Run speed test.
net.speed_test_interval(minutes=10)
net.export_speed_results(f'{date}_speed.csv')
net.report_speed(f'{date}_speed.png')
For similar situations I use a simple workaround:
run_number = 0
save_name = "Connection" + str(run_number)
net.export_speed_results(save_name + '.csv')
net.report_speed(save_name + '.png')
run_number += 1
Maybe that works for your problem as well?
Recently I have been trying to make a makeshift "disk space" reader. I made a library that stores values in a list "the disk" and when I subprocess a new script to write to the "disk" to see if the values change on the display nothing happens. I realized that any time you import a module the module sort of clones itself to only that script.
I want to be able to have scripts import the same module and so that if 1 script changes a value another script can see that value.
Here is my code for the "disk" system
import time
ram = []
space = 256000
lastspace = 0
for i in range(0,space + 1):
ram.append('')
def read(location):
try:
if ram[int(location)] == '':
return "ERR_NO_VALUE"
else:
return ram[int(location)]
except:
return "ERR_OUT_OF_RANGE"
def write(location, value):
try:
ram[int(location)] = value
except:
return "ERR_OUT_OF_RANGE"
def getcontents():
contents = []
for i in range(0, 256001):
contents.append([str(i)+ '- ', ram[i]])
return contents
def getrawcontents():
contents = []
for i in range(0, 256001):
contents.append(ram[i])
return contents
def erasechunk(beg, end):
try:
for i in range(int(beg), int(end) + 1):
ram[i] = ''
except:
return "ERR_OUT_OF_RANGE"
def erase(location):
ram[int(location)] = ''
def reset():
ram = []
times = space/51200
tc = 0
for i in range(0,round(times)):
for x in range(0,51201):
ram.append('')
tc += 1
print("Byte " + str(tc) + " of " + " Bytes")
for a in range(0,100):
print('\a', end='')
return [len(ram), ' bytes']
def wipe():
for i in range(0,256001):
ram[i] = ''
return "WIPED"
def getspace():
x = 0
for i in range(0,len(ram)):
if ram[i] != "":
x += 1
return [x,256000]
The shortest answer to your question, which I'm understanding as "if I import the same function into two (or more) Python namespaces, can they interact with each other?", is no. What actually happens when you import a module is that Python uses the source script to 'build' those functions in the namespace you're importing them to; there is no sense of permanence in "where the module came from" since that original module isn't actually running in a Python process anywhere! When you import those functions into multiple scripts, it's just going to create those pseudo-global variables (in your case ram) with the function you're importing.
Python import docs: https://docs.python.org/3/reference/import.html
The whole page on Python's data model, including what __globals__ means for functions and modules: https://docs.python.org/3/reference/datamodel.html
Explanation:
To go into a bit more depth, when you import any of the functions from this script (let's assume it's called 'disk.py'), you'll get an object in that function's __globals__ dict called ram, which will indeed work as you expect for these functions in your current namespace:
from disk import read,write
write(13,'thing')
print(read(13)) #prints 'thing'
We might assume, since these functions are accurately accessing our ram object, that the ram object is being modified somehow in the namespace of the original script, which could then be accessed by a different script (a different Python process). Looking at the namespace of our current script using dir() might support that notion, since we only see read and write, and not ram. But the secret is that ram is hidden in those functions' __globals__ dict (mentioned above), which is how the functions are interacting with ram:
from disk import read,write
print(type(write.__globals__['ram'])) #<class 'list'>
print(write.__globals__['ram'] is read.__globals__['ram']) #True
write(13,'thing')
print(read(13)) #'thing'
print(read.__globals__['ram'][13]) #'thing'
As you can see, ram actually is a variable defined in the namespace of our current Python process, hidden in the functions' __globals__ dict, which is actually the exact same dictionary for any function imported from the same module; read.__globals__ is write.__globals__ evaluates to True (even if you don't import them at the same time!).
So, to wrap it all up, ram is contained in the __globals__ dict for the disk module, which is created separately in the namespace of each process you import into:
Python interpreter #1:
from disk import read,write
print(id(read.__globals__),id(write.__globals__)) #139775502955080 139775502955080
Python interpreter #2:
from disk import read,write
print(id(read.__globals__),id(write.__globals__)) #139797009773128 139797009773128
Solution hint:
There are many approaches on how to do this practically that are beyond the scope of this answer, but I will suggest that pickle is the standard way to send objects between Python interpreters using files, and has a really standard interface. You can just write, read, etc your ram object using a pickle file. To write:
import pickle
with open('./my_ram_file.pkl','wb') as ram_f:
pickle.dump(ram,ram_f)
To read:
import pickle
with open('./my_ram_file.pkl','rb') as ram_f:
ram = pickle.load(ram_f)
I am struggling with properly organising Python code. My struggle stems from namespaces in Python. More specifically, as I understand, every module has its own specific namespace and if not stated specifically, elements from one namespace will not be available in another namespace.
To give an example what I mean, consider the function subfile1.py:
def foo();
print(a)
Now if I want the foo() function to print a from the namespace of another module/script, I will have to set it explicitly for that subfile1 module; e.g. like in main.py:
import subfile1
a = 4
subfile1.a = 4
subfile1.foo()
When I code (in R), it’s generally to provide data analytics reports. So I have to import data, clean data and perform analyses on this data. My coding projects (to be re-run every couple of months) are then organised as follows: I have a main script from which I run other sub-scripts. I find this very handy I have separate sub-scripts for:
Setting paths
Loading data
Data cleaning
Specific analyses
Simply looking at the names of the sub-scripts listed in the main script gives me then a good overview of what is being done and where I need to add additional sub-scripts if needed.
With Python I don’t find this an easy way of working because of the namespaces. If I would like to keep this way of working in Python, it seems to me that I would have to import all sub-scripts as modules and set the necessary variables for each module explicitly (either in the module or in the main script as in the example above).
So I thought that maybe my way of working is simply not optimal, at least in a Python setting. Could anyone tell me how best to organise the project I have in mind? Or put differently, what is a good way of organising code if the same data/variables has to be used at various places in the code?
Update:
After reading the comments, I made another example. Now with classes. Any comments on whether this is an appropriate way of structuring code or not would be very welcome. My main aim is to split the code in bits and pieces so a third party can quickly gauge what is going on and make changes to the code where necessary.
I have a master_file.py in which I run the code:
from sub_file_1 import DoSomething
from sub_file_2 import DoSomethingElse
from sub_file_3 import CombineTheStuff
x1 = DoSomething.c #could e.g. be import from a data source and transform the data
x2 = DoSomethingElse.d #could e.g. be import from another data source and transform the data taking into its particularities
y = CombineTheStuff(x1, x2) # e.g. with the data imported and cleaned (x1 and x2) I can now estimate a particular model
print(x1)
print(x2)
print(y.fin_result())
sub_file_1.py contains (in reality these sub_files are very lengthy)
from param_file import GeneralPar
class DoSomething():
c = GeneralPar.a + GeneralPar.b
sub_file_2.py contains
from param_file import GeneralPar
class DoSomethingElse():
d = GeneralPar.a * 4
sub_file_3.py contains
class CombineTheStuff():
def __init__(self, input1, input2):
self.input1 = input1
self.input2 = input2
def fin_result(self):
k = self.input1 * self.input2 - 0.01
return(k)
This is my first attempt at using IPython.parallel so please bear with me.
I read this question
Parfor for Python
and am having trouble implementing a simple example as follows:
import gmpy2 as gm
import numpy as np
from IPython.parallel import Client
rc = Client()
lview = rc.load_balanced_view()
lview.block = True
a = 1
def L2(ii,jj):
out = []
out.append(gm.fac(ii+jj+a))
return out
Nloop = 100
ii = range(Nloop)
jj = range(Nloop)
R2 = lview.map(L2, zip(ii, jj))
The problems I have are:
a is defined outside the loop and I think I need to do something like "push" but am a bit confused by that. Do I need to "pull" after?
there are two arguments that are required for the function and I don't know how to pass them correctly. I tried things like zip(ii,jj) but got some errors.
Also,, I assume the fact that I'm using a random library gmpy2 shouldn't affect things. Is this correct? Do I need to do anything special for this?
Ideally I would like your help so on this simple example the code runs error free.
If you think it would be beneficial to post my failed attempts at #2 let me know. I'm in the dark with #1.
I found two ways that make this work:
One is pushing the variable to the cores. There is no need to pull it. The variable will simply be defined in the namespace of each process-engine.
rc.client[:].push({'a':a})
R2 = lview.map(L2, ii, jj)
The other way is as to redefine L2 to take a as an input and pass an array of a's to the map function:
def L2(ii,jj,a):
out = []
out.append(gm.fac(ii+jj+a))
return out
R2 = lview.map(L2, ii, jj, [a]*Nloop)
With regards to the import as per this website:
http://ipython.org/ipython-doc/dev/parallel/parallel_multiengine.html#non-blocking-execution
You simply import the required libraries in the function:
Note the import inside the function. This is a common model, to ensure
that the appropriate modules are imported where the task is run. You
can also manually import modules into the engine(s) namespace(s) via
view.execute('import numpy')().
Or you can do as per this link
http://ipython.org/ipython-doc/dev/parallel/parallel_multiengine.html#remote-imports
First, sorry for my stupid title :) And here is my problem.. Actually it's not a problem. Everything works, but I want to have better structure...
I have a python script with a loop "looped" each second.
In the loop there are many many IFs. Is it possible to put each IF in a separate file and then to include it in the loop? So this way every time the loop is "looped", all the IFs will be passed, too..
There are too many conditions in my script and all of them are different generally from the otheres so I want to have some kind of folder with modules - mod_wheather.py, mod_sport.py, mod_horoscope.py, etc..
Thanks in advance. I hope I wrote everything understandable..
EDIT:
Here is a structural example of what I have now:
while True:
if condition=='news':
#do something
if condition=='sport':
#so something else
time.sleep(1)
It will be good if I can have something like this:
while True:
import mod_news
import mod_sport
time.sleep(1)
And these IFs from the first example to be separated in files mod_news.py, mod_sport.py...
perhaps you wonder how to work with your own modules in general.
make one file named 'weather.py' and have it contain the appropriate if-statements like:
""" weather.py - conditions to check """
def check_all(*args, **kwargs):
""" check all conditions """
if check_temperature(kwargs['temperature']):
... your code ...
def check_temperature(temp):
-- perhaps some code including temp or whatever ...
return temp > 40
same for sport.py, horoscope.py etc
then your main script would look like:
import time, weather, sport, horoscope
kwargs = {'temperature':30}
condition = 'weather'
while True:
if condition == 'weather':
weather.check_all(**kwargs)
elif condition == 'sport':
sport.check_all()
elif condition == 'horoscope':
horoscope.check_all()
time.sleep(1)
edit: edited according to the edit in your question. Note that I suggest importing all modules only one time, at the beginning of the script, and using its functions. This is better than executing code by importing. But if you insist, you could use reload(weather), which actually performs a reload including code execution. But I cannot stress too much that using functions of external modules is a better way to go!
Put them in functions in separate files and then Import them:
"""thing1.py
A function to demonstrate
"""
def do_things(some_var):
print("Doing things with %s" % (some_var))
``
"""thing2.py
Demonstrates the same thing with a condition
"""
def do_things(some_var):
if len(some_var) < 10:
print("%s is < 10 characters long" % (some_var))
else:
print("too long")
``
"""main_program.py"""
import thing1, thing2
myvar = "cats"
thing1.do_things(myvar)
thing2.do_things(myvar)
I believe you are looking for some kind of PHP-like include() or C prepocessor #include. You would have a file such as the included.py below:
a = 2
print "ok"
and another file which has the following code:
for i in values:
import included
and you want the result to be equivalent to
for i in values:
a = 2
print "ok"
Is it what you are looking for? If so... no, it is not possible. Once Python imports a module, the code of the module is executed and following imports of the same mode only retrieve the already imported instance of the module. The code of a module is not executed everytime it is imported.
I can invent some crazy ways of doing it (let us say, file.read() + eval(), or calling reload() in an imported module.) but it would be a bad idea anyway. I bet we can think of a better solution to your real problem :)
Perhaps all you need is to call functions in your loop; and have those functions in other modules, which you import as needed.
while true:
if condition:
from module_a import f
f()
if condition2
from module_b import g
g()
Though the above is legal Python, and so answers your question, you should in practice write all the imports at the top of your file.
You could import the needed modules if they're needed, for example:
if condition:
import weather
... do something
However I'm not sure if that's what you really want.
I have a python script with a loop "looped" each second. In the loop
there are many many IFs.
Then you must optimize the repeatedly executed tests. Suppose there are 50 IFs blocks in your code and that in a turn of the for-loop, the N th condition is True: that means that the N-1 other conditions must be tested before the N th is tested and triggers the execution of the corresponding code.
It would be preferable to do so:
# to_include.py
def func_weather(*args,**kwargs):
# code
return "I'm the weather"
def func_horoscope(*args,**kwargs):
# code
return "Give me your birth'date"
def func_gastronomy(*args,**kwargs):
# code
return 'Miam crunch'
def func_sports(*args,**kwargs):
# code
return 'golf, swimming and canoeing in the resort station'
didi = {'weather':func_weather, 'horoscope':func_horoscope,
'gastronomy':func_gastronomy, 'sports':func_sports}
and the main module:
# use_to_include.py
import to_include
x = 'sports'
y = to_include.didi[x]()
# instead of
# if x =='weather' : y = func_weather()
# elif x=='horoscope' : y = func_horoscope()
# elif x=='gastronomy': y = func_gastronomy()
# elif x=='sports' : y = func_sports()
print y
result
golf, swimming and canoeing in the resort station