I'm trying to implement a code that will generate a lot of scenarios. This scenarios will be executed after by a 3 different simulators.
I'm saving my scenarios in a file using shelve. My code is like this:
def save_variables(I, T, R, C, lambd, K, iteraction):
filename='/folder/shelve_{}.out'.format(iteraction)
my_shelf = shelve.open(filename,'n')
for key in dir():
try:
my_shelf[key] = locals()[key]
except TypeError:
pass
my_shelf.close()
and I'm loading data in each simulators like this:
my_shelf = shelve.open(filename)
my_shelf = shelve.open(filename)
for key in my_shelf:
globals()[key]=my_shelf[key]
my_shelf.close()
This part works great. My problem is:
If I run all the code together, in the same terminal of execution, for example, first the scenario_generator, after the simulator_1, after simulator_2 and at least simulator_3, in the same line of execution, the codes works great.
But if I run only the scenario_generator in one terminal and start each simulation in other 3 differents terminals of execution, I receive the follow error:
Number of arguments: 2 arguments.
Argument List: on
Iteraction 0
Traceback (most recent call last):
File "main.py", line 53, in <module>
onets.onets(i)
File "/Users/simulator_1.py", line 31, in simulator_1
n = [[0 for x in range(I+1)] for y in range(T+1)]
NameError: name 'T' is not defined
I understand with this notification that was not possible to read the data saved by shelved. But the files was in folder. Someone knows how I can fix this problem?
P.S.: Is important to me execute in this way (separately) because I should start the 3 simulator as the same time, in different terminals of execution. This will provide a big economy of execution time to have my results.
thanks everyone.
Without being sure, I suspect you are not storing all the variables you think you are in the shelve.
When the steps are together in a single script, if some values are not written in the shelve, you won't notice because you explicitely silent the TypeError` exception.
This is not recommended as errors go unnoticed. You probably made it exactly to avoid some error that was happening.
Because the scripts are together in the same file they share the same global scope.
When you read the values back into the global scope, and some values are missing, you are probably covered by the original values in that same global scope.
So you miss some values, but when you need them they were already there in global scope to be read from.
With separate files there is no more a common global scope. Any missing value will cause an error.
Remove the pass and make the except clause print the current key and value to check for missing keys.
Other than that I see no problem in generating file data to read later from whatever different processes/scripts like you described (given there are no cache/sync trouble and you open the files for reading only)
Related
My supervisor asked me the other day if a log file in a Python script is created at runtime or not. Is that another way of asking: is a new log file created every time the script is run? Because to me everything happens "at runtime" and it can be no other way. How else could it be?
The phrase is usually used when you cannot figure out the content of some variable/data by just analyzing the code and other associated files (such as configs). The easiest example I can think of is
x = 'hi'
print(x)
versus
x = input()
print(x)
In the second example "x is created at runtime". (Or more precisely, the data which the name x points to is dynamic and can be different in each execution.)
Yeah i think you got it right! However if you for example would only append to an already existing log file it would not be considered as "created at runtime".
Here is the lifecycle of a program, source Wikipedia.
You can create the log file before the run time, just create a new file in your project, or let the program create one during the runtime, if the file already not exist. The extreme case is to save all the log data in memory during the execution end create the file during the end time, but I think it's a really bad idea XD...
I have 2 files, 1 file with the classes and 1 file with the 'interface'
In the first file I have:
from second_file import *
class Catalog:
def ListOfBooks(self):
more_20 = input()
# If press 1 starts showing from 20 until 40
if more_20 == '1':
for item in C[20:41:1]:
print("ID:",item['ID'],"Title:",item['title']," Author: ", item['author'])
elif more_20 == '2':
return librarian_option()
test = Catalog()
test.ListOfBooks()
What I try to achieve is when the user presses 2, I want to go back to the function in my other file.
Second file:
def librarian_option():
.......
I don't want to use globals and I have read that the librarian_option() is in the scope of the second file and that's why I can't call it directly. I can't find a solution to it.
I get the following error:
NameError: name 'librarian_option' is not defined
Have you tried? It's just best practice to be explicit instead of using * wildcard
from second_file import librarian_option
Make sure the second_file is in the same directory.
Note: this is not an answer, but an example to help improve the question with more details.
You need to reduce your problem to the minimum amount of code (and actions) necessary to produce your problem. You also need provide how exactly you are running your script, and what version (of Python and your OS) you are using.
For example, I have created the following two scripts (named exactly as shown):
first_file.py:
from second_file import *
class Catalog:
def ListOfBooks(self):
return librarian_option()
test = Catalog()
a = test.ListOfBooks()
print(a)
second_file.py:
def librarian_option():
return 1
These two files are located in the same, random, directory on my computer (MacOS). I run this as follows:
python3.7 first_file.py
and my output is
1
Hence, I can't reproduce your problem.
See if you can still produce your problem with such simplified scripts (i.e., no extra functions or classes, no __init__.py file, etc). You probably want to do this in a temporary directory elsewhere on your system.
If your problem goes away, slowly build back up to where it reappears again. By then, possibly, you've discovered the actual problem as well. If you then don't understand why this (last) change you made caused the problem, feel free to update your question with all the new information.
I have a Python script that runs 24hrs a day.
A module from this script is using variables values that I wish to change from time to time, without having to stop the script, edit the module file, then launch the script again (I need to avoid interruptions as much as I can).
I thought about storing the variables in a separate file, and the module would, when needed, fetch the new values from the file and use them.
Pickle seemed a solution but is not human readable and therefore not easily changeable. Maybe a JSON file, or another .py file I import over again ?
Another advantage of doing so, for me, is that in case of interruption (eg. server restart), I can resume the script with the latest variable values if I load them from a separate file.
Is there a recommended way of doing such things ?
Something along the lines :
# variables file:
variable1 = 10
variable2 = 25
# main file:
while True:
import variables
print('Sum:', str(variable1+variable2))
time.sleep(60)
An easy way to maintain a text file with variables would be the YAML format. This answer explains how to use it, basically:
import yaml
stream = open("vars.yaml", "r")
docs = yaml.load_all(stream)
If you have more than a few variables, it may be good to check the file descriptor to see if the file was recently updated, and only re-load variables when there was a change in the file.
import os
last_updated = os.path.getmtime('vars.yaml')
Finally, since you want avoid interruption of the script, it may be good to have the script catch any errors in the YAML file and warn the user, instead of just throwing an exception and die. But also remember that "errors should never pass silently". What is the best approach here would depend on your use-case.
Please excuse what I know is an incredibly basic question that I have nevertheless been unable to resolve on my own.
I'm trying to switch over my data analysis from Matlab to Python, and I'm struggling with something very basic: in Matlab, I write a function in the editor, and to use that function I simply call it from the command line, or within other functions. The function that I compose in the matlab editor is given a name at the function definition line, and it's generally best for the function name to match the .m file name to avoid confusion.
I don't understand how functions differ in Python, because I have not been successful translating the same approach there.
For instance, if I write a function in the Python editor (I'm using Python 2.7 and Spyder), simply saving the .py file and calling it by its name from the Python terminal does not work. I get a "function not defined" error. However, if I execute the function within Spyder's editor (using the "run file" button), not only does the code execute properly, from that point on the function is also call-able directly from the terminal.
So...what am I doing wrong? I fully appreciate that using Python isn't going to be identical to Matlab in every way, but it seems that what I'm trying to do isn't unreasonable. I simply want to be able to write functions and call them from the python command line, without having to run each and every one through the editor first. I'm sure my mistake here must be very simple, yet doing quite a lot of reading online hasn't led me to an answer.
Thanks for any information!
If you want to use functions defined in a particular file in Python you need to "import" that file first. This is similar to running the code in that file. Matlab doesn't require you to do this because it searches for files with a matching name and automagically reads in the code for you.
For example,
myFunction.py is a file containing
def myAdd(a, b):
return a + b
In order to access this function from the Python command line or another file I would type
from myFunction import myAdd
And then during this session I can type
myAdd(1, 2)
There are a couple of ways of using import, see here.
You need to a check for __main__ to your python script
def myFunction():
pass
if __name__ == "__main__":
myFunction()
then you can run your script from terminal like this
python myscript.py
Also if your function is in another file you need to import it
from myFunctions import myFunction
myFunction()
Python doesn't have MATLAB's "one function per file" limitation. You can have as many functions as you want in a given file, and all of them can be accessed from the command line or from other functions.
Python also doesn't follow MATLAB's practice of always automatically making every function it can find usable all the time, which tends to lead to function name collisions (two functions with the same name).
Instead, Python uses the concept of a "module". A module is just a file (your .py file). That file can have zero or more functions, zero or more variables, and zero or more classes. When you want to use something from that file, you just import it.
So say you have a file 'mystuff.py':
X = 1
Y = 2
def myfunc1(a, b):
do_something
def myfunc2(c, d):
do_something
And you want to use it, you can just type import mystuff. You can then access any of the variables or functions in mystuff. To call myfunc2, you can just do mystuff.myfunc2(z, w).
What basically happens is that when you type import mystuff, it just executes the code in the file, and makes all the variables that result available from mystuff.<varname>, where <varname> is the name of the variable. Unlike in MATLAB, Python functions are treated like any other variable, so they can be accessed just like any other variable. The same is true with classes.
There are other ways to import, too, such as from mystuff import myfunc.
You run python programs by running them with
python program.py
I am working on a large-scale software system that is written in Python right now.
The thing is, I am not sure how to make sure if each individual .py file in the system is correct. The only way for me to run the software is to run the main.py file, which uses all the other .py files.
So either everything works, or one thing doesn't (causing everything to not work).
I keep getting a NameError even when importing the correct file. I think this may have to do with the fact that the class associated with that name in the NameError may have errors in it. Any suggestions? NameError is giving me this:
File "<string>", line 1, in <module>
NameError: name 'RGBox' is not defined
It's not a very helpful error message, and I'm not sure why it's giving "string" and 'module' instead of actual values.....
[EDIT]- I am working through ssh into a remote unix machine
This is a straight-forward error message which indicates that the execution flow has not yet encountered class/module/variable RGBox prior to it being called.
RGBox is either being called out of sequence or has been mispelt.
Perform a commandline search through the app files for the name 'RGBox' or its regex equivalents. for example with grep you can do a case-insensitive search:
$ grep -lsri 'rgbox' ./my_project_folder
which will output any file which contains the patterns 'RGBox', 'rgBox', etc.
If you are unfamiliar with the code and its structure, then you may as well insert strategic logging (or print) statements at significant locations in the code to understand its flow and execution logic.