This is going to be a very long question. So, pardon me
I have the following scenario, I guess it will be better to give a pseudo code to explain things better
A python file say test.py
def test(i):
from rpy2.robjects import r
r.source('r_file.R')
r.call_function(with some arguments)
#Some Operations
del r
File: r_file.R
rm(list=ls(all=TRUE))
#some global variables
#some reference class
#creating an object of reference class
call_function = function(some arguments)
{
Do some processing
call few methods on a reference class
call some more methods and do some operations
rm(list=ls(all=TRUE))
gc()
return(0)
}
The call to the the function test in python happens for some values of 'i' i.e the function gets called for some values of i which is always greater than 1 i.e the function gets invoked multiple times from main. Hence, we source the R file more than once. I wanted a new R interpreter every time I invoke the python function. Therefore, I import r every time the function is called and also delete the rpy2 object.
Within the r function "call_function", I invoke some methods, which in turn creates reference class objects.
Within the R code, I use rm in the beginning of the code and also when the function some_function exits.
Given this background, the problem which I'm facing now is that the rm does not remove any of the reference class in the code and I keep getting some warning like this
In .removePreviousCoerce(class1, class2, where, prevIs) :
methods currently exist for coercing from "Rev_R5" to "envRefClass"; they will be replaced
Here, Rev_R5 is a reference class. I do not want this to happen, is there a way to remove all the methods, objects related to the reference classes using rm ?
Removing all objects from R's global environment does not mean that you are back to a freshly started R process (class and method definitions may remain, as you discovered it).
R functions such as removeClass(), removeMethod(), or removeGeneric could be considered but unless there are objective requirements to do so (like avoid the loading of large objects over and over again), creating R processes each time might just be the safest way to go (starting an R process is relatively fast).
Since it is not possible to terminate and restart an embedded R (limitation coming from R, not rpy2), so you'll have to start and stop Python processes embedding R.
One way to do so is to use the Python package multiprocessing (included in Python's standard library). An added bonus is that the processes can be run in parallel.
Simple examle using Doug Hellmann's excellent tutorial as a base:
import multiprocessing
def R_worker(i):
"""worker function"""
print('Worker %i started' % i)
from rpy2.robjects import r
r.source('r_file.R')
r.call_function(with some arguments)
#Some Operations
del r
return
if __name__ == '__main__':
jobs = []
for i in range(5):
p = multiprocessing.Process(target = R_worker, args=(i,))
jobs.append(p)
p.start()
Related
This is my first post here so please let me know if I'm doing it wrong. I tried looking for an existing answer, but wasn't sure what to search for.
Consider the following simple example, a python module called mymath.py which uses only built-in python operations and modules. This custom module is portable, so anyone can execute the code without installing anything other than stock python.
# mymath.py
import sys
def minus(a, b):
return a-b
def mult(a, b):
return a*b
def div(a, b):
return a/b
def plus(a, b):
return a+b
def sum_series(int_list):
sum = 0
for i in int_list:
sum = plus(sum, i)
return sum
def main():
my_list = [2, 4, 6]
value = sum_series(my_list)
sys.stdout.write("your list total = {}".format(value))
Notice, main() only calls sum_series() which in turn calls plus(). The other functions may be required elsewhere in this fictional code base, but we're only concerned with main().
Now, I would like to copy only the relevant source code to another object as a text string. In other words, gather main() and all it's dependencies (recursively), resulting in a string of executable code.
My current solution:
import inspect
import mymath
# copy the source code in to the callback
source = inspect.getsource(mymath)
# then append a call to the main function
source += "\nmain()"
This works, producing a local copy of the module as a string that can run main() without requiring an import of mymath. The problem is that knob is now bloated with all the extra unused functions, although it is able to pick up any changes I make to mymath.py by rerunning my current solution.
So, the question is - is there a way to do the equivalent of:
source = getFunctionSourcerRecursively(mymath.main)
source += "\nmain()"
resulting in source =
# mymath.py
import sys
def plus(a, b):
return a+b
def sum_series(int_list):
sum = 0
for i in int_list:
sum = plus(sum, i)
return sum
def main():
my_list = [2, 4, 6]
sys.stdout.write("your list total = {}".format(sum_series(my_list)))
main()
So, basically "source" now contains only the relevant code and is portable, no longer requiring people offsite to have mymath installed.
If you're curious, my real-world case involves using The Foundry Nuke (compositing application) which has an internal callback system that can run code when callback events are triggered on a knob (property). I want to be able to share these saved Nuke files (.nk or .nknc) with offsite clients, without requiring them to modify their system.
You might try informal interfaces (a.k.a. protocols). While protocols work fine in many cases, there are situations where informal interfaces or duck typing in general can cause confusion. For example, an Addition and Multiplication both are mathFunc(). But they ain't the same thing even if they implement the same interfaces/protocols. Abstract Base Classes or ABCs can help solve this issue.
The concept behind ABCs is simple - user defines base classes which are abstract in nature. We define certain methods on the base classes as abstract methods. So any objects deriving from these bases classes are forced to implement those methods. And since we’re using base classes, if we see an object has our class as a base class, we can say that this object implements the interface. That is now we can use types to tell if an object implements a certain interface.
import mymath
class MathClass(mymath.ABC):
#mymath.abstractmethod
def mathFunc(self):
pass
I have two .py script files. The "main" script will import the second script containing misc "helper" functions.
In the main script, I have set up an object for a SPI interface. I would like to write functions in the imported file that use the SPI interface directly. I'm a noob at this and tried writing and passing in various ways but always get errors.
mainscript.py
import helperfunctions.py as helper
spi = spidev.SpiDev()
spi.open(0, 0)
response = spi.xfer([ ... some data ...]) #this works when
#called from mainscript.py
helper.sendOtherStuff() #this doesn't work (see helper script below)
helperfunctions.py
def sendOtherStuff():
#need to somehow reference 'spi.' object from mainscript.py file
otherData = ([... some different data ...])
resp = spi.xfer([otherData]) #this fails because helperfunctions
#apparently doesn't know spi. object
return resp
I have the same general question often regarding global variable values as well. I'm sure there is a "better" way to do it, but out of convenience for now, I often wish to define some global variables in mainscript.py then reference those globals inside functions of helperfunctions.py. I can't figure a way to do this. Going the other way is easy - declare the globals inside helperfunctions.py then reference them from mainscript.py as helper.variableName, but I don't know how to go the other direction.
Any direction is much appreciated. Thank you.
By my lights the easiest thing to do would be to pass the spi object to the helper function as a parameter:
def sendOtherStuff(spi):
otherData = ([... some different data ...])
return spi.xfer([otherData])
Once it's passed in, you can call methods on it in the body of the function. I removed your variable assignment because it seemed redundant.
I am trying to better understand Pythons modules, coming from C background mostly.
I have main.py with the following:
def g():
print obj # Need access to the object below
if __name__ == "__main__":
obj = {}
import child
child.f()
And child.py:
def f():
import main
main.g()
This particular structure of code may seem strange at first, but rest assured this is stripped from a larger project I am working on, where delegation of responsibility and decoupling forces the kind of inter-module function call sequence you see.
I need to be able to access the actual object I create when first executing main python main.py. Is this possible without explicitly sending obj as parameter around? Because I will have other variables and I don't want to send these too. If desperate, I can create a "state" object for the entire main module that I need access to, and send it around, but even that is to me a last resort. This is global variables at its simplest in C, but in Python this is a different beast I suppose (module global variables only?)
One of the solutions, excluding parameter passing at least, has turned to revolve around the fact that when executing the main Python module main as such - via f.e. python main.py where if clause suceeds and subsequently, obj is bound - the main module and its state exist and are referenced as __main__ (inspected using sys.modules dictionary). So when the child module needs the actual instance of the main module, it is not main it needs to import but __main__, otherwise two distinct copies would exist, with their own distinct states.
'Fixed' child.py:
def f():
import __main__
__main__.g()
This is a newbie question:
A class is an object, so I can create a class called pippo() and inside of this add function and parameter, I don't understand if the functions inside of pippo are executed from up to down when I assign x=pippo() or I must call them as x.dosomething() outside of pippo.
Working with Python's multiprocessing package, is it better to define a big function and create the object using the target argument in the call to Process(), or to create your own process class by inheriting from Process class?
I often wondered why Python's doc page on multiprocessing only shows the "functional" approach (using target parameter). Probably because terse, succinct code snippets are best for illustration purposes. For small tasks that fit in one function, I can see how that is the preferred way, ala:
from multiprocessing import Process
def f():
print('hello')
p = Process(target=f)
p.start()
p.join()
But when you need greater code organization (for complex tasks), making your own class is the way to go:
from multiprocessing import Process
class P(Process):
def __init__(self):
super(P, self).__init__()
def run(self):
print('hello')
p = P()
p.start()
p.join()
Bear in mind that each spawned process is initialized with a copy of the memory footprint of the master process. And that the constructor code (i.e. stuff inside __init__()) is executed in the master process -- only code inside run() executes in separate processes.
Therefore, if a process (master or spawned) changes it's member variable, the change will not be reflected in other processes. This, of course, is only true for bulit-in types, like bool, string, list, etc. You can however import "special" data structures from multiprocessing module which are then transparently shared between processes (see Sharing state between processes.) Or, you can create your own channels of IPC (inter-process communication) such as multiprocessing.Pipe and multiprocessing.Queue.
This question already has answers here:
How do I forward-declare a function to avoid `NameError`s for functions defined later?
(17 answers)
Closed 8 months ago.
Is it possible to call a function without first fully defining it? When attempting this I get the error: "function_name is not defined". I am coming from a C++ background so this issue stumps me.
Declaring the function before works:
def Kerma():
return "energy / mass"
print Kerma()
However, attempting to call the function without first defining it gives trouble:
print Kerma()
def Kerma():
return "energy / mass"
In C++, you can declare a function after the call once you place its header before it.
Am I missing something here?
One way that is sort of idiomatic in Python is writing:
def main():
print Kerma()
def Kerma():
return "energy / mass"
if __name__ == '__main__':
main()
This allows you to write you code in the order you like as long as you keep calling the function main at the end.
When a Python module (.py file) is run, the top level statements in it are executed in the order they appear, from top to bottom (beginning to end). This means you can't reference something until you've defined it. For example the following will generate the error shown:
c = a + b # -> NameError: name 'a' is not defined
a = 13
b = 17
Unlike with many other languages, def and class statements are executable in Python—not just declarative—so you can't reference either a or b until that happens and they're defined. This is why your first example has trouble—you're referencing the Kerma() function before its def statement has executed and body have been processed and the resulting function object bound to the function's name, so it's not defined at that point in the script.
Programs in languages like C++ are usually preprocessed before being run and during this compilation stage the entire program and any #include files it refers to are read and processed all at once. Unlike Python, this language features declarative statements which allow the name and calling sequence of functions (or static type of variables) to be declared (but not defined), before use so that when the compiler encounters their name it has enough information to check their usage, which primarily entails type checking and type conversions, none of which requires their actual contents or code bodies to have been defined yet.
This isn't possible in Python, but quite frankly you will soon find you don't need it at all. The Pythonic way to write code is to divide your program into modules that define classes and functions, and a single "main module" that imports all the others and runs.
For simple throw-away scripts get used to placing the "executable portion" at the end, or better yet, learn to use an interactive Python shell.
If you are willing to be like C++ and use everything inside a functions. you can call the first function from the bottom of the file, like this:
def main():
print("I'm in main")
#calling a although it is in the bottom
a()
def b():
print("I'm in b")
def a():
print("I'm in a")
b()
main()
That way python is first 'reading' the whole file and just then starting the execution
Python is a dynamic programming language and the interpreter always takes the state of the variables (functions,...) as they are at the moment of calling them. You could even redefine the functions in some if-blocks and call them each time differently. That's why you have to define them before calling them.