I'm using python to communicate with a certain piece of software through the provided API. As the API-code is written in C#, I use pythonnet to import the DLL and subsequently use it. It is beneficial to do so using e.g. Jupyter Lab or Jupyter Notebook when optimizing the code, as you can easily compare the results within your code and within the software. However, I run into an issue with cleanup. The API requires you to establish connection by running the following code
import clr
clr.AddReference('API')
api = __import__('API', globals(), locals(), [], 0)
connection = api.connection()
app = connection.connect()
Now you can communicate with the software using app. The main reason for my issue it that you are only allowed to have one app in the CLR. If you want to create a new one, you should call app.close() and subsequently newapp = connection.connect(). What happens when you create newapp without calling app.close() is not clearly defined. I'm unsure how C# would handle this, would it overwrite the app in memory, would app now also point to newapp, or something else? With that, I'm even more unsure how python+CLR handles it.
To ensure that your would always work with a properly connected app, I have created a class that only allows one instance of app to be present. This restriction is implemented by evaluating connection.Alive through the API, which is True when an app is spawned and not properly closed yet. The class is similar to:
class APIWrapper:
def __init__(self, api):
self.API = api
self.Connection = api.connection()
def connect():
if self.Connection.Alive:
raise RunTimeError('Only one live application is allowed at runtime')
app = self.Connection.connect()
return app
While this works fine, my issue arises when I accidentally do something like:
wrap = APIWrapper()
wrap.connect()
When doing so, the app goes live and wrap.Connection.Alive evaluates to True. However, as I do not assign the return of wrap.connect() to a variable, I cannot close it using app.close(). For example, if I do:
wrap = APIWrapper()
print(wrap.Connection.Alive) # -> False
app = wrap.connect()
print(wrap.Connection.Alive) # -> True
app.close()
print(wrap.Connection.Alive) # -> False
wrap.connect()
print(wrap.Connection.Alive) # -> True
I cannot close the connection anymore. I have thought about altering the class so the to just bind wrap.connect() to wrap.App and allow access through the attribute. This would solve the problem of losing the app, but I prefer to not have to call wrap.App continuously for code readability. Additionally, I am just wondering if there is a proper way of handling these finalization problems?
First of all, if calling wrap.connect() without storing the return value anywhere is the problem, then there is an easy solution for it: don't do it! It looks like the connection is a resource, so you must keep track of it to release it properly when the time comes.
In your example, what should happen to the previously created connection, when somebody calls connect() again?
Second, in Python there are two ways to explicitly keep track of resources:
with statements + context managers (highly recommended). In this case you would need to implement a context manager on your wrappers.
__del__ function, that you can define, that will be called when the object is no longer needed. This one you should avoid, because it will execute at arbitrary time, meaning when you try to create a new connection, the old one might still be around, because Python did not realize it should call __del__ yet.
Another alternative is to make a singleton.
Related
I've got a LibreOffice python script that uses serial IO. On my systems, opening a serial port is a very slow process (around 1 second), so I'd like to keep the serial port open, and just send stuff as required.
But LibreOffice python apparently reloads the python framework every time a call is made. Unlike most python implementations, where the process is persistent, and un-enclosed code in a module is run once, when the module is imported.
Is there a way in LibreOffice python to persist objects between calls?
SerialObject=None
def return_global():
return str(SerialObject) #always returns "None"
def init_serial_object():
SerialObject=True
It looks like there is a simple bug. Add global and then it works.
However, it may be that the real problem is your setup. Here is a working example. Put the code in a file called inctest.py under $LIBREOFFICE_USER_DIR/Scripts/python/pythonpath/incmod/.
def init_serial_object():
global SerialObject
SerialObject=True
def moduleVersion():
return "2.0" #change to verify that this is the most recently updated code
The code to call it should be located in the user profile directory (that is, location=user).
from incmod import inctest
inctest.init_serial_object()
msgbox("{}: {}".format(
inctest.moduleVersion(), inctest.return_global()))
Run the script by going to Tools > Macros > Run Macro and find it under My Macros.
Be aware that inctest.py will not always get reloaded. There are two ways to reload it: restart LibreOffice, or force python to reload it by doing del sys.modules[mod].
The moduleVersion() function is not necessary but helps you see when the module is getting reloaded — make changes to that line and then see whether the output changes.
I have a program that, for data security reasons, should never persist anything to local storage if deployed in the cloud. Instead, any input / output needs to be written to the connected (encrypted) storage instead.
To allow deployment locally as well as to multiple clouds, I am using the very useful fsspec. However, other developers are working on the project as well, and I need a way to make sure that they aren't accidentally using local File I/O methods - which may pass unit tests, but fail when deployed to the cloud.
For this, my idea is to basically mock/replace any I/O methods in pytest with ones that don't work and make the test fail. However, this is probably not straightforward to implement. I am wondering whether anyone else has had this problem as well, and maybe best practices / a library exists for this already?
During my research, I found pyfakefs, which looks like it is very close what I am trying to do - except I don't want to simulate another file system, I want there to be no local file system at all.
Any input appreciated.
You can not use any pytest addons to make it secure. There will always be ways to overcome it. Even if you patch everything in the standard python library, the code always can use third-party C libraries which can't be patched from the Python side.
Even if you, by some way, restrict every way the python process can write the file, it will still be able to call the OS or other process to write something.
The only ways are to run only the trusted code or to use some sandbox to run the process.
In Unix-like operating systems, the workable solution may be to create a chroot and run the program inside it.
If you're ok with just preventing opening files using open function, you can patch this function in builtins module.
_original_open = builtins.open
class FileSystemUsageError(Exception):
pass
def patched_open(*args, **kwargs):
raise FileSystemUsageError()
#pytest.fixture
def disable_fs():
builtins.open = patched_open
yield
builtins.open = _original_open
I've done this example of code on the basis of the pytest plugin which is written by the company in which I work now to prevent using network in pytests. You can see a full example here: https://github.com/best-doctor/pytest_network/blob/4e98d816fb93bcbdac4593710ff9b2d38d16134d/pytest_network.py
I use python and bokeh to implement streamed live graphing. I want to include several live graphs into a gridplot and run into a kind of "deathlock".
The graphs (there are a lot of them) are created by different classes and the figure objects are returned and then used as input to the gridplot() function.
For live graphing curdoc().add_periodic_callback(update1, 300) references the update routine. I call the update routines of the other graphs directly from update1(). This works but gives me following error continuously:
`raise RuntimeError("_pending_writes should be non-None when we have a document lock, and we should have the lock when the document changes")
This is expected behavior since data of the other graphs is altered from the 'outside' of their object and from an 'unregistered update routine'. I want to get rid of this error.
In my main object (where the layout is pieced together and curdoc().add_root() is called) I intend to register the other graphs update routines (which have to be regular object routines, so that they can be referenced.) via curdoc().add_periodic_callback(). the problem with this approach is, that the objects update functions take the self parameter and bokeh does not accept that.
Yet I can not do it without self, cause update() needs to reference the source.stream object.
I have no clue how to solve this or do it the 'correct' way. Suggestions are appreciated.
thanks
for clarification:
main object:
def graph(self):
.... bokeh code
#count()
def update(t):
.... update code
curdoc().add_root(gridplot([[l[0]], [l[1]]], toolbar_location="left", plot_width=1000))
curdoc().add_periodic_callback(update, 300)
this works
generic other object
def graph(self):
.... bokeh code
def update(self,t): ....
main object:
curdoc().add_periodic_callback(other_object.update, 300)
this does NOT work.
"_pending_writes should be non-None when we have a document lock, and we should have the lock when the document changes"
Disclaimer: I've been dealing with this error in my own work for two weeks now, and finally resolved the issue today. (: It's easy when every sample you see comes with a csv file that's read and pushed to the doc, all in the same thread, but when things get real and you have a streaming client doing the same thing, suddenly everything stops working.
The general issue is, Bokeh server needs to keep its version of the document model in sync with what Bokeh client has. This happens through a series of events and communication happening between the client (Javascript running in the browser) and the server (inside an event loop that we will get to later on).
So every time you need to change the document, which essentially affects the model, the document needs to be locked (the simplest reason I could think of is concurrency). The simplest way to get around this issue, is to tell the Bokeh document instance you hold, that you are in need of making a change - and request a callback, so Bokeh manages when to call your delegate and allow you to update the document.
Now, with that said, there are few methods in bokeh.plotting.Document that help you request a callback.
The method you would want to probably use based on your use case, for example, if you need an ASAP callback, is add_next_tick_callback.
One thing to remember is that, the reference/pointer to your doc must be correct.
In order to make sure of that, I did wrap all my charting application into a class, and kept an instance of doc internally to access its add_next_tick_callback when new data is received. The way I could point at the right instance, was to initialize my Bokeh app using bokeh.server.server.Server - when it initializes the app, you will receive a doc variable that it's created before starting the server - that would be the right reference to the doc you present in the app. One benefit for having this "chart initializer" in a class, is that you can instantiate it as many times as you may need to construct more charts/documents.
Now, if you are a fan of data pipelines, and streaming, and use something like StreamZ to stream the data to the Pipe or Buffer instance you may have, you must remember one thing:
Be aware of what happens asynchronously, in a thread, or outside of it. Bokeh relies on tornado.ioloop.IOLoop for the most part, and if you are anywhere near running things asynchronously, you must have come across asyncio.
The event loops on these two modules can conflict, and will affect how/when you can change the document.
If you running your streaming in a thread (as the streaming client I wrote did..), make sure that thread has a current loop, otherwise you will face other similar issues. Threads can cause conflicts with internally created loops, and affect how they interact with each other.
With something like the following:
asyncio.set_event_loop(asyncio.new_event_loop())
Finally, be aware of what #gen.coroutine does in tornado. Your callbacks for the streaming, the way I understood, must be decorated with #gen.coroutine if you are doing things asynchronously.
I am trying to create a callback in my Maya plugin that runs every time a new object is created. The callback works but the data object passed is of NoneType. I want to access the newly created object in my callback function, how do i do that?
g_Callbacks = list()
def initializePlugin( obj ):
g_Callbacks.append( OpenMaya.MEventMessage.addEventCallback( "DagObjectCreated", callback ) )
...
def callback( data ):
print data.apiTypeStr()
From the comments it sounds like there are two separate issues.
Notificiations on object creation are 'free', in the sense that you can do this without the API by using a scriptJob. Here's a trivial example
def new_object_callback():
print "created", cmds.ls(sl=True)
cmds.scriptJob(e=('DagObjectCreated', new_object_callback))
The callback here is just a plain python script , but it could be a function created by an MPXCommand (that link is a good intro to a very simple command plugin, btw).
There is one limitation here: the creation callback will fire once per undo block. A menu item, button or script creates a single undo --- which means that an action which creates multiple objects will only get that notification once (and the example code above will only print out the message for the last created object).
Another way of reading the question is how to fire a callback when you create the object yourself in a plugin. That's a simpler problem, although plugins for creating objects are kind of wordy to write. (A decent intro here -- the python stuff is interleaved with C++, because the docs for all of this are still written for C++) Every plugin node class has to have a function called initialize, which will be called every time the plugin creates a new node, so you can use that to call any code you need to run at startup.
OpenMaya is a real pain in the butt, since you're basically writing C++ through Python. Here's a couple of decent references:
http://www.amazon.com/Maya-Python-Games-Film-Reference/dp/0123785782
http://www.amazon.com/Practical-Programming-Python-Robert-Galanakis/dp/1849694729
My django app contains a loop, which is launched by the following code in urls.py:
def start_serial():
rfmon = threading.Thread(target=rf_clicker.RFMonitor().run)
rfmon.daemon = True
rfmon.start()
start_serial()
The loop inside this subthread references a global variable defined in global_vars.py. I would like to change to value of this variable in a view, but it doesn't seem to work.
from views.py:
import global_vars
def my_view(request):
global_vars.myvar = 2
return httpResponse...
How can a let the function inside the loop know that this view has been called?
The loop listens for a signal from a remote, and based on button presses may save data to the database. There are several views in the web interface, which change the settings for the remotes. While these settings are being changed the state inside the loop needs to be such that data will not be saved.
I agree with Ignacio Vazquez-Abrams, don't use globals.
Especially in your use case. The problem with this approach is that, when you deploy your app to a wsgi container or what have you, you will have multiple instances of your app running in different processes, so changing a global variable in one process won't change it in others.
And I would also not recommend using threads. If you need a long running process that handles tasks asynchronously(which seems to be the case), consider looking at Celery( http://celeryproject.org/). It's really good at it.
I will admit to having no experience leveraging them, but if you haven't looked at Django's signaling capabilities, it seems like they would be a prime candidate for this kind of activity (and more appropriate than global variables).
https://docs.djangoproject.com/en/dev/ref/signals/