I want to write a function 'backup(filename)' to store all the working data (objects?) under current environment of Python, and 'restore(filename)' to restore the data/object again. Just like R's save.image(file="workspace21.RData") and load(file="workspace21.RData"), which can snapshot the system.
How to write the "backup" & "restore" ? Or is there any package existed can do that ?
A bit out of the box, but if this is important and you really need a full solution, you can run python inside a virtual machine and use snapshots to save session state. Whether it is practical or not depends on your use case.
pickle module seems like a solution but it cannot really save the whole environment for you. Try for example this:
import pickle
def backup(fileName):
pickle.dump(globals(), open(fileName,'w'), pickle.HIGHEST_PROTOCOL)
def restore(fileName):
globals().update(pickle.load(open(fileName,"rb")))
This will not work because module objects are not picklable. You will also have problems with open file descriptors and other objects. See answers on this question for (partial) solutions:
How can I save all the variables in the current python session?
So, while you cannot have a general solution to your problem, you can write something which will save some of your (global) objects, using the snippet above as a starting point, if that helps.
You should take a look on pickle documentation. You also have an example that can help you to do what you're thinking on.
Related
I am new to Pycharm; however, I want to take advantage of my R and Python knowledge. I am a big fan of both languages, and I am constantly learning more about them.
I am hoping to pass an R variable to a Python variable, similar to Jupyter Notebook.
I could not find any example code anywhere of doing this.
R code
x <- 5
python code
# Some conversion method needs to be added
print(x)
Python Console
>>>5
This is possible because Jupyter provides its own Kernel that code runs in. This kernel is able to translate variables between languages.
Pycharm does not provide a kernel, and instead executes Python code directly through an interpreter on your system. It's similar to doing python my_script.py. AFAIK vanilla Pycharm does not execute R at all.
There are plugins for Pycharm that support R and Jupyter notebooks. You might be able to find one that does what you want.
I usually solve this problem by simply adhering to the Unix philosophy:
Rscript foo.R | python bar.py
where foo.R prints the data to STDOUT and bar.py expects to read it from STDIN (you could also read/write from files). This approach of course requires you to define a clear interface of data between the scripts, rather than passing all the variables indiscriminately, which I don't personally mind because I do that anyway as a matter of design best practices.
There are also packages like reticulate which will allow you to call one language from the other. This might be more useful if you like to switch languages a lot.
Thanks for the above answer! That is great. I did find solution that could help other people that use pycharm.
If you have installed the R plug in option, you can use the following code.
Python code to save to feather file:
```
import pyarrow.feather as feather
data = pd.read_csv('C:/Users/data.csv')
feather.write_feather(data, 'C:/Users/data.feather')
```
R code to retrieve feather file
```
library(tidyverse)
library(feather)
library(arrow)
data <- arrow::read_arrow('C:/Users/data.feather')
print(data)
```
However, this process seems very similar to writing a file to csv in Python, then uploading the csv into R. There could be some type of lightweight storage difference, speed processing, and language variable agnostic/class translation. Below is the official documentation: Git: https://github.com/wesm/feather
Apache Arrow: https://arrow.apache.org/docs/python/install.html
RStudio: https://www.rstudio.com/blog/feather/
Context:
I currently have a program which loads a set of plugins from their file paths (in a mapped network drive) using the method shown in another SO thread. These plugins are designed to be rolling release, which means I need constant access to writing them. The current mechanism locks the files so I have to ask everyone to close the software so I can update the files.
The question:
I was wondering if there was a way to, possibly using a similar method to that linked above, import a file from an io.BytesIO object of the plugin's raw contents (hence unlocking the file for me to make changes as I please).
More generally:
More specifically, can I keep the raw module contents in memory without touching a physical disk? If such a thing is not possible, is there a way to fully load these modules into memory so I can then unlock the files being imported?
As I have stated in my comment, I understand you can mount a virtual-filesystem on a Linux-based OS (which could have solved my problem), though sadly I developing for Windows and Microsoft can never make your life easy! :-)
Note:
I am not asking where I can copy these files to import them from a local version (e.g. temp, cache, etc.).
I understand this is quite a specialist question so any help is much appreciated
While not being from an io.BytesIO object as I originally asked for, I was able to import a module from its source after finding this incredibly helpful article. I have not copied the code here as it is quite large, though I was able to get it to successfully import the virtual module.
The following code is after I modified the loader to remove the common prefix, and creates a class of the module by first executing the source, getting the globals from it and finally using Python's type method to create the module class.
It is not particularly pretty and definitely breaks some Python style recommendations, so I am definitely open to improvements!
source = """def hello():
print("I don't want to say hi to the world")"""
name = "my_module"
glo = {}
exec(source, glo)
injector = DependencyInjector()
injector.provide(name, type(name, (), glo))
injector.install()
foo = __import__(name)
foo.hello()
I've got a kind of weird question--but would be immensely useful if it is possible--in Maya using Python, can I take in several points of user input and have Python create a separate script for me? In this instance, I want to take in controller and locator names and have Python spit out a complete IKFK match script also in Python (it's really just a lot of getAttr and setAttr commands, although with 6 if statements per limb for PV matching.) The only other wrinkle there is that it has to be able to prefix hierarchy names in the script if necessary if the rig is imported into a new scene rather than just opened. There's an expression component to my switches that it would be nice if Python could make for me, too.
Is this possible or am I crazy?
That's no problem. Just write a textfile with a .py extension into a path where maya can find it. Then you have to import it somewhere. Creating expressions is not a problem either.
Maybe it could make sense to think about the approach you choose. Imagine you have written a dozen of these new python files and you discover a problem in the script, you will have to redo it. I'd try to collect all data and only write the required informations into a textfile e.g. in json format. Then you can read the data and rebuild your skeletons.
Is there a way to save the current Python session? I would like to be able to save all my Python variables, functions, and imported libraries. If possible, having the history saved would be nice. I have looked all over Google and Stackoverflow for a simple command to do this, but have found nothing quite fitting what I want. Some suggested downloading new software, which I don't want. Others had big scripts that they wrote to do this for them.
I'm looking for a no-nonsense easy to use command. Is there one? If not, how would you suggest to make one?
UPDATE: readline.write_history_file() and readline.read_history_file() save the history.
I think the closest thing you're going to find is ipython. I'm not sure if it does the variable saving thing, but I believe it does everything else you're asking for.
1.) After typing all commands just do CTRL+S.
2) Give a file name and say OK.
This works for Python Shell.
I am about to get a bunch of python scripts from an untrusted source.
I'd like to be sure that no part of the code can hurt my system, meaning:
(1) the code is not allowed to import ANY MODULE
(2) the code is not allowed to read or write any data, connect to the network etc
(the purpose of each script is to loop through a list, compute some data from input given to it and return the computed value)
before I execute such code, I'd like to have a script 'examine' it and make sure that there's nothing dangerous there that could hurt my system.
I thought of using the following approach: check that the word 'import' is not used (so we are guaranteed that no modules are imported)
yet, it would still be possible for the user (if desired) to write code to read/write files etc (say, using open).
Then here comes the question:
(1) where can I get a 'global' list of python methods (like open)?
(2) Is there some code that I could add to each script that is sent to me (at the top) that would make some 'global' methods invalid for that script (for example, any use of the keyword open would lead to an exception)?
I know that there are some solutions of python sandboxing. but please try to answer this question as I feel this is the more relevant approach for my needs.
EDIT: suppose that I make sure that no import is in the file, and that no possible hurtful methods (such as open, eval, etc) are in it. can I conclude that the file is SAFE? (can you think of any other 'dangerous' ways that built-in methods can be run?)
This point hasn't been made yet, and should be:
You are not going to be able to secure arbitrary Python code.
A VM is the way to go unless you want security issues up the wazoo.
You can still obfuscate import without using eval:
s = '__imp'
s += 'ort__'
f = globals()['__builtins__'].__dict__[s]
** BOOM **
Built-in functions.
Keywords.
Note that you'll need to do things like look for both "file" and "open", as both can open files.
Also, as others have noted, this isn't 100% certain to stop someone determined to insert malacious code.
An approach that should work better than string matching us to use module ast, parse the python code, do your whitelist filtering on the tree (e.g. allow only basic operations), then compile and run the tree.
See this nice example by Andrew Dalke on manipulating ASTs.
built in functions/keywords:
eval
exec
__import__
open
file
input
execfile
print can be dangerous if you have one of those dumb shells that execute code on seeing certain output
stdin
__builtins__
globals() and locals() must be blocked otherwise they can be used to bypass your rules
There's probably tons of others that I didn't think about.
Unfortunately, crap like this is possible...
object().__reduce__()[0].__globals__["__builtins__"]["eval"]("open('/tmp/l0l0l0l0l0l0l','w').write('pwnd')")
So it turns out keywords, import restrictions, and in-scope by default symbols alone are not enough to cover, you need to verify the entire graph...
Use a Virtual Machine instead of running it on a system that you are concerned about.
Without a sandboxed environment, it is impossible to prevent a Python file from doing harm to your system aside from not running it.
It is easy to create a Cryptominer, delete/encrypt/overwrite files, run shell commands, and do general harm to your system.
If you are on Linux, you should be able to use docker to sandbox your code.
For more information, see this GitHub issue: https://github.com/raxod502/python-in-a-box/issues/2.
I did come across this on GitHub, so something like it could be used, but that has a lot of limits.
Another approach would be to create another Python file which parses the original one, removes the bad code, and runs the file. However, that would still be hit-and-miss.