I want to save a function using dill.
This function is using a function from a specific library (e.g re)
To save the dill/pickle file, I use the following :
import dill
import re
def fct(a):
#The function uses the re library
return ...
filename = 'test.pickle'
dill.dump(fct, open(filename, 'wb'))
However, when I try to load this function on a different notebook (using dill.load), I got the following error:
name 're' is not defined
I am guessing that dill.dump did not save the import with the function, how could I solve that? I tried including the import inside the function but that does not work.
Adding the following in the first script before dumping the function in the dill file seems to work :
dill.settings['recurse'] = True
Related
I am working on a program in python that uses both the pandas and io library's and I am running into a issue with image = Image.open(io.BytesIO(imageData)) where the io.BytesIO is trying to be used as pandas.io instead of the .BytesIO function from the io library. This leads me to ask if their is a way to change witch function corresponds to this line. I am working in pyCharm 2022.1.4.
I imported the io package as ior
import io as ior
then just changed io.BytesIO to ior.BytesIO
I currently have the problem that I cant use inspect.getsource (OSError: could not get source code) inside code executed through exec().
For example when i feed the following code as a string into exec(code)
import inspect
def sample(p1):
print(p1)
return 1
print(inspect.getsource(sample))
So does anyone know how i could get the source inside the exec()?
I use python 3.8 64 Bit
Thank you for your answers.
Edit: First of all, thanks for the answers and the help to get the right formating for my question, but the function and the inspect.getsource() have to be inside the exec() string.
Regrettably, as I'm sure you've realised, this simply is not possible using exec...but it is possible using exec_module.
If you are not required to use exec specifically, this may work for you:
import tempfile
from importlib import util
import importlib.machinery
raw = """
def sample(p1):
print(p1)
return 1
import inspect
print(inspect.getsource(sample))
"""
# Create a temporary source code file
with tempfile.NamedTemporaryFile(suffix='.py') as tmp:
tmp.write(raw.encode())
tmp.flush()
# Now load that file as a module
spec = util.spec_from_file_location('tmp', tmp.name)
module = util.module_from_spec(spec)
spec.loader.exec_module(module)
# ...or, while the tmp file exists, you can query it externally
import inspect
print(inspect.getsource(module.sample))
The TLDR is, without a source file, it is just flat out not-possible to use getsource to read the source code of a function, because the implementation reads the source code from the file defined in f.__code__.co_filename on the function (read-only, so monkey patching is impossible).
However, if you write the source code to a temporary file, you can load that file as a module (exec_module is practically identical to the effect of exec, you just have to do a few other steps first).
Note: tmp.flush() <-- don't forget this, tmp.write() doesn't write until you flush, and you'll be 'loading' an empty file if you don't flush it first.
If your code is like this:
exec("""\
import inspect
def sample(p1):
print(p1)
return 1
print(inspect.getsource(sample))\
""")
And the exception is OSError: could not get source code then that probably means that inspect.getsource() requires a
file's source to inspect. exec is a dynamic execution of python code so the sample function that's being defined from exec is not on the file's source!
Something like this:
def sample(p1):
print(p1)
return 1
exec("""\
import inspect
print(inspect.getsource(sample))\
""")
would work because the function sample is defined in the source!
Hope that makes sense and answers your question!
~ PanTrakX
I want to use NumPy in a Python script that uses pandas to process an Excel file. However, one of my constraints is that my file must be named keyword.py, which causes an import error. The import error is traced back to a line from keyword import iskeyword as _iskeyword in C:\Python27\lib\collections.py, which I assume causes an error because my own keyword.py is overriding the default keyword module. Is there any way to avoid this collision?
Not pretty, but a keyword.py of
if True:
import imp, sys
keyword_loc = imp.find_module("keyword", sys.path[1:])[1]
imp.load_source("keyword", keyword_loc)
import collections
print(collections.Counter)
fails with an AttributeError if we replace True with False, but gives me
(2.7) dsm#notebook:~/coding/kw$ python keyword.py
<class 'collections.Counter'>
as is. This works by finding out where the original keyword library is and manually importing it. After this, any following attempts to import keyword will see that it's already there.
For working with a single script, you can remove the current directory from the import search path. That might be sufficient for working on your TopCoder problem, but I wouldn't recommend it as a long-term solution. (Long-term: don't use file names that mirror the standard library.)
If the following script is called keyword.py, it can be run and the import of collections will not trigger an error.
# keyword.py
# Remove the current directory from the import search path
# This is a hack, but it will be sufficient for working with a
# single script that doesn't import any other modules from the
# current directory.
import sys
sys.path = sys.path[1:]
import collections
print(collections)
I have a vaguely defined function of a class Graph of a module I call gt (it is graph-tool). so i declare g = gt.graph() then want to use g.degree_property_map but do not know how. Therefore I want to see where in code g.degree_property_map or in this case just the function, is defined. How can I find that? I'm working on command line on a vm.
Thanks
For reference the library in question is graph-tool - http://projects.skewed.de/graph-tool/
Also I am currently importing it using from graph_tool.all import * . that is of course somewhat of a problem.
You could use inspect.getsource(gt.Graph.degree_property_map). (You have to import inspect.)
Of course, what you pass into getsource() will change depending on how you imported Graph. So if you used from graphtools.all import *, you'd just need to use inspect.getsource(Graph.degree_property_map).
If you open interactive python (type python and hit ENTER on the command line), you should be able to run the command help(<graph's module name>), then, under the FILE section of the help documentation that is generated, you should see the absolute path to the code you are interested in.
For example, I just ran:
import numpy
help(numpy)
# Returned documentation containing:
# FILE
# /usr/lib/python2.7/dist-packages/numpy/__init__.py
Also,
import my_module # A module I just created that contains the "Line" class
help(my_module)
# Returned documentation containing:
# FILE
# /home/<my user name>/Programming/Python/my_module.py
If it is a normal function (not a builtin, ufunc, etc) you can try using the func_code attribute
For example:
>>> inspect.iscode
<function iscode at 0x02EAEF30>
>>> inspect.iscode.func_code
<code object iscode at 02EB2B60, file "C:\Python27\lib\inspect.py", line 209>
Never mind I just did help(graph_tool) and manually jumped through code. Thanks for the help though!
I created a function to read in a csv file and then write some of the data from the csv file into another file. I had to manipulate some of the data in the original csv file before I write it. I will probably have to do that manipulation a lot during the next couple months so I wrote another function to just do that manipulation, but I am having trouble calling in the function in my other function.
this is the function I am trying to call in:
import sys
import math
def convLatLon(measurement): # should be in '##.####' format
tpoint=float(measurement)
point_deg=math.floor(measurement) # find the degree for lat and lon
dpoint=tpoint-point_deg #subtract the lat value from just the degs to get the decimal fraction
pointmin=dpoint * 60 # find the degree mins
npoint= str(point_deg) + str(pointmin)
print(npoint)
How do I call in this function in another function? They are currently in the same directory. I am used to Matlab and thought it would be a simple call in command but I can not seem to figure it out. Any help will be greatly apprectiated.
Shay
You can import the file (same as you imported sys and math). If your function is in a file called util.py:
import util
util.convLatLon(37.76)
If the file is in another directory, the directory must be in your PYTHONPATH.
Is:
from <filename> import convLatLon
What you're looking for?
Sounds like you need to import the file.
If your function is defined in file myfile.py and you wan`t to use it from myotherfile.py, you should import myfile.py, like this:
import myfile
and then you can use the function like this:
result = myfile.myfunc(myparms)
If you want to get rid of the myfile prefix, import it like this:
from myfile import myfunc
If you have this function saved in a file called myconv.py
from myconv.py import convLatLon
convLatLon("12.3456")