Develop Reference

python import a module with source code in temporary file

import tempfile
tmp = tempfile.NamedTemporaryFile(delete=True)
try:
# do stuff with temp
tmp.write(b'def fun():\n\tprint("hello world!")\n')
if __name__ == '__main__':
func = __import__(tmp.name)
func.fun()
finally:
tmp.close() # deletes the file
So I want to create a temporary file, add some source code to it and then import the module and call the function, but I always run into this error:
ModuleNotFoundError: No module named '/var/folders/3w/yyp887lx4018h9s5sr0bwhkw0000gn/T/tmp84bk0bic'
It doesn't seem to find the module of the temporary file. How do I solve this?

There are a few problems with your code:
Your filename does not end with .py, but Python modules are expected to. You can fix this by setting suffix='.py' in NamedTemporaryFile().
__import__() is not the right way to load a module from a full path. See here: How to import a module given the full path?
You do not flush after writing and before importing, so even if Python does find the file, it may well be empty. Add tmp.flush() after writing to fix this.

Importing can only be done from certain directories which are part of the PYTHON_PATH. You can extend that. Then you will have to use __import__() with a module name (not a path in the file system). You will have to deal with the suffix for the temp file.
I implemented a simple version using the local directory for the temp module file and a version using a proper tempfile:
#!/usr/bin/env python3
import sys
import os
import tempfile
SCRIPT = '''\
def fun():
print("hello world!")
'''
# simple version using the local directory:
with open('bla.py', 'w') as tmp_module_file:
tmp_module_file.write(SCRIPT)
import bla
bla.fun()
# version using the tempfile module:
tmpfile = tempfile.NamedTemporaryFile(suffix='.py', delete=True)
try:
tmpfile.write(SCRIPT.encode('utf8'))
tmpfile.flush()
tmpmodule_path, tmpmodule_file_name = os.path.split(tmpfile.name)
tmpmodule_name = tmpmodule_file_name[:-3] # strip off the '.py'
sys.path.append(tmpmodule_path)
tmpmodule = __import__(tmpmodule_name)
finally:
tmpfile.close()
tmpmodule.fun()

Python ConfigParser cannot search .ini file correctly (Ubuntu 14, Python 3.4)

Problem:
The code compiles fine but when ever i call the read_db_config function i get "Exception: mysql not found in the mysql_config.ini file"
The file is in the same directory but the main script runs two directories up using
import sys
from Config.MySQL.python_mysql_dbconfig import read_db_config
I am new to python and have searched everywhere but i cannot seem to pinpoint my issue
Code:
from ConfigParser import ConfigParser
def read_db_config(filename='mysql_config.ini', section='mysql'):
# create parser and read ini configuration file
parser = ConfigParser()
parser.read(filename)
# get section, default to mysql
db = {}
if parser.has_section(section):
items = parser.items(section)
for item in items:
db[item[0]] = item[1]
else:
raise Exception('{0} not found in the {1} file'.format(section, filename))
return db
mysql_config.ini:
[mysql]
database = testdba
user = root
password = test
unix_socket = /opt/lampp/var/mysql/mysql.sock

if you use relative paths for file or directory names python will look for them (or create them) in your current working directory (the $PWD variable in bash).
if you want to have them relative to the current python file, you can use (python 3.4)
from pathlib import Path
HERE = Path(__file__).parent.resolve()
CONFIG_PATH = HERE / '../etc/mysql_config.ini'
or (python 2.7)
import os.path
HERE = os.path.abspath(os.path.dirname(__file__))
CONFIG_PATH = os.path.join(HERE, '../etc/mysql_config.ini')
if your mysql_config.ini file lives in the etc directory below your python script.
you could of course always use absolute paths (starting with a /; i.e. /home/someuser/mysql_config.ini).

I ran it again but with the modification of adding
parser = configparser.ConfigParser()
parser['mysql'] = {'database': 'testdba',
'user' : 'root',
'password' : 'test',
'unix_socket' : '/opt/lampp/var/mysql/mysql.sock'}
with open('mysql_config.ini', 'w') as configfile:
parser.write(configfile
and I found that this created the file "mysql_config.ini" not in the directory where the python "read_db_config" was stored but the parent directory of main python module which calls this module. I searched it up a bit and figured out a perment solution the lets me keep the "mysql_config.ini" where I wish.
import configparser
def read_db_config(dirname = '/opt/Python_MySQL_Email_Receipt_Client/Config/MySQL/', filename='mysql_config.ini', section='mysql'):
# create parser and read ini configuration file
parser = configparser.ConfigParser()
parser.read(dirname + filename)
# get section, default to mysql
db = {}
if parser.has_section(section):
items = parser.items(section)
for item in items:
db[item[0]] = item[1]
else:
raise Exception('{0} not found in the {1} file'.format(section, filename))
return db

How to read a config file using python

I have a config file abc.txt which looks somewhat like:
path1 = "D:\test1\first"
path2 = "D:\test2\second"
path3 = "D:\test2\third"
I want to read these paths from the abc.txt to use it in my program to avoid hard coding.

In order to use my example, your file "abc.txt" needs to look like this.
[your-config]
path1 = "D:\test1\first"
path2 = "D:\test2\second"
path3 = "D:\test2\third"
Then in your code you can use the config parser.
import ConfigParser
configParser = ConfigParser.RawConfigParser()
configFilePath = r'c:\abc.txt'
configParser.read(configFilePath)
As human.js noted in his comment, in Python 3, ConfigParser has been renamed configparser. See Python 3 ImportError: No module named 'ConfigParser' for more details.

You need a section in your file:
[My Section]
path1 = D:\test1\first
path2 = D:\test2\second
path3 = D:\test2\third
Then, read the properties:
import ConfigParser
config = ConfigParser.ConfigParser()
config.readfp(open(r'abc.txt'))
path1 = config.get('My Section', 'path1')
path2 = config.get('My Section', 'path2')
path3 = config.get('My Section', 'path3')

If you need to read all values from a section in properties file in a simple manner:
Your config.cfg file layout :
[SECTION_NAME]
key1 = value1
key2 = value2
You code:
import configparser
config = configparser.RawConfigParser()
config.read('path_to_config.cfg file')
details_dict = dict(config.items('SECTION_NAME'))
This will give you a dictionary where keys are same as in config file and their corresponding values.
details_dict is :
{'key1':'value1', 'key2':'value2'}
Now to get key1's value :
details_dict['key1']
Putting it all in a method which reads sections from config file only once(the first time the method is called during a program run).
def get_config_dict():
if not hasattr(get_config_dict, 'config_dict'):
get_config_dict.config_dict = dict(config.items('SECTION_NAME'))
return get_config_dict.config_dict
Now call the above function and get the required key's value :
config_details = get_config_dict()
key_1_value = config_details['key1']
Generic Multi Section approach:
[SECTION_NAME_1]
key1 = value1
key2 = value2
[SECTION_NAME_2]
key1 = value1
key2 = value2
Extending the approach mentioned above, reading section by section automatically and then accessing by section name followed by key name.
def get_config_section():
if not hasattr(get_config_section, 'section_dict'):
get_config_section.section_dict = collections.defaultdict()
for section in config.sections():
get_config_section.section_dict[section] = dict(config.items(section))
return get_config_section.section_dict
To access:
config_dict = get_config_section()
port = config_dict['DB']['port']
(here 'DB' is a section name in config file
and 'port' is a key under section 'DB'.)

A convenient solution in your case would be to include the configs in a yaml file named
**your_config_name.yml** which would look like this:
path1: "D:\test1\first"
path2: "D:\test2\second"
path3: "D:\test2\third"
In your python code you can then load the config params into a dictionary by doing this:
import yaml
with open('your_config_name.yml') as stream:
config = yaml.safe_load(stream)
You then access e.g. path1 like this from your dictionary config:
config['path1']
To import yaml you first have to install the package as such: pip install pyyaml into your chosen virtual environment.

This looks like valid Python code, so if the file is on your project's classpath (and not in some other directory or in arbitrary places) one way would be just to rename the file to "abc.py" and import it as a module, using import abc. You can even update the values using the reload function later. Then access the values as abc.path1 etc.
Of course, this can be dangerous in case the file contains other code that will be executed. I would not use it in any real, professional project, but for a small script or in interactive mode this seems to be the simplest solution.
Just put the abc.py into the same directory as your script, or the directory where you open the interactive shell, and do import abc or from abc import *.

Since your config file is a normal text file, just read it using the open function:
file = open("abc.txt", 'r')
content = file.read()
paths = content.split("\n") #split it into lines
for path in paths:
print path.split(" = ")[1]
This will print your paths. You can also store them using dictionaries or lists.
path_list = []
path_dict = {}
for path in paths:
p = path.split(" = ")
path_list.append(p)[1]
path_dict[p[0]] = p[1]
More on reading/writing file here.
Hope this helps!

For Pyhon 3.X:
Notice the lowercase import configparser makes a big difference.
Step 1:
Create a file called "config.txt" and paste the below two lines:
[global]
mykey = prod/v1/install/
Step 2:
Go to the same directory and create a testit.py and paste the code below into a file and run it. (FYI: you can put the config file anywhere you like you, you just have to change the read path)
#!/usr/bin/env python
import configparser
config = configparser.ConfigParser()
config.read(r'config.txt')
print(config.get('global', 'mykey') )

Include .pyd module files in py2exe compilation

I'm trying to compile a python script. On executing the exe I got:-
C:\Python27\dist>visualn.exe
Traceback (most recent call last):
File "visualn.py", line 19, in <module>
File "MMTK\__init__.pyc", line 39, in <module>
File "Scientific\Geometry\__init__.pyc", line 30, in <module>
File "Scientific\Geometry\VectorModule.pyc", line 9, in <module>
File "Scientific\N.pyc", line 1, in <module>
ImportError: No module named Scientific_numerics_package_id
I can see the file Scientific_numerics_package_id.pyd at the location "C:\Python27\Lib\site-packages\Scientific\win32". I want to include this module file into the compilation. I tried to copy the above file in the "dist" folder but no good. Any idea?
Update:
Here is the script:
from MMTK import *
from MMTK.Proteins import Protein
from Scientific.Visualization import VRML2; visualization_module = VRML2
protein = Protein('3CLN.pdb')
center, inertia = protein.centerAndMomentOfInertia()
distance_away = 8.0
front_cam = visualization_module.Camera(position= [center[0],center[1],center[2]+distance_away],description="Front")
right_cam = visualization_module.Camera(position=[center[0]+distance_away,center[1],center[2]],orientation=(Vector(0, 1, 0),3.14159*0.5),description="Right")
back_cam = visualization_module.Camera(position=[center[0],center[1],center[2]-distance_away],orientation=(Vector(0, 1, 0),3.14159),description="Back")
left_cam = visualization_module.Camera(position=[center[0]-distance_away,center[1],center[2]],orientation=(Vector(0, 1, 0),3.14159*1.5),description="Left")
model_name = 'vdw'
graphics = protein.graphicsObjects(graphics_module = visualization_module,model=model_name)
visualization_module.Scene(graphics, cameras=[front_cam,right_cam,back_cam,left_cam]).view()

Py2exe lets you specify additional Python modules (both .py and .pyd) via the includes option:
setup(
...
options={"py2exe": {"includes": ["Scientific.win32.Scientific_numerics_package_id"]}}
)
EDIT. The above should work if Python is able to
import Scientific.win32.Scientific_numerics_package_id

There is a way to work around this types of issues that I have used a number of times. In order to add extra files to the py2exe result you can extend the media collector in order to have a custom version of it. The following code is an example:
import glob
from py2exe.build_exe import py2exe as build_exe
def get_py2exe_extension():
"""Return an extension class of py2exe."""
class MediaCollector(build_exe):
"""Extension that copies Scientific_numerics_package_id missing data."""
def _add_module_data(self, module_name):
"""Add the data from a given path."""
# Create the media subdir where the
# Python files are collected.
media = module_name.replace('.', os.path.sep)
full = os.path.join(self.collect_dir, media)
if not os.path.exists(full):
self.mkpath(full)
# Copy the media files to the collection dir.
# Also add the copied file to the list of compiled
# files so it will be included in zipfile.
module = __import__(module_name, None, None, [''])
for path in module.__path__:
for f in glob.glob(path + '/*'): # does not like os.path.sep
log.info('Copying file %s', f)
name = os.path.basename(f)
if not os.path.isdir(f):
self.copy_file(f, os.path.join(full, name))
self.compiled_files.append(os.path.join(media, name))
else:
self.copy_tree(f, os.path.join(full, name))
def copy_extensions(self, extensions):
"""Copy the missing extensions."""
build_exe.copy_extensions(self, extensions)
for module in ['Scientific_numerics_package_id',]:
self._add_module_data(module)
return MediaCollector
I'm not sure which is the Scientific_numerics_package_id module so I've assumed that you can import it like that. The copy extensions method will get a the different module names that you are having problems with and will copy all their data into the dir folder for you. Once you have that, in order to use the new Media collector you just have to do something like the following:
cmdclass['py2exe'] = get_py2exe_extension()
So that the correct extension is used. You might need to touch the code a little but this should be a good starting point for what you need.

I encountered similar probelm with py2exe and the only solution I can find ,is to use another tool to convert python to exe - pyinstaller
Its very easy tool to use and more important , it works!
UPDATE
As I understood from your comments below , running your script from command line is not working also , due to import error (My recommendation is to first check your code from command line ,and than try to convert it to EXE)
It looks like PYTHONPATH problem.
PYTHONPATH is list of paths (similar of Windows PATH) that python programs use to find import modules.
If your script run from your IDE , that means the PYTHONPATH is set correctly in the IDE ,so all imported modules are found.
In order to set PYTHONPATH you can use :
import sys|
sys.path.append(pathname)
or use the following code that add the all folders under path parameter to PYTHONPATH:
import os
import sys
def add_tree_to_pythonpath(path):
"""
Function: add_tree_to_pythonpath
Description: Go over each directory in path and add it to PYTHONPATH
Parameters: path - Parent path to start from
Return: None
"""
# Go over each directory and file in path
for f in os.listdir(path):
if f == ".bzr" or f.lower() == "dll":
# Ignore bzr and dll directories (optional to NOT include specific folders)
continue
pathname = os.path.join(path, f)
if os.path.isdir(pathname) == True:
# Add path to PYTHONPATH
sys.path.append(pathname)
# It is a directory, recurse into it
add_tree_to_pythonpath(pathname)
else:
continue
def startup():
"""
Function: startup
Description: Startup actions needed before call to main function
Parameters: None
Return: None
"""
parent_path = os.path.normpath(os.path.join(os.getcwd(), ".."))
parent_path = os.path.normpath(os.path.join(parent_path, ".."))
# Go over each directory in parent_path and add it to PYTHONPATH
add_tree_to_pythonpath(parent_path)
# Start the program
main()
startup()

The ImportError is rectified by using "Gil.I" and "Janne Karila" suggestion by setting pythonpath and by using include function. But before this I had to create __init__.py file in the win32 folder of both the modules.
BTW I still got another error for the above script - link

How to update the variable list in the config file dynamically in to python exe file

I have a python file that converted into exe using pyinstaller
but i need to get the data from another file which is to be updated and without another pyinstaller execution it needed to be worked like getting a data from the config file to exe
from configfile import variables
variables that is loaded from the configfile.py file but after converting into exe i cant able to update configfile.py variables
any suggestion will be welcomed

function from the python config file updated and loaded dynamically outside the environment of exe
import os
extDataDir = os.getcwd()
ext_config = os.path.join(extDataDir, '', 'configfile.py')
def importCode(code,name,add_to_sys_modules=0):
import sys,imp
module = imp.new_module(name)
exec(code,module.__dict__)
if add_to_sys_modules:
sys.modules[name] = module
return module
configfile_rd = open(ext_config, "r")
configfile_code = configfile_rd.read()
configfile = importCode(configfile_code,"configfile") #dynamically readed the config file from exe and execute the python file configfile.py and working as well as import configfile
constant = configfile.variables()