Develop Reference

How do you resolve 'hidden imports not found!' warnings in pyinstaller for scipy?

I'm working on using pyinstaller to create an .exe for a python program that uses pandas and sklearn. The pyinstaller process completes and produces the dist folder with the executable as expected. However, when I run the .exe I get module import errors related to sklearn and scipy.
I created a test script (test.py) to test imports, which only imports pandas and sklearn and then prints a success message:
import time
import pandas as pd
import sklearn
def main():
print('hello world!')
time.sleep(5)
if __name__ == '__main__':
main()
I'm aware of pyinstaller hooks and I was able to resolve the pandas errors by adding a hook to the pyinstaller hooks directory. I added similar hooks for sklearn and scipy it looks like they're running, but in the pyinstaller output I'm getting warnings that 'Hidden import "sklearn.utils.sparsetools._graph_validation" not found!' and similar one for '._graph_tools'.
Here's the hook for scipy (hook-scipy.py):
print('loading custome hook for scipy')
from PyInstaller.utils.hooks import collect_submodules
hiddenimports = collect_submodules('scipy')
Here's a snapshot of the warnings generated from running pyinstaller
Here's a snapshot of the error when running test.exe
I'm working in a virtual environment where pyinstaller, pandas, sklearn, scipy and all dependencies are installed (at least I can get the regular test.py script running in this venv). Using PyInstaller 3.3.1, Python 3.6.4 on Windows 10.10.0.
Any help is appreciated!

You need to go into the hook-scipy.py (or create one) and have it look like this:
from PyInstaller.utils.hooks import collect_submodules
from PyInstaller.utils.hooks import collect_data_files
hiddenimports = collect_submodules('scipy')
datas = collect_data_files('scipy')
then go into the hook-sklearn.metrics.cluster.py file and modify it to look like this:
from PyInstaller.utils.hooks import collect_data_files
hiddenimports = ['sklearn.utils.sparsetools._graph_validation',
'sklearn.utils.sparsetools._graph_tools',
'sklearn.utils.lgamma',
'sklearn.utils.weight_vector']
datas = collect_data_files('sklearn')
I do not know if this part is necessary but I also created a hook-sklearn.py file that looks like this:
from PyInstaller.utils.hooks import collect_submodules
hiddenimports = collect_submodules('sklearn')
In the cmd I used pyinstaller test.py -F to create one file.
Then it should work:

importing an external c module to python

I'm trying to run a python code (test.py) which uses an external module written in C (swapped). Here is the directory structure,
test.py
setup.py
swapped.so
ext_src/
c_swapped.c
c_swapped.h
swapped.c
In order to use the swapped module, I first run setup.py, which is configured as following:
from distutils.core import setup
from distutils.extension import Extension
from distutils.command import build_ext
import numpy as np
import sys
sys.argv[1:] = ['build_ext', '--inplace']
ext_modules = [Extension(
name="swapped",
sources=["ext_src/swapped.c", "ext_src/c_swapped.c"],
language="c", include_dirs=[np.get_include()])
]
setup(
name = 'RLScore',
ext_modules = ext_modules,
)
And in test.py, swapped is imported as:
import swapped
So when I run these on python 2.7.6, test.py works just fine, but on python 3.4.5, I get the following error:
import swapped
File "swapped.pyx", line 2, in init rlscore.utilities.swapped (ext_src/swapped.c:3533)
System error parent module '' not loaded cannot perform relative import
I wish to run this code on python 3.4.5, so is there a way that I can make swapped module working?
Thanks a lot!
Edit: I installed virtual environment as python=2.7, so it works now. But still, if there is a way to run the package without using virtualenv, I'd like to learn.

ImportError: No module named 'tkinter' after pyInstaller

I want to do an executable, but ervery time I run the .exe it writes ImportError: No module named 'tkinter', and all I read on Stackowerflow do not help me !
My python program is simple (ODE solver) and requests only :
from math import*
from pylab import*
import numpy as np
import matplotlib
import numpy as np
import matplotlib.pyplot as plt
I paste a copy of my prog.py into the C:\Python\Scripts folder where pyInstaller is. I compute the command line pyinstaller -F eulersolver.py, this creates a prog.exe in the dist folder. When I run this code I have
ImportError: No module named 'tkinter'
Failed to execute script prog
But my program do not use this module... do you have any proposition or help for me ?
OS : Windows64
Python : 3.5 for Win64
Note : I already unistall/install python 3 times today (after reading documentation on this webside and abroad).
Note 2 : I use Python only for scientific issues. I am no computer scientist, so be kind to me when explaining computer stuff :S

FINALLY WORKED FOR pyinstaller -F --hidden-import=tkinter --hidden-import=tkinter.filedialog prog.py Thanks a lot !!!

You should use hidden import
pyinstaller eulersolver.py --hidden-import=tkinter -y

The problem is that pyinstaller won't see second level imports. So if you import module A, pyinstaller sees this. But any additional module that is imported in A will not be seen.
There is no need to change anything in your python scripts. You can directly add the missing imports to the spec file (prog.spec in your case).
Just change the following line:
hiddenimports=[],
to
hiddenimports=["tkinter"],
After that run pyinstaller prog.spec to create the prog.exe.

How do i convert a python3.4 script(that has installed modules) into a windows executable using py2exe?

I have a python script that i have to import all these modules, which some of them require downloading:
import datetime
from dateutil import parser
from tkinter import filedialog
import tkinter
import mailbox
import pprint
import json
import urllib.request
from tkinter import *
Is there a way, using py2exe, i can convert the script into a windows executable. If so, how?

To simplify the procedure, you will want to have all your modules already downloaded. This can be accomplished with a simple "pip install " from command prompt. Then it is as simple as writing and running a setup.py file - detailed directions for how to do this can be found here.

Having an issue creating an exe with py2exe and script importing xlrd

My goal is to create a python script that loops over cells of an excel document. This is my python script called reader.py, and it works just fine.
import xlrd
import os
exceldoc = raw_input("Enter the path to the doc [C:\\folder\\file.xlsx]: ")
wb = xlrd.open_workbook(exceldoc,'rb')
.... some code....
The problem I'm encountering is attempting to use py2exe to create an executable so this script can be used elsewhere.
Here is my setup.py file:
from distutils.core import setup
import py2exe
import sys
from glob import glob
setup(name='Excel Document Checker',console=['reader.py'])
I run the following command: python setup.py py2exe
It appears to run fine; it creates the dist folder that has my reader.exe file, but near the end of the command I get the following:
The following modules appear to be missing
['cElementTree', 'elementtree.ElementTree']
I did some searching online, and tried the recommendations here Re: Error: Element Tree not found, this changing my setup.py file:
from distutils.core import setup
import py2exe
import sys
from glob import glob
options={
"py2exe":{"unbuffered": True,"optimize": 2,
'includes':['xml.etree.ElementPath', 'xml.etree.ElementTree', 'xml.etree.cElementTree'],
"packages": ["elementtree", "xml"]}}
setup(name='Excel Document Checker',options = options,console=['reader.py'])
I'm now getting an error:
ImportError: No module named elementtree
I'm sort of at an impasse here. Any help or guidance is greatly appreciate.
Just some information - I'm running Python 2.6 on a 32 bit system.

You explicitly told setup.py to depend on a package named elementtree here:
"packages": ["elementtree", "xml"]}}
There is no such package in the stdlib. There's xml.etree, but obviously that's the same name.
The example you found is apparently designed for someone who has installed the third-party package elementtree, which is necessary if you need features added after Python 2.6's version of xml.etree, or if you need to work with Python 1.5-2.4, but not if you just want to use Python 2.6's version. (And anyway, if you do need the third-party package… then you have to install it or it won't work, obviously.)
So, just don't do that, and that error will go away.
Also, if your code—or the code you import (e.g., xlrd) is using xml.etree.cElementTree, then, as the py2exe FAQ says, you must also import xml.etree.ElementTree before using it to get it working. (And you also may need to specify it manually as a dependency.)
You presumably don't want to change all the third-party modules you're using… but I believe that making sure to import xml.etree.ElementTree before importing any of those third-party modules works fine.