I'm trying to create simple executable with pyinstaller from script I use for testing so I do not have to install everything on server I'm testing from.
#! /usr/bin/env python
from scapy.all import *
sourceport=int(raw_input('Soruce port:'))
destinationport=int(raw_input('Destination port:'))
destinationip=raw_input('Destination IP:')
maxttl=int(raw_input('MAX TTL:'))
for i in range(1,maxttl):
udptrace = IP(dst=destinationip,ttl=i)/UDP(dport=destinationport,sport=sourceport,len=500)
received=sr1(udptrace,verbose=0,timeout=2)
try:
print received.summary()
except AttributeError:
print "** TIMEOUT **"
Then I make executable:
pyinstaller -F udp.py
However when I run it and I have the following error:
Soruce port:500
Destination port:500
Destination IP:4.4.4.4
MAX TTL:3
Traceback (most recent call last):
File "<string>", line 16, in <module>
NameError: name 'IP' is not defined
user#:~/2/dist$
I have spent some time researching but did not find any answers.
The problem
First, we need to pinpoint the exact problem.
The PyInstaller manual specifies that:
Some Python scripts import modules in ways that PyInstaller cannot detect: for example, by using the __import__() function with variable data.
Inspecting Scapy's source code reveals that this is exactly how the various networking layers are imported:
scapy/layers/all.py:
def _import_star(m):
mod = __import__(m, globals(), locals())
for k,v in mod.__dict__.iteritems():
globals()[k] = v
for _l in conf.load_layers:
log_loading.debug("Loading layer %s" % _l)
try:
_import_star(_l)
except Exception,e:
log.warning("can't import layer %s: %s" % (_l,e))
Note that __import__ is invoked for each module in conf.load_layers.
scapy/config.py:
class Conf(ConfClass):
"""This object contains the configuration of scapy."""
load_layers = ["l2", "inet", "dhcp", "dns", "dot11", "gprs", "hsrp", "inet6", "ir", "isakmp", "l2tp",
"mgcp", "mobileip", "netbios", "netflow", "ntp", "ppp", "radius", "rip", "rtp",
"sebek", "skinny", "smb", "snmp", "tftp", "x509", "bluetooth", "dhcp6", "llmnr", "sctp", "vrrp",
"ipsec" ]
Note that Conf.load_layers contains "inet".
The file scapy/layers/inet.py defines the IP class, which was not imported successfully in the enclosed example.
The solution
Now, that we have located the root cause, let's see what can be done about it.
The PyInstaller manual suggests some workarounds to such importing issues:
You can give additional files on the PyInstaller command line.
You can give additional import paths on the command line.
You can edit the myscript.spec file that PyInstaller writes the first time you run it for your script. In the spec file you can tell PyInstaller about code and data files that are unique to your script.
You can write "hook" files that inform PyInstaller of hidden imports. If you "hook" imports for a package that other users might also use, you can contribute your hook file to PyInstaller.
A bit of googling reveals that an appropriate "hook" was already added to the default PyInstaller distribution, in this commit, which introduced the file PyInstaller/hooks/hook-scapy.layers.all.py.
The PyInstaller manuals indicates that such built-in hooks should run automatically:
In summary, a "hook" file tells PyInstaller about hidden imports called by a particular module. The name of the hook file is hook-<module>.py where "<module>" is the name of a script or imported module that will be found by Analysis. You should browse through the existing hooks in the hooks folder of the PyInstaller distribution folder, if only to see the names of the many supported imports.
For example hook-cPickle.py is a hook file telling about hidden imports used by the module cPickle. When your script has import cPickle the Analysis will note it and check for a hook file hook-cPickle.py.
Bottom Line
Therefore, please verify that you're running the latest version of PyInstaller. If you can't upgrade to the latest version, or if it doesn't contain the file PyInstaller/hooks/hook-scapy.layers.all.py, then create it with the following content:
#-----------------------------------------------------------------------------
# Copyright (c) 2013, PyInstaller Development Team.
#
# Distributed under the terms of the GNU General Public License with exception
# for distributing bootloader.
#
# The full license is in the file COPYING.txt, distributed with this software.
#-----------------------------------------------------------------------------
from PyInstaller.hooks.hookutils import collect_submodules
# The layers to load can be configured using scapy's conf.load_layers.
# from scapy.config import conf; print(conf.load_layers)
# I decided not to use this, but to include all layer modules. The
# reason is: When building the package, load_layers may not include
# all the layer modules the program will use later.
hiddenimports = collect_submodules('scapy.layers')
Related
I'm trying to compile a usable .dll file from Julia to be used in Python as I've already written a large GUI in Python and need some fast optimization work done. Normally I would just call PyJulia or some "live" call, however this program needs to be compiled to distribute within my research team, so whatever solution I end up with needs to be able to run on its own (without Julia or Python actually installed).
Right now I'm able to create .dll files via PackageCompiler.jl, something I learned from previous posts on StackOverflow, however when trying to run these files in Python via the following code
Julia mock package
module JuliaFunctions
# Pkg.add("BlackBoxOptim")
Base.#ccallable function my_main_function(x::Cfloat,y::Cfloat)::Cfloat
z = 0
for i in 1:x
z += i ^ y
end
return z
end
# function julia_main()
# print("Hello from a compiled executable!")
# end
export my_main_function
end # module
Julia script to use PackageCompiler
# using PackageCompiler
using Pkg
# Pkg.develop(path="JuliaFunctions") # This is how you add a local package
# include("JuliaFunctions/src/JuliaFunctions.jl") # this is how you add a local module
using PackageCompiler
# Pkg.add(path="JuliaFunctions")
#time create_sysimage(:JuliaFunctions, sysimage_path="JuliaFunctions.dll")
Trying to use the resulting .dll in CTypes in Python
import ctypes
from ctypes.util import find_library
from ctypes import *
path = os.path.dirname(os.path.realpath(__file__)) + '\\JuliaFunctions.dll'
# _lib = cdll.LoadLibrary(ctypes.util.find_library(path)) # same error
# hllDll = ctypes.WinDLL(path, winmode=0) # same error
with os.add_dll_directory(os.path.dirname(os.path.realpath(__file__))):
_lib = ctypes.CDLL(path, winmode=0)
I get
OSError: [WinError 127] The specified procedure could not be found
With my current understanding, this means that CTypes found the dll and imported it, but didn't find.. something? I've yet to fully grasp how this behaves.
I've verified the function my_main_function is exported in the .dll file via Nirsoft's DLL Export Viewer. Users from previous similar issues have noted that this sysimage is already callable and should work, but they always add at the end something along the lines of "Note that you will also in general need to initialize the Julia runtime."
What does this mean? Is this even something that can be done independently from the Julia installation? The dev docs in PackageCompiler mention this, however they just mention that julia_main is automatically included in the .dll file and gets called as a sort of launch point. This function is also being exported correctly into the .dll file the above code creates. Below is an image of the Nirsoft export viewer output for reference.
Edit 1
Inexplicably, I've rebuilt this .dll on another machine and made progress. Now, the dll is imported correctly. I'm not sure yet why this worked on a fresh Julia install + Python venv, but I'm going to reinstall them on the other one and update this if anything changes. For anyone encountering this, also note you need to specify the expected output, whatever it may be. In my case this is done by adding (after the import):
_lib.testmethod1.restype = c_double # switched from Cfloat earlier, a lot has changed.
_lib.testmethod1.argtypes = [c_double, c_double] # (defined by ctypes)
The current error is now OSError: exception: access violation writing 0x0000000000000024 when trying to actually use the function, which is specific to Python. Any help on this would also be appreciated.
I'm developing a plugin for Inkscape. Some versions:
Inkscape v0.92.3
Windows 10, version 1803 (build 17134.165)
Python 3.7 explicitly installed
MonoDevelop Version 7.7 Preview (7.7) Extra versions below
Installation Locations:
Inkscape: C:\Program Files\Inkscape
Extension: C:\Program Files\Inkscape\share\extensions
Contains: myplugin.inx, myplugin.py, MyPlugin.exe
I've made a plugin which, for development reasons, works as currently intended.
Most important of all, it runs when I run it either from MonoDevelop, or the built exe itself (both with the generated .dll's etc in the same location, or with only the exe copied to a different location).
I use (a slightly edited version of) SugarPillStudio's python script to run the .exe file. However, when I run that python script by invoking the extension, the .exe is not launched. Inkscape blinks a message that says 'MyPlugin is launching...' and closes that as fast as it opens.
I know that the python script works, because I have it print debugging lines to a .log file on my desktop. I know that the .exe doesn't launch because I have it also writing lines to the same .log file, first thing when the main() is invoked. When I (successfully) run the .exe it does print to the file, when I run the extension it doesn't.
This leads me to believe there's a problem with the python script in invoking the .exe. Any help?
Python Script:
#!/usr/bin/env python
'''
sugarpillstudios.com/wp/?p=142
'''
import os, sys, subprocess, datetime
f=open("C:\Users\Diamundo\Documents\plugin.log", "a+")
f.write("[PYT] %s Python script called at: %s.\n" % (datetime.datetime.now().isoformat(), os.getcwd() ) )
argv = []
for arg in sys.argv[:]:
if arg.startswith("--executable="):
executable = arg.split("=")[1]
else:
argv.append(arg)
argv[0] = executable
f.write("[PYT] %s %s\n" % ( datetime.datetime.now().isoformat(), executable ) )
process = subprocess.Popen(argv,shell=False,stdout=subprocess.PIPE)
print process.communicate()[0]
Plugin.inx:
<inkscape-extension>
<name>MyPlugin</name>
<id>name.space.plugin.main</id>
<param name="executable" type="string" gui-hidden="true">MyPlugin.exe</param>
<effect>
<object-type>all</object-type>
<effects-menu>
<submenu _name="MyPlugin"/>
</effects-menu>
</effect>
<script>
<command reldir="extensions" interpreter="python">myplugin.py</command>
</script>
</inkscape-extension>
Extra Monodevelop versions:
Runtime:
Microsoft .NET 4.0.30319.42000
GTK+ 2.24.26 (Light theme)
GTK# 2.12.45
NuGet
Version: 4.3.1.4445
.NET Core
Runtime: C:\Program Files\dotnet\dotnet.exe
Runtime Versions:
2.0.9
2.0.5
SDK: C:\Program Files\dotnet\sdk\2.1.202\Sdks
SDK Versions:
2.1.202
2.1.4
MSBuild SDKs: Not installed
Inkscape uses Python 2.7, which it brings with it, unless you set that differently in the settings file (edit manually).
If you want to write an Inkscape extension, you can learn how to do this by:
reading https://inkscape.org/develop/extensions/
following the examples in other extensions that work (e.g. for running additional Inkscape instances, you could follow this one: https://gitlab.com/su-v/inx-pathops/blob/master/src/pathops.py)
Loosely based on the pathops.py file, linked by Moini in her answer, I've come up with the following file.
About
It uses the inkex.py (source on GitLab) library to declare an Inkscape Effect. The Effect class uses the OptionParser library to parse the default given parameters (e.g. --id=$$ for selected nodes where $$ is the XML node's 'id' tag's value). By adding the custom executable option, we can also parse this.
Parsing arguments
After the OptionParser is done parsing, the values will be visible in self.options, i.e. our executable now lives in self.options.executable (because of the action="store" and dest="executable" parameters).
Furthermore, the temporary SVG-file as created by Inkscape, can be found in self.svg_file.
Saving edits
As previously said, Inkscape makes a temporary file with the contents of the SVG in its then current state. Any edits you(r plugin) make(s) should not be saved back to this file, but returned to Inkscape itself - this is the premise of the Effect class: it edits an SVG and returns the edit to Inkscape. Further reading here.
Instead, in your plugin you should (readonly) open the file, read its contents, and then edit it. When you're done editing, write the entire SVG to your commandline.
Then, the line out, err = process.communicate(None) will grab your plugin's output and error-output. These are used to return information to Inkscape.
Notes
The structure of the cmd array is of no importance, except the fact that the executable should come as the very first element. All other array-elements can be anything in any order, I just added '--id=$$' to every ID because that's the way Inkscape uses, and this way it looks the same as if there's no Python middleware present. The same goes for the self.svg_file which I placed last, Inkscape does the same in its arguments - you could also make '--file='+self.svg_file from it for clarity.
Source
#!/usr/bin/env python
import os
from subprocess import Popen, PIPE
import time
try:
import inkex_local as inkex
except ImportError:
import inkex
#import simplestyle
class MyPlugin(inkex.Effect):
def __init__(self):
inkex.Effect.__init__(self)
self.OptionParser.add_option("--executable", action="store", type="string", dest="executable", default="MyPlugin.exe")
def effect(self):
out = err = None
cmd = []
cmd.append(self.options.executable)
for id in self.options.ids:
cmd.append("--id=" + id)
cmd.append(self.svg_file)
#inkex.debug(cmd);
process = Popen(cmd, shell=False, stdin=PIPE, stdout=PIPE, stderr=PIPE)
out, err = process.communicate(None)
if process.returncode == 0:
print out
elif err is not None:
inkex.errormsg(err)
if __name__ == '__main__':
myplugin = MyPlugin()
myplugin.affect()
As the title says.
When I execute the converted python file (the .exe) I get the following output:
Traceback (most recent call last):
File "background.py", line 10, in <module>
File "site-packages\praw\reddit.py", line 129, in __init__
File "site-packages\praw\config.py", line 72, in __init__
File "site-packages\praw\config.py", line 98, in _initialize_attributes
File "site-packages\praw\config.py", line 31, in _config_boolean
AttributeError: '_NotSet' object has no attribute 'lower'
[1692] Failed to execute script background
I did not use a praw.ini file, instead hardcoding the values for the logon as such:
import praw
import praw.models
import urllib.request
from random import randint
from os import getcwd
import ctypes
r = praw.Reddit(client_id='client',
client_secret='secret',
user_agent='user')
sub = r.subreddit('earthporn')
choose = []
for i in sub.hot(limit=20):
a = str(i.url)
if "i.redd" in a or "i.imgur" in a:
choose.append(i.url)
x = choose[randint(0,len(choose)-1)]
resource = urllib.request.urlopen(x)
output = open("daily_image.jpg","wb")
output.write(resource.read())
output.close()
direc = getcwd() + "\\daily_image.jpg"
ctypes.windll.user32.SystemParametersInfoW(20, 0, direc , 0)
The above file works in just python but not when converted to an exe. (obviously with the client,secret and user id's set, feel free to steal it idrc)
Any help is really appreciated!
I had this error and found that to resolve it you need to have a copy of the praw.ini in the directory where you a running the executable (your_app.exe) from. You can find your praw.ini in your installed \praw directory.
Right.
So, pyinstaller isn't a perfect .py to .exe converter, so some things get lost in translation.
I first commented out all updating in praw as this caused a crash in the .exe created by pyinstaller (note everything I did here caused errors in the .exe but never in the .py).
I then had to manually set some variables along the way because they weren't set when they were called in the .exe version. Either threading is used in PRAW and in the .exe version it can't keep up, or there's some seriously weird shit going on.
Yeh so I basically just modified code all throughout praw so this thing would run. If anyone comes across this issue like me and can't find the answer anywhere (because trust me I scoured planet Earth and it's no where to be found) message me and I can send you my praw version.
May god forbid you get this error
I can't imagine anyone is still looking at this, but I solved this by adding the praw.ini as a data file using a pyinstaller .spec file. The steps I used were:
Create a .spec file by using the normal pyinstaller <name>.py with whatever flags you like (e.g., --onefile).
Modify the generated .spec file by adding the following:
import os
import importlib
proot = os.path.dirname(importlib.import_module('praw').__file__)
datas = [(os.path.join(proot, 'praw.ini'), 'praw')]
a = Analysis(['<name>.py'],
...
datas=datas,
Run pyinstaller against the newly edited .spec file: pyinstaller <name>.spec
Have no fear! In lieu of a properly-packaged praw.ini, the configuration it contains can also be explicitly provided as args to the Reddit constructor:
reddit = praw.Reddit(
client_id="CHANGEME",
client_secret="CHANGEME",
user_agent="CHANGEME",
check_for_updates=False,
comment_kind="t1",
message_kind="t4",
redditor_kind="t2",
submission_kind="t3",
subreddit_kind="t5",
trophy_kind="t6",
oauth_url="https://oauth.reddit.com",
reddit_url="https://www.reddit.com",
short_url="https://redd.it",
ratelimit_seconds=5,
timeout=16,
)
The above code works great for me, even after being packaged with PyInstaller. Note that future PRAW updates may add more defaults to praw.ini, so you'll have to copy them over or you'll get similar errors about unset config options.
See this GitHub issue: https://github.com/praw-dev/praw/issues/1016.
I'm not a programmer, so I don't even know in what terms should I ask this. Let's say I've compiled a Python script to have an .exe (I use py2exe to do this). This is the major program. Now, I want to add some extra functionality to it, but I don't want to recompile the entire script with the added functionality. I tried to search something on the web, and I found examples of extending a C++ or other application with Python scripts (like a sort of plugin). But I can't figure out how to do it with an application already written in Python.
I tried this: I wrote major.py (this is the script from where I build the executable) and stuff.py. In major I wrote this:
def generic():
import stuff
while True:
param=input('what did you say? ')
stuff.speak(param)
generic()
And in stuff I wrote this:
def speak(param):
print(param)
Then I created a .exe with py2exe. It works as expected, when I run the program in the command line says "what did you say?" and waits until I type something, then it prints what I typed.
Then, I changed stuff.py with this:
def speak(param):
print('I said '+param)
Hoping that now upon the execution of the .exe created earlier it would print "I said.." plus whatever I typed. Obviously, it didn't work, the program continued to behave like before. So I'm guessing that once I imported stuff and created the .exe file, that import is permanent, not allowing me to change whatever is in stuff. What should I do?
py2exe packs the compiled scripts in the executable.
You need to recreate the executable (which will recompile any changed script) to see the changes take effect.
EDIT following the comments:
You can do it if you dynamically import/reimport the module from inside the executable.
In your main script you do (see code below)
mod, error = loadmodule('mystuff.py')
if mod is not None:
# loading succeeded you can now proceed and do things with it
pass
Of course you have to leave mystuff.py out of the scripts that py2exe packs into the executable. In the above example mystuff.py would be in the same directory as the executable.
The loading code:
def loadmodule(modpath, modname=''):
if not modpath.endswith('.py'):
modpath += '.py'
# generate a random name for the module
if not modname:
modpathbase = os.path.basename(modpath)
modname, _ = os.path.splitext(modpathbase)
version = (sys.version_info[0], sys.version_info[1])
if version < (3, 3):
mod, e = loadmodule2(modpath, modname)
else:
mod, e = loadmodule3(modpath, modname)
return mod, e
def loadmodule2(modpath, modname):
import imp
try:
mod = imp.load_source(modname, modpath)
except Exception as e:
return (None, e)
return (mod, None)
def loadmodule3(modpath, modname):
import importlib.machinery
try:
loader = importlib.machinery.SourceFileLoader(modname, modpath)
mod = loader.load_module()
except Exception as e:
return (None, e)
return (mod, None)
If you run your script from Python, instead of compiling it as an executable, you can make changes and run them without having to recompile. The py2exe is mostly for allowing you to distribute your application to other Windows computers that don't have Python installed. After you have finished developing it, then compile it as an executable so you can run it on other computers.
There is no way to do what you want. py2exe builds a standalone python interpreter (In the file named python.dll) with just the dependencies of your project. Then the .exe file runs your script using that interpreter
I suggest you that if you really need to provide regular upgrades then you should recompile it, or install python in the target machine, or make an updating routine that checks for updates and compiles it in the target machine (With py2exe)
I've written a setup.py script for py2exe, generated an executable for my python GUI application and I have a whole bunch of files in the dist directory, including the app, w9xopen.exe and MSVCR71.dll. When I try to run the application, I get an error message that just says "see the logfile for details". The only problem is, the log file is empty.
The closest error I've seen is "The following modules appear to be missing" but I'm not using any of those modules as far as I know (especially since they seem to be of databases I'm not using) but digging up on Google suggests that these are relatively benign warnings.
I've written and packaged a console application as well as a wxpython one with py2exe and both applications have compiled and run successfully. I am using a new python toolkit called dabo, which in turn makes uses of wxpython modules so I can't figure out what I'm doing wrong. Where do I start investigating the problem since obviously the log file hasn't been too useful?
Edit 1:
The python version is 2.5. py2exe is 0.6.8. There were no significant build errors. The only one was the bit about "The following modules appear to be missing..." which were non critical errors since the packages listed were ones I was definitely not using and shouldn't stop the execution of the app either. Running the executable produced a logfile which was completely empty. Previously it had an error about locales which I've since fixed but clearly something is wrong as the executable wasn't running. The setup.py file is based quite heavily on the original setup.py generated by running their "app wizard" and looking at the example that Ed Leafe and some others posted. Yes, I have a log file and it's not printing anything for me to use, which is why I'm asking if there's any other troubleshooting avenue I've missed which will help me find out what's going on.
I have even written a bare bones test application which simply produces a bare bones GUI - an empty frame with some default menu options. The code written itself is only 3 lines and the rest is in the 3rd party toolkit. Again, that compiled into an exe (as did my original app) but simply did not run. There were no error output in the run time log file either.
Edit 2:
It turns out that switching from "windows" to "console" for initial debugging purposes was insightful. I've now got a basic running test app and on to compiling the real app!
The test app:
import dabo
app = dabo.dApp()
app.start()
The setup.py for test app:
import os
import sys
import glob
from distutils.core import setup
import py2exe
import dabo.icons
daboDir = os.path.split(dabo.__file__)[0]
# Find the location of the dabo icons:
iconDir = os.path.split(dabo.icons.__file__)[0]
iconSubDirs = []
def getIconSubDir(arg, dirname, fnames):
if ".svn" not in dirname and dirname[-1] != "\\":
icons = glob.glob(os.path.join(dirname, "*.png"))
if icons:
subdir = (os.path.join("resources", dirname[len(arg)+1:]), icons)
iconSubDirs.append(subdir)
os.path.walk(iconDir, getIconSubDir, iconDir)
# locales:
localeDir = "%s%slocale" % (daboDir, os.sep)
locales = []
def getLocales(arg, dirname, fnames):
if ".svn" not in dirname and dirname[-1] != "\\":
mo_files = tuple(glob.glob(os.path.join(dirname, "*.mo")))
if mo_files:
subdir = os.path.join("dabo.locale", dirname[len(arg)+1:])
locales.append((subdir, mo_files))
os.path.walk(localeDir, getLocales, localeDir)
data_files=[("resources", glob.glob(os.path.join(iconDir, "*.ico"))),
("resources", glob.glob("resources/*"))]
data_files.extend(iconSubDirs)
data_files.extend(locales)
setup(name="basicApp",
version='0.01',
description="Test Dabo Application",
options={"py2exe": {
"compressed": 1, "optimize": 2, "bundle_files": 1,
"excludes": ["Tkconstants","Tkinter","tcl",
"_imagingtk", "PIL._imagingtk",
"ImageTk", "PIL.ImageTk", "FixTk", "kinterbasdb",
"MySQLdb", 'Numeric', 'OpenGL.GL', 'OpenGL.GLUT',
'dbGadfly', 'email.Generator',
'email.Iterators', 'email.Utils', 'kinterbasdb',
'numarray', 'pymssql', 'pysqlite2', 'wx.BitmapFromImage'],
"includes": ["encodings", "locale", "wx.gizmos","wx.lib.calendar"]}},
zipfile=None,
windows=[{'script':'basicApp.py'}],
data_files=data_files
)
You may need to fix log handling first, this URL may help.
Later you may look for answer here.
My answer is very general because you didn't give any more specific info (like py2exe/python version, py2exe log, other used 3rd party libraries).
See http://www.wxpython.org/docs/api/wx.App-class.html for wxPyton's App class initializer. If you want to run the app from a console and have stderr print to there, then supply False for the redirect argument. Otherwise, if you just want a window to pop up, set redirect to True and filename to None.