I am trying to CX_Freeze an application for the Linux Platform. The Windows MSI installer works perfectly but the Linux counter-part doesn't really function the way I want it.
When the package is build it runs perfectly on the original system, but when ported to a different system (although same architecture) it generates a segfault. First thing I did was check the librarys and there are some huge version differences with libc, pthread and libdl. So I decided to include these in the build, like so:
if windows_build:
build_exe_options['packages'].append("win32net")
build_exe_options['packages'].append("win32security")
build_exe_options['packages'].append("win32con")
pywintypes_dll = 'pywintypes{0}{1}.dll'.format(*sys.version_info[0:2]) # e.g. pywintypes27.dll
build_exe_options['include_files'].append((os.path.join(GetSystemDirectory(), pywintypes_dll), pywintypes_dll))
else:
build_exe_options['packages'].append("subprocess")
build_exe_options['packages'].append("encodings")
arch_lib_path = ("/lib/%s-linux-gnu" % os.uname()[4])
shared_objects = ["libc.so.6", "libpthread.so.0", "libz.so.1", "libdl.so.2", "libutil.so.1", "libm.so.6", "libgcc_s.so.1", "ld-linux-x86-64.so.2"]
lib_paths = ["/lib", arch_lib_path, "/lib64"]
for so in shared_objects:
for lib in lib_paths:
lib_path = "%s/%s" % (lib, so)
if os.path.isfile(lib_path):
build_exe_options['include_files'].append((lib_path, so))
break
After checking the original cx_frozen bin it seems the dynamic libraries play there part and intercept the calls perfectly. Although now I am at the part where pthread segfaults due to the fact that he tries the systems libc instead of mine (checked with ldd and gdb).
My question is quite simple, this method I am trying is horrible as it doesn't do the recursive depency resolving. Therefore my question would be "what is the better way of doing this? or should I write recursive depency solving within my installer?"
And in order to beat the Solution: "Use native Python instead", we got some hardware appliances (think 2~4U) with Linux (and Bash access) where we want to run this aswell. porting entire python (with its dynamic links etcetc) seemd like way to much work when we can cx_freeze it and ship the librarys with.
I don't know about your other problems, but shipping libc.so.6 to another system the way you do it can not possibly work, as explained here.
Related
I have compiled with success a rather big app ,but the size of the app is way too big(about 1 GB).I realized that almost the half (400MB) was taken from just 2 modules(PyQt5 and mpl_toolkits(basemap)).Of course the one approach is to delete the side-services you dont need but i cant in my case as i use about all of these in my app.
Then i was thinking about ziping these modules but i don't know if this is safe or execute time viable.
So is there a suficient way to minimize the size of my app, and if the ZIP is the only answer is it viable to do this on a lib such PyQt5 witch is imported many times in my script?
It is entirely possible to do this; Python has always supported "zipapps", and as of 3.5, provides a module, zipapp, to assist with creating them.
The only concern is whether your dependencies run properly zipped; most do, but some take file-system layer dependencies that zipping (and eggs and the like) don't handle (the setup.py file for these would declare zip_safe=False to tell tools not to allow it). Packages with C extensions would have to assert zip_safe=True, as the default analyzer assumes C extensions can't be safely zipped (though many can).
It will slow down startup a bit (since it has to decompress the modules), but it shouldn't dramatically slow runtime, no matter how many times you import it; the first import is the only one that really matters, since subsequent imports just pull the cached copy of the module from an internal cache (sys.modules).
I have started to work with ITK for a week thanks to SimpleITK in Python. Even though, currently, I am satisfied with SimpleITK, I have noticed that some features such as the Powell optimization scheme or the OnePlusEvolutionary one are not available in SimpleITK. It seems to be the same with landmark-based registration methods.
So, I was wondering if there is a way to retrieve all the features available in ITK (in C++) in SimpleITK or if it is necessary to perform my own wrapping?
If not I will later learn C++ to do so!
Thanks!
You are correct. SimpleITK is limited in functionality. It is however possible to access the entire ITK library in Python using the WrapITK interface. This is an old interface which I don't believe has been updated for a while. However, it is still possible to compile new builds from source and use WrapITK.
The process is not exactly smooth. I have done the build on a Windows machine in the past and had then noted some of the not-so-smooth-steps that I had to encounter. I'll lay those down here. Since you haven't mentioned your OS, I'm going to go ahead explain the setup for Windows. Let's see if you're able to get it up and running.
ITK-4.6 + Python2.7 + CMake + VS2008 professional (all 32-bit)
Nothing later than VS2008 can compile GCCXML, which is required for generating python bindings
After configuring CMAKE, the following flags need to be additionally set:
ITK_WRAP_PYTHON
ITK_WRAP_* (all types: float, double, etc.)
ITK_BUILD_SHARED_LIBS (gets automatically set if first flag is set)
In VS2008, build in Release mode only. Number of projects in the project explorer will be more than 500. Around 300-350 should get built.
When building, make sure that you have an accessible internet conection for downloading GCCXML (which will likely get downloaded after you have started the build in VS2008). There should be no error while verifying the download. If there is, it might be because of directory creation permission errors.
The project should get built with NO ERRORS
Copy the WrapITK.pth file from the ITK build/Wrapping/Generators/Python/Release to Python/Lib/site-packages
Add the following to your Path variable:
C:\ProgramLibs\ITK\build2008\lib\Release
C:\ProgramLibs\ITK\build2008\bin\Release
C:\ProgramLibs\ITK\build2008\lib
Now ITK should work properly (below is a test python file that you can use to run a sanity check on the build). Except every time you call itk.Image the first time in your program or on the Python interpreter, there are a dozen warnings and they take time to execute that. This is a known issue. Once you are past this, it's smooth.
Test file
import itk
pixelType = itk.UC
imageType = itk.Image[pixelType, 2]
readerType = itk.ImageFileReader[imageType]
writerType = itk.ImageFileWriter[imageType]
reader = readerType.New()
writer = writerType.New()
reader.SetFileName("<input image file location>")
writer.SetFileName("D:/Output.png")
writer.SetInput( reader.GetOutput() )
writer.Update()
There is also WrapITK, which is a python wrapping of ITK that you can enable to compile with ITK (so, you'll have to compile it by yourself but you will not need to code the warpping at least). See http://kitware.com/blog/home/post/888, http://www.itk.org/Wiki/ITK/Wrapping , http://www.itk.org/Wiki/ITK/Release_4/Wrapping/WrapITK_Installation.t .
Note however that probably not all filters are wrapped (http://www.itk.org/Wiki/Proposals:Increasing_WrapITK_Coverage#List_of_Unwrapped_Filters , last updated in 2009 so probably now the situation is better).
The only compiled wrapITK package I came across is Devide-RE https://www.youtube.com/watch?v=-b1zS536R2M (with an older version of ITK, 3.2 if I remember correctly), but maybe also Slicer and Vistrail have it (http://www.itk.org/pipermail/insight-users/2009-August/031910.html)
I'm using github's linguist to identify unknown source code files. Running this from the command line after a gem install github-linguist is insanely slow. I'm using python's subprocess module to make a command-line call on a stock Ubuntu 14 installation.
Running against an empty file: linguist __init__.py takes about 2 seconds (similar results for other files). I assume this is completely from the startup time of Ruby. As #MartinKonecny points out, it seems that it is the linguist program itself.
Is there some way to speed this process up -- or a way to bundle the calls together?
One possibility is to just adapt the linguist program (https://github.com/github/linguist/blob/master/bin/linguist) to take multiple paths on the command-line. It requires mucking with a bit of Ruby, sure, but it would make it possible to pass multiple files without the startup overhead of Linguist each time.
A script this simple could suffice:
require 'linguist/file_blob'
ARGV.each do |path|
blob = Linguist::FileBlob.new(path, Dir.pwd)
# print out blob.name, blob.language, blob.sloc, etc.
end
I've seen numerous ways of retrieving installed programs on WinXP+ in python. What is the proper and most robust way of doing this?
Currently I'm accessing HKLM\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall and reading each of the keys from there to get a list. (I've been told this isn't the proper way of doing things) I've seen examples of using WMI/Win32com to do this as well but have seen comments along with those implementations that WMI might be turned off on certain machines and that it's not a very reliable solution.
Is there a method which is both proper, and reliable to get a list of installed programs? None of the WMI examples I've seen have worked on this machine (hence my reluctance to use it, I'm only running WinFLP; which is a stripped vers of XP.)
I seem to have also found the TechNet article which my searches have turned up which is provided to a similar answer on my question: http://gallery.technet.microsoft.com/ScriptCenter/en-us/154dcae0-57a1-4c6e-8f9f-b215904485b7 Note that Vista/7 listed under Platforms very clearly says "Nope"...won't work. So the WMI deal seems like it's a no-go...
Being able to retrieve the installed path would be an upside as well, as right now my current code does not account for someone installing on another drive, or in a non-default directory.
The technet script you refer to perfectly works under Win 7 (with Python 2.5 32bits), and I really cannot see why it shouldn't.
Actually, the real weakness of the WMI approach is that it only lists products installed through the Windows Installer. So it's will not give you the full list. Many programs use different installers. Just compare the results between the (Select * from Win32_Product) and what is displayed in the Control Panel. So, unless you are sure that the program that interset you in your listing are installed with MSI, WMI is definitely not an answer.
So it may be not very pythonic, but the best way, as far as I know, is to use the registry as you've done. This is actually how the control panel works, so at least Windows considers it to be the most robust way to do it.
WMI is the correct way to look for installed programs as it will work across different versions of the OS and will be supported going forward. Looking for specific regkeys may work fine for specific versions of Windows but is not guaranteed to work in the future. Here is some simple python code to check for Box Sync which I just tried on Windows 7. Note that not all fields will be available for every product so be aware these will be 'None.'
import wmi
w = wmi.WMI()
for p in w.Win32_Product():
if 'Box, Inc.' == p.Vendor and p.Caption and 'Box Sync' in p.Caption:
print 'Installed {}'.format(p.Version)
The downside I have seen with WMI is it is very slow to start up.
Currently the buildout recipe collective.recipe.omelette uses junction.exe on all versions of Windows to create symlinks. However junction.exe does not come with Windows by default and most importantly does not support creating symlinks to files (only directories) which causes a problem with quite a few Python packages.
On NT6+ (Vista and 7) there is now the mklink utility that not only comes by default but is also capable of creating symlinks to files as well as directories. I would like to update collective.recipe.omelette to use this if available and have done so except for one otherwise simple feature; detecting whether a file or folder is actually a symlink. Since this is a small buildout recipe, requiring Pywin32 in my opinion is a bit too much (unless setuptools could somehow only download it on Windows?).
Currently on Windows what omelette does is call junction.exe on the folder and then grep the response for "Substitute Name:" but I can't find anything as simple for mklink.
The only method I can think of is to call "dir" in the directory and then to go through the response line by line looking for "<SYMLINK>" and the folder/filename on the same line. Surely there is something better?
See jaraco.windows.filesystem (part of the jaraco.windows package) for extensive examples on symlink operations in Windows without pywin32.
Could you use ctypes to access the various needed functions and structures? this patch, currently under discussion, is intended to add symlink functionality to module os under Vista and Windows 7 -- but it won't be in before Python 2.7 and 3.2, so the wait (and the requirement for the very latest versions when they do eventually come) will likely be too long; a ctypes-based solution might tide you over and the code in the patch shows what it takes in C to do it (and ctypes-based programmed in Python is only a bit harder than the same programming in C).
Unless somebody's already released some other stand-alone utility like junction.exe for this purpose, I don't see other workable approaches (until that patch finally makes it into a future Python's stdlib) beyond using ctypes, Pywin32, or dir, and you've already ruled out the last two of these three options...!-)
On windows junctions and symbolic links have the attribute FILE_ATTRIBUTE_REPARSE_POINT (0x400) for reparse points. If you get the file's attributes, then detect this on?
You could use ctypes (as stated in the other answer) to access Kernel32.dll and GetFileAttributes, and detect this value.
You could leverage the Tcl you have available with Tkinter, as that has a 'file link' command that knows about junctions, unlike Pythons os module.
I have searched widely for a better solution, however python 2.7 just does not have a good solution for this. So eventually, I ended up with this (code below), its admittedly ugly, but it's downright pretty compared to all the ctypes hacks I've seen.
import os, subprocess
def realpath(path):
# not a folder path, ignore
if (not os.path.isdir(path)):
return path
rootpath = os.path.abspath(path)
oneup, foldername = os.path.split(rootpath)
output = subprocess.check_output("dir " + oneup, shell=True)
links = {}
for line in output.splitlines():
pos = line.find("<SYMLINKD>")
if not pos == -1:
link = line[pos+15:].split()
links[link[0]] = link[1].strip("[]")
return links[foldername] if links.has_key(foldername) else rootpath