I have written an application that gives the user an option to specify a particular network scanning tool (say nmap,xprobe,p0f) and then scans a given subnet with that tool (I do not re-implement the tool, simply call it in shell and parse its response for my app), parses the result and stores in a particular format in DB. This application is basically a feeder application for another app that will use the data in DB.
I have written my code so that all Scan Tool interfaces are implemented as plug-ins. The structure is
Project/
DBWriter.py
Scanner/
__init__.py
network_mapper.py
Scanner.py
Plugins/
__init__.py
nmap.py
p0f.py
Others/
So to add a new scan interface, all the developer needs to do is write a tool.py file (in correct semantics ofcourse) and drop it into Plugins folder.
To achieve this I need to be able to import correct python module at run-time.
from Plugins import nmap
or
from Plugins import xprobe
Depending on what the user enters
Here's the code that I have added
f,p,d = imp.find_module("Plugins")
x = imp.load_module("Plugins",f,p,d)
path_n = x.__path__
f,p,d = imp.find_module("%s"%(tool),path_n)
tool_mod = imp.load_module("%s"%(tool),f,p,d)
tool_mod.scan() #Irrespective of what user enters or what new plugin is developed, this line does not need to change and will work as long as plug-in has a scan function
The code works correctly. However, when I try to bundle it into one executable using PyInstaller (for Windows/*Nix platforms), it has a lot of trouble in find_module and load_module and consequently I get an ImportError (maybe its because the paths are not correctly unrolled in executable decompression).
My question is - Is there an alternate way in Python using which I can achieve the same functionality (that might hopefully work well with PyInstaller) ?
I had earlier asked this question looking for workarounds in PyInstaller, but no responses have forced me to look for workarounds in Python
If you only need to run on Windows, then I recommend trying py2exe.
py2exe is a Python Distutils extension which converts Python scripts into executable Windows programs, able to run without requiring a Python installation.
We use it at work to speed up a very large Python program, and it copes with imported modules.
Related
I've tried looking online and I'm honestly lost at this point.
I'm trying to find if there's a way to import python scripts and run them AFTER my Python program has been compiled.
For an example, let's say I have a main.py such that:
import modules.NewModule
a = NewModuleClass()
a.showYouWork()
then I compile main.py with pyinstaller so that my directory looks like:
main.exe
modules/NewModule.py
My end goal is to make a program that can dynamically read new Python files in a folder (this will be coded in) and use them (the part I'm struggling with). I know it's possible, since that's how add-ons work in Blender 3D but I've struggled for many hours to figure this out. I think I'm just bad at choosing the correct terms in Google.
Maybe I just need to convert all of the Python files in the modules directory to .pyc files? Then, how would I use them?
Also, if this is a duplicate on here (it probably is), please let me know. I couldn't find this issue on this site either.
You may find no detailed answer simply because there is no problem. PyInstaller does not really compile Python scripts into machine code executables. It just assembles then into a folder along with an embedded Python interpretor, or alternatively creates a compressed single file executable that will automatically uncompress itself at run time into a temporary folder containing that.
From then on, you have an almost standard Python environment, with normal .pyc file which can contain normal Python instructions like calls to importlib to dynamically load other Python modules. You have just to append the directory containing the modules to sys.path before importing them. An other possible caveat, is that pyinstaller only gets required modules and not a full Python installation, so you must either make sure that the dynamic modules do not rely on missing standard modules, or be prepared to face an ImportError.
I would like to distribute my python application to the end-user as single-file executable. However, the end-user has the possibility to add functionalities to the application. For this he can write funtions to a python file which is then imported at runtime.
However, if I freeze the Application for distribution, any changes made by the user to the python file afterwards will have no effect.
How can I distribute my Application as a more or less single-file executable to the end-user with him still being able to add/remove functions?
Or can you give the user the possibility to add functionality in another way than by importing his Python functions at runtime?
I think there is a misunderstanding of how a programme written in Python is actually executed. If we want to "import Python code at runtime" into a programme, this imported Python code would have to be compiled at runtime. If the end-user does not have a Python compiler installed, how should that be possible? Unless your programme effectively contains a Python compiler, I think, what you want to achieve is impossible by design.
I want to run a python script as part of a jenkins pipline triggered from a github repo. If I store the script directly in the repo itself, I can just do sh 'python path/to/my_package/script.py' which work perfectly. However since I want to use this from multiple pipelines from multiple repos, I want to put this in a jenkins shared library.
I found this question which suggested storing the python file in the resources directory and copying it to a temp file before use. That only works if the script is one standalone file. Unfortunately, mine is a package with multiple python files and imports between them, so thats a no go. I also tried to copy the entire folder containing the python package from the answer to this question, which suggests getting the location of the library with
import groovy.transform.SourceURI
import java.nio.file.Path
import java.nio.file.Paths
class ScriptSourceUri {
#SourceURI
static URI uri
}
but its gives me the following error:
Scripts not permitted to use staticMethod java.net.URI create java.lang.String. Administrators can decide whether to approve or reject this signature.
It seems that some additional permissions are required, which I don't think I'll be able to acquire (its a shared machine).
So? Does anyone know how I can run a python package from jenkins shared library? Right now the only solution I can think of is to manually recreate the directory structure of the python package, which is obviously very messy and non-generic.
PS: There is no particular reason for using the python script over writing the same script in groovy. Its just that the python script is well tested, well understood and well supported. Rewriting the whole thing in groovy just isn't feasible right now.
You can go to http://host:8080/jenkins/scriptApproval/ page of your Jenkins installation and approve the request for your scripts, please see below:-
And follow the link for more information.
I have made a python interactive script project containing a few directories with project files and a main python script.
The script does the work of batch processing scientific images for biological systematics.
I wrote os agnostic code, but I was thinking about trying to package/freeze the script as a cli utility that a (more or less)lay person could download and use.
I have been reading about packaging and freezing techniques in python and the more I read the more I feel I'm confused. (I'm linux user)
Am I conceiving of this script as a utility correctly? Is is worth it/possible to pass command line args to an .exe, and should I package/freeze the files for this kind of interactive cli script?
I don't have a lot of experience with windows. I'm looking for advice/pointer where to look next/search.
You are on the correct path, but you need to understand a little more about python packaging to know how to proceed.
You are right that you need to package your code with setup tools. This will make it cake to install the package for others and will let python decide where to properly put things. -including script utilities/non-code files-
Setuptools has special support built in JUST FOR SCRIPTS! that will let you pass arguments, ect..
However, in packaging the code, you must understand that what you are really doing is making it a library that can be installed with pip. Therefore, that python script utility that you wrote will be treated as a library by setuptools. Setuptools will have you make a separate script file to import your new library and call the script main() function.
I know that you were already using this manual, and that's fantastic. Everything you need to first set up and understand a python package is in there. RTFM up a storm.
As I understand it, you can then look more into freezing the package (and register it with Pypi) for other systems once you have a handle on packaging.
note: use scripts keyword in setup tools to specify your script
Currently, I'm deploying a full python distribution (the original python 2.7 msi) with my app. Which is an embedded web server made with delphi.
Reading this, I wonder if is possible to embed the necessary python files with my app, to decrease load files and avoid conflict with several python versions.
I have previous experience with python for delphi so I only need to know if only shipping the python dll + zip with the distro + own scripts will work (and if exist any caveats I must know or a sample where I can look)
zipimport should work just fine for you -- I'm not familiar with Python for Delphi, but I doubt it disables that functionality (an embedding application can do that, but it's an unusual choice). Just remember that what you can zip up and import directly are the Python-coded modules (or just their corresponding .pyc or .pyo byte codes) -- DLLs (even if renamed as .pyds;-) need to be on disk to be loaded (so if you have a zipfile with them it will need to be unzipped at the start of the app, e.g. into a temporary directory).
Moreover, you don't even need to zip up all modules, just those you actually need (by transitive closure) -- and you can easily find out exactly which modules those are, with the modulefinder module of the standard Python library. The example on the documentation page I just pointed to should clarify things. Happy zipping!
Yes it is possible.
I'm actually writing automatisation script in Python with the Zipimport library. I actually included every .py files in my zip as well as configuration or xml files needed by those script.
Then, I call a .command file targeting a __main__.py class that redirect towards the desired script according to my sys.argv parameters which is really useful!