Is there any way to create a virtual import path in Python?
My directory structure is like this:
/
native
scripts
some.py
another.py
[Other unrelated dirs]
The root is the directory from where the program is executed. Atm I add native/scripts/ to the search path so I can do import some, another instead of from native.scripts import some, another, but I'd like to be able to do it like this:
from native import some
import native.another
Is there any way to achieve this?
Related questions:
Making a virtual package available via sys.modules
Why not move some.py and another.py out into the native directory so that everything Just Works and so that people returning to the source code later won't be confused about why things are and aren't importable? :)
Update:
Thanks for your comments; they have usefully clarified the problem! In your case, I generally put functions and classes that I might want to import inside, say, native.some where I can easily get to them. But then I get the script code, and only the script code — only the thin shim that interprets arguments and starts everything running by passing those to a main() or go() function as parameters — and put that inside of a scripts directory. That keeps external-interface code cleanly separate from code that you might want to import, and means you don't have to try to fool Python into having modules several places at once.
In /native/__init__.py, include:
from scripts import some, another
Related
So bascically I have a filestructure lile
directory
/subdirecrory
>view.py
>view_model.py
>controller.py
>main.py
My controller looks something like:
import view
startChat()
#^ for testing, to see if the import works when directly calling file
def startChat(socket):
#datahandling
view.startGui()
My view is simply:
import tkinter
def startGui():
gui = tkinter.Tk()
gui.mainloop()
And lastly, the main is:
from subdirectory import controller
if __name__ == '__main__':
controller.startChat(s)
I removed all the meat to reduce myself to the GUI starting. When I run the controller everything works as it should, if I put the main into the subdirectory it also works.
Important note: This is a fix, but not what I want. I will soon again need different files, folders and directories and unless there is no way to do this would like to know a solution to this problem that doesn't involve putting everything in the same folder.
If I run the program as it is right now, it will execute the controller.py, if i put a print() above the imports (of which i have a few, like sys and time, which all work), it will only fail once it reaches the import view
The errormessage is a:
Exception has occurred: ModuleNotFoundError
No module named 'chat_view
My theory is that when calling from another directory the runtime has no info about the folder it is put into, and can't do what would happen if I started it from the directory. Which is what I tried to fix in the first and third solution of the other things I have tried so far:
Putting "from subdirectory import view" but that didn't work
Looking up this question on Google to no success
Importing the view in the main
Adding an init.py into the /subdirectory
As you might see I am more trying around and guessing and I think it's unlikely I find the solution to this anytime soon, so I thought to ask here. I wouldn't be surprised if this is a duplicate, but I wasn't able to find what I was looking for.
Instead of an absolute import, you can use a relative import in your controller
from . import view
the dot means it will search in the same folder.
A weird thing worked that I hope someone else can EXPLAIN, but here for future reference what I did was:
I put all the files into a directory names "src" like
directory/src #this is how it is displayed in VSC, idk why it is not displayed as extra directory but like this?
main.py
/subdirectory
__init__.py
#other files
I did this to clean up my code but then in the controller the import import view didn't work anymore, so when I changed it to from subdirectory import view (which earlier caused an error) it now works calling it from the main.py
This is bizarre to me as all I did was add a directory, but it works, so, yes. This is the accepted answer for now, until someone explains it better than me then I will switch it
I've looked on many sites and many related questions, but following the solutions to those questions still didn't seem to help. I figured maybe I am missing something, so here goes.
My project is to create a DM's tool for managing table top role playing games. I need to be able to split my project into many different files in order to keep everything organized. (so far) I have only three files I'm trying to work with. I have my main file which I called dmtool.py3, I have a file for class definitions called classdef.py3, and I have a file for creating race objects called races.py3.
1] The first of my questions is regarding importing singular files. I've tried organizing the files in several different ways, so for this lets assume all of my three files are in the same directory.
If I want to import my class definitions from classdef.py3 into my main file dmtool.py3, how would I do that? import classdef and import classdef.py3 do not seem to work properly saying their is no module with that name.
2] So I then made a module, and it seemed to work. I did this by creating a sub-directory called defs and putting the classdef.py3 and races.py3 files into it. I created the __init__.py3 file, and put import defs in dmtool.py3. As a test I put x = 1 at the very top of races.py3 and put print("X =", defs.x) in dmtool.py3. I get an error saying that module doesn't have an attribute x.
So I guess my second question is if it is possible to just use variables from other files. Would I use something like defs.x or defs.races.x or races.x or maybe simply x? I can't seem to find the one that works. I need to figure this out because I will be using specific instances of a class that will be defined in the races.py3 file.
3] My third question is a simple one that kind of spawned from the previous two. Now that races.py3 and classdef.py3 are in the same module, how do I make one access the other. races.py3 has to use the classes defined in classdef.py3.
I would really appreciate any help. Like I said I tried looking up other questions related to importing, but their simple solutions seemed to come up with the same errors. I didn't post my specific files because other than what I mentioned, there is just very simple print lines or class definitions. Nothing that should affect the importing.
Thanks,
Chris
Firstly, do not use .py3 as a file extension. Python doesn't recognize it.
Python 3's import system is actually quite simple. import foo looks through sys.path for a package (directory) or module (.py file) named foo.
sys.path contains various standard directories where you would normally install libraries, as well as the Python standard library. The first entry of sys.path is usually the directory in which the __main__ script lives. If you invoke Python as python -m foo.bar, the first entry will instead be the directory which contains the foo package.
Relative imports use a different syntax:
from . import foo
This means "import the foo module or package from the package which contains the current module." It is discussed in detail in PEP 328, and can be useful if you don't want to specify the entire structure of your packages for every import.
Start python and type these commands:
>>> import sys
>>> sys.path
The path is the list of directories where python looks for libraries. If your modules are not on the list, none are found.
The reason I want to this is I want to use the tool pyobfuscate to obfuscate my python code. Butpyobfuscate can only obfuscate one file.
I've answered your direct question separately, but let me offer a different solution to what I suspect you're actually trying to do:
Instead of shipping obfuscated source, just ship bytecode files. These are the .pyc files that get created, cached, and used automatically, but you can also create them manually by just using the compileall module in the standard library.
A .pyc file with its .py file missing can be imported just fine. It's not human-readable as-is. It can of course be decompiled into Python source, but the result is… basically the same result you get from running an obfuscater on the original source. So, it's slightly better than what you're trying to do, and a whole lot easier.
You can't compile your top-level script this way, but that's easy to work around. Just write a one-liner wrapper script that does nothing but import the real top-level script. If you have if __name__ == '__main__': code in there, you'll also need to move that to a function, and the wrapper becomes a two-liner that imports the module and calls the function… but that's as hard as it gets.) Alternatively, you could run pyobfuscator on just the top-level script, but really, there's no reason to do that.
In fact, many of the packager tools can optionally do all of this work for you automatically, except for writing the trivial top-level wrapper. For example, a default py2app build will stick compiled versions of your own modules, along with stdlib and site-packages modules you depend on, into a pythonXY.zip file in the app bundle, and set up the embedded interpreter to use that zipfile as its stdlib.
There are a definitely ways to turn a tree of modules into a single module. But it's not going to be trivial. The simplest thing I can think of is this:
First, you need a list of modules. This is easy to gather with the find command or a simple Python script that does an os.walk.
Then you need to use grep or Python re to get all of the import statements in each file, and use that to topologically sort the modules. If you only do absolute flat import foo statements at the top level, this is a trivial regex. If you also do absolute package imports, or from foo import bar (or from foo import *), or import at other levels, it's not much trickier. Relative package imports are a bit harder, but not that big of a deal. Of course if you do any dynamic importing, use the imp module, install import hooks, etc., you're out of luck here, but hopefully you don't.
Next you need to replace the actual import statements. With the same assumptions as above, this can be done with a simple sed or re.sub, something like import\s+(\w+) with \1 = sys.modules['\1'].
Now, for the hard part: you need to transform each module into something that creates an equivalent module object dynamically. This is the hard part. I think what you want to do is to escape the entire module code so that it can put into a triple-quoted string, then do this:
import types
mod_globals = {}
exec('''
# escaped version of original module source goes here
''', mod_globals)
mod = types.ModuleType(module_name)
mod.__dict__.update(mod_globals)
sys.modules[module_name] = mod
Now just concatenate all of those transformed modules together. The result will be almost equivalent to your original code, except that it's doing the equivalent of import foo; del foo for all of your modules (in dependency order) right at the start, so the startup time could be a little slower.
You can make a tool that:
Reads through your source files and puts all identifiers in a set.
Subtracts all identifiers from recursively searched standard- and third party modules from that set (modules, classes, functions, attributes, parameters).
Subtracts some explicitly excluded identifiers from that list as well, as they may be used in getattr/setattr/exec/eval
Replaces the remaining identifiers by gibberish
Or you can use this tool I wrote that does exactly that.
To obfuscate multiple files, use it as follows:
For safety, backup your source code and valuable data to an off-line medium.
Put a copy of opy_config.txt in the top directory of your project.
Adapt it to your needs according to the remarks in opy_config.txt.
This file only contains plain Python and is exec’ed, so you can do anything clever in it.
Open a command window, go to the top directory of your project and run opy.py from there.
If the top directory of your project is e.g. ../work/project1 then the obfuscation result will be in ../work/project1_opy.
Further adapt opy_config.txt until you’re satisfied with the result.
Type ‘opy ?’ or ‘python opy.py ?’ (without the quotes) on the command line to display a help text.
I think you can try using the find command with -exec option.
you can execute all python scripts in a directory with the following command.
find . -name "*.py" -exec python {} ';'
Wish this helps.
EDIT:
OH sorry I overlooked that if you obfuscate files seperately they may not run properly, because it renames function names to different names in different files.
So at work i'm developing a way for us to push out tools/plugins to the team as a whole. I actually have the system up and running and it's completely dynamic except for the topic i'm about to talk about (this is all done in python). On start up Maya checks the local folder against a folder on the server and checks to see if they are different and handles and copying down files/dirs that are needed as well as the deleting of old plugins that we delete on the server. The system is flexible enough that users can create custom shelves of all the plugins and we can re organize the folders in the back end without breaking the shelves of all the users. The plugins are accessed through a drop down menu in Maya's main interface and we can add sub folders into the system and plugins freely without messing with the code. We can also arrange the order at which menu items and plugins can be displayed through a simple numbering system of the folders.
This is all working fine until I get to making plugins, when they import a module in their folder, dynamic as well. So when I start moving the plugins folder around the root directory, if I have an imported module that I created the path for, the imported modules path in the plugin script is now wrong at that point. I already have a way of getting the proper path info to the plugin through my menu setup. I'm having issue with the import of the module and accessing classes with in that module.
so If the standard for importing a module's class
from fileName import className
and the __import__ way that i'm using looks like.
className = __import__("folderN.folderN.folderN.fileName", {}, {}, ["className"])
But with that method I loose the ability to just call on that class name like I can with the regular from import method. I got around that by doing
className = className.className
but this is a rather ugly method and i'd prefer to be able to just import and call on the name without doing that extra step. I do not know this import method very well and I know i'm missing some things with it.
Am I just going about this import process the wrong way? Is there a way to make it look into the local directory for the plugin without appending to maya's paths that way i can just do the regular way of importing method without a weird path that has to change anytime I move the plugin?
__import__ doesn't work they way you are assuming. It returns a module object for the import path provided, with a guarantee that all the children you specify in the list have been explicitly imported. For classes, it doesn't really make a difference.
mod = __import__('a.b', {}, {}, ['c', 'd'])
Is more accurately the equivalent of
import a.b
try:
import a.b.c
except ImportError:
pass
try:
import a.b.d
except ImportError:
pass
mod = a.b
What you actually probably want here is something like
child = getattr(__import__(import_path, {}, {}, [child_name]), child_name)
As to your versioning and distribution system, have you looked at using an SCM/VCS like SVN or GIT, and/or a package management system? These give you much better change tracking and synchronization than a straight file share, and you could easily integrate them with a sync/install script on your client machines that could be customizable as needed for the client's specific configuration demands.
More specifically let's say I have a number of .py files, with main.py importing stuff like os, pygame, math and all my other .py files, mymodule01.py etc.
My problem is that whenever main.py calls on one of my .py files and that file contains something like an os.listdir() I keep getting an error saying stuff like 'os is not defined'.
Should I just import all the required modules in each .py file I write, or is there a better way, like a centralized import that every file can recognize? With pygame especially this would be very confusing since I'd have to init pygame in each file just to use it's functions, not to mention if I want to blit something on the screen.
The python modules and packages documentation didn't help much, that or I'm really slow, also considering that after following the doc I keep getting a not found error after adding e.g. import mymodule01.py in the init.py file in the containing folder.
I think you may be under the impression that "import" acts like "include" in other languages. It doesn't.
Each module object is a singleton. There's no performance degradation or danger of initializing a module's code more than once.
Furthermore, each file has its own scope, so in your example if you define foo = 1 in main.py, foo won't be visible in mymodule01.py. You would have to import main; main.foo to see it (not that you should)
You grumble, but this is a much better system than include
Should I just import all the required modules in each .py file I write
Yes.
With pygame especially this would be very confusing since I'd have to init pygame in each file just to use it's functions
No, only init it once. There's only one copy of the module.