More specifically let's say I have a number of .py files, with main.py importing stuff like os, pygame, math and all my other .py files, mymodule01.py etc.
My problem is that whenever main.py calls on one of my .py files and that file contains something like an os.listdir() I keep getting an error saying stuff like 'os is not defined'.
Should I just import all the required modules in each .py file I write, or is there a better way, like a centralized import that every file can recognize? With pygame especially this would be very confusing since I'd have to init pygame in each file just to use it's functions, not to mention if I want to blit something on the screen.
The python modules and packages documentation didn't help much, that or I'm really slow, also considering that after following the doc I keep getting a not found error after adding e.g. import mymodule01.py in the init.py file in the containing folder.
I think you may be under the impression that "import" acts like "include" in other languages. It doesn't.
Each module object is a singleton. There's no performance degradation or danger of initializing a module's code more than once.
Furthermore, each file has its own scope, so in your example if you define foo = 1 in main.py, foo won't be visible in mymodule01.py. You would have to import main; main.foo to see it (not that you should)
You grumble, but this is a much better system than include
Should I just import all the required modules in each .py file I write
Yes.
With pygame especially this would be very confusing since I'd have to init pygame in each file just to use it's functions
No, only init it once. There's only one copy of the module.
Related
I am moving common functions shared between a couple of Python files in to a third common_commands.py file. To test the functions I imported a couple of functions using:
from common_commands import func_one, func_two
But got this error:
ImportError: cannot import name 'func_two'
So I tried only importing func_one and that worked fine but then importing just func_two gives me the same error again! Why?! And to make things even confusing, the exact same import line from above works perfectly fine when I put it in the scripts I am refactoring.
What is causing this odd behavior?
TL;DR:
I had renamed func_twosince starting my interactive shell. Starting a new shell got things working.
What I learned:
I don't understand all the inner workings of the interactive shell and what happens when you call import, but after quitting and starting a new shell the exact same import call worked.
When I started the shell func_two was old_func_two but then I decided to rename it, and then I tried to import it with the new name, and that failed. After scratching my head and doing some google foo I found nothing that helped in my case and tried starting a new shell and it worked!
So I decided to do a little more experimenting before asking this question and learned that I could rename the function as much as I wanted after starting the shell but only until I first imported the file in some way.
That is to say, as soon as I called from common_commands import func_one I can no longer rename any functions and import them with the new name, since the file has already been imported. I can, however, still import old_func_two. I also tried changing the 'guts' of func_two after importing it and then importing it again and it kept the original behavior. So from what I can tell, the first time you import a file (not a function or class, but the whole file) it is cached and all future imports are run on the cached version, not the real file on disk.
So, even if you only import func_one i.e. from common_commands import func_one and then rename or change func_two and then import it, you'll have to use the original name of func_two and you'll get the original functionality as well, even though you didn't explicitly import it earlier.
Often I write command line utilities that are only meant to be run as main. For example, I might have a file that looks like this:
#!/usr/bin/env python
if __name__ == '__main__':
import sys
# do stuff
In other words, there is nothing going on that isn't under the if statement checking that this file is being run as main. I tried importing a file like this to see what would happen, and the import was successful.
So as I expected, one is allowed to import files like this, but what is the convention surrounding this practice? Is one supposed to throw an error telling the user that there is nothing to be imported? Or if all the contents of the file are supposed to be run as main, does one need to check if the program is being run as main? Or is the conditional not necessary?
Also, if I have import statements, should they be at the top of the file, or under the conditional? If the modules are only being used under the conditional, it would seem to me that they should be imported under the conditional and not at the top of the file.
If you are writing simple utilities that you are entirely certain that you will never import as a module in another program, then you really do not need to include the if __name__ == '__main__' stuff. The fundamental point of that construct is to allow a module to be developed that can both be imported as a module for use, and run as a stand-alone program for some other purpose. For example, if you had a module and had some test vectors you wanted to run on it regularly, you would put the trigger mechanism for your test vectors in the if __name__ block.
Another example might be if you have a stand-alone program that you develop, that would also provide useful functions for others. If you have a look at the pip module, this is an excellent example of this technique.
I've looked on many sites and many related questions, but following the solutions to those questions still didn't seem to help. I figured maybe I am missing something, so here goes.
My project is to create a DM's tool for managing table top role playing games. I need to be able to split my project into many different files in order to keep everything organized. (so far) I have only three files I'm trying to work with. I have my main file which I called dmtool.py3, I have a file for class definitions called classdef.py3, and I have a file for creating race objects called races.py3.
1] The first of my questions is regarding importing singular files. I've tried organizing the files in several different ways, so for this lets assume all of my three files are in the same directory.
If I want to import my class definitions from classdef.py3 into my main file dmtool.py3, how would I do that? import classdef and import classdef.py3 do not seem to work properly saying their is no module with that name.
2] So I then made a module, and it seemed to work. I did this by creating a sub-directory called defs and putting the classdef.py3 and races.py3 files into it. I created the __init__.py3 file, and put import defs in dmtool.py3. As a test I put x = 1 at the very top of races.py3 and put print("X =", defs.x) in dmtool.py3. I get an error saying that module doesn't have an attribute x.
So I guess my second question is if it is possible to just use variables from other files. Would I use something like defs.x or defs.races.x or races.x or maybe simply x? I can't seem to find the one that works. I need to figure this out because I will be using specific instances of a class that will be defined in the races.py3 file.
3] My third question is a simple one that kind of spawned from the previous two. Now that races.py3 and classdef.py3 are in the same module, how do I make one access the other. races.py3 has to use the classes defined in classdef.py3.
I would really appreciate any help. Like I said I tried looking up other questions related to importing, but their simple solutions seemed to come up with the same errors. I didn't post my specific files because other than what I mentioned, there is just very simple print lines or class definitions. Nothing that should affect the importing.
Thanks,
Chris
Firstly, do not use .py3 as a file extension. Python doesn't recognize it.
Python 3's import system is actually quite simple. import foo looks through sys.path for a package (directory) or module (.py file) named foo.
sys.path contains various standard directories where you would normally install libraries, as well as the Python standard library. The first entry of sys.path is usually the directory in which the __main__ script lives. If you invoke Python as python -m foo.bar, the first entry will instead be the directory which contains the foo package.
Relative imports use a different syntax:
from . import foo
This means "import the foo module or package from the package which contains the current module." It is discussed in detail in PEP 328, and can be useful if you don't want to specify the entire structure of your packages for every import.
Start python and type these commands:
>>> import sys
>>> sys.path
The path is the list of directories where python looks for libraries. If your modules are not on the list, none are found.
The reason I want to this is I want to use the tool pyobfuscate to obfuscate my python code. Butpyobfuscate can only obfuscate one file.
I've answered your direct question separately, but let me offer a different solution to what I suspect you're actually trying to do:
Instead of shipping obfuscated source, just ship bytecode files. These are the .pyc files that get created, cached, and used automatically, but you can also create them manually by just using the compileall module in the standard library.
A .pyc file with its .py file missing can be imported just fine. It's not human-readable as-is. It can of course be decompiled into Python source, but the result is… basically the same result you get from running an obfuscater on the original source. So, it's slightly better than what you're trying to do, and a whole lot easier.
You can't compile your top-level script this way, but that's easy to work around. Just write a one-liner wrapper script that does nothing but import the real top-level script. If you have if __name__ == '__main__': code in there, you'll also need to move that to a function, and the wrapper becomes a two-liner that imports the module and calls the function… but that's as hard as it gets.) Alternatively, you could run pyobfuscator on just the top-level script, but really, there's no reason to do that.
In fact, many of the packager tools can optionally do all of this work for you automatically, except for writing the trivial top-level wrapper. For example, a default py2app build will stick compiled versions of your own modules, along with stdlib and site-packages modules you depend on, into a pythonXY.zip file in the app bundle, and set up the embedded interpreter to use that zipfile as its stdlib.
There are a definitely ways to turn a tree of modules into a single module. But it's not going to be trivial. The simplest thing I can think of is this:
First, you need a list of modules. This is easy to gather with the find command or a simple Python script that does an os.walk.
Then you need to use grep or Python re to get all of the import statements in each file, and use that to topologically sort the modules. If you only do absolute flat import foo statements at the top level, this is a trivial regex. If you also do absolute package imports, or from foo import bar (or from foo import *), or import at other levels, it's not much trickier. Relative package imports are a bit harder, but not that big of a deal. Of course if you do any dynamic importing, use the imp module, install import hooks, etc., you're out of luck here, but hopefully you don't.
Next you need to replace the actual import statements. With the same assumptions as above, this can be done with a simple sed or re.sub, something like import\s+(\w+) with \1 = sys.modules['\1'].
Now, for the hard part: you need to transform each module into something that creates an equivalent module object dynamically. This is the hard part. I think what you want to do is to escape the entire module code so that it can put into a triple-quoted string, then do this:
import types
mod_globals = {}
exec('''
# escaped version of original module source goes here
''', mod_globals)
mod = types.ModuleType(module_name)
mod.__dict__.update(mod_globals)
sys.modules[module_name] = mod
Now just concatenate all of those transformed modules together. The result will be almost equivalent to your original code, except that it's doing the equivalent of import foo; del foo for all of your modules (in dependency order) right at the start, so the startup time could be a little slower.
You can make a tool that:
Reads through your source files and puts all identifiers in a set.
Subtracts all identifiers from recursively searched standard- and third party modules from that set (modules, classes, functions, attributes, parameters).
Subtracts some explicitly excluded identifiers from that list as well, as they may be used in getattr/setattr/exec/eval
Replaces the remaining identifiers by gibberish
Or you can use this tool I wrote that does exactly that.
To obfuscate multiple files, use it as follows:
For safety, backup your source code and valuable data to an off-line medium.
Put a copy of opy_config.txt in the top directory of your project.
Adapt it to your needs according to the remarks in opy_config.txt.
This file only contains plain Python and is exec’ed, so you can do anything clever in it.
Open a command window, go to the top directory of your project and run opy.py from there.
If the top directory of your project is e.g. ../work/project1 then the obfuscation result will be in ../work/project1_opy.
Further adapt opy_config.txt until you’re satisfied with the result.
Type ‘opy ?’ or ‘python opy.py ?’ (without the quotes) on the command line to display a help text.
I think you can try using the find command with -exec option.
you can execute all python scripts in a directory with the following command.
find . -name "*.py" -exec python {} ';'
Wish this helps.
EDIT:
OH sorry I overlooked that if you obfuscate files seperately they may not run properly, because it renames function names to different names in different files.
Is there any way to create a virtual import path in Python?
My directory structure is like this:
/
native
scripts
some.py
another.py
[Other unrelated dirs]
The root is the directory from where the program is executed. Atm I add native/scripts/ to the search path so I can do import some, another instead of from native.scripts import some, another, but I'd like to be able to do it like this:
from native import some
import native.another
Is there any way to achieve this?
Related questions:
Making a virtual package available via sys.modules
Why not move some.py and another.py out into the native directory so that everything Just Works and so that people returning to the source code later won't be confused about why things are and aren't importable? :)
Update:
Thanks for your comments; they have usefully clarified the problem! In your case, I generally put functions and classes that I might want to import inside, say, native.some where I can easily get to them. But then I get the script code, and only the script code — only the thin shim that interprets arguments and starts everything running by passing those to a main() or go() function as parameters — and put that inside of a scripts directory. That keeps external-interface code cleanly separate from code that you might want to import, and means you don't have to try to fool Python into having modules several places at once.
In /native/__init__.py, include:
from scripts import some, another