How should I handle having a local dependency shared by two projects? - python

Here's the theoretical scenario:
Project_A and Project_B are two independent processes for the client. Both projects rely on a custom module called Module_X.
I have the following folder structure currently :
project_a
project_a/module_x
project_b
project_b/module_x
This functionally works but has the drawback of if I do a change in project_b/module_x I then have to manually copy over the updated contents of module_x to project_a/module_x so that they are congruent and I get consistent results between the two projects.
What is a better way in which I can handle this?
module_x is not able to be put up on pypi as it contains logic for sensitive information resources.

This will add that path to paths where python searches for modules
import sys
sys.path.insert(1, '/home/user/projectb')

Related

Get resource from python resource root

I am using the PyCharm IDE. I marked a folder as a resource root and wanted to get a file from its directory and was wondering the appropriate way to do so.
In Java, you can use getClass().getResource("/resourceName.extension")
Is there some way to get a path from a python in said manner?
Based off what you have said it sounds like you just need to include the directory of the file with a simple include statement.
for instance, if your files are set up as such:
c:program\main
c:program\resources
then you can just do a simple
import resources
However, you could run into coupling issues if you have any sub-packages. Solving the coupling issue involving resources has been gone over in more detail in another thread I have linked below.
Managing resources in a Python project
What I want can be accomplished by this answer.
I used the code as follows:
os.path.join(os.path.dirname(__file__), '../audio/music.wav'

I'm having trouble understanding importing in python3

I've looked on many sites and many related questions, but following the solutions to those questions still didn't seem to help. I figured maybe I am missing something, so here goes.
My project is to create a DM's tool for managing table top role playing games. I need to be able to split my project into many different files in order to keep everything organized. (so far) I have only three files I'm trying to work with. I have my main file which I called dmtool.py3, I have a file for class definitions called classdef.py3, and I have a file for creating race objects called races.py3.
1] The first of my questions is regarding importing singular files. I've tried organizing the files in several different ways, so for this lets assume all of my three files are in the same directory.
If I want to import my class definitions from classdef.py3 into my main file dmtool.py3, how would I do that? import classdef and import classdef.py3 do not seem to work properly saying their is no module with that name.
2] So I then made a module, and it seemed to work. I did this by creating a sub-directory called defs and putting the classdef.py3 and races.py3 files into it. I created the __init__.py3 file, and put import defs in dmtool.py3. As a test I put x = 1 at the very top of races.py3 and put print("X =", defs.x) in dmtool.py3. I get an error saying that module doesn't have an attribute x.
So I guess my second question is if it is possible to just use variables from other files. Would I use something like defs.x or defs.races.x or races.x or maybe simply x? I can't seem to find the one that works. I need to figure this out because I will be using specific instances of a class that will be defined in the races.py3 file.
3] My third question is a simple one that kind of spawned from the previous two. Now that races.py3 and classdef.py3 are in the same module, how do I make one access the other. races.py3 has to use the classes defined in classdef.py3.
I would really appreciate any help. Like I said I tried looking up other questions related to importing, but their simple solutions seemed to come up with the same errors. I didn't post my specific files because other than what I mentioned, there is just very simple print lines or class definitions. Nothing that should affect the importing.
Thanks,
Chris
Firstly, do not use .py3 as a file extension. Python doesn't recognize it.
Python 3's import system is actually quite simple. import foo looks through sys.path for a package (directory) or module (.py file) named foo.
sys.path contains various standard directories where you would normally install libraries, as well as the Python standard library. The first entry of sys.path is usually the directory in which the __main__ script lives. If you invoke Python as python -m foo.bar, the first entry will instead be the directory which contains the foo package.
Relative imports use a different syntax:
from . import foo
This means "import the foo module or package from the package which contains the current module." It is discussed in detail in PEP 328, and can be useful if you don't want to specify the entire structure of your packages for every import.
Start python and type these commands:
>>> import sys
>>> sys.path
The path is the list of directories where python looks for libraries. If your modules are not on the list, none are found.

importing modules/scripts in python

So at work i'm developing a way for us to push out tools/plugins to the team as a whole. I actually have the system up and running and it's completely dynamic except for the topic i'm about to talk about (this is all done in python). On start up Maya checks the local folder against a folder on the server and checks to see if they are different and handles and copying down files/dirs that are needed as well as the deleting of old plugins that we delete on the server. The system is flexible enough that users can create custom shelves of all the plugins and we can re organize the folders in the back end without breaking the shelves of all the users. The plugins are accessed through a drop down menu in Maya's main interface and we can add sub folders into the system and plugins freely without messing with the code. We can also arrange the order at which menu items and plugins can be displayed through a simple numbering system of the folders.
This is all working fine until I get to making plugins, when they import a module in their folder, dynamic as well. So when I start moving the plugins folder around the root directory, if I have an imported module that I created the path for, the imported modules path in the plugin script is now wrong at that point. I already have a way of getting the proper path info to the plugin through my menu setup. I'm having issue with the import of the module and accessing classes with in that module.
so If the standard for importing a module's class
from fileName import className
and the __import__ way that i'm using looks like.
className = __import__("folderN.folderN.folderN.fileName", {}, {}, ["className"])
But with that method I loose the ability to just call on that class name like I can with the regular from import method. I got around that by doing
className = className.className
but this is a rather ugly method and i'd prefer to be able to just import and call on the name without doing that extra step. I do not know this import method very well and I know i'm missing some things with it.
Am I just going about this import process the wrong way? Is there a way to make it look into the local directory for the plugin without appending to maya's paths that way i can just do the regular way of importing method without a weird path that has to change anytime I move the plugin?
__import__ doesn't work they way you are assuming. It returns a module object for the import path provided, with a guarantee that all the children you specify in the list have been explicitly imported. For classes, it doesn't really make a difference.
mod = __import__('a.b', {}, {}, ['c', 'd'])
Is more accurately the equivalent of
import a.b
try:
import a.b.c
except ImportError:
pass
try:
import a.b.d
except ImportError:
pass
mod = a.b
What you actually probably want here is something like
child = getattr(__import__(import_path, {}, {}, [child_name]), child_name)
As to your versioning and distribution system, have you looked at using an SCM/VCS like SVN or GIT, and/or a package management system? These give you much better change tracking and synchronization than a straight file share, and you could easily integrate them with a sync/install script on your client machines that could be customizable as needed for the client's specific configuration demands.

Reliable way to get path to py file of a module

I'm trying to figure out the best way to reliably discover at runtime the location on the file system of the py file for a given module. I need to do this because I plan to externalize some configuration data about some methods (in this case, schemas to be used for validation of responses from service calls for which interfaces are defined in a module) for cleanliness and ease of maintenance.
An simplified illusutration of the system:
package
|
|-service.py
|
|-call1.scm
|
|-call2.scm
service.py (_call() is a method on the base class, though that's irrelevant to the question)
class FooServ(AbstractService):
def call1(*args):
result = self._call('/relative/uri/call1', *args)
# additional call specific processing
return result
def call2(*args):
result = self._call('/relative/uri/call2', *args)
# additional call specific processing
return result
call1.scm and call2.scm define the response schemas (in the current case, using the draft JSON schema format, though again, irrelevant to the question)
In another place in the codebase, when the service calls are actually made, I want to be able to detect the location of service.py so that I can traverse the file structure and find the scm files. At least on my system, I think that this will work:
# I realize this is contrived here, but in my code, the method is accessed this way
method = FooServ().call1
module_path = sys.modules[method.__self__.__class__.__module__].__file__
schema_path = os.path.join(os.path.dirname(module_path), method.__name__ + '.scm')
However, I wanted to make sure this would be safe on all platforms and installation configurations, and I came across this while doing research, which made me concerned that trying to do this this way will not work reliably. Will this work universally, or is the fact that __file__ on a module object returns the location of the pyc file, which could be in some location other than along side the py file, going to make this solution ineffective? If it will make it ineffective, what, if anything, can I do instead?
In the PEP you link, it says:
In Python 3, when you import a module, its __file__ attribute points to its source py file (in Python 2, it points to the pyc file).
So in Python 3 you're fine because __file__ will always point to the .py file. In Python 2 it might point to the .pyc file, but that .pyc will only ever be in the same directory as the .py file for Python 2.
Okay, I think you're referring to this bit:
Because these distributions cannot share pyc files, elaborate mechanisms have been developed to put the resulting pyc files in non-shared locations while the source code is still shared. Examples include the symlink-based Debian regimes python-support [8] and python-central [9]. These approaches make for much more complicated, fragile, inscrutable, and fragmented policies for delivering Python applications to a wide range of users.
I believe those mechanisms are applied only to Python modules that are packaged by the distribution. I don't think they should affect modules installed manually outside of the distribution's packaging system. It would be the responsibility of whoever was packaging your module for that distribution to make sure that the module isn't broken by the mechanism.

Python: Create virtual import path

Is there any way to create a virtual import path in Python?
My directory structure is like this:
/
native
scripts
some.py
another.py
[Other unrelated dirs]
The root is the directory from where the program is executed. Atm I add native/scripts/ to the search path so I can do import some, another instead of from native.scripts import some, another, but I'd like to be able to do it like this:
from native import some
import native.another
Is there any way to achieve this?
Related questions:
Making a virtual package available via sys.modules
Why not move some.py and another.py out into the native directory so that everything Just Works and so that people returning to the source code later won't be confused about why things are and aren't importable? :)
Update:
Thanks for your comments; they have usefully clarified the problem! In your case, I generally put functions and classes that I might want to import inside, say, native.some where I can easily get to them. But then I get the script code, and only the script code — only the thin shim that interprets arguments and starts everything running by passing those to a main() or go() function as parameters — and put that inside of a scripts directory. That keeps external-interface code cleanly separate from code that you might want to import, and means you don't have to try to fool Python into having modules several places at once.
In /native/__init__.py, include:
from scripts import some, another

Categories

Resources