Basically I'm using virtualenv and pip to manage my all 3rd-party modules, and so far it's going great. However as I'm developing, I found out that these 3rd-party modules got various very little bugs and I've been fixing them right at my 'virtualenv' folder in which obviously isn't under version control and those fixes will get lost if I ever do pip --upgrade or re-create the 'virtualenv'.
I've proposed fixes to their respective repo but some of them are not very active and it will took a while before my fix can be implemented.
My questions is, what is a good workflow in case like this? Should I just put the 3rd-party modules right under my project folder thus I can ensure my fixes would stay but I've read doing that is bad?
What you are describing is a difficult problem that does not have a good solution. I would generally copy the module into my own project, but I would only do that as a last resort, and I have not had to do so yet.
Instead, if I'm using a module and it has a class Foo that isn't quite what I need. I will do:
class MyFoo(Foo):
...
and override whatever methods I need to. Then, I just use MyFoo in my project. This generally gives me the behavior I need without resorting to modifying the module's code.
But in general, if I'm inclined to modify the module's source code, I will first look extensively for alternatives. So far, I've been able to avoid it.
Related
There’s an open source python library, let’s call it foo, and several other open source projects that depend on it, foo-configurator, foo-manager, etc.
I want to make a fork of foo to change the way it works under the hood (in a way that it needs to be a fork, it’s a library that interfaces with industrial hardware and we’re making one that interacts with our stuff that’s pretty different than what’s existing). So my fork would be called something like other-foo. The problem is I still want the other open source projects which use foo to be able to work with my other-foo as well, but they will fail on installation since their dependency check is looking foo and not other-foo.
Is there a way for me to handle this as the creator of other-foo, to spoof or somehow present as foo in terms of dependencies, but still have a package name of other-foo?
These foo libraries deal with industrial hardware devices, so there’s no chance that foo and other-foo will be installed at the same time.
The only way I can think to solve this would be to change the setup configurations of the other utility projects, (foo-configurator, etc.) to use dependency options and extras_require settings, so someone could install them in an environment with other-foo by using pip install foo-configurator[other-foo] or whatever. But that would require me to get these other projects on board, and also I think the extras_require can only add package dependencies to what’s already there, so them adding and extras_require for other-foo won’t remove the need for foo, so maybe that’s not the correct route?
The only other way I can think to do this is to fork all those other projects too, and change just the setup config to require other-foo instead of foo, but then I would need to maintain forks and that seems like a huge pain.
So, any ideas for the best, most pythonic way to handle this situation? Thanks!
Would it be possible to create a python module that lazily downloads and installs submodules as needed? I've worked with "subclassed" modules that mimic real modules, but I've never tried to do so with downloads involved. Is there a guaranteed directory that I can download source code and data to, that the module would then be able to use on subsequent runs?
To make this more concrete, here is the ideal behavior:
User runs pip install magic_module and the lightweight magic_module is installed to their system.
User runs the code import magic_module.alpha
The code goes to a predetermine URL, is told that there is an "alpha" subpackage, and is then given the URLs of alpha.py and alpha.csv files.
The system downloads these files to somewhere that it knows about, and then loads the alpha module.
On subsequent runs, the user is able to take advantage of the downloaded files to skip the server trip.
At some point down the road, the user could run a import magic_module.alpha ; alpha._upgrade() function from the command line to clear the cache and get the latest version.
Is this possible? Is this reasonable? What kinds of problems will I run into with permissions?
Doable, certainly. The core feature will probably be import hooks. The relevant module would be importlib in python 3.
Extending the import mechanism is needed when you want to load modules that are stored in a non-standard way. Examples include [...] modules that are loaded from a database over a network.
Convenient, probably not. The import machinery is one of the parts of python that has seen several changes over releases. It's undergoing a full refactoring right now, with most of the existing things being deprecated.
Reasonable, well it's up to you. Here are some caveats I can think of:
Tricky to get right, especially if you have to support several python versions.
What about error handling? Should application be prepared for import to fail in normal circumstances? Should they degrade gracefully? Or just crash and spew a traceback?
Security? Basically you're downloading code from someplace, how do you ensure the connection is not being hijacked?
How about versionning? If you update some of the remote modules, how can make the application download the correct version?
Dependencies? Pushing of security updates? Permissions management?
Summing it up, you'll have to solve most of the issues of a package manager, along with securing downloads and permissions issues of course. All those issues are tricky to begin with, easy to get wrong with dire consequences.
So with all that in mind, it really comes down to how much resources you deem worth investing into that, and what value that adds over a regular use of readily available tools such as pip.
(the permission question cannot really be answered until you come up with a design for your package manager)
New to Python, so excuse my lack of specific technical jargon. Pretty simple question really, but I can't seem to grasp or understand the concept.
It seems that a lot of modules require using pip or easy_install and running setup.py to "install" into your python installation or your virtualenv. What is the difference between installing a module and simply taking it and importing the into another script? It seems that you access the modules the same way.
Thanks!
It's like the difference between:
Uploading a photo to the internet
Linking the photo URL inside an HTML page
Installing puts the code somewhere python expects those kinds of things to be, and the import statement says "go look there for something named X now, and make the data available to me for use".
For a single module, it usually doesn't make any difference. For complicated webs of modules, though, an installation program may do many things that wouldn't be immediately obvious. For example, it may also copy data files into locations the new modules can find them, put executables (binary libraries, or DLLs on Windws, for example) where the new modules can find them, do different things depending on which version of Python you have, and so on.
If deploying a web of modules were always easy, nobody would have written setup programs to begin with ;-)
I made a Python module (https://github.com/Yannbane/Tick.py) and a Python program (https://github.com/Yannbane/Flatland.py). The program imports the module, and without it, it cannot work. I have intended for people to download both of these files before they can run the program, but, I am concerned about this a bit.
In the program, I've added these lines:
sys.path.append("/home/bane/Tick.py")
import tick
"/home/bane/Tick.py" is the path to my local repo of the module that needs to be included, but this will obviously be different to other people! How can I solve this situation better?
What suggested by #Lattyware is a viable option. However, its not uncommon to have core dependencies boundled with the main program (Django and PyDev do this for example). This works fine especially if the main code is tweaked against a specific version of the library.
In order to avoid the troubles mentioned by Lattyware when it comes to code maintenance, you should look into git submodules, which allow precisely this kind of layout, keeping code versioning sane.
From the structure of your directory it seems that both files live in the same directory. This might be the tell-tale than they might be two modules of a same package. In that case you should simply add an empty file called __init__.py to the directory, and then your import could work by:
import bane.tick
or
from bane import tick
Oh, and yes... you should use lower case for module names (it's worth to take a in-depth look at PEP8 if you are going to code in python! :)
HTH!
You might want to try submitting your module to the Python Package Index, that way people can easily install it (pip tick) into their path, and you can just import it without having to add it to the python path.
Otherwise, I would suggest simply telling people to download the module as well, and place it in a subdirectory of the program. If you really feel that is too much effort, you could place a copy of the module into the repository for the program (of course, that means ensuring you keep both versions up-to-date, which is a bit of a pain, although I imagine it may be possible just to use a symlink).
It's also worth noting your repo name is a bit misleading, capitalisation is often important, so you might want to call the repo tick.py to match the module, and python naming conventions.
Recently I started working on a personal project in my notebook that, all going OK, it will be placed in a server elsewhere. The problem is that I make use of modules. Some were installed from apt-get, others from easy_install and one or two of those were placed directly under a subdirectory since I changed them a bit. My question is: is there a way to move all those things together? Moreover, I don't want any of those modules being updated since it may break something. How to handle that?
Finally, I'm pretty sure that I've done things the wrong way since the beginning. How do you guys work to avoid those problems?
Have a look at virtualenv. Virtualenv is a tool to create isolated Python environments.