Recently I started working on a personal project in my notebook that, all going OK, it will be placed in a server elsewhere. The problem is that I make use of modules. Some were installed from apt-get, others from easy_install and one or two of those were placed directly under a subdirectory since I changed them a bit. My question is: is there a way to move all those things together? Moreover, I don't want any of those modules being updated since it may break something. How to handle that?
Finally, I'm pretty sure that I've done things the wrong way since the beginning. How do you guys work to avoid those problems?
Have a look at virtualenv. Virtualenv is a tool to create isolated Python environments.
Related
There probably isn't one "right answer" to this question. I'm interested in thoughts and opinions.
We have a couple hundred RHEL7/Centos7/Rocky8 nodes. Many of them have python modules installed via pip/pip3.
I've been searching for a best practices on routine/monthly patching these modules...so I far haven't found any. Obviously things installed with rpm/yum/dnf are pretty easy to deal with.
From the pip man page:
pip install --upgrade SomePackage
Great!
But how do you update all of them?
Sure. It is possible to do a "pip list/freeze" pipe that to awk...etc..
Surely, there's a better way. Ideally, one that captures things like "boto3 V1.2 replaced with boto3 V1.3"
Right now it feels like I'm the only one thinking about this. Maybe I am and it is stupid. I'm ok with that response as well (but please tell me why).
A common solution is to deploy the application code inside a Docker container - the container image contains its own version of Python and all the dependency modules, so you don't have to update each module on all the host machines individually. It also means that the combination of OS, Python and modules that you deploy can be tested and then "frozen" into an immutable image which is then deployed the same everywhere.
Right now it feels like I'm the only one thinking about this.
I realise the above answer is probably not helpful in your situation as you already have a fairly large system deployed... but it might help to explain why not many people are developing solutions to your problem!
I want to know if I can create a python script with a folder in the same directory with all the assets of a python module, so when someone wants to use it, they would not have to pip install module, because it would import from the directory.
Yes, you can, but it doesn't mean that you should.
First, ask yourself who is suposed to use that code.
If you plan to give it to consumers, it would be a good idea to use a tool like py2exe and create executable file which would include all modules and not allow for code to be changed.
If you plan to share it with another developer, you might want to look into virtual environments and requirements.txt file.
There are multiple reasons why sharing modules is bad idea:
It is harder to update modules later, at least without upgrading whole project.
It uses more space on version control, which can create issues on huge projects with hundreds of modules and branches
It might be illegal as some licenses specifically forbid including their code in your source code.
The pip install of some module might do different things depending on operating system version or installed packages. The modules on your machine might be suboptimal on someone else's machine, and in some instances might not even work.
And probably more that I can't think of right now.
The only situation where I saw this being unavoidable was when the module didn't support python implementation the application was running on. The module was changed, and its source was put under lib folder with the rest of the libraries.
I think you can add the directory with python modules into PYTHONPATH. Then people want to use those modules just need has this envvar set.
https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPATH
This may well stink of newbie but...
I'm working on my first Django project, and am reading a lot that using virtualenv is considered Best Practice. I understand that virtualenv sandboxes all my python dependencies but I just don't know if this is necessary if I'm working in sandboxed VM's anyway? I'm developing in Vagrant, and won't be using these VM's for anything else, and I'll be deploying to a VM server that will only have this Django project on it. Is it possible that in the future further Django apps in this project will require different dependencies and so need to be in different virtualenv's? (Not sure if it works like that tbh?)
Am I just showing my inexperience and shortsightedness?
I would always recommend you use a virtualenv as a matter of course. There is almost no overhead in doing so, and it just makes things easier. In conjunction with virtualenvwrapper you can easily just type workon myproject to activate and cd to your virtualenv in one go. You avoid any issues with having to use sudo to install things, as well as any possible version incompatibilities with system-installed packages. There's just no reason not to, really.
I don't have any knowledge on Vagrant but I use virtualenvs for my Django projects. I would recommend it for anyone.
With that said, if you're only going to be using one Django project on a virtual machine you don't need to use a virtualenv. I haven't come across a situation where apps in the same project have conflicting dependencies. This could be a problem if you have multiple projects on the same machine however.
There are many benefit of working with virtual environment on your development machine.
You can go to any version of any supported module to check for issues
Your project runs under separate environment without conflicting with your system wide modules and settings
Testing is easy
Muliple version of same project can co-exist.
if you develop multiple projects with different django versions, virtualenv is just a must thing, there is no other way (not that i know). you feel in heaven in virtualenv if you once experience the dependency hell. Even if you develop one project I would recommend to code inside virtualenv, you never know what comes next, back in the days, my old laptop was almost crashing because of so many dependency problems, after i discovered virtualenv, my old laptop became a brand new laptop for my eyes..
No, in your case, you don't need to bother with virtualenv. Since you're using a dedicated virtual machine it's just a layer of complexity you, as a noob, don't really need.
Virtualenv is pretty simple, in concept and usage, so you'll layer it on simply enough when the need arises. But, imho, there is added value in learning how a python installation is truly laid out before adding indirection. When you hit a problem that it can solve, then go for it. But for now, keep it simple: don't bother.
Basically I'm using virtualenv and pip to manage my all 3rd-party modules, and so far it's going great. However as I'm developing, I found out that these 3rd-party modules got various very little bugs and I've been fixing them right at my 'virtualenv' folder in which obviously isn't under version control and those fixes will get lost if I ever do pip --upgrade or re-create the 'virtualenv'.
I've proposed fixes to their respective repo but some of them are not very active and it will took a while before my fix can be implemented.
My questions is, what is a good workflow in case like this? Should I just put the 3rd-party modules right under my project folder thus I can ensure my fixes would stay but I've read doing that is bad?
What you are describing is a difficult problem that does not have a good solution. I would generally copy the module into my own project, but I would only do that as a last resort, and I have not had to do so yet.
Instead, if I'm using a module and it has a class Foo that isn't quite what I need. I will do:
class MyFoo(Foo):
...
and override whatever methods I need to. Then, I just use MyFoo in my project. This generally gives me the behavior I need without resorting to modifying the module's code.
But in general, if I'm inclined to modify the module's source code, I will first look extensively for alternatives. So far, I've been able to avoid it.
New to Python, so excuse my lack of specific technical jargon. Pretty simple question really, but I can't seem to grasp or understand the concept.
It seems that a lot of modules require using pip or easy_install and running setup.py to "install" into your python installation or your virtualenv. What is the difference between installing a module and simply taking it and importing the into another script? It seems that you access the modules the same way.
Thanks!
It's like the difference between:
Uploading a photo to the internet
Linking the photo URL inside an HTML page
Installing puts the code somewhere python expects those kinds of things to be, and the import statement says "go look there for something named X now, and make the data available to me for use".
For a single module, it usually doesn't make any difference. For complicated webs of modules, though, an installation program may do many things that wouldn't be immediately obvious. For example, it may also copy data files into locations the new modules can find them, put executables (binary libraries, or DLLs on Windws, for example) where the new modules can find them, do different things depending on which version of Python you have, and so on.
If deploying a web of modules were always easy, nobody would have written setup programs to begin with ;-)