Newbie Python Learner here,
A question for programmers,
English is not main language, will be hard to explain the question I wish to convey.
How do you programmers know which modules exist and which don't?
Say you are writing a script/program etc.
There are modules/functions etc. which you may need to create or use to either write your program, or perhaps complete it quicker, how will you know a required module/function etc. exists that may help you write your program? What prevents you from wasting your time in writing an entire module/function etc. which you might need to use which may already exist without you knowing so?
If you're asking how to find Python packages, in general, that can help you, usually a quick Google search or few will show you packages other people have used for similar problems as yours.
Experience is the short answer for this.
A more detailed explanation is understanding the scope of your needs. If you believe that the problem you are faced with is a common one, chances are that someone has come up with a solution for it and put it into a module. You become aware of modules by running into a challenge and then searching up how others have solved it. You will most likely run into others who have come across the same thing and have used others modules to solve it.
The more specific your problem is the less likely there will be a module already made for it. For example, plotting data is a widely common need, which is why the Matplotlib module is known by most python programmers. Searching the PyPi website will show you a lot of modules that can come in handy later.
Good Luck and have fun looking at all the oddly specific modules out there!
The main sources of info are
docs.python.org (The Tutorial also introduces important modules in the standard lib)
pypi.org
StackOverflow (of course)
Google
You can be almost sure that basic functionalities are provided by standard lib, pypi.org allows then to search by several criteria.
Related
I've just finished my course of Python, so now I can write my own script. So to do that I started to write a script with the module Scapy, but the problem is, the documentation of Scapy is used for the interpreter Scapy, so I don't know how to use it, find the functions, etc.
I've found a few tutorials in Internet with a few examples but it's pretty hard. For example, I've found in a script the function "set_payload" to inject some code in the layer but I really don't know where he found this function.
What's your suggestion for finding how a module works and how to write correctly with it? Because I don't really like to check and pick through other scripts on Internet.
If I have understood the question correctly, roughly what you are asking is how to find the best source to understand a module.
If you are using an inbuilt python module, the best source is the python documentation.
Scapy is not a built-in python module. So you may have some issues with some of the external modules (by external I mean the ones you need to explicitly install).
For those, if the docs aren't enough, I prefer to look at some of the github projects that may use that module one way or the other and most of the times it works out. If it doesn't, then I go to some blogs or some third party tutorials. There is no right way to do it, You will have to put in the effort where its needed.
I've never used Scapy but it seems well documented.
https://buildmedia.readthedocs.org/media/pdf/scapy/latest/scapy.pdf
This version appearing to have been released at the time of writing this.
Basically I'm using virtualenv and pip to manage my all 3rd-party modules, and so far it's going great. However as I'm developing, I found out that these 3rd-party modules got various very little bugs and I've been fixing them right at my 'virtualenv' folder in which obviously isn't under version control and those fixes will get lost if I ever do pip --upgrade or re-create the 'virtualenv'.
I've proposed fixes to their respective repo but some of them are not very active and it will took a while before my fix can be implemented.
My questions is, what is a good workflow in case like this? Should I just put the 3rd-party modules right under my project folder thus I can ensure my fixes would stay but I've read doing that is bad?
What you are describing is a difficult problem that does not have a good solution. I would generally copy the module into my own project, but I would only do that as a last resort, and I have not had to do so yet.
Instead, if I'm using a module and it has a class Foo that isn't quite what I need. I will do:
class MyFoo(Foo):
...
and override whatever methods I need to. Then, I just use MyFoo in my project. This generally gives me the behavior I need without resorting to modifying the module's code.
But in general, if I'm inclined to modify the module's source code, I will first look extensively for alternatives. So far, I've been able to avoid it.
New to Python, so excuse my lack of specific technical jargon. Pretty simple question really, but I can't seem to grasp or understand the concept.
It seems that a lot of modules require using pip or easy_install and running setup.py to "install" into your python installation or your virtualenv. What is the difference between installing a module and simply taking it and importing the into another script? It seems that you access the modules the same way.
Thanks!
It's like the difference between:
Uploading a photo to the internet
Linking the photo URL inside an HTML page
Installing puts the code somewhere python expects those kinds of things to be, and the import statement says "go look there for something named X now, and make the data available to me for use".
For a single module, it usually doesn't make any difference. For complicated webs of modules, though, an installation program may do many things that wouldn't be immediately obvious. For example, it may also copy data files into locations the new modules can find them, put executables (binary libraries, or DLLs on Windws, for example) where the new modules can find them, do different things depending on which version of Python you have, and so on.
If deploying a web of modules were always easy, nobody would have written setup programs to begin with ;-)
I have a directory with several python modules in it. Each module is mutually exclusive of all the others but after lots of trial and error, I have decided that each module horks out when using the multiprocessing functionality in Python. I have used the join() function on each process and its just not working like I want.
What I am really looking for is the ability to drop new mutually exclusive python modules in to the directory and have them invoked when the directory is launched. Does anyone know how to do this?
It sounds to me like you are asking about plugin architecture and sandboxing. Does that sound right?
The plugin component has been done and written about else where. SO has code examples on basic ways to import all the files.
The sandbox part is going to be a harder. Have a look at RestrictedPython and the Restricted Execution docs and the generally older but nevertheless helpful discussion of sandboxing.
If you aren't worried about untrusted code but rather want to isolate errors you could just wrap each module in a generic try/except that handles all errors. This would make debugging hard but would ensure that an error in one module didn't bring down the whole system.
If you aren't worried about untrused code but do need to have each file totally isolated then you might be best off looking into various systems of interprocess communication. I've actually had some luck using Redis for this (which sounds ridiculous but actually has been very easy and effective).
Anyway hopefully some of that helps you. Without more information it's hard to provide more than general thoughts and a guide to better googling.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I have write a python library app(which contains several *.py files). And several of my python projects need to reuse the code in the library app. What's the recommended best practice for reusing python code? Currently I have thought out three options:
Copy and paste. This is far away from best practice. It violates the
DRY principle.(Don't repeat yourself.)
Add the folder of the library app to the environment variable PYTHONPATH: export PYTHONPATH=/path/to/library/app. Then every projects on the same computer can reference the code in the library app.
And the folder of the library app to sys.path in python code: sys.path.append('/path/to/library/app')
Among the three options above which one do you prefer? What advantage does it have compared to the other two options? Do you have any other better options? It is much appreciated that if some one with years of python development experiences could answer this question.
Allow me to propose a fourth alternative: take the time to learn how to package your library and install it in your site-packages; it's easier than one may think and I'm convinced it's time well spent. This is a very good starting point: https://packaging.python.org/en/latest/
Of your three options, PYTHONPATH is the way to go. Copy & paste is clearly out, and adding code to your other projects to modify sys.path simply pollutes those files with knowledge about their environment.
A fourth option is, create a true installable package from your common code, and install it into your Python installation. Then you can simply import those modules like any other 3rd-party install code.
If its a shared library you should package it up and add it to site-packages and then you wont have to worry about setting anything up. This is the best option.
If you dont want to use site-packages, then use PYTHONPATH. Its why it exists, and it is the way to do what you want.
You might want to look into using site.addsitedir, path.append does not prevent duplicates. It will also allow you to leverage .pth files.
Dynamically setting/adding things to PYTHONPATH via sys.path achieves the same result, but it can complicate things if you choose to reuse your new library. It also causes issues if your paths change, you have to change code versus an environment variable. Unless it is completely necessary, you should not dynamically setup your python path.
Copy & Paste is not a re-use pattern. You aren't reusing anything you are duplicating and increasing maintenance.
The first way is, as you noted yourself, hardly acceptable as it has countless well-known problems.
The other two have their own problems. For starters, they require manual work when files move (particular bad if you mix it with the application logic, i.e. put it in *.py files instead of leaving it to the computer it runs on) and requires either fixed installation locations (absolute paths) or at least a certain directory structure (relative paths). IMHO, these ways are only acceptable if the applications aren't ever going to move anywhere else. As soon as that becomes required, you should give up and use the existing solution as the one reason not to use it, being overkill for small and local scripts, doesn't apply any more:
Make the common parts, which you apparently already treat as a free-standing library (good!), a fully-fledged project in its own right, with a setup.py that allows installing and adding to PYTHONPATH in a cross-platform way with a single command. You don't need to actually publish it at PyPI, but it makes doing so easier if you should change your mind in the future. Should you do so and also publish some of your projects on PyPI, you also made installing the project in question and its dependencies easier for every potential user.