So recently I have found about a NEAT algorithm and wanted to give it a try using NEAT-Python(not sure if this is even the correct source :| ). So I created my virtual environment activated it and installed the neat-python using pip in the VE. When I then tried to run one of the examples from their GitHub page it threw an error like this:
ImportError: No module named visualize
So I checked my source files, and actually the neat-python doesn't include the visualize.py script, however it is in their GitHub repository. I then tried to add it myself by downloading just the visualize.oy script dragging it inside my VE and adding it to all the textfiles the NEAT brought with it, like the installed-filex.txt etc. However it still threw the same error.
I'm still fairly new to VE and GitHub so please don't be too hard on me :] thanks in advance.
-Jorge
I think you could simply copying the visualize.py into the same directory as the script you are running.
If you wanted it in your lib/site-packages directory so you could import it with the neat module:
copy visualize.py into lib/site-packages/neat/ and modify __init__.py to add the line import neat.visualize as visualize. Delete the __pycache__ directory. Make sure you have modules installed: Numpy, GraphViz, and Matplotlib. When you've done the above, you should be able to import neat and access neat.visualize.
I don't recommend doing this though for several reasons:
Say you wanted to update your neat module. Your visualize.py file is technically not part of the module. So it wouldn't be updated along with your neat module.
the visualize.py file seems to be written in the context of the examples as opposed to being for general use with the module, so contextually, it doesn't belong there.
At some point in the future, you might also forget that this wasn't a part of the module, but your code acts as if it was part of the API. So your code will break in some other neat installation.
Related
I have made a few Python projects for work that all revolve around extracting data, performing Pandas manipulations, and exporting to Excel. Obviously, there are common functions I've been reusing. I've saved these into utils.py, and I copy paste utils.py into each new project.
Whenever I change utils.py, I need to ensure that I change it in my other project, which is an error-prone process
What would you suggest?
Currently, I create a new directory for each project, so
/PyCharm Projects
--/CollegeBoard
----/venv
----/CollegeBoard.py
----/Utils.py
----/Paths.py
--/BoxTracking
----/venv
----/BoxTracking.py
----/Utils.py
----/Paths.py
I'm wondering if this is the most effective way to structure/version control my work. Since I have many imports in common, too, would a directory like this be better?
/Projects
--/Reporting
----/venv
----/Collegeboard
------/Collegeboard.py
------/paths.py
----/BoxTracking
------/BoxTracking.py
------/paths.py
----/Utils.py
I would appreciate any related resources.
Instead of putting a copy of utils.py into each of your projects, make utils.py into a package with it's own dedicated repository/folder somewhere. I'd recommend renaming it to something less generic, such as "zhous_utils".
In that dedicated repository for zhous_utils, you can create a setup.py file and you can use that setup.py file to install the current version of the zhous_utils into your python install. That way you can import zhous_utils into any other python script on your PC, just like you would import pandas or any other package you've installed to your computer.
Check out this stackoverflow thread: What is setup.py?
When you understand setup.py, then you will understand how to make and install your own packages so that you can import those installed packages to any python script on your PC. That way all source code for zhous_utils is centralized to just one folder on your PC, which you can update whenever you want and re-install the package.
Now, of course, there are some potential challenges/downsides to this. If you install zhous_utils to your computer and then import and use zhous_utils in one of your other projects, then you've just made zhous_utils into a dependency of that project. That means that if you want to share that project with other people and let them work on it as well or use it in some way, then they will need to install zhous_utils. Just be aware of that. This won't be an issue if you're the only one interacting/developing the source code of the projects you intend to import zhous_utils into.
Users should install our python package via pip or it can be cloned from a github repo and installed from source. Users should not be running import Foo from within the source tree directory for a number of reasons, e.g. C extensions are missing (numpy has the same issue: read here). So, we want to check if the user is running import Foo from within the source tree, but how to do this cleanly, efficiently, and robustly with support for Python 3 and 2?
Edit: Note the source tree here is defined as where the code is downloaded too (e.g. via git or from the source archive) and it contrasts with the installation directory where the code is installed too.
We considered the following:
Check for setup.py, or other file like PKG-INFO, which should only be present in the source. It’s not that elegant and checking for the presence of a file is not very cheap, given this check will happen every time someone import Foo. Also there is nothing to stop someone from putting a setup.py outside to the source tree in their lib/python3.X/site-packages/ directory or similar.
Parsing the contents of setup.py for the package name, but it also adds overhead and is not that clean to parse.
Create a dummy flag file that is only present in the source tree.
Some clever, but likely overcomplicated and error-prone, ideas like modifying Foo/__init__.py during installation to note that we are now outside of the source tree.
Since you mention numpy in your comments and wanting to do it like they do but not fully understanding it, I figured I would break that down and see if you could implement a similar process.
__init__.py
The error you are seeking starts here which is what you linked to in your comments and answers so you already know that. It's just attempting an import of __config__.py and failing if it isn't there or can't be imported.
try:
from numpy.__config__ import show as show_config
except ImportError:
msg = """Error importing numpy: you should not try to import numpy from
its source directory; please exit the numpy source tree, and relaunch
your python interpreter from there."""
raise ImportError(msg)
So where does the __config__.py file come from then and how does that help? Let's follow below...
setup.py
When the package is installed, setup is called to run and it in turn does some configuration actions. This is essentially what ensures that the package is properly installed rather than being run from the download directory (which I think is what you are wanting to ensure).
The key here is this line:
config.make_config_py() # installs __config__.py
misc_util.py
That is imported from distutils/misc_util.py which we can follow all the way down to here.
def make_config_py(self,name='__config__'):
"""Generate package __config__.py file containing system_info
information used during building the package.
This file is installed to the
package installation directory.
"""
self.py_modules.append((self.name, name, generate_config_py))
Which is then running here which is writing in that __config__.py file with some system information and your show() function.
Summary
The import of __config__.py is attempted and fails which generates the error you are wanting to raise if setup.py wasn't run, which is what triggers that file to be properly created. That ensures not only that a file check is being done but that the file only exists in the installation directory. It is still some overhead of importing an additional file on every import but no matter what you do you're adding some amount of overhead making this check in the first place.
Suggestions
I think that you could implement a much lighter weight version of what numpy is doing while accomplishing the same thing.
Remove the distutils subfunction and create the checked file within your setup.py file as part of the standard install. It would only exist in the installed directory after installation and never elsewhere unless a user faked that (in which case they could get around just about anything you try probably).
As an alternative (without knowing your application and what your setup file is doing) possibly you have a function that is normally imported anyway that isn't key to the running of the application but is good to have available (in numpy's case the functions are information about the installation like version(). Instead of keeping those functions where you put them now, you make them part of this file that is created. Then you are at least loading something that you would otherwise load anyway from somewhere else.
Using this method you are importing something no matter what, which has some overhead, or raising the error. I think as far as methods to raise an error because they aren't working out of the installed directory, it's a pretty clean and straightforward way to do it. No matter what method you use, you have some overhead of using that method so I would focus on keeping the overhead low, simple, and not going to cause errors.
I wouldn't do something that is complicated like parsing the setup file or modifying necessary files like __init__.py somewhere. I think you are right that those methods would be more error prone.
Checking if setup.py exists could work but I would consider it less clean than attempting to import which is already optimized as a standard Python function. They accomplish similar things but I think implemented the numpy style is going to be more straight forward.
I am quite new to Python and Django. I have a problem with integrating a python package (openpyxl) to my django app. I'd like to use the methods of these files into my views.py file.
My problem is first that I don't know where's the best place to put the openpyxl folder containing all the files in my file hierarchy.
My hierarchy looks like this:
http://imgur.com/t4iOX98
Is it well placed? Should I put it outside the international folder? inside the carte_interactive folder?
And my biggest problem is inside the __init__.py of openpyxl. I get errors lines like this one:
from openpyxl.xml import LXML
Where there is no resolved reference to LXML, but is actually defined in the xml file of openpyxl.
Is it my bad file placement that caused this? or is it Django?, or is it openpyxl's fault? Do anyone have an idea?
You can see openpyxl's source files here, where I downloaded them:
https://bitbucket.org/openpyxl/openpyxl/src
If you need any more details, please ask!
Thanks in advance!
I applaud your enthusiasm for wanting to learn Django while being new to Python. That said, the way you have things set up right now will make your life unnecessarily difficult to manage.
I would first recommend reading up on best practices for setting up a Django project. Just doing a quick google search for "Django project layout best practices" will give you a lot of resources, but they'll all essentially tell you to do what's in the SO answer above.
The second very basic thing is using pip to install and use other python packages. This is especially important for a django project, where you often have a lot of dependencies outside of Django. Pip is a program to install additionaly python packages. They get installed in your PYTHONPATH, which is just a list of filepaths on disk where python will look for additional packages. If you're on a *NIX system, this is usually in something like /usr/lib/python2.7/. Once you have something in your python path, you can from any piece of code, use other libraries you've installed via the python import system. Essentially, all this more or less does is look through each location in your PYTHONPATHs for the library you're trying to import.
Finally, in regards specifically to lxml, you will want to install it via apt or some other package installer. (e.g. on ubuntu, apt install python-lxml
In order to keep track of all your external python-dependencies, stuff them in a file named "requirements.txt" in the top level directory. This is a pretty standard thing to do for Django projects, so don't worry about shipping code with ALL dependencies inside the project.
Thanks to all of you! I'm using Jetbrains Pycharm and when I wrote import openpyxl, it gave me the choice to install the package. I suppose it does it with pip, which would certainly have worked the same. And I put the package in requirements.txt, so that other users would only have to install this requirement!
It works now! And thanks for the link on the best practices. I'll read that!
I don't know how many duplicates of this are out there but none of those I looked at solved my problem.
To practice writing and installing custom modules I've written a simple factorial module. I have made a factorial folder in my site-packages folder containing factorial.py and an empty __init__.py file.
But typing import factorial does not work. How can I solve this? I also tried pip install factorial but that didn't work either.
Do I have to save my code intended to use factorial in the same folder inside site-packages or can I save it whereever I want?
Greetings
holofox
EDIT: I solved it. Everything was correct as I did it. Had some problems at importing and using it properly in my code...
I think there are several different things to take into consideration here.
When you're importing a module, (doing import factorial), Python will look in the defined PATH and try to find the module you're trying to import. In the simplest case, if your module is in the same folder where your script is trying to import it, it will find it. If it's somewhere else, you will have to specify the path.
Now, site-packages is where Python keep the installed libraries. So for example when you do pip install x, Python will put the module x in your site-packages destination and when you try to import it, it will look for it there. In order to manage site-packages better, try to read about virtualenv.
If you want your module to go there, first you need to create a package that you can install. For that look at distutils or all the different alternatives for packaging that involve some type of building process based on a setup file.
I don't want to go into details in any of these points because all of them have been covered before. Just wanted to give you a general idea of where to look for.
Given that every library is a python code itself, I think its logical that instead of using the import command, we can actually copy the whole code of that library and paste it to the top of our main.py.
I'm working on a remote pc, I cannot install libraries, can I use a library by just doing this?
Forgive me if this a very silly question.
Thanks
In some cases yes, you can. But there are (a lot of) libraries that have some of their functionality written in C and compiled to binary (e.g. the famous numpy). You can't just paste those.
Another thing that the pasting might introduce are naming colisions. If you use
import module
than any name in the module module can be safely used in the importing module using module.name even if the name name is already defined somewhere in the importing module. If you just paste the code this won't work.
While pasting the entire library at the top of your main file can work, I don't think it's the best way to go about solving your problem.
One option is to move the library and put it in to the same folder as your main.py file. I believe the import statement will check the current working directory for the library before looking elsewhere.
Another option is to use a virtual environment(virtualenv) and then install all the required libraries within this virtual environment. I'm not sure that this will work for you since you said you cannot install on this libraries and virtualenv requires pep. If you are interested in learning more about python virtual environments, take a look here.
Most modules are actually written in C, like Pygame for example. Python itself is based on C. You can't jump to conclusions, but if the library is pure Python, I'd suggest copying the package into your project directory and importing, rather than copying and pasting code snippets.