I made some modifications to the code for a deep learning model implemented in MxNet.
On my local computer, I installed MxNet by conda/pip, so I could just go to the installation folder, where I found the files where the model architecture is specified and made my changes. The structure is like:
.../environment_folder/lib/python3.8/site-packages/
- gluoncv/model_zoo/action_recognition/i3d_resnet.py
- mxnet/gluon/block.py
and I made my changes to these files.
Now, I need to do the same on another machine, where MxNet has been compiled from source. I looked into the analogous installation folder, and I found the following structure:
.../environment_folder/lib/python3.8/site-packages/
- gluoncv/model_zoo/action_recognition/i3d_resnet.py
- mxnet.egg-link
i.e., I found the gluoncv folder, but instead of the mxnet one there is a egg-link. I honestly didn't know about egg files, I've been searching around and found it was an old way of packaging python files before wheels and pip. Is there any way I can open the link and get to the folder it is presumably pointing to?
If you can open a python shell prompt (with the environment loaded) on the machine in question, try:
import mxnet
mxnet.__file__
Related
I have installed pyarrow version 0.14.0. I'm creating a package to run that from lambda.
While executing from lambda i'm getting error - No module named 'pyarrow.lib'
I have incorporated pyarrow package to my deployment zip file as well. My python version used is 3.7.
Can someone please help on this issue?
The underlying problem is that modules like pyarrow port their code from C/ C++. When you check pyarrow codebase, you will find in fact two pyarrow.lib files exist, but they have .pyx and .pxd file extensions. This is not pure Python code and therefore depends on underlying CPU architecture.
Bundling pyarrow with my code in the same zip did not work, irrespective of the .whl file I used. I was able to solve the problem in two ways.
1. Lambda Layer
I had to manually download .whl files for my required version for pyarrow and its dependency numpy. From http://pypi.org/project/pyarrow/, click on Download files and search for your matching version. cp39 means cpython 3.9. and x86 represents the CPU architecture. Follow the same steps for Numpy. I ended up downloading these files: pyarrow-8.0.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl and numpy-1.22.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.
You then have to unzip them and create an archive where both sit together in a folder named python. This folder can be used to create a layer in Lambda. Attach this layer to your project and import pyarrow should now work.
I followed this guide for creating a pyarrow layer.
2. Docker container
The other solution is to use custom Docker images. This worked smoothly for me. I believe the AWS docs are exhaustive on that topic. I have written a PoC and all the steps that I followed here.
I'm extremely new to python and I'm recently trying to understand more about Machine Learning and Neural Nets
I know this is a trivial question but I seem to be having problem importing data_utils on jupyter notebook. Can anyone please help
Note: I am not using Keras, and I am following the tutorial in this video.
I was also following the video and went through the same issue, after searching for a while; Here is the link from Github tensorflow_chatbot_required_files. You can download from here and copy it to your working directory (python file directory).
Now you will be able to import both.
Based on the link that you provided for the video you are using, go to
this link and download both files into your working directory.
The files you need to download are data_utils and seq2seq_model.
But before doing this tutorial try the tensorflow tutorials found on the tensorflow website, to get you started with this library.
Also, if you are very new to python, I recommend this tutorial first.
Since you mentioned about Machine Learning and Neural Nets, I'll assume you're referring to Keras.
I'll also assume that you installed it via
pip install keras
In that case, you need to uninstall it first by running
pip uninstall keras
Then, clone it from Github, cd into its directory and run
sudo python setup.py install
More information here
I'm assuming you are using code from tensorflow's github, in that case you need to download https://github.com/tensorflow/models/blob/master/tutorials/rnn/translate/data_utils.py into your folder also.
what you are trying to do is importing a self-defined module, in order to do that do as follow:
in your case the module is called 'data_util' which contains functions that will be called later as data_util.'function name'.
let say that the data_util which is a python file (.py) is in this directory (C:/Users/xxx/modules), so all what you have to do is to run this line of code in order for python to find your modul when you call import data_util:
import sys
sys.path.append('C:/Users/xxx/modules')
import data_util
I too was using same files.
Just open seq2seq_model.py and in line 35 remove from TensorFlow and keep it just import dat_
I know that this has been asked in some similar ways before. However, all questions I found on this dealt with some very specific system setups which were not applicable for me (because so is mine).
System:
Windows 7 64bit
Python 3.4 64bit
sqlite3 2.6.0 (shipped with Python I guess)
Spatialite Windows binaries 2.3.1
(anything else of importance?)
How can I activate the spatialite extension for the ´sqlite3´ module?
What I tried (the way that other people in similar questions say it works):
Downloading from https://www.gaia-gis.it/spatialite-2.3.1/binaries.html :
libspatialite-win-x86-2.3.1.zip
proj-win-x86-4.6.1.zip
geos-win-x86-3.1.1.zip
libiconv-win-x86-1.9.2.zip
unzipping all of them into the same folder on C:\
(also tried only putting the DLLs into that folder)
putting that folder into my system PATH variable
Then, running
import sqlite3
conn = sqlite3.connect(":memory:")
conn.enable_load_extension(True)
conn.execute('SELECT load_extension("libspatialite-2.dll")')
gives
conn.execute("SELECT load_extension('libspatialite-2.dll')")
sqlite3.OperationalError: The specified module could not be found.
What more can I try to make this work?
you probably don't have the folder in which libspatialite-2.dll is placed in your PATH.
Perhaps you can add the folder from within your Python script (I don't know any Python).
Or else you could add it from the Windows properties interface.
BTW you are using a very old version of spatialite: have a look here for newer versions:
https://www.gaia-gis.it/fossil/libspatialite/index
I have recently faced this problem with mod_spatialite.dll with Spatialite 5.0.1, Python 3.8 (with Anaconda) and Windows 10. I fixed the problem with the following steps:
Installed OSGeo4W;
Copied all the files and folders in OSGeo4W64\bin to anaconda3\Library\bin (without replacing the existing files, as this may break other things in the Python installation). I backed up this folder beforehand so I could undo everything, just in case things go wrong;
Copied mod_spatialite.dll (which I had already downloaded) into anaconda3\Library\bin.
The reason why I think I had this problem is because of missing recursive dll dependencies; since just getting the direct dependencies (through dumpbin /dependents mod_spatialite.dll) and copying them to the anaconda3\Library\bin folder did not solve the problem, so I came to the conclusion that I was missing a dependency of some dependency. Given that all dependencies are already neatly placed in OSGeo4W64\bin, copying everything from there solved it for me.
I had to reinstall my OS and now I can't load my old projects on Eclipse IDE. At first it didn't find the GAE source folder, as it was not on the same location where it was when I created the project, so I just changued the variable $GOOGLE_APP_ENGINE to where it really was.
But now it seems that on the new version of the GAE source the library folders have different names, and Eclipse won't find them.
How can I make Eclipse modify the info on my old projects so it updates the library folders names?
Thank you very much.
If you are using python, you need to set the correct paths on Project -> Properties -> External Libraries or edit the .pydevproject file. I did it a few days ago, just replace the ones that can't be found with the ones below.
<pydev_project>
....
<pydev_variables_property name="org.python.pydev.PROJECT_VARIABLE_SUBSTITUTION">
<key>GOOGLE_APP_ENGINE</key>
<value>/usr/Local/google_appengine</value>
</pydev_variables_property>
<pydev_pathproperty name="org.python.pydev.PROJECT_EXTERNAL_SOURCE_PATH">
<path>${GOOGLE_APP_ENGINE}/lib/antlr3</path>
<path>${GOOGLE_APP_ENGINE}/lib/enum</path>
<path>${GOOGLE_APP_ENGINE}/lib/fancy_urllib</path>
<path>${GOOGLE_APP_ENGINE}/lib/google-api-python-client</path>
<path>${GOOGLE_APP_ENGINE}/lib/graphy</path>
<path>${GOOGLE_APP_ENGINE}/lib/grizzled</path>
<path>${GOOGLE_APP_ENGINE}/lib/httplib2</path>
<path>${GOOGLE_APP_ENGINE}/lib/ipaddr</path>
<path>${GOOGLE_APP_ENGINE}/lib/oauth2</path>
<path>${GOOGLE_APP_ENGINE}/lib/prettytable</path>
<path>${GOOGLE_APP_ENGINE}/lib/protorpc</path>
<path>${GOOGLE_APP_ENGINE}/lib/python-gflags/tests</path>
<path>${GOOGLE_APP_ENGINE}/lib/simplejson</path>
<path>${GOOGLE_APP_ENGINE}/lib/sqlcmd</path>
<path>${GOOGLE_APP_ENGINE}/lib/yaml/lib</path>
<path>${GOOGLE_APP_ENGINE}/lib/django-1.4</path>
<path>${GOOGLE_APP_ENGINE}/lib/webapp2-2.5.2</path>
<path>${GOOGLE_APP_ENGINE}/lib/markupsafe-0.15</path>
<path>${GOOGLE_APP_ENGINE}/lib/jinja2-2.6</path>
<path>${GOOGLE_APP_ENGINE}/lib/webob-1.1.1</path>
<path>${GOOGLE_APP_ENGINE}</path>
</pydev_pathproperty>
</pydev_project>
I need to keep a large number of Windows XP machines running the same version of python, with an assortment of modules, one of which is python-win32. I thought about installing python on a network drive that is mounted by all the client machines, and just adjust the path on the clients. Python starts up fine from the network, but when importing win32com I get a pop-up error saying:
The procedure entry point ?PyWinObject_AsHANDLE##YAHPAU_object##PAPAXH#Z could not be located in the dynamic link library pywintypes24.dll
after dismissing the message dialog I get in the console:
ImportError: DLL load failed: The specified procedure could not be found.
I searched the python directory for the pywintypes24.dll and it is present in "Lib\site-packages\pywin32_system32" .
What am I missing and is there another way in which I can install Python + Python-Win32 + additional module once and have them running on many machines? I don't have access to the Microsoft systems management tools, so I need to be a bit more low-tech than that.
On every machine you have to basically run following pywin32_postinstall.py -install once. Assuming your python installation on the network is N:\Python26, run following command on every client:
N:\Python26\python.exe N:\Python26\Scripts\pywin32_postinstall.py -install
Another important thing is Good Luck!. The reason is that you might need to do this as admin. In my case such setup worked for all but one computer. I still did not figure out why.
Python (or precisely, the OS) searches the DLLs using os.environ["PATH"] and not by searching sys.path.
So you could start Python using a simple .cmd file instead which adds \server\share\python26 to the path (given the installer (or you) copied the DLLs from \server\share\python26\lib\site-packages\pywin32-system32 to \server\share\python26).
Or, you can add the following code to your scripts before they try to import win32api etc:
# Add Python installation directory to the path,
# because on Windows 7 the pywin32 installer fails to copy
# the required DLLs to the %WINDIR%\System32 directory and
# copies them to the Python installation directory instead.
# Fortunately, in Python it is possible to modify the PATH
# before loading the DLLs.
os.environ["PATH"] = sys.prefix + ";" + os.environ.get("PATH")
import win32gui
import win32con
You could use batch files running at boot to
Mount the network share (net use \\server\share)
Copy the Python and packages installers from the network share to a local folder
Check version of the msi installer against the installed version
If different, uninstall Python and all version dependent packages
Reinstall all packages
This would be pretty much a roll your own central management system for that software.