Background: I have been working in Python to create a lambda function that will hit the Binance API to fetch balances and transactions. To deploy, I have been using the Serverless Framework (https://serverless.com/) and virtualenv which has made it a breeze up until this point. I have 2 other functions working perfect with other exchanges.
Error: When I deploy, I am getting the following:
Unable to import module 'getBinanceTransactions': No module named '_regex'
getBinanceTransactions being the function I created to return what I want. Nothing crazy, just following the python-binance documentation (https://github.com/sammchardy/python-binance) to grab all transactions and then data wrangling.
Note that this works on my local machine!
I do a serverless deploy, and everything updates just fine using serverless-python-requirements to package everything separately. Here are my imports (only 2 external packages):
from __future__ import print_function
import json
from binance.client import Client
import pymysql
And my requirements.txt, with both this and the code separated in a directory just like the other I have working in a similar format:
PyMySQL==0.9.2
python_binance==0.6.9
I have been searching for the solution but no one has seem to run into this problem. It also seems that _regex is a code method to Python which makes the situation even more strange.
I have tried wiping my virtualenv out and rebuilding, rebuilding the entire file structure, pip freeze >> requirements.txt on installing both packages to make sure nothing was missed, changing the name of the imports and requirements, importing regex/re in my function, even switching to Python 2.7 for a hail mary. Nothing seems to work (despite the others working) and I get the same error every time.
Does anyone have any ideas?
Related
So I've cloned this
https://bitbucket.org/romildo/django-jqgrid-demo.git
as I am looking for a working example of jqgrid with django.
I've been updating the code (as this seems like it was written for a version 2 of django and I'm workng on 4.1)
I'm completely stumped by the lines
from jqgrid import JqGrid
giving me this error
ModuleNotFoundError: No module named 'jqgrid'
I cannot find a reference to jqgrid within pip and I cannot install one (jqgrid is not a python package)
I understand that jqgrid is a javascript component around jquery but how do I get that to work in Python
I have google for django-jqgrid and on youtube. None of the answers provide enough information to get a simple working example up. There seems to be an assumption that everything is installed and I'd like to understand what is required where and how to reference
What am I missing?
Simply you can install this library:
pip install js.jqgrid
And now your above error will solve
I want to run my code on AWS lambda function. To do so, i need to import some python packages (i.e. pandas, numpy, sklearn, scipy)
I have two problems:
First of all, the size of (unzip) packaged python zip files is greater than 250MB.
Secondly, I got some error using scipy as well as sklearn as:
Unable to import module 'lambda_function': cannot import name
'_ccallback_c'
of
Unable to import module 'lambda_function': No module named
'sklearn.check_build._check_build'
___________________________________________________________________________ Contents of /var/task/sklearn/__check_build:
__pycache _check_build.cpython-35m-x86_64-linux-gnu.sosetup.py
init.py
___________________________________________________________________________ It seems that scikit-learn has not been built correctly.
I tried to reinstall many times...
But still problems in sklearn and scipy.
Any idea?
sample code in AWS LambdaFunction:
import json
import numpy
import pandas
import sklearn
import scipy
def lambda_handler(event, context):
# TODO implement
print(event)
return
You appear to have two issues.
The first (and easiest to solve) is that you need to install the relevant modules on a Linux distro comparable to Amazon Linux.
You can either do this using EC2 or in a Docker container with Amazon Linux on it.
The second issue (which is a bit trickier if not impossible to solve given the size of the modules you want to use) is that you need to get your deployment size down to under 250MB unzipped and under 50MB zipped.
Using relevant CFLAG when installing may get you some of the way there. See here for an idea of what might work.
If you are still over limit (which I suspect you will be) your only choice left will be to delete some of the files in the modules which you believe will not be used in your particular program. This is risky, often error prone and usually takes many attempts to get right. Using code coverage tools may help you here, as they can indicate which files are actually being used.
I am following the tutorial here:
http://www.prokopyshen.com/create-custom-zipline-data-bundle
and trying to set up a custom bundle to get price from custom, non US financial assets. I am stuck on the line that says:
Advise zipline of our bundle by registering it via .zipline/extension.py
My extension.py file is located in the .zipline/ directiory and has the following code:
from zipline.data.bundles import register
from zipline.data.bundles.viacsv import viacsv
eqSym = {
"CBA"
}
register(
'CBA.csv', # name this whatever you like
viacsv(eqSym),
)
I don't get what it means to register the bundle via .zipline/extension.py though? I thought it might mean to just run the extension.py file from my terminal via a:
python extenion.py
but that fails and says:
ImportError: No module named viacsv
How do i register this bundle?
I also followed this tutorial and I must confess this part is a little confusing.
First of all, I don't think it's necessary to run:
$ python extension.py
The error message you get probably comes from the fact that Python cannot find the viacsv.py file in sys.path (the places where it looks for modules, etc.). In the tutorial you mentioned, it's not really clear what to do with this file. As far as I am concerned, I just saved the viacsv.py file in my local site-packages directory. As I am on Linux I put it there ~/.local/lib/python2.7/site-packages but it might different for you. You can run the following python script to find out:
import sys
for dr in sys.path:
print dr
Then I just substituted from zipline.data.bundles.viacsv import viacsv with from viacsv import viacsv in extension.py.
I suspect you might be looking for the wrong place for the extension.py file.
For windows machine, the file is under "~\.zipline\extension.py". In my case, it's under "C:\Users\XXXX\.zipline\extension.py".
I had been looking at zipline folder under conda's site-packages folder, and couldn't find it. Then created an extension.py myself wondering why it's not called.
Check a related post here https://www.quantopian.com/posts/zipline-issue-while-creating-custom-bundle-to-bring-yahoo-data.
Same issue here, #Gillu13 pointed me to this solution.
I installed zipline through conda. So zipline is installed in
home/me/anaconda3/envs/krakenex/lib/python3.6/site-packages
in there you will find zipline/data/bundles and you can put viacsv.py in there...
then
from zipline.data.bundles.viacsv import viacsv
works
Well I'm trying to use Datastore in a personal project using the Google App Engine. Though, I can't import the datastore module, no matter how hard I try.
I've been using the online console during the whole time (in order to avoid to have to solve problems first on my PC and then on GCloud...)
So, I'm using
from google.cloud import datastore
Unfortunately, that's not working at all. The last error I have is
ImportError: No module named google.protobuf
But before I had things like Can't import Datastore.
What I did was removing the integrality of /lib, and reinstalling every dependancy with pip. Here is my requirements.txt:
# This requirements file lists all third-party dependencies for this project.
#
# Run 'pip install -r requirements.txt -t lib/' to install these dependencies
# in `lib/` subdirectory.
#
# Note: The `lib` directory is added to `sys.path` by `appengine_config.py`.
Flask==0.10
google.cloud==0.25.0
protobuf==3.3.0
(The last line was added to try to resolve the last error I got). Before having this error, I got
Also, a little clarification question: I've seen (while looking for answers) people using gcloud and some using google.cloud. What's the difference? What should I use?
Also, pip show google.cloud shows nothing.
What am I missing?
Thank you
Well, if anyone is wondering, here's how I solved the problem. What fixed it was changing the Flask version to 0.12 (don't know why, but that's what happened).
I deleted lib to be sure I was starting from scratch. Then, I used this requirements.txt file:
Flask==0.12
google-cloud==0.25.0
click==5.1
(click is needed by Flask 0.12).
I started writing python codes two weeks ago and until now I have manipulated some excel data after converting it to txt file. Now, I want to manipulate excel data directly, so I need to install openpyxl package. However, my script will be used in many different places and computers (Note that: all of them use either OS X or a Linux distrubution) which might (probably do not) contain openpyxl package installed. I want my script to check whether this package exist, and if it does not exit, then download and install it.
For that purpose, as I searched and found that I could check the existence of the package by first importing the pip module and then pip.get_installed_distributions() method. However, I am not sure whether I am in a wrong way or not. Besides, I have no idea how to download and install openpyxl package "without leaving the script".
Would you like to help me ?
You are getting ahead of yourself by trying to solve a deployment problem when you don't have the software to deploy.
Having a script trying to manage its own deployment is generally a bad idea because it takes responsibility away from package managers.
Here is my answer to your question:
Build your software under the assumption that all packages are installed/deployed correctly. I recommend creating a virtualenv (virtual environment) on your development computer, using pip to install openpyxl and any other packages you need.
Once your software is built and functional, read up on how to deploy your software package to PyPi: https://pypi.python.org/pypi. Create your package by defining openpyxl as a dependency, ensure your package can be installed/run properly (I recommend adding in tests), then deploy to PyPi so anyone can use your software.
If you want to add a nice layer of validation in your script just in case, add the following to your main method:
#!/usr/bin/env python3
import sys
try:
import openpyxl
except ImportError as iE:
print("Python Excel module 'openpyxl' not found")
sys.exit(1)
... rest of script ...