I created a lambda function which upload data to snowflake. I installed a all requirements in folder and zipped along with my main python file. While running in AWS it shows an error:
no module found. Cryptography.hamtaz.bindings._constant_time.
But I have this module at specified path. I don't know why it shows an error. I don't know why the error is arise.
Here is the code:
main(event, context):
import snowflake.connector
cnx = snowflake.connector.connect( user='xxx', password='yyyyy', account='zzzz', database="db Name", schema = "schema Name" )
try:
query = "SELECT * FROM Table_Name"
cnx.cursor().execute(query)
finally:
cnx.close()
I recently encountered the same issue. It turned out my Lambda function runtime was Python 3.8 but the 'cffi' library had been compiled for Python 3.6. I created a new Lambda function with the Python 3.6 runtime and uploaded my deployment package to it and it began working right away.
I faced same issue recently and found it is a problem with windows environment, try to create linux environment, install Python, packages, zip your code with all libraries and then throw back to AWS lambda, hopefully it will work.
i needed to set up a virtualenv for my lambda package to work. i also found pip install snowflake-connector-python did not install some cryptography libraries, although if i navigated to the directory i wanted them to be put in, adding --target . did cause those libraries to get installed.
For python 3.6, when I encountered the error "Unable to import module 'main': No module named '_cffi_backend'" in an AWS Lambda Function, I was able to run mv _cffi_backend.cpython-36m-x86_64-linux-gnu.so _cffi_backend.so in my linux docker image with virtualenv and the issue was resolved. Like mentioned above, some dependencies might be better placed with --target, to get them where you need them
Related
I am attempting to import the google-cloud-firestore module in a python script on AWS Lambda. I have the module installed in my virtual environment and upload the code/packages in a zip folder, but receive the following error.
I have successfully imported and executed a script with the requests module using the same approach.
I have attempted to uninstall and reinstall the following modules to my virtual environment and still have no luck: google-cloud-firestore, grpcio, google-cloud-core.
Does anyone have a solution to this problem? What could I be missing here?
This thread gives me the impression that the google-cloud-firestore module cannot be used in a zip file. If this is the case, can I install the module at runtime?
Thank you for any and all advice!
My resources:
Python 2.7, Ubunutu 18.04, Pycharm, virtual box oracle
I have an automation solution built in python.
The solution can be run from both cmd or pycharm of course.
2 options to run automation solution.
python main.py args a,b,c...(run 1 suite of tests)
python jenkinsRun.py arg a,b,c...(run main.py with diff args each time -lets say 5 time for instance)
Once jenkinsRun.py is runnig it will execute each main.py like this:
os.system('python main.py %s %s %s %s %s %s'%(STD,config.VpcStackName, '-dryrun', 'false', '-tenant' ,config.PROD_STAGE_Tenant))
Note that this is how I implemented it 3 years ago..could be better ways like using __import__, but need way to pass arguments, etc...
Anyway, when run:
python main.py arg a,b,c..
All good.
When run:
jenkinsRun.py
which should run main each time with diff args I get exception:
"/home/ohad/.local/lib/python2.7/site-packages/botocore/httpsession.py", line 7, in <module>
from urllib3.util.ssl_ import (
ImportError: cannot import name ssl
This happend only when I run the code on my new environment (see resources above)
last week I had old virtul box with ubuntu 15.04 (old) which everything worked well (didn't touch the vode ever since).
I have installed on new virtual box from scratch libaries, drivers, etc, etc.
Any ideas?
Could be some issue with installation. I did re-installed on MAC and it worked
sudo pip install awscli --ignore-installed six
Just to make sure: are you certain that you are invoking Python 2.x ?
Ubuntu 18.04 has Python 3.x as default, so make sure that you are not accidentally starting the script using another python version.
I had a similar error after creating a new environment (which also uses Boto3). It turned out to be a DLL error (ImportError: DLL load failed), which was caught by SSL module resulting in the error from the question: ImportError: cannot import name ssl.
Solution for me was to add an additional folder to the path: path_to_anaconda/Anaconda3/Library/bin. In that way, DLL load succeeds and the given ImportError is resolved.
I was working in PyCharm when I hit this wall.
Solved it by redirecting the path to my Anaconda environment, which I keep better provisioned and up to date.
Update the latest version of awscli resolved on my Mac by the below command line.
curl "https://awscli.amazonaws.com/AWSCLIV2-2.0.30.pkg" -o
"AWSCLIV2.pkg" sudo installer -pkg AWSCLIV2.pkg -target /
Reference:
https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-mac.html#cliv2-mac-install-cmd
After uninstalling, installing, even creating environments... this worked for me!
https://stackoverflow.com/a/60405693
In my case this issue came apparently from having colliding versions of boto3, botocore and awscli. This fixed the issue in my case:
pip install boto3 botocore awscli aiobotocore --ignore-installed
I was getting the same error on Win 10 and VS Code was pointing at the Conda interpreter. The issue was solved by installing Python 3.11 outside of Conda and pointing at the new interpreter. Don't forget to add the new Python to PATH and install boto3 afterwards.
If you are using boto3 as a dependency, there was a bug with how the boto3 dependencies were managed.
try running
pip3 install boto3 --upgrade to update boto3 and the issue should be resolved!
I am not sure why it worked. But, I had this issue in AWS Glue, and I was able to get around this problem by using Glue 3.0 instead of Glue 2.0.
Please update the latest urllib package:
run :
pip3 uninstall urllib3
pip3 install urllib3
I have followed all the steps in the documentation:
https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html
create a directory.
Save all of your Python source files (the .py files) at the root level of this directory.
Install any libraries using pip at the root level of the directory.
Zip the content of the project-dir directory)
But after I uploaded the zip-file to lambda function, I got the error message when I test the script
my code:
import psycopg2
#my code...
the error:
Unable to import module 'myfilemane': No module named 'psycopg2._psycopg'
I don't know where is the suffix '_psycopg' from...
Any help regarding this?
You are using native libraries with lambda. We had this similar problem and here is how we solved it.
Spin a machine with AWS supported AMI that runs your real lambda.
https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html
As this writing, it is,
AMI name: amzn-ami-hvm-2017.03.1.20170812-x86_64-gp2
Full documentation in installing native modules your python lambda.
https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html
Install the required modules required for your lambda,
pip install module-name -t /path/to/project-dir
and prepare your package to upload along with the native modules under lambda ami environment.
Hope this helps.
I believe this is caused because psycopg2 needs to be build an compiled with statically linked libraries for Linux. Please reference Using psycopg2 with Lambda to Update Redshift (Python) for more details on this issue. Another [reference][1] of problems of compiling psycopg2 on OSX.
There are a few solutions, but basically it comes down to installing the library on a Linux machine and using that as the Psycopg2 Library in your upload package.
I have successfully installed mysql-connector using pip.
Installing collected packages: mysql-connector
Running setup.py install for mysql-connector ... done
Successfully installed mysql-connector-2.1.6
However, in PyCharm when I have a script that uses the line:
import mysql-connector
PyCharm gives me an error saying there isn't a package called "mysql" installed. Is there some sort of syntax that should be used to indicate that the entire package name contains the "-" and is not just "mysql"?
When I run my script in IDLE, mysql.connector imports just fine. (I changed it to mysql-connector after seeing the "-" in the name of the package and having trouble in PyCharm.)
EDIT: per #FlyingTeller's suggestions, in the terminal, "where python" returns C:...Programs\Python\Python36-32\python.exe. "where pip" returns ...Python\Python36-32\Scripts\pip.exe. The interpreter in PyCharm for this project is this same filepath & exe as "where python" in the terminal.
Per #Tushar's comment, this program isn't using a virtual environment and the mysql-connector library is already present in the Preferences->Project->Python Interpreter.
Thanks for the feedback and any additional guidance you may be able to provide.
You need to import the connector as
import mysql.connector
Check the examples in the docs for this
If that doesn't work, then there might be an inconsistency between the interpreter that pycharm uses and the one you installed the package for. In pycharm, go to
File->Settings->Project Interpreter
In the terminal, enter
where python #Windows
which python #Linux
and also
where/which pip
make sure that the interpreter configured in pycharm is the same that appears when typing which/where python in the command line/shell. Also make sure that pip also points to the same python distribution.
It may be because you are using a virtual environment inside pyCharm, while you might have installed the library using system's default pip.
Check Preferences->Project->Python Interpreter inside Pycharm, and see if your library is listed there. If not, install it using + icon. Normally, if you use pyCharm's inbuilt terminal, it is already using the same virtual env as your project. So using pip there may help.
Usage syntax is as below:
import mysql.connector
conn = mysql.connector.connect(
user='root',
password='#####',
host='127.0.0.1',
database='some_db')
conn.close()
Go to project interpreter and download mysql-connector.You need to install it also in pycharm
I was having this exact issue as well, after a while i solved it by simply changing the name of my script in PyCharm, turns out i had named my script mysql.py (because it was my first time attempting to connect it to python) and it was critically interfering with the import.
TLDR: Change the name file if it's asssigned as mysql.py, as it takes priority over the mysql-connector and prevents it from being imported, even if installed correctly.
People have commented with reasonable responses (and I'm sure OP is good on this by now) but they weren't super clear to me...
Don't use "pip3 install" in terminal in your pycharm project. In fact, uninstall any mysql connectors you have already using this method.
So now that you have verified there are no other mysql connector packages... Add the package "mysql-connector-python" using the Python Interpreter only (in preferences) in your pycharm project. Now the package should work!
How do we create a pymssql package for lambda. I tried creating it using
pip install pymssql -t . When I run my lambda function it complaints saying that
Unable to import module 'lambda_function': No module named lambda_function
I follow the steps on this link
http://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html
I have a windows machine
Glad that it worked for you, could you please share the working process for your, me too tried different trial and error steps and ended up with the following one which is working fine in AWS Lambda, I am using PYMSSQL package only.
1) did 'pip install pymssql' on amazon EC2 instance as under the hood Amazon uses Linux AMIs to run their Lambda functions.
2) copied the generated .so files and packaged inside the Lambda deployment package hope this will helps others as well who are searching for the solution.
Hope this will help you further, can you please share what you did to connect to MSSQL server using AWS Lambda.
Below is the folder structure of my lambda deployment package
Windows "pymssql" file not supported in AWS lambda since lambda running on Amazon Linux 2.
So we can get Linux-supported wheel files from the following official "pymssql" website download link. You look for "manylinux_2_24_x86_64" version file with the required python version. At the time of writing python 3.9 is the latest version. So download file will be "pymssql-2.2.2-cp39-cp39-manylinux_2_24_x86_64.whl".
Once download the wheel file execute the below command
pip install {path of downloaded wheel file} -t {target folder to store this package}
Example
pip install pymssql-2.2.2-cp39-cp39-manylinux_2_24_x86_64.whl -t /package
Finally i could do it. It didnt worked with windows packages so used ubuntu to package freetds.so file and it worked.
Along with pymssql try to import cypthon.
Just a quick update for new changes.
There seems to be some issue with python 3.9 AWS lambda, it throws the below error on pymssql==2.2.3.
{
"errorMessage": "Unable to import module 'index': No module named
'pymssql._pymssql'",
"errorType": "Runtime.ImportModuleError",
"stackTrace": []
}
but if I change the python version to 3.8, the error vanishes.