I am writing a python script which will run in AWS as lambda function. Since it needs to connect to a Postgres database, library psycopg2 is required. It seems the standard psycopg2 does not work with AWS lambda. I downloaded it from this git repo.
I am using virtualenv for all the dependencies, so I copied the psycopg2-3.6 folder from the downloaded package to [myproject]/env/Lib/site-packages. In my main script this library is imported
import psycopg2
However when I run it in PyCharm, I got error:
File "C:\Users\dxx0111\WorkSpace\iq-iot-lambda\app.py", line 2, in <module>
import psycopg2
File "C:\Users\dxx0111\WorkSpace\iq-iot-lambda\env\lib\site-packages\psycopg2\__init__.py", line 50, in <module>
from psycopg2._psycopg import ( # noqa
ModuleNotFoundError: No module named 'psycopg2._psycopg'
Based on the error message, it looks like it was able to locate the directory of psycopg2 under virtual environment package folder. It just couldn't find psycopg2._psycopg. What am I missing here?
As it turns out, the psycopg2 library downloaded from that link only works on Amazon Linux because that is where the package was compiled on. It doesn't work on my Windows machine. In order to make it work locally, I had to install with pip install psycopg2 to my virtual env. When I deploy to AWS Lambda though, I make the zip with the downloaded library. So different psycopg2 in different platform. Now it works in both places.
Related
I have a small python app I've developed for file validation. I need to deploy it to a server that has no outside internet access. It utilizes a number of libraries (pandas, glob, os, datetime, pyodbc and numpy).
I was able to install Python 3.8.5 on the server and I attempted to use Pyinstaller to wrap everything including the libraries into a nice exe that could run on the server but it did not work. I am trying that again using the --onefile flag. Error message below:
Traceback (most recent call last):
File "BDWtoStuckyValidation.py", line 2, in <module>
import pandas as pd
File "C:\Program Files\Python38\lib\site-packages\pandas\__init__.py", line 16, in <module>
raise ImportError(
ImportError: Unable to import required dependencies:
dateutil: No module named 'dateutil'
The server is completely offline - I cannot simply use PIP to install the libraries that are missing. Additionally, my work computer is VERY locked down so I also cannot simply download WHL files or similar and manually install them on the server. Anything that has to be transferred to the server would need to be downloaded on my personal laptop, transferred to my work laptop via bluetooth and then to the server via the network. I did just run pip download for my needed modules on a machine with internet access, bundled the whl files into a tar.gz file, transferred that to the server in question and it still wouldn't install when attempting to run pip install tar.gz Update: I unzipped the tar.gz file and attempted to install some WHL files manually, it got stuck on pandas, attempting to download something or connect to pypi.org, which it obviously can't.
Any help would be appreciated.
The process you described should work, but you need to ensure that you get all of the indirect dependencies (the things that your direct dependencies depend on). A good way to do that is via a tool like pip-compile (part of pip-tools), where you can declare your direct requirements in a requirements.in file and have it figure out the complete list of dependencies and versions (written in to requirements.txt, generally). Once you have that list, you can download all the appropriate wheel files and proceed just as you described.
The reason that it would not work with your .tar.gz file is that pip expects a .tar.gz file to be a source distribution for a single package, rather than a tarball of a bunch of different wheel files for many packages.
I'm trying to create a development package for AWS Lambda. One of the modules I am using is praw which has a lot of dependencies such as urllib3 and requests.
I use the command pip install praw -t . in the folder where I need to use praw this installs all the dependencies as well. But whenever I try to import praw i get the error message
File "c:\Script\Modules\praw\reddit.py", line 23, in
from prawcore import ( ModuleNotFoundError: No module named 'prawcore
This happens despite prawcore being installed. If I go into that reddit.py file and replace that code with from Modules.prawcore import ( then it works but then another dependencies fails and I have to come in and put the Modules. bit in. Is there a way to get around this and make sure the dependencies are detected? Or is it some other problem?
Ah I think you need to deploy Lambda and all code packages as zip folder and upload them in AWS. Did you check that when you do pip install praw -t . you will get library folder? So, just zip that with lambda code and upload as a zip folder
I have followed all the steps in the documentation:
https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html
create a directory.
Save all of your Python source files (the .py files) at the root level of this directory.
Install any libraries using pip at the root level of the directory.
Zip the content of the project-dir directory)
But after I uploaded the zip-file to lambda function, I got the error message when I test the script
my code:
import psycopg2
#my code...
the error:
Unable to import module 'myfilemane': No module named 'psycopg2._psycopg'
I don't know where is the suffix '_psycopg' from...
Any help regarding this?
You are using native libraries with lambda. We had this similar problem and here is how we solved it.
Spin a machine with AWS supported AMI that runs your real lambda.
https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html
As this writing, it is,
AMI name: amzn-ami-hvm-2017.03.1.20170812-x86_64-gp2
Full documentation in installing native modules your python lambda.
https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html
Install the required modules required for your lambda,
pip install module-name -t /path/to/project-dir
and prepare your package to upload along with the native modules under lambda ami environment.
Hope this helps.
I believe this is caused because psycopg2 needs to be build an compiled with statically linked libraries for Linux. Please reference Using psycopg2 with Lambda to Update Redshift (Python) for more details on this issue. Another [reference][1] of problems of compiling psycopg2 on OSX.
There are a few solutions, but basically it comes down to installing the library on a Linux machine and using that as the Psycopg2 Library in your upload package.
I am trying to install python package cassandra driver in Azure Machine Learning studio. I am following this answer from here. Unfortunately i don't see any wheel file for cassandra-driver https://pypi.python.org/pypi/cassandra-driver/ so i downloaded the .tar file and converted to zip.
I included this .zip file as dataset and connected to python script
But when i run it, it says No module named cassandra
Does this work only with wheel file? Any solution is much appreciated.
I am using Python Version : Anoconda 4.0/Python 3.5
I got it working. Changed the folder inside .zip file to "cassandra" (just like cassandra package).
And in the Python script, i added
from cassandra import *
I am on a Windows 7 64bit machine, using Python 2.7 and I am trying to use the python database library in robotframework. I have previously used a java library file but now I want to use the python library.
I have gone to github and downloaded version 0.6.
I have also installed a setup file and MySQL-python from here
However when I try to install the database library (using python setup.py build) I get the following error:
Traceback (most recent call last):
File "setup.py", line 25, in <module>
from DatabaseLibrary import __version__
File "src\DatabaseLibrary\__init__.py", line 15, in <module>
from connection_manager import ConnectionManager
File "src\DatabaseLibrary\connection_manager.py", line 16, in <module>
from robot.api import logger
ImportError: No module named api
Why do I not have robot.api and how do I get it and install it? Or is there an easier way to install the python database library?
It seems that the Database library uses Robot Framework internals, but does not list Robot Framework as it's dependency. The robot.api package was introduced in RF 2.6, so upgrading/installing the latest Robot Framework (from project pages) should resolve your issue
First ensure the integrity of your module before trying to install. In order to install a module using distutils (setup.py) you need to run this command as an administrator:
python setup.py install
That should run the setup and report back to you any missing dependencies.
Alternatively, you can install PIP from this location: PIP Project home page. Their page provides instructions how to install PIP, it's a package manager for Python, similar to PEAR for PHP, CPAN for Perl or gem for ruby. When you have it installed you can install packages with this command:
pip install <module>
The issue was I did not have the "API" folder in the "Robot" folder in "Python27\Lib\site-packages" as I did not have the latest version of RF. And logger is a new logging API for Robot Framework 2.6 since Oct 2011. (As janne as pointed out)
Two fixes for this issue seem to be:
Tested and worked but not recommended unless you dont want to update RF: Edit the 2 files "connection_manager.py" and "query.py" in "robotframework-databaselibrary-0.6" so that there is no dependency on the Robotframework logger. This is a easy and quick edit, where you replace the "from robot.api import logger" to "import logging" and "logger" to "logging"
See "http://robotframework.googlecode.com/hg/doc/userguide/RobotFrameworkUserGuide.html#programmatic-logging-apis" for more detail.
Reinstall Robotframework and ensure the "API" folder is created. This is recommended as it is the best approach.
(Added as an answer as too long for a comment)