Hello fellow AWS contributors, I’m currently working on a project to set up an example of connecting a Lambda function to our PostgreSQL database hosted on RDS. I tested my Python + SQL code locally (in VS code and DBeaver) and it works perfectly fine with including only basic credentials(host, dbname, username password). However, when I paste the code in Lambda function, it gave me all sorts of errors. I followed this template and modified my code to retrieve the credentials from secret manager instead.
I’m currently using boto3, psycopg2, and secret manager to get credentials and connect to the database.
List of errors I’m getting-
server closed the connection unexpectedly. This probably means the server terminated abnormally before or while processing the request
could not connect to server: Connection timed out. Is the server running on host “db endpoint” and accepting TCP/IP connections on port 5432?
FATAL: no pg_hba.conf entry for host “ip:xxx”, user "userXXX", database "dbXXX", SSL off
Things I tried -
RDS and Lambda are in the same VPC, same subnet, same security group.
IP address is included in the inbound rule
Lambda function is set to run up to 15 min, and it always stops before it even hits 15 min
I tried both database endpoint and database proxy endpoint, none of it works.
It doesn’t really make sense to me that when I run the code locally, I only need to provide the host, dbname, username, and password, that’s it, and I’m able to write all the queries and function I want. But when I throw the code in lambda function, it’s requiring all these secret manager, VPC security group, SSL, proxy, TCP/IP rules etc. Can someone explain why there is a requirement difference between running it locally and on lambda?
Finally, does anyone know what could be wrong in my setup? I'm happy to provide any information in related to this, any general direction to look into would be really helpful. Thanks!
Following the directions at the link below to build a specific psycopg2 package and also verifying the VPC subnets and security groups were configured correctly solved this issue for me.
I built a package for PostgreSQL 10.20 using psycopg2 v2.9.3 for Python 3.7.10 running on an Amazon Linux 2 AMI instance. The only change to the directions I had to make was to put the psycopg2 directory inside a python directory (i.e. "python/psycopg2/") before zipping it -- the import psycopg2 statement in the Lambda function failed until I did that.
https://kalyanv.com/2019/06/10/using-postgresql-with-python-on-aws-lambda.html
This the VPC scenario I'm using. The Lambda function is executing inside the Public Subnet and associated Security Group. Inbound rules for the Private Subnet Security Group only allow TCP connections to 5432 for the Public Subnet Security Group.
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.Scenarios.html#USER_VPC.Scenario1
Related
I am trying to connect to a MySQL server through ssh tunnel on one of my google cloud functions. This works fine in my home environment. I assume it is some port issue on cloud function.
Edit: For clarification the MySQL server sits on a Namecheap shared hosting web server. Not Google Cloud SQL
Every time I run this I timeout with "unknown error". The tunnel appears to be successful. I am however unable to get the mysql connection to work.
import base64
import sshtunnel
import mysql.connector
def testing(event, context):
"""
Testing function
"""
with sshtunnel.SSHTunnelForwarder(
("server address", port),
ssh_username="user",
ssh_password="password",
remote_bind_address=("127.0.0.1",3306),
) as server:
print(server.local_bind_port)
with mysql.connector.connect(
user="user",
password="password",
host="localhost",
database="database",
port=server.local_bind_port
) as connection:
print(connection)
There's too many steps to list, but I'm wondering if the "connector" setup plays a difference even for SSH. Maybe you have to create a connector as shown here (notice how the instructions in "Private IP" tab are different than on your local computer). Then, configure Cloud Functions to use that connector. Make sure you also use the right port.
A Serverless VPC Access connector handles communication to your VPC
network. To connect directly with private IP, you need to:
Make sure that the Cloud SQL instance created above has a private IP
address. If you need to add one, see the Configuring private IP page
for instructions.
Create a Serverless VPC Access connector in the same
VPC network as your Cloud SQL instance. Unless you're using Shared
VPC, a connector must be in the same project and region as the
resource that uses it, but the connector can send traffic to resources
in different regions. Serverless VPC Access supports communication to VPC networks connected via Cloud VPN and VPC Network Peering. Serverless VPC Access does not support legacy networks.
Configure Cloud Functions to use the connector.
Connect using your
instance's private IP and port 3306.
Keep in mind, this "unknown" error could also very well be due to the Cloud SQL Admin API not being enabled here. As a matter of fact, make sure you follow that entire page as it's a broad question.
Let us know what worked for this type of error.
I wrote a simple lambda function in python to fetch some data from AWS RDS. PostgreSQL is the database engines.
conn = psycopg2.connect(host=hostname, user=username, password=password, dbname=db_name, connect_timeout=50)
I did like this. But it didn't work. Always returns an error like this
Response:
{
"errorMessage": "2018-06-06T11:28:53.775Z Task timed out after 3.00 seconds"
}
How can I resolve this??
It is most probably timing-out because the network connection cannot be established.
If you wish to connect to the database via a public IP address, then your Lambda function should not be connected to the VPC. Instead, the connection will go from Lambda, via the internet, into the VPC and to the Amazon RDS instance.
If you wish to connect to the database via a private IP address, then your Lambda function should be configured to use the same VPC as the Amazon RDS instance.
In both cases, the connection should be established using the DNS Name of the RDS instance, but it will resolve differently inside and outside of the VPC.
Finally, the Security Group associated with the Amazon RDS instance needs to allow the incoming connection. This, too, will vary depending upon whether the request is coming from public or private space. You can test by opening the security group to 0.0.0.0/0 and, if it works, then try to restrict it to the minimum possible range.
I have a problem with an AWS Lambda Function which depends upon DynamoDB and SQS to function properly. When I try to run the lambda stack, they time out when trying to connect to the SQS service. The AWS Lambda Function lies inside a VPC with the following setup:
A VPC with four subnets
Two subsets are public, routing their 0.0.0.0/16 traffic to an internet gateway
A MySQL server sits in a public subnet
The other two contain the lambdas and route their 0.0.0.0/16 traffic to a NAT which lives in one of the public subnets.
All route tables have a 10.0.0.0/16 to local rule (is this the problem because Lambdas use private Ip's inside a VPC?)
The main rout table is the one with the NAT, but I explicitly associated the public nets with the internet gateway routing table
The lambdas and the mysql server share a security group which allows for inbound internal access (10.x/16) as well as unrestricted outbound traffic (0.0.0.0/16).
Traffic between lambdas and the mysql instance is no problem (except if I put the lambdas outside the VPC, then they can't access the server even if I open up all ports). Assume the code for the lambdas is also correct, as it worked before I tried to mask it in a private net. Also the lambda execution roles have been set accordingly (or do they need adjustments after moving them to a private net?).
Adding a dynamodb endpoint solved the problems with the database, but there are no VPC endpoints available for some of the other services. Following some answers I found here, here, here and in the announcements / tutorials here and here, I am pretty sure I followed all the recommended steps.
I would be very thankful and glad for any hints where to check next, as I have currently no idea what could be the problem here.
EDIT: The function don't seem to have any internet access at all, since a toy example I checked also timed out:
import urllib.request
def lambda_handler(event, context):
test = urllib.request.urlopen(url="http://www.google.de")
return test.status
Of course the problem was sitting in front of the monitor again. Instead of routing 0.0.0.0/0 (any traffic) to the internet gateway, I had just specified 0.0.0.0/16 (traffic from machines with an 0.0.x.x ip) to the gate. Since no machines with such ip exists any traffic was blocked from entering leaving the VPC.
#John Rotenstein: Thx, though for the hint about lambdash. It seems like a very helpful tool.
Your configuration sounds correct.
You should test the configuration to see whether you can access any public Internet sites, then test connecting to AWS.
You could either write a Lambda function that attempts such connections or you could use lambdash that effectively gives you a remote shell running on Lambda. This way, you can easily test connectivity from the command line, such as curl.
When the python lambda function is executed I get "Task timed out after 3.00 seconds" error. I am trying the same example function.
When I try to run the same code from eclipse it works fine and I can see the query result. Same way I can connect to the db instance from local-machine Mysql workbench without any issues.
I tried creating a role with with full administrator access policy for this lambda function and even then its not working fine. The db instance has a vpc and I just added my local ip address there using the edit CIDR option so I can access the instance through my local machine workbench. For VPC, subnet and security group parameter in lambda function I gave the same values as I have in the RDS db instance.
I have also increased the timeout for lambda function and still I see the timeout error.
Any input would be appreciated.
For VPC, subnet and security group parameter in lambda function I gave the same values as I have in the RDS db instance.
Security groups don't automatically trust their own members to access other members.
Add a rule to this security group for "MySQL" (TCP port 3306) but instead of specifying an IP address, start typing s g into the box and select the id of the security group that you are adding the rule to -- so that the group is self-referential.
Note that this is probably not the correct long-term fix, because if your Lambda function needs to access the Internet or most AWS services, the Lambda function needs to be on a private subnet behind a NAT device. That does not describe the configuration of the subnet where your RDS instance is currently configured, because you mentioned adding your local IP to allow access to RDS. That suggests your RDS is on a public subnet.
See also Why Do We Need Private Subnets in VPC for a better understanding of public vs. private subnets.
While I had executed command edit() connecting to managed instance I was ended-up with the following error. How & What I have to do in order to come out of this problem.
wls:/offline> connect('Admin60000','sun1rise','t3://my-comm-app-serv:60001')
Connecting to t3://my-comm-app-serv:60001 with userid Admin60000 ...
Successfully connected to managed Server "MiCommApp" that belongs to domain "MiBeaDir".
Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.
wls:/MiBeaDir/serverConfig>cd('/Servers/MiCommApp/SSL/MiCommApp')
wls:/MiBeaDir/serverConfig/Servers/MiCommApp/SSL/MiCommApp> edit()
Edit MBeanServer is not enabled on a Managed Server.
60001 is managed instance port which is one among the managed instance that runs in admin server. Admin server runs in 60000 port
That is because for managed servers, WLST functionality is limited to browsing the configuration bean hierarchy. Read below excerpt from WL official documentation.
To edit configuration beans, you must be connected to an
Administration Server, and you must navigate to the edit tree and
start an edit session, as described in edit and startEdit,
respectively.
If you connect to a Managed Server, WLST
functionality is limited to browsing the configuration bean hierarchy.
While you cannot use WLST to change the values of MBeans on Managed
Servers, it is possible to use the Management APIs to do so. BEA
Systems recommends that you change only the values of configuration
MBeans on the Administration Server. Changing the values of MBeans on
Managed Servers can lead to an inconsistent domain configuration.
So, basically you need to connect with your Admin server (current you are getting connected with your managed server, as per logs you have provided - Successfully connected to managed Server "MiCommApp" that belongs to domain "MiBeaDir".) and then issue edit configurations using edit() and startEdit() WLST commands.
BTW, I connect to my server using following command:
If HTTPS - connect(url='t3s://abc.xyz.com:37001',adminServerName='AdminServer')
If HTTP - connect(url='t3://abc.xyz.com:37001',adminServerName='AdminServer')