I am new to the AWS Lambda. i have a requirement like upload csv file to S3 through Lambda.
this is web based application. from UI User will upload the CSV File and submit. this request should be handled by Lambda function(Python) and upload the csv content to S3 Bucket.
Can anyone help me on this.
If you want to use lambda to upload to S3 you'll need to use API Gateway and map the endpoint to Lambda.
An easier way is to upload directly to S3, bypassing lambda. You should use presigned urls to do the upload.
Related
My goal is to upload a .zip file containing a python project, process it and return a json. To do that I have implemented a webapp, which takes the .zip file via drag and drop. I want to process this .zip file with AWS Lambda and return a json to the webapp, but since Lambda has to receive, unzip and read the files it forces me to use S3 I guess. I can make a direct upload from my webapps javascript to S3 and let S3 trigger Lambda for each uploaded file. But then I lose this typical request response semantic between my webapp and Lambda, since S3 trigger to Lambda are async and this way Lambda wouldn't know who to respond.
I guess my question is somehow conceptual, so how can I basically send a request from my webapp in any way shape or form so that lambda can process a file sent by my webapp and send a response, in form of a json containing some info, back to the webapp?
You can not really have synchronous processing here. API Gateway will limit the Lambda response time to 29 seconds, so if your files are really small, you most likely wont be able to fit into this period.
What I suggest doing is the following:
Upload the file to S3 and trigger a Lambda after the the upload did successfully finished. Process the file with a Lambda function (this can run for 15 minutes). When the processing did finish, save the output into a known location, for example a DynamoDB table. You can poll this location from the front-end for being able to see if the processing did finished.
I am making a website to get to know aws and Django better. The idea is to let a user upload an excel file, convert it to csv and then let the user download the converted csv file.
I am using amazon s3 for file storage. My question is, what is the best way to make the conversion? Is there any way to access the excel file once it is stored in the s3 bucket and convert it to csv via Django? Sorry if my question is silly but I haven’t been able to find much information on that online. Thanks in advance
AWS Lambda is the best way. It has many types of Event triggers. You can specify the bucket and the event(put, delete, copy, etc).
So what you have to do is, create a lambda function which will be
triggered only when an object gets inserted into the S3 bucket. In that
lambda function you can do your coding such as getting the file from
the S3 bucket and the conversion.
Since you are familiar with python already, I suggest to use Boto 3 to get files from the S3 bucket
Check out my blog about AWS lambda with s3 if you want to get a more clearer idea and more about permissions when work with S3 bucket.
My Blog
On every put event of Bucket you can trigger a AWS Lambda function which will convert your File format and save in desired bucket location.
I created an endpoint in aws sagemaker and it works well, I created a lambda function(python3.6) that takes files from S3, invoke the endpoint and then put the output in a file in S3.
I wonder if I can create the endpoint at every event(a file uploaded in an s3 bucket) and then delete the endpoint
Yes you can Using S3 event notification for object-created and call a lambda for creating endpoint for sagemaker.
This example shows how to make object-created event trigger lambda
https://docs.aws.amazon.com/lambda/latest/dg/with-s3.html
You can use python sdk to create endpoint for sagemaker
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_endpoint
But it might be slow for creating endpoint so you may be need to wait.
I know how to upload a file to s3 buckets in Python. I am looking for a way to upload data to a file in s3 bucket directly. In this way, I do not need to save my data to a local file, and then upload the file. Any suggestions? Thanks!
AFAIK standard Object.put() supports this.
resp = s3.Object('bucket_name', 'key/key.txt').put(Body=b'data')
Edit: it was pointed out that you might want the client method, which is just put_object with the kwargs differently organized
client.put_object(Body=b'data', Bucket='bucket_name', Key='key/key.txt')
Creating a file (key) into Amazon S3 using Python (and boto) is not a problem.
With this code, I can connect to a bucket and create a key with a specific content:
bucket_instance = connection.get_bucket('bucketname')
key = bucket_instance.new_key('testfile.txt')
key.set_contents_from_string('Content for File')
I want to upload a file via the browser (file dialogue) into Amazon S3.
How can I realize this with boto?
Thanks in advance
You can't do this with boto, because what you're asking for is purely client-side - there's no direct involvement from the server except to generate the form to post.
What you need to use is Amazon's browser-based upload with POST support. There's a demo of it here.
do you mean this one? Upload files in Google App Engine