Develop Reference

How can I access the created folder in s3 to write csv file into it?

I have created the folder code but how can i access the folder to write csv file into that folder?
# Creating folder on S3 for unmatched data
client = boto3.client('s3')
# Variables
target_bucket = obj['source_and_destination_details']['s3_bucket_name']
subfolder = obj['source_and_destination_details']['s3_bucket_uri-new_folder_path'] + obj['source_and_destination_details']['folder_name_for_unmatched_column_data']
# Create subfolder (objects)
client.put_object(Bucket = target_bucket, Key = subfolder)
Folder is getting created succesfully by above code but how to write csv file into it?
Below is the code which i have tried to write but its not working
# Writing csv on AWS S3
df.reindex(idx).to_csv(obj['source_and_destination_details']['s3_bucket_uri-write'] + obj['source_and_destination_details']['folder_name_for_unmatched_column_data'] + obj['source_and_destination_details']['file_name_for_unmatched_column_data'], index=False)

An S3 bucket is not a file system.
I assume that the to_csv() method is supposed to do write to some sort of file system, but this is not the way it works with S3. While there are solutions to mount S3 buckets as file systems, this is not the preferred way.
Usually, you would interact with S3 via the AWS REST APIs, the AWS CLI or a client library such as Boto, which you’re already using.
So in order to store your content on S3, you first create the file locally, e.g. in the system’s /tmp folder. Then use Boto’s put_object() method to upload the file. Remove from your local storage afterwards.

Automatically Upload New Files in SharePoint to S3 with Python

I'm very new to AWS, and relatively new to python. Please go easy on me.
I want to upload files from a Sharepoint location to an S3 bucket. From there, I'll be able to perform analysis on those files.
The below code uploads a file in a local directory to an example S3 bucket. I'd like to modify this to only upload new files from a Sharepoint location (and not upload new files).
import boto3
BUCKET_NAME = "test_bucket"
s3 = boto3.client("s3")
with open("./burger.jpg", "rb") as f:
s3.upload_fileobj(f, BUCKET_NAME, "burger_new_upload.jpg", ExtraArgs={"ACL": "public-read"})
Would I find use of AWS Lambda via Python code? Thank you for sharing your knowledge.

AWS Boto3 upload files issue

I am facing a weird issue.
I am trying to upload few parquet files from local PC to S3 bucket. Below is the script I used.
It ran well for the first file , but as soon as I change the folder and try loading a different file to the same s3 bucket. It doesn't load. The code doesn't fail. But the 2nd file is not visible in the s3 bucket. I have no clue why its behaving this way.
s3 = boto3.resource('s3', aws_access_key_id='*****', aws_secret_access_key='****')
bucket = s3.Bucket(BUCKET)
bucket.upload_file("****.parquet", "****.parquet")

Reading Data from AWS S3

I have some data with very particular format (e.g., tdms files generated by NI systems) and I stored them in a S3 bucket. Typically, for reading this data in python if the data was stored in my local computer, I would use npTDMS package. But, how should is read this tdms files when they are stored in a S3 bucket? One solution is to download the data for instance to the EC2 instance and then use npTDMS package for reading the data into python. But it does not seem to be a perfect solution. Is there any way that I can read the data similar to reading CSV files from S3?

Some Python packages (such as Pandas) support reading data directly from S3, as it is the most popular location for data. See this question for example on the way to do that with Pandas.
If the package (npTDMS) doesn't support reading directly from S3, you should copy the data to the local disk of the notebook instance.
The simplest way to copy is to run the AWS CLI in a cell in your notebook
!aws s3 cp s3://bucket_name/path_to_your_data/ data/
This command will copy all the files under the "folder" in S3 to the local folder data
You can use more fine-grained copy using the filtering of the files and other specific requirements using the boto3 rich capabilities. For example:
s3 = boto3.resource('s3')
bucket = s3.Bucket('my-bucket')
objs = bucket.objects.filter(Prefix='myprefix')
for obj in objs:
obj.download_file(obj.key)

import boto3
s3 = boto3.resource('s3')
bucketname = "your-bucket-name"
filename = "the file you want to read"
obj = s3.Object(bucketname, filename)
body = obj.get()['Body'].read()

boto3 is the default option, however, as an alternative awswrangler provides some nice wrappers.

using python boto to copy json file from my local machine to amazon S3

I have a json file with file name like '203456_instancef9_code323.json' in my C:\temp\testfiles directory and want to copy the file to Amazon s3 bucket and my bucket name is 'input-derived-files' using python and boto library but throwing exceptions at all times saying the file does not exist.I have a valid access id and secret key and could establish connection to AWS. Could someone help me with the best code to script for this please. Many thanks for your contribution

Here is the code that you need based on boto3, it is the latest boto library and is maintained. You need to make sure that you use the forward slash for directory path. I have tested this code on windows and it works.
import boto3
s3 = boto3.resource('s3')
s3.meta.client.upload_file('C:/temp/testfiles/203456_instancef9_code323.json',
'input-derived-files', '203456_instancef9_code323.json')