Develop Reference

503 when trying to access boto3 from kubernetes

I wrote a python script to automatically download a file from a DB and upload it to an S3 account we own. The script works from the PC and I'm successfully pinging Amazon S3 from within the Kubernetees we are working on, but I'm getting 503 when the script tries to upload/download a file from the S3. I'm using the following installation: 'python3.6 -m pip install boto3'
and getting the following error: "otocore.exceptions.ClientError: An error occurred (503) when calling the GetObject operation (reached max retries: 15): Service Unavailable"
I tried adding/removing SSL, changing timeout and max retries, and nothing seems to help. Also tried different boto3 objects (client, session etc.)
The code that crashes is the following: (the line that crashes is the one marked with **)
def write_to_s3():
s3 = get_s3()
object1 = s3.Object(BUCKET_NAME, FILENAME)
print(object1)
**test = object1.get()**
latest_num = int(str(object1.get()['Body'].read())[2:-1])
print(str(latest_num))
...
def get_s3():
my_config = Config(
region_name=REGION,
connect_timeout=25,
retries={
'max_attempts': 15,
'mode': 'standard'
}
)
return boto3.resource('s3', use_ssl=False, config=my_config, aws_access_key_id=os.environ.get("ACCESS_KEY_ID"),
aws_secret_access_key=os.environ.get("SECRET_ACCESS_KEY"))
I really do not understand why this happens and found no answers or similar errors on the web. Please help!

Send command to Amazon EC2 instance using Boto3 in Python

I am able to create an instance from an image on Amazon EC2 using boto3 using the following code:
ec2 = boto3.client('ec2',region_name = 'eu-west-2')
instance = ec2.run_instances(
ImageId='ami-011c936382e4e2g9c',
MinCount=1,
MaxCount=1,
InstanceType = 't2.micro',
SecurityGroupIds= ['sg-0c08ad7b130e3hf3',],)
id = str(instance['Instances'][0]['InstanceId'])
This works fine, but I then wish to send a simple command to the instance which will execute a python script stored on the instance. From what I can gather, boto3 has the AWS command line functionality built in and so I shouldn't have to SSH to the instance; I should be able to send a command through boto3. However I'm strugging to do this, after trying different variations of the below code:
client = boto3.client('ssm',region_name='eu-west-2')
commands = ['echo "hello world" > hello.txt'] #this would be replaced by the command to execute the python script
instance_id = [id] #id being the instance id established from above
response = client.send_command(DocumentName='AWS-RunShellScript',
Parameters= 'commands':commands},
InstanceIds=instance_id,)
I'm aware that it takes time for the server to fire up etc but this isn't the problem. I have executed this second chunk of code after a large delay when I know the server is indeed ready to go.
As mentioned, I think this might be to do with the pem file that I normally need to use in order to putty/ssh into an instance as this isn't configured anywhere into my code. Any clues would be greatly appreciated!

Python get request hangs for long duration api calls even though the api server side response is completed

I have looked at every related article or answer but could not get this resolved.I have a python script which calls a set of apis sequentially and write data to a file. Some apis take time to return response in ~30-40 mins and some return results in milliseconds.
When I run this python script on my local mac machine, it runs smooth and as intended.
When I run the same script on a remote Linux server, the python scripts initiates the call to the apis, it starts the first api, completes successfully and then starts the second api but never completes after that and just hangs forever.
The apis are completely accessible from the linux server, so there is no access issue.
I juggled the sequence of api calls to check if there's a problem with certain api but all the apis are working. It's just that if the python script executes for too long it just hangs.
Here's my python file test.py
import requests
import json, collections
import time
sourceDataApiBaseUrl = "https://apiserver.com/"
dataSinkBaseUrl = "/outputFolder"
jobs = [
{"jobTitle":"API1","dataSourceApi":"api1/", "dataSinkFile": "1.csv"},
{"jobTitle":"API2","dataSourceApi":"api2/", "dataSinkFile": "2.csv"},
{"jobTitle":"API3","dataSourceApi":"api3/", "dataSinkFile": "3.csv"},
{"jobTitle":"API4","dataSourceApi":"api4/", "dataSinkFile": "4.csv"},
{"jobTitle":"API5","dataSourceApi":"api5/", "dataSinkFile": "5.csv"}
]
for job in jobs:
print '\n'+job['jobTitle']+' job started..'
# fetch data from the api
print ' '+job['dataSourceApi']+' api started ...'
step1Seconds = time.time()
response = requests.get(sourceDataApiBaseUrl+job['dataSourceApi'])
jsonResponse = json.loads(response.text, object_pairs_hook=collections.OrderedDict)
print ' '+job['dataSourceApi']+' api completed, time taken in seconds='+str(time.time() - step1Seconds)
csvContent = ""
for myrow in jsonResponse:
csvContent += '"'+'","'.join(myrow.values())+'"\n'
csvContent = csvContent.rstrip("\n")
print ' '+'file writing started ...'
step2Seconds = time.time()
f= open(dataSinkBaseUrl+"/"+job['dataSinkFile'],"w+")
f.write(csvContent)
f.close()
print ' '+'file writing completed, records='+str(len(jsonResponse))+', time taken in seconds='+str(time.time() - step2Seconds)
P.S
I also tried adding timeout to the python requests.get method.I tried putting a longer timeout say 3600 seconds while the actual time for api completion is around 2400 seconds. In this case python throws a read timeout after 1h (3600s). I checked the api calling logs on the api server and it finishes in around 2400 seconds but the client never got the response and times out :(

Uploading large files to Google Storage GCE from a Kubernetes pod

We get this error when uploading a large file (more than 10Mb but less than 100Mb):
403 POST https://www.googleapis.com/upload/storage/v1/b/dm-scrapes/o?uploadType=resumable: ('Response headers must contain header', 'location')
Or this error when the file is more than 5Mb
403 POST https://www.googleapis.com/upload/storage/v1/b/dm-scrapes/o?uploadType=multipart: ('Request failed with status code', 403, 'Expected one of', <HTTPStatus.OK: 200>)
It seems that this API is looking at the file size and trying to upload it via multi part or resumable method. I can't imagine that is something that as a caller of this API I should be concerned with. Is the problem somehow related to permissions? Does the bucket need special permission do it can accept multipart or resumable upload.
from google.cloud import storage
try:
client = storage.Client()
bucket = client.get_bucket('my-bucket')
blob = bucket.blob('blob-name')
blob.upload_from_filename(zip_path, content_type='application/gzip')
except Exception as e:
print(f'Error in uploading {zip_path}')
print(e)
We run this inside a Kubernetes pod so the permissions get picked up by storage.Client() call automatically.
We already tried these:
Can't upload with gsutil because the container is Python 3 and gsutil does not run in python 3.
Tried this example: but runs into the same error: ('Response headers must contain header', 'location')
There is also this library. But it is basically alpha quality with little activity and no commits for a year.
Upgraded to google-cloud-storage==1.13.0
Thanks in advance

The problem was indeed the credentials. Somehow the error message was very miss-leading. When we loaded the credentials explicitly the problem went away.
# Explicitly use service account credentials by specifying the private key file.
storage_client = storage.Client.from_service_account_json(
'service_account.json')

I found my node pools had been spec'd with
oauthScopes:
- https://www.googleapis.com/auth/devstorage.read_only
and changing it to
oauthScopes:
- https://www.googleapis.com/auth/devstorage.full_control
fixed the error. As described in this issue the problem is an uninformative error message.

The API tells me ,I will get a json response ,but i am getting None

I am trying to create a droplet in Digital Ocean.The API tells me the response of .create is a JSON object. As I run this query a droplet is created succesfully in my DigitalOcean account, but I am getting only None as the response. What is the problem?
import digitalocean
droplet = digitalocean.Droplet(token="<<<TOKEN>>>",
name='DanyshOcean',
region='nyc2', # New York 2
image='ubuntu-14-04-x64', # Ubuntu 14.04 x64
size_slug='512mb', # 512MB
backups=True)
jsonobj=droplet.create()
print(jsonobj)
Output:
None

The Digital Ocean REST API is different than the python library (which probably uses the API under the covers).
If you read the library docs you'll see that you need to use the status:
actions = droplet.get_actions()
for action in actions:
action.load()
# Once it shows complete, droplet is up and running
print(action.status)
Once there's a complete status, your droplet is up. I don't know if this blocks and waits for your droplet to complete, so you might need to do something like this:
import time
running = False
while not running:
print('Waiting 1s for droplet to start up...')
time.sleep(1)
actions = droplet.get_actions()
for action in actions:
action.load()
if action.status == 'complete':
running = True

droplet.create() will create the droplet and droplet itself is an object that has the attributes of the new droplet.
Try for example:
print(droplet.id)
or
droplet.destroy()