I have a use case, where I have an EC2 instance with Fedora Linux and some applications running. When this instance fails, I have to spin up a new instance with the same OS and install the applications. I am trying to do in Ansible (and Python), I'm a complete novice and have no idea how to do it.
For my applications, I have a variable (a structure) that tells me how many of each type of server I need in each of three subnets. The environment creation playbook loops through that structure and builds however many are needed to fill the requirements. So if I need five (5) and only have three (3), it builds two (2). I use exact_count for that in the ec2 module.
So if one fails, I can delete that instance and re-run my create playbook, which will also re-write all the configuration files on the other servers that they use to communicate with each other. For instance, if I delete a JBoss server create a new one to replace it, the load balancer has to know about it.
Good practise here would be to have a base image that covers what you need, use that as a feeder for an AMI, and then plug it into an Auto-scaling group. As part of the auto-scaling group, you can use user-data to load specific updates/etc onto the instance at boot time.
Autoscale group min 1 max 1 will do exactly what you want, if you can configure it the above way.
Related
I am using Python, Docker, and Locust to load test a service I am working on. I have some test data I need to partition uniquely between how ever many docker containers I spin up dynamically. Locust gives all of it's worker containers a name styled as "worker_N". How can I get the worker name assigned to the container by checking inside the python script itself? I am using docker compose to spin up the containers, so I can't use the similar solution provided here.
Depending on exactly what it is you're trying to do, you could set a hostname or maybe an alias for each worker. (Locust uses hostname + a GUID for the name of the workers.) If you need to access each worker somehow, you could set the hostname to include an IP address. Just an idea.
To address your original question more directly, there are also other ways to access hostname, such as in Python code (like Locust does).
One last option would be to use the master do push the data you need to each worker using Locust's messages feature. Then you only need to ensure the master has access to whatever data you want distributed to the workers and it can divvy it up to however many workers are connected.
I have created 15 servers which contain the same machine learning program. However, each server has different arguments at the runtime. It is determined by the hostname. Each server also has a copy of a 5 1gb pkl files which contain training data.
So for example right now, I have created 15 servers in the cloud with the following names.
ml-node-1
ml-node-2
ml-node-3
..
ml-node-15
So when my program runs on ml-node-1 it looks like this, python3 mlscript.py $HOSTNAME.csv and it will run python3 mlscript.py ml-node-1.csv. Each server will run the script which is meant for its hostname.
My problem is that I have to create 15 copies of the 5gb pkl data in each server before they are run. I find this very inefficient and costly therefore I am looking up Kubernetes as a solution. From the documentation, I can see that containers within a pod can share a persistent volume. This way I might be able to mitigate copying the 5gb pkl data 15 times.
However, I am trying to figure out the naming of the servers/containers. I figure that I would need 1 pod with a shared persistent volume and 15 containers. According to what I can understand from the documentation, https://kubernetes.io/docs/concepts/containers/container-environment/, all the containers within the pod will share the same hostname.
How do I differentiate them and give them specific hostnames so that I can still have a different argument running within each container? My docker images are the standard debian image with my machine learning script.
Rather than relying on custom hostnames for a scalable multi-host environment like Kubernetes (which is not designed to allow for this), as a more feasible solution (suggested by MatsLindh) you could write some code in your mlscript.py that generates a unique random key on startup and then watches an input directory for changes in a continuous loop: This directory would contain the available files to be picked up by the different containers, and the script would rename a given file with the generated key when it assigns it to the running server instance in the container, with "unassigned" files (not containing a key with the same format in the name) being considered available for other instances.
This would allow you to scale the workload up or down from Kubernetes as well as have several files processed by the same container instance.
I want to use a very small EC2 instance that is always ready, but when a request is made, I will have very heavy computations to run and would like to use a more expensive instance to run the task and then shut down. What is the best way to go about doing this?
Edit: By heavy computations, I mean multithreaded image manipulations. The time it takes to run scales linearly with the number of CPUs.
There are a few different use cases that could lead someone to this question, so here are a few options:
1. Auto Scaling Group in Elastic Beanstalk
Something like AWS Elastic Beanstalk would make sense if you're trying to handle load balancing / capacity provisioning for an app across EC2 instances. You can use an auto scaling group and triggers to automatically provision more nodes as needed on the fly.
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.managing.as.html
2. boto3 ec2.create_instances
To answer your question more literally, you could use boto3 to provision an EC2 instance using the create_instances call whenever the condition you have in mind exists in an incoming request. Here's an example and the docs.
EC2 billing used to be 1-hour minimum but they've recently changed that to be 1-minute minimum, so this could be a viable route depending on your workload.
3. boto3 Lambda.Client.invoke
If you wanted to do something similar on AWS Lambda, you can also do that with boto3 via the invoke call on the Lambda client (docs).
I am new to AWS EC2 so that I make this post for some questions.
1) Right now, I am considering running some script on the server. I use two tools usually. One is a software can only be used in Windows. The other is just python. Should I open two instances, one for windows, one for ubuntu? Or just one instance of Windows with Git Bash installed? I want to be cost and performance efficiently.
2) I am not going to use the script very often (usually 2-3 hours per day or 10-12 hours per week). Therefore, is it easy to schedule those jobs automatically across the instances? I mean it can automatically turn off and restart given appropriate time.
3) Some of the script involves web scraping. I am also wondering if it is ok to switch IP address every time I run the script. Mainly, it is for python script.
Thanks.
1) Well, off course, the less instances you have, the less you will pay. Python can run on Windows, I just don't know how tricky it would be to make it work in your case. It all depends on what you are running and what are your management requirements. Those script languages were originally designed for Unix environments, so people usually runs it on those kind of systems, so running it in Windows may be a little unpleasant. Anyway, I don't think you should ask someone else it, you should figure it out yourself what suits you best.
2) AWS doesn't have a scheduler for EC2 (stop, starting, etc, given date/times/recurrence). It's something that I miss on it too. So, to achieve something like this you have some options.
Turning your temporary instance into an auto-scaling group of 1 instance, and scheduling policies to scale it in to zero instances and scale it out to 1 instance again when you want. The problem with this approach is: if you can't be sure how long it will take for your job to be completed, then you have a problem, off course, because those scheduled actions are based in fixed date/times. One solution for this would be the temporary instance itself changing the autoscaling group configuration to zero instances via API when it has finished. (In this case, you would just have a scale out scheduled policy, to launch the instance, leaving the termination of it to be done 'manually', via auto-scaling group configuration handling from inside the temporary instance). But be aware that auto-scaling is very tricky for begginers, and you should go throught the documentation before using it. (For exemple, each time you scale in and out you instances, they're terminated, not just stopped, and you lose every data on it.)
Not using auto-scaling group, having a regular instance, and scheduling all those actions from outside it via API. It could be from your Windows (master) instance. In this case, the master would start the temporary instance via API, which would run its things and then turn itself off when it had finished. Otherwise, the master instance would have to keep polling the temporary one somehow to know when the jobs are done and it can be shutdown from outside.
There are probably more complicated ways for doing this (Elastic Beanstalk crons, maybe).
I think, in this case, the more simple, the better. So, I would stick to the option 2). You will only need to figure how to install and use AWS CLI on Windows and manage IAM credentials and permissions to provide your CLI access enough for it to do what it needs.
3) If you don't assign an Elastic IP to your instance, you will get a different IP each time you stop and start it, so this is, by default, what you want. In auto-scaling, this is the only way, you can't even assign a fixed IP to instances.
I hope I could help you a little bit.
(ubuntu 12.04). I envision some sort of Queue to put thousands of tasks, and have the ec2 isntances plow through it in parallel (100 ec2 instances) where each instance handles one task from the queue.
Also, each ec2 instance to use the image I provide, which will have the binaries and software installed on it for use.
Essentially what I am trying to do is, run 100 processing (a python function using packages that depend on binaries installed on that image) in parallel on Amazon's EC2 for an hour or less, shut them all off, and repeat this process whenever it is needed.
Is this doable? I am using Python Boto to do this.
This is doable. You should look into using SQS. Jobs are placed on a queue and the worker instances pop jobs off the queue and perform the appropriate work. As a job is completed, the worker deletes the job from the queue so no job is run more than once.
You can configure your instances using user-data at boot time or you can bake AMIs with all of your software pre-installed. I recommend Packer for baking AMIs as it works really well and is very scriptable so your AMIs can be rebuilt consistently as things need to be changed.
For turning on and off lots of instances, look into using AutoScaling. Simply set the group's desired capacity to the number of worker instances you want running and it will take care of the rest.
This sounds like it might be easier to with EMR.
You mentioned in comments you are doing computer vision. You can make your job hadoop friendly by preparing a file where each line a base64 encoding of the image file.
You can prepare a simple bootstrap script to make sure each node of the cluster has your software installed. Hadoop streaming will allow you to use your image processing code as is for the job (instead of rewriting in java).
When your job is over, the cluster instances will be shut down. You can also specify your output be streamed directly to an S3 bucket, its all baked in. EMR is also cheap, 100 m1.medium EC2 instances running for an hour will only cost you around 2 dollars according to the most recent pricing: http://aws.amazon.com/elasticmapreduce/pricing/