I'm developing an Android App and I want the users to upload some images to an EC2 instance in order to process them using some fancy python code, and then return the final images to the user. I am new with servers and AWS, it's a little bit confusing, there are too much things that seem related, but I am not sure where to look. What can I do to achieve my goal? Thanks.
Long story short, I am gonna put things into the list in sequential order to make things easier for you.
Spin off an EC2 instance (Instance class as per your CPU/Network/Memory Requirements).
Install required dependencies and Publish code into your EC2 Instance (Via GitHub or CI/CD. Github will be a straightforward way for you as a beginner.)
Run your code.
Configure security groups which will allow the internet to communicate with your instance. Security groups help you define which port to open to the internet and make it accessible from your android app.
Connect your code via EC2 IP or public IP domain exposed by every EC2 instance (If configured with the public IP option).
These are the basic things you need to do for a simple DIY set-up as a beginner. This will be a good start but definitely not recommended for production usage. For production usage, you need to familiarise yourself with the concept of VPCs, Subnets, NATs, IGWs, Security Groups, Load Balancers, AutoScaling Groups, ACM for SSL certificate management and Cloudwatch for alerting and logging.
I hope this will help you to kickstart.
Note: If you are a beginner and this is just a hobby learning project then I would recommend not to get into advanced concepts like VPCs, subnets etc. and start with humble beginnings. Once you achieve that, look into more security + availability of your existing setup. Do things in smaller increments rather than everything all at once. Happy Coding.
Related
If you're building a system that has three databases - dev, testing and live, and you write a module to connect to the database, how do you make sure that a developer can not connect to the live database?
For example, you have different database clusters which are identified by variables. How do you stop the developer from simply replacing the "dev" variable with the "live" one?
Some potential solutions:
Use an environment variable (but couldn't the developer just change the environment variable on their local machine?)
Use some sort of config file which you then replace with a script when deploying to production. This makes sense and is quite simple...but is it the way these things are normally done?
The assumptions in this question are:
The database connection module is just part of the codebase, so any developer can see it and use it, and potentially change it. (Is this in itself bad?)
It would be great to know how to approach this issue - the stack is a Python server connecting to a Cassandra DB cluster, but where the cluster changes depending on whether it's dev, testing or live.
Many thanks in advance.
Most common 2 solutions for this are (normally used together):
Firewall, production servers should have strict access rules so that at the very least apart from intra-cluster communication that might be free, all other exterior channels need to be pre-approved, so for examples only IPs assigned to the devops team or DBAs can even try to access the machines.
Use credentials. Most frameworks support some form or the other of application.properties/application.yaml/application.conf. Spring boot for example can read files with the name application.properties that are in the same folder as the jar and then use those values to override the ones bundled into the jar. So that you can override the user and password for the database in the production environment, where the dev should not have access.
So:
Dev has access trough the firewall to the dev server and also the user/password to it. This way he can experiment and develop with no problems. Depending on the organization tho' he might sue a local database so this may not apply.
When you go up to test/preprod/prod, the application should be configured to read the connection details from a file or as starting parameters so that the administrator or the dev-ops team can change them to be medium specific. This is also important as not to have the same credentials across all DBs.
For specific info regarding authentication on Cassandra, you can start here: docs
Hope this helped,
Cheers!
I'm tasked with creating a web app (I think?) for my job that will tracker something in our system. It'll be an internal tool that staff uses to keep track of the status of one of the things we do. It should look like trello, with cards that drag from step to step. That frontend exists, but my job is to make the system update when the cards are dragged. This requires using an API in Python and isn't that complicated to grab from/update. I have no idea how to put all of this together. My job is almost completely nontechnical and there's no one internally who knows what I'm doing except for me. I'm in so over my head here and have no idea where to begin. Is this something I should deploy on Elastic Beanstalk? EC2? How do I tie this together and put it somewhere?
Are you trying to pull in live data from Trello or from your companies own internal project management tool?
An EC2 might be useful, but honestly, it may be completely unnecessary if your company has its own servers. An EC2 is basically just a collection of rental computers to help with scaling. I have never used beanstalk so my input would be useless there.
From what I can assume from the question, you could have a python script running to pull from the API and make the changes without an EC2.
First thing you should do is gather as much information about what the end product should look like. From your question, I have the feeling that you have only a vague idea of what the stakeholders want. Don't be afraid to ask more clarification about an unclear task. It's better to spend 30 minutes discussing and taking note than to show the end-product after a month and realizing that's not what your boss/team wanted.
Question I would Ask
Who is going to be using this app? (technical or non-technical person)
For what purpose is this being developed?
Does it need to be on the web or can it be used locally?
How many users need to have access to this application?
Are we handling sensitive information with this application?
Will this need to be augmented with other functionality at some point?
This is just a sample of what I would ask, during the conversation with the stakeholder a lot more will pop up for sure.
What I think you have to do
You need to make a monitoring system for the tasks that need to be done by your development team (like a Kanban)
What I think you already have
A frontend with the card that are draggable to each bin. I also assume that you can create a new card and delete one in the frontend. The frontend is most likely written in React, Angular or Vue.js. You might also have no frontend framework (a mix of jQuery and vanilla js), but usually frontend developper end up picking a framework of sort to help the development.
A backend API in Python (in Flask or with Django-rest-framework most likely) that is communicating with a SQL database like postgresql or a Document database like MongoDB.
I'm making a lot of assumption here, but your aim should be to understand the technology you will be working with in order to check which hosting would work best. For instance, if the database that is setup is a MySQL database you might have some trouble with some hosting provider.
What I think you are missing
Currently the frontend and the backend don't communicate to each other. When you drag a card it won't persist if you refresh the page. Also, all of this is sitting in your computer and cannot be used by any one from your staff. You need to first connect the frontend with the backend so that the application has persistance. Then you need to deploy this application somewhere so that it is reachable by your staff.
What I would do is first work locally to make sure that the layer of persistance is working. This imply having the API server, the frontend server and the database server running simultaneously on your computer to develop. You should then fetch data from the API to know which cards are there in the database and then create them visually in your frontend at the right spot.
When you drop a card to a new spot after having dragging it should trigger a POST request to your API server in order to update the status of this particular card (look at the documentation of your API to check what you need to send).
The server should be sending back an updated version of the cards status if the POST request was sucessful, so your application should then just redraw the card at the right spot (it won't make a difference for you since they are already at the right spot and your frontend framework will most likely won't act on this change since the state hasn't changed). That's all I would do for that part.
I would then move to the deployment phase to make sure that whatever you did locally can still work online. I would use Heroku to start instead of jumping directly to AWS. Heroku is a service built on top of AWS which manage a lot of the complexity of AWS for you. This is great for prototyping and it means that when your stuff is ready you can migrate to AWS easily and be confident that a setup exist to make your app work. You might also be tied up to your company servers, which is another thing I would ask to the stakeholder (i.e. where can I put this application and where I can't put it).
The flow for a frontend + api + database application on Heroku is usually as follow. You create a github repo for your frontend (make it private) and you create an app on Heroku that will watch this repository for changes. It will re-deploy the application for you when it sees a change at a specific subdomain of Heroku hosting. You will need to configure some procfiles that will tell Heroku what to do with a given application type. This is where you need to double check what frontend you are using since that might change the procfiles used. It's most likely a node.js based frontend (React, Angular or Vue) so head over here for the documentation of how to put that online.
You will need to make a repo for the backend also that is separate from the frontend, these two entities are distinct and they only communicate through HTTP request (frontend->backend) and JSON (backend->frontend). You will need to follow the same idea as with the frontend to deploy, head over here.
Once you have these two online, you need to create a database on Heroku. This is done by adding a datastore to your api, head over here. There are some framework specific configuration you need to do to make the API talk to an online database, but then you will need to find that configuration on the framework documentation. The database could also be already up and living on your server, if this is the case you just need to configure your online backend to talk to that particular database at a particular address.
Once all of the above is done, re-test your application to check if you get the same behavior as before. This is a usable MVP, however there are no layer of security. Anyone with the right URL could just fetch your frontend and start messing around with your data.
There is more engineering that need to be done to make this a viable end product. This leads us to my final remark: why you are not using a product like Trello, Jira, or even Github Project? If it is to save some money on not paying for a subscription I think you should factor in the cost of development, security and maintenance of this application.
Hope it helps!
One simple option is Heroku for deploy your API and your frontend application.
I am a web backend developer. In the past, I've used a lot of Python and specifically django to create custom APIs to serve data, in JSON for instance, to web frontends.
Now, I am facing the task of developing a mobile backend that needs to provides services such as push notifications, geolocating etc. I am aware of the existing mBaaS providers which could definitely address a lot of the issues with the task at hand, however, the project requires a lot of custom backend code, async tasks, algorithms to perform calculations on the data that in response trigger additional behavior, as well as an extensive back office.
Looking at the features of the popular mBaaS provider, I feel like they are not able to meet all my needs, however it would be nice to use some of their features such as push notifications, instead of developing my own. Am I completely mistaken about mBaaS providers? Is this sort of hybrid approach even possible?
Thanks!
There are a ton of options out there. Personally, I'm still looking for the holy grail of mBaaS providers. I've tried Parse, DreamFactory, and most recently Azure Mobility Services.
All three are great getting started from PoC to v1, but the devil is always in the details. There are a few details to watch out for:
You sacrifice control and for simplicity. Stay in the lanes and things should work. The moment you want to do something else is when complexity creeps in.
You are at the mercy of their infrastructure. Yes -- even Amazon and Azure go down from time to time. Note -- Dreamfactory is a self-hosted solution.
You are locked into their platform. Any extra code customizations
you make with their hooks (ie - Parse's "CloudCode" and Azure's API
scripts) will most likely not port to another platform.
Given the learning curve and tradeoffs involved I think you should just play the strong hand you already have. Why not host an Django app on Heroku? Add on DjangoRestFramework and you basically can get a mBaas up and running in less than a day.
Heroku has plenty of third party providers for things like Push notifications, Authentication mechanisms, and even search engines (Elasticsearch).
All that is required is to drop the right "pip install" code into your controllers and you are off an running.
Currently I am working on a portal which is exposed to end users. This portal is developed using Python 2.7, Django 1.6 and MySQL.
Now we want to expose this portal as a mobile app. But current design does not support that as templates, views and database are tightly coupled with each other. So we decided to re-architect the whole portal. After some research I found following:
Client side: AngularJS for all client side operations like show data and get data using ajax.
Server side: Rest API exposed to AngularJS. This Rest API can be developed using either Tastypie or Django Rest Framework (still not decided). Rest API will be exposed over Django.
I have few questions:
What you guys think about architecture? Is this is a good or bad design? How it can be improved?
Will performance of portal will go down after adding above layers in architecture?
In the above architecture whether 2 servers should be used to run this (like one for client and other for serving the API's) or one server will be enough. Currently Heroku is used for deployment.
Currently portal is getting 10K hits in a day and it is expected to go to 100K a day in 6 months. Will be happy to provide more information if needed.
If i got an opportunity to architect the portal which you mentioned than i would really love to design the architecture which i have already explained here.
What you guys think about architecture?
This is a common Service Oriented Architecture with decoupled Clients. You just have REST endpoints on your backend, and any Client can consume those endpoints.
You should also think about:
Do you need RESTful service (RESTful == stateless, will you store any state on the server?)
How to scale the service in the future? (this is a legit thing as you already aware of huge traffic increase and assume 2 servers)
How it can be improved?
Use scala instead of python :)
Will performance of portal will go down after adding above layers in architecture?
It depends.
It will get some performance penalty (any additional abtract layer has it's tax), but most probably you won't event notice it. But still, you should measure it using some stress tests.
In the above architecture whether 2 servers should be used to run this (like one for client and other for serving the API's) or one server will be enough. Currently Heroku is used for deployment.
Well, as usual, it depends.
It depends on the usage profile you have right now and on the resources available.
If you are interested in whether the new design will perform better than the old one? - there are a number of parameters.
Resume
This is a good overall approach for the system with different clients.
It will allow you:
Totally decouple mobile app and frontend development from backend development. (It could be different independent teams, outsourceable)
Standardize your API layer (as all clients will consume the same endpoints)
Make you service scalable easier (this includes the separate webserver for static assets and many more).
I wanted to know if there was a way I can get my python script located on a shared web hosting provider to read the contents of a folder on my desktop and list out the contents?
Can this be done using tempfiles?
Server-side web scripts have no access to the client other than through requests. If you can somehow break through the browser's protection settings to get JavaScript, Java, or Flash to read the contents of the client then you stand a fighting chance. But doing so will make many people angry and is generally considered a bad idea.
Unless your desktop computer has a public, accessible IP, neither your app running on a shared web hosting provider, nor any other app and host on the internet, can get information from your desktop computer. Does your desktop computer fall within the tiny minority that does have such a public, accessible IP?
If not, and if you're willing to run the obvious risks involved of course, you can try turning the (probably dynamically assigned) IP address that your ISP gives you into a resolvable domain name, by working with such DNS providers as DynDNS -- it can be done for free.
Once you're past the hurdle of public accessibility, you need to run on your computer some server that can respond to properly authenticated requests by supplying the information you desire. For example, you could run a web server such as Apache (which is powerful indeed but perhaps a bit hard for you to set up), or the like -- and a custom app on top of it to check authentication and provide the specific information you want to make available.
If you have no privacy worry (i.e., you don't mind that any hacker in the world can look at that folder's contents), you can skip the authentication, which is the really delicate and potentially fragile part (given that there's really no way for your app, running on a shared web hosting provider, to hold "secrets" very effectively).
If you can clarify each of these issues, then we can help pinpoint the best approach (what to install and how on both your desktop computer, and that shared web hosting provider).