I need to get an App Engine app talking to and sharing data with an external database,
The best option i can come up with is outputting the external database data to an xml file and then processing this in my app engine app and storing it inside the datastore,
although the data being shared is sensitive data such as login details so outputting this to an xml file is not exactly a great idea, is it possible for the app engine app to directly query the database? or is there a secure option for using xml files?
oh and im using python/django and the external database will be hosted on another domain
Google Apps' Secure Data Connector (SDC) is designed for this kind of tasks -- indeed, it even works when the "other database" lives behind a firewall (a common case for enterprise data), and for other Google Apps (Docs, Spreadsheets, ...) as well as App Engine.
As the docs summarize things, the flow is:
Google Apps forwards authorized data
requests from users who are within
the Google Apps domain to the Google
tunnel protocol servers.
The tunnel servers validate that a
user is authorized to make the
request to the specified resource.
Google tunnel servers are connected
by an encrypted tunnel to SDC, which
runs within a company's internal
network.
The tunnel protocol allows SDC to
connect to a Google tunnel server,
authenticate, and encrypt the data
that flows across the Internet.
SDC uses resource rules to validate
if a user is authorized to make a
request to a specified resource.
An optional intranet firewall can be
used to provide extra network
security.
SDC performs a network request to
the specified resource or services.
The service validates the signed
request, checks the credentials, and
if the user is authorized, returns
the data.
If you don't have to worry about firewalls, and have no security worries whatsoever, you can simplify things (as Daniel's answer suggests) by just using urlfetch directly (no tunnels, no validation, no encryption, no filtering, ...) -- but your worry about "the data being shared is sensitive data such as login details" suggests that this is not the case.
It's not a problem of XML vs other formats -- the problem is that sensitive data should not travel "in clear" over unprotected channels, nor be made available to all and sundry, and it's often nicer to have specialized infrastructure deal with encryption, filtering, and authorization problems, as the SDC does, rather than having to code all of this (and make it totally secure and locked-down) in your own app or specialized infrastructure middleware. For these purposes, the SDC can be very helpful, even if you only need a fraction of its functionality.
You may want to consider exposing a set of web services on the external domain where your database is hosted, and then use the App Engine's URL Fetch API to communicate with your external domain via HTTPS.
Related
I have a general architectural question regarding a new project with microservices based on FastAPI. Looking at it now it’s really not FastAPI specific but here it goes:
How would you make JWT based authentication work between multiple services assuming the following:
You cannot use managed services from AWS etc. like API Gateway (security requirement)
There is a service for user management, which contains username, password etc. and ideally it would allow for JWT authentication
There are other microservices containing app logic for the users stored in the first microservice (from 2.)
In a monolith app this is easy and in a microservice architecture with API gateways it’s also easy but:
how would you approach this (high level) if things kind of hinge on the user management microservice to hold the relevant user data (and in effect being a gateway of its own) where other services should perform actions for those users (based on the JWT token)
tl;dr How would you extend authentication to the other microservices holding the app logic? I think I am overthinking this one and potentially introducing too much coupling :)
If your authentication/user management service issues a JWT, then your other services can rely on that to provide only the allowed data.
The token issued should contain all relevant security info. Let's say you have three types of user: admin, manager, employee. For sake of simplicity let's say this is the only security criteria (you may have multiple criteria in place).
The token can have a field indicating the user permission. As per the OAuth2 spec this is usually done in a scope field. You can create your own custom fields for it anyways.
With the token you can then call your backend services using it. The services will then:
first check the integrity and validity of the token
check the token payload for the scope/permissions and provide the data according to it.
See also the official FastAPI tutorial about using scopes for this purpose with a simple dependency injection.
Answering your comment:
JWT is sadly only within Traefik Enterprise
Maybe you're talking about handling the JWT directly in Traefik, but you should have no trouble passing on the authorization header and deal with the authorization check in the backend service. Plus, you can always embed the token in the request body and totally bypass the reverse proxy.
I'm building a PWA with django/python on the server-side and vue on the client-side and want to use firebase as a database as well as make use of the firebase authentication.
After some thorough research I realised that I had to make a few choices.
Question 1: Authentication
I can do authentication on the client-side or server-side. Which one would be best (more secure) ?
Question 2: Database
In terms of CRUDS I am a bit conflicted. Do I write all my data to firestore from the client-side?
Do I rather use api's to communicate with my backend and then write data to firestore from the backend? What are the security implications of doing this?
Should I just use both in terms of context? If there are no security implications I would do my authentication client-side and my CRUDS from the server-side. I think I would also have to check authentication to write to the database from the backend.
Authentication of a user's credentials should always happen on a server, as it can't be securely done on the client's computer. What Firebase Authentication allows however, is that the authentication runs on Google's servers, while you control it from a simple client-side API call.
There are multiple mobile apps.
I want people using one app to login with their same login credentials into all other apps.
What is the best approach to implement this?
I'm thinking to create a separate authorization server that will issue tokens/secrets on registering and logins. It will have a validation API that will be used by mobile app servers to validate requests.
There is a protocol for handling this called Central Authentication Service(http://www.jasig.org/cas), or CAS. CAS utilizes OAuth2 protocol to provide a secure way to manage a single sign on procedure -- users have a single login credential that is used via tokens and proxy tickets to authenticate the user across a variety of applications using a single CAS authentication server. This is a procedure the company I work for uses to authenticate users across multiple applications (we use Java with Spring Security and Spring CAS) however this summer we built out a proof of concept with Django and an internal company API in which we implemented CAS procedure with the Django application -- Django authenticated users via the CAS server we run and returned them as valid, authenticated Django users. To do this we used the django-cas middleware and custom backend library which can be found here: https://bitbucket.org/cpcc/django-cas/overview.
CAS is definitely the way to go if you're looking to handle user authentication with a single sign on across multiple applications -- it was the reason CAS was created in the first place. The learning curve with CAS can be a little steep but django-cas does a good enough job making it easy to get set up with CAS in the Django project.
First check if OAuth could be adapted to using this, that would save you a lot of work. Of course all the services and apps would have to talk to some backend network server to sync tokens issued to apps.
Half-secure/maybe-abusable solution: have symmetric cipher encrypted cookie that webpages (and apps?) hold and use it for authorization with different network services (which again have to verify cookie for authorization with authorization service that knows the passphrase used to encrypt the cookie)
I've used approach #2 on internal systems but I am not sure if it is advisable to use it in in the wild - this may pose some security risks.
The authentification method in every application connects to the same webservice for autentification.
Here is the situation:
We use Flask for a website application development.Also on the website sever, we host a RESTful service. And we use Flask-login for as the authentication tool, for BOTH the web application access and the RESTful service (access the Restful service from browsers).
Later, we find that we need to, also, access the RESTful from client calls (python), so NO session and cookies etc. This gives us a headache regarding the current authentication of the RESTful service.
On the web, there exist whole bunch of ways to secure the RESTful service from client calls. But it seems no easy way for them to live together with our current Flask-login tool, such that we do not need to change our web application a lot.
So here are the question:
Is there a easy way(framework) so the RESTful services can support multiple authentication methods(protocols) at the same time. Is this even a good practice?
Many thanks!
So, you've officially bumped into one of the most difficult questions in modern web development (in my humble opinion): web authentication.
Here's the theory behind it (I'll answer your question in a moment).
When you're building complicated apps with more than a few users, particularly if you're building apps that have both a website AND an API service, you're always going to bump into authentication issues no matter what you're doing.
The ideal way to solve these problems is to have an independent auth service on your network. Some sort of internal API that EXCLUSIVELY handles user creation, editing, and deletion. There are a number of benefits to doing this:
You have a single authentication source that all of your application components can use: your website can use it to log people in behind the scenes, your API service can use it to authenticate API requests, etc.
You have a single service which can smartly managing user caching -- it's pretty dangerous to implement user caching all over the place (which is what typically happens when you're dealing with multiple authentication methods: you might cache users for the API service, but fail to cache them with the website, stuff like this causes problems).
You have a single service which can be scaled INDEPENDENTLY of your other components. Think about it this way: what piece of application data is accessed more than any other? In most applications, it's the user data. For every request user data will be needed, and this puts a strain on your database / cache / whatever you're doing. Having a single service which manages users makes it a lot nicer for you to scale this part of the application stack easily.
Overall, authentication is really hard.
For the past two years I've been the CTO at OpenCNAM, and we had the same issue (a website and API service). For us to handle authentication properly, we ended up building an internal authentication service like described above, then using Flask-Login to handle authenticating users via the website, and a custom method to authenticate users via the API (just an HTTP call to our auth service).
This worked really well for us, and allowed us to scale from thousands of requests to billions (by isolating each component in our stack, and focusing on user auth as a separate service).
Now, I wouldn't recommend this for apps that are very simple, or apps that don't have many users, because it's more hassle than it's worth.
If you're looking for a third party solution, Stormpath looks pretty promising (just google it).
Anyhow, hope that helps! Good luck.
Is there a way to specify a proxy server when using urlfetch on Google App Engine?
Specifically, every time I make a call using urlfetch, I want GAE to go through a proxy server. I want to do this on production, not just dev.
I want to use a proxy because there are problems with using google's outbound IP addresses (rate limiting, no static outbound IP, sometimes blacklisted, etc.). Setting a proxy is normally easy if you can edit the http message itself, but GAE's API does not appear to let you do this.
You can always roll your own:
In case of fixed destination: just setup a fixed port forwarding on a proxy server. Then send requests from GAE to proxy. If you have multiple destinations, then set forwarding on separate ports, one for each destination.
In case of dynamic destination (too much to handle via fixed port forwarding), your GAE app adds a custom http header (X-Something) containing final destination and then connects to custom proxy. Custom proxy inspects this field and forwards the request to the destination.
We ran into this issue and reached out to Google Cloud support. They suggested we use Google App Engine flexible with some app.yaml settings, custom network, and an ip-forwarding NAT gateway instance.
This didn't work for us because many core features from App Engine Standard are not implemented in App Engine Flexible. In essence we would need to rewrite our product.
So, to make applicable URL fetch requests appear to have a static IP we made a custom proxy: https://github.com/csgactuarial/app-engine-proxy
For redundancy reasons, I suggest implementing this as a multi region, multi zone, load balanced system.