Hi I am struggling to understand the following example.
I want to add functionality for a user to reset password for my website.
As a lot of sites do I want to send a token to the users email that will let them reset their password.
I am following a guide that suggests using a python module called itsdangerous.
I have been given the following code as a simple example from the tutorial to understand how the module works before deploying to my website:
from itsdangerous import TimedJSONWebSignatureSerializer as Serializer
s = Serializer('secret_key',30)
token = s.dumps({'usr_id': 1}).decode('utf-8')
s.loads(token)
Now if I run this here is what happens:
I use s to create a token that allows me to take a dictionary {'usr_id':1} then if I run s.loads(token) within 30 seconds I can get this dictionary {'usr_id':1} back otherwise I get an error.
Can anyone explain (in a simple way for a beginner) what is going on here?
I don't really understand what is happening and I don't see what the secret_key argument to the Serializer is doing?
Also if someone could explain how this kind of code can help me with allowing users to get an email to reset their password that would be great. Thanks!
So this kind of serialization provided by itsdangerous library is JSON Web Token based. In order to create a JSON Web Token you need a secret key, which is a signature to certify that the information tokenized was provided by you and can only be edited with this signature.
A JSON Web Token can be read by anyone but can only be edited by someone who knows the secret key - so you shouldn't tokenize sensitive information like a password, but a user id is ok, which alongside a expiry time set is just enough to check its identity. You must hide your secret key and keep it safe.
A good usage in your case would be to send a expiring token - let's say one day - to the user email, to proof it has authorization to change the password. And after one day it'll be invalid, so you won't compromise your system.
What is the JSON Web Token structure?
In its compact form, JSON Web Tokens consist of three parts separated by dots (.), which are:
Header: The header typically consists of two parts: the type of the token, which is JWT, and the signing algorithm being used, such as HMAC SHA256 or RSA.
Payload: The information and the expiry time is held here.
Signature: The signature is used to verify the message wasn't changed along the way, and, in the case of tokens signed with a secret key, it can also verify that the sender of the JWT is who it says it is.
Therefore, a JWT typically looks like the following. xxxxx.yyyyy.zzzzz
You can play around seeing what is inside a JWT by pasting it on the link below:
https://jwt.io/
More info on:
https://jwt.io/introduction/
I'm using pyotp https://github.com/pyotp/pyotp to integrate my application with Google Authenticator.
The documentation suggest using qrious https://github.com/neocotic/qrious this is fine and works well. Essentially qrious is able to generate a QR code purely in the browser. In this case, the provisioning URI is passed to the QR code generator and a QR code is made from that.
The thing that puzzles me is that the provisioning URI contains the secret key, and yet we send this URI to the client end to be turned into a QR code by qrious. So the secret key isn't secret because it has been sent to the client.
I would have expected that the secret key must never be sent out of the back end - what am I failing to understand?
# generate a base32 secret key
>>> pyotp.random_base32()
'55OZSEMXLL7VAUZP'
# make a provisioning_URI
>>> provisioning_URI = pyotp.totp.TOTP('55OZSEMXLL7VAUZP').provisioning_uri('someperson#example.org',issuer_name="FooCorporation")
>>> provisioning_URI
'otpauth://totp/FooCorporation:someperson%40example.org?secret=55OZSEMXLL7VAUZP&issuer=FooCorporation'
>>>
The provisioning_URI gets sent to the browser to be turned into a QR code - but it includes the secret key - surely that's not secure?
Yes, you should keep it secret from others and share the QR code with the intended users only so they will scan it using their phone app. And you are right. Sending it to the client-side is risky but even if you are going to generate the QR code on the server and show it to the user, still chances are that it may get compromised anytime. The user's phone may get stolen too. So there are a lot of risks. But it is the best idea to generate the QR code on the server and present it to the user so they will scan it. Just don't use any remote solution or any browser extension to generate the code to make your life easier.
In short, you should never consider this form of verification alone and this verification should be always implemented only after the user enters and verify their password.
When it comes to storing the secret key on your server, you should always encrypt it using a technique like this one and then decrypt it on the fly when you need the secret for verification. And obviously, the secret should be generated for each user separately, otherwise, this doesn't make sense. If you will use the same secret, then secrets can be generated for any user by just swapping their emails.
I will once again repeat here that 2FA should never be considered as a standalone security measure. It should be implemented always as the second layer of the authentication.
The context is testing of a web app with selenium while using a number of virtual user accounts we created for this very purpose. And so the testing process needs to access our sites and log-on with the virtual user's id and password.
None of these accounts are critical and they are flagged as testing accounts so no damage can be done. Still, it would probably be a good idea to encrypt the passwords and decrypt them prior to use.
If it matter, our test app is written in Python, Django and uses PostgreSQL for the database. It runs on a small Linode instance.
What might best practices be for something like this?
EDIT 1
The other thought I had was to store the credentials on a second machine and access them through and API while only allowing that access to happen from a known server's non-public IP. In other words, get two instances at Linode and create a private machine-to-machine connection within the data center.
In this scenario, access to the first machine would allow someone to potentially make requests to the second machine if they are able to de-obfuscate the API code. If someone really wants the data they can certainly get it.
We could add two factor authentication as a way to gate the tests. In other words, even if you had our unencrypted test_users table you couldn't do anything with them because of the 2FA mechanism in place just for these users.
Being that this is for testing purposes only I am starting to think the best solution might very well be to populate the test_users table with valid passwords only while running a test. We could keep the data safe elsewhere and have a script that uploads the data to the test server when we want to run a test suite. Someone with access to this table could not do thing with it because all the passwords would be invalid. In fact, we could probably use this fact to detect such a breach.
I just hate the idea of storing unencrypted passwords even if it is for test users that can't really do any damage to the actual app (their transactions being virtual).
EDIT 2
An improvement to that would be to go ahead and encrypt the data and keep it in the test server. However, every time the tests are run the system would reach out to us for the crypto key. And, perhaps, after the test is run the data is re-encrypted with a new key. A little convoluted but it would allow for encrypted passwords (and even user id's, just to make it harder) on the test server. The all-important key would be nowhere near the server and it would self-destruct after each use.
What is generally done in a case like this is to put the password through a cryptographic hash function, and store the hashed password.
To verify a login, hash the provided password and compare the calculated hash to the stored version.
The idea behind this is that it is considered impossible to reverse a good cryptographic hash function. So it doesn't matter if an attacker could read the hashed passwords.
Example in Python3:
In [1]: import hashlib
In [2]: hashlib.sha256('This is a test'.encode('utf8')).hexdigest()
Out[2]: 'c7be1ed902fb8dd4d48997c6452f5d7e509fbcdbe2808b16bcf4edce4c07d14e'
In [3]: hashlib.sha256('This is a tist'.encode('utf8')).hexdigest()
Out[3]: 'f80b4162fc28f1f67d1a566da60c6c5c165838a209e89f590986333d62162cba'
In [4]: hashlib.sha256('This is a tst.'.encode('utf8')).hexdigest()
Out[4]: '1133d07c24ef5f46196ff70026b68c4fa703d25a9f12405ff5384044db4e2adf'
(for Python2, just leave out the encode.)
As you can see, even one-letter changes lead to a big change in the hash value.
If app.secret_key isn't set, Flask will not allow you to set or access the session dictionary.
This is all that the flask user guide has to say on the subject.
I am very new to web development and I have no idea how/why any security stuff works. I would like to understand what Flask is doing under the hood.
Why does Flask force us to set this secret_key property?
How does Flask use the secret_key property?
The answer below pertains primarily to Signed Cookies, an implementation of the concept of sessions (as used in web applications). Flask offers both, normal (unsigned) cookies (via request.cookies and response.set_cookie()) and signed cookies (via flask.session). The answer has two parts: the first describes how a Signed Cookie is generated, and the second is presented as a series of Question/Answer that address different aspects of the scheme. The syntax used for the examples is Python3, but the concepts apply also to previous versions.
What is SECRET_KEY (or how to create a Signed Cookie)?
Signing cookies is a preventive measure against cookie tampering. During the process of signing a cookie, the SECRET_KEY is used in a way similar to how a "salt" would be used to muddle a password before hashing it. Here's a (widely) simplified description of the concept. The code in the examples is meant to be illustrative. Many of the steps have been omitted and not all of the functions actually exist. The goal here is to provide a general understanding of the main idea, but practical implementations will likely be a bit more involved. Also, keep in mind that Flask already provides most of this for you in the background. So, besides setting values to your cookie (via the session API) and providing a SECRET_KEY, it's not only ill-advised to re-implement this yourself, but there's no need to do so:
A poor man's cookie signature
Before sending a Response to the browser:
( 1 ) First a SECRET_KEY is established. It should only be known to the application and should be kept relatively constant during the application's life cycle, including through application restarts.
# choose a salt, a secret string of bytes
>>> SECRET_KEY = 'my super secret key'.encode('utf8')
( 2 ) create a cookie
>>> cookie = make_cookie(
... name='_profile',
... content='uid=382|membership=regular',
... ...
... expires='July 1 2030...'
... )
>>> print(cookie)
name: _profile
content: uid=382|membership=regular...
...
...
expires: July 1 2030, 1:20:40 AM UTC
( 3 ) to create a signature, append (or prepend) the SECRET_KEY to the cookie byte string, then generate a hash from that combination.
# encode and salt the cookie, then hash the result
>>> cookie_bytes = str(cookie).encode('utf8')
>>> signature = sha1(cookie_bytes+SECRET_KEY).hexdigest()
>>> print(signature)
7ae0e9e033b5fa53aa....
( 4 ) Now affix the signature at one end of the content field of the original cookie.
# include signature as part of the cookie
>>> cookie.content = cookie.content + '|' + signature
>>> print(cookie)
name: _profile
content: uid=382|membership=regular|7ae0e9... <--- signature
domain: .example.com
path: /
send for: Encrypted connections only
expires: July 1 2030, 1:20:40 AM UTC
and that's what is sent to the client.
# add cookie to response
>>> response.set_cookie(cookie)
# send to browser -->
Upon receiving the cookie from the browser:
( 5 ) When the browser returns this cookie back to the server, strip the signature from the cookie's content field to get back the original cookie.
# Upon receiving the cookie from browser
>>> cookie = request.get_cookie()
# pop the signature out of the cookie
>>> (cookie.content, popped_signature) = cookie.content.rsplit('|', 1)
( 6 ) Use the original cookie with the application's SECRET_KEY to recalculate the signature using the same method as in step 3.
# recalculate signature using SECRET_KEY and original cookie
>>> cookie_bytes = str(cookie).encode('utf8')
>>> calculated_signature = sha1(cookie_bytes+SECRET_KEY).hexdigest()
( 7 ) Compare the calculated result with the signature previously popped out of the just received cookie. If they match, we know that the cookie has not been messed with. But if even just a space has been added to the cookie, the signatures won't match.
# if both signatures match, your cookie has not been modified
>>> good_cookie = popped_signature==calculated_signature
( 8 ) If they don't match then you may respond with any number of actions, log the event, discard the cookie, issue a fresh one, redirect to a login page, etc.
>>> if not good_cookie:
... security_log(cookie)
Hash-based Message Authentication Code (HMAC)
The type of signature generated above that requires a secret key to ensure the integrity of some contents is called in cryptography a Message Authentication Code or MAC.
I specified earlier that the example above is an oversimplification of that concept and that it wasn't a good idea to implement your own signing. That's because the algorithm used to sign cookies in Flask is called HMAC and is a bit more involved than the above simple step-by-step. The general idea is the same, but due to reasons beyond the scope of this discussion, the series of computations are a tad bit more complex.
If you're still interested in crafting a DIY, as it's usually the case, Python has some modules to help you get started :) here's a starting block:
import hmac
import hashlib
def create_signature(secret_key, msg, digestmod=None):
if digestmod is None:
digestmod = hashlib.sha1
mac = hmac.new(secret_key, msg=msg, digestmod=digestmod)
return mac.digest()
The documentaton for hmac and hashlib.
The "Demystification" of SECRET_KEY :)
What's a "signature" in this context?
It's a method to ensure that some content has not been modified by anyone other than a person or an entity authorized to do so.
One of the simplest forms of signature is the "checksum", which simply verifies that two pieces of data are the same. For example, when installing software from source it's important to first confirm that your copy of the source code is identical to the author's. A common approach to do this is to run the source through a cryptographic hash function and compare the output with the checksum published on the project's home page.
Let's say for instance that you're about to download a project's source in a gzipped file from a web mirror. The SHA1 checksum published on the project's web page is 'eb84e8da7ca23e9f83....'
# so you get the code from the mirror
download https://mirror.example-codedump.com/source_code.tar.gz
# you calculate the hash as instructed
sha1(source_code.tar.gz)
> eb84e8da7c....
Both hashes are the same, you know that you have an identical copy.
What's a cookie?
An extensive discussion on cookies would go beyond the scope of this question. I provide an overview here since a minimal understanding can be useful to have a better understanding of how and why SECRET_KEY is useful. I highly encourage you to follow up with some personal readings on HTTP Cookies.
A common practice in web applications is to use the client (web browser) as a lightweight cache. Cookies are one implementation of this practice. A cookie is typically some data added by the server to an HTTP response by way of its headers. It's kept by the browser which subsequently sends it back to the server when issuing requests, also by way of HTTP headers. The data contained in a cookie can be used to emulate what's called statefulness, the illusion that the server is maintaining an ongoing connection with the client. Only, in this case, instead of a wire to keep the connection "alive", you simply have snapshots of the state of the application after it has handled a client's request. These snapshots are carried back and forth between client and server. Upon receiving a request, the server first reads the content of the cookie to reestablish the context of its conversation with the client. It then handles the request within that context and before returning the response to the client, updates the cookie. The illusion of an ongoing session is thus maintained.
What does a cookie look like?
A typical cookie would look like this:
name: _profile
content: uid=382|status=genie
domain: .example.com
path: /
send for: Encrypted connections only
expires: July 1 2030, 1:20:40 AM UTC
Cookies are trivial to peruse from any modern browser. On Firefox for example go to Preferences > Privacy > History > remove individual cookies.
The content field is the most relevant to the application. Other fields carry mostly meta instructions to specify various scopes of influence.
Why use cookies at all?
The short answer is performance. Using cookies, minimizes the need to look things up in various data stores (memory caches, files, databases, etc), thus speeding things up on the server application's side. Keep in mind that the bigger the cookie the heavier the payload over the network, so what you save in database lookup on the server you might lose over the network. Consider carefully what to include in your cookies.
Why would cookies need to be signed?
Cookies are used to keep all sorts of information, some of which can be very sensitive. They're also by nature not safe and require that a number of auxiliary precautions be taken to be considered secure in any way for both parties, client and server. Signing cookies specifically addresses the problem that they can be tinkered with in attempts to fool server applications. There are other measures to mitigate other types of vulnerabilities, I encourage you to read up more on cookies.
How can a cookie be tampered with?
Cookies reside on the client in text form and can be edited with no effort. A cookie received by your server application could have been modified for a number of reasons, some of which may not be innocent. Imagine a web application that keeps permission information about its users on cookies and grants privileges based on that information. If the cookie is not tinker-proof, anyone could modify theirs to elevate their status from "role=visitor" to "role=admin" and the application would be none the wiser.
Why is a SECRET_KEY necessary to sign cookies?
Verifying cookies is a tad bit different than verifying source code the way it's described earlier. In the case of the source code, the original author is the trustee and owner of the reference fingerprint (the checksum), which will be kept public. What you don't trust is the source code, but you trust the public signature. So to verify your copy of the source you simply want your calculated hash to match the public hash.
In the case of a cookie however the application doesn't keep track of the signature, it keeps track of its SECRET_KEY. The SECRET_KEY is the reference fingerprint. Cookies travel with a signature that they claim to be legit. Legitimacy here means that the signature was issued by the owner of the cookie, that is the application, and in this case, it's that claim that you don't trust and you need to check the signature for validity. To do that you need to include an element in the signature that is only known to you, that's the SECRET_KEY. Someone may change a cookie, but since they don't have the secret ingredient to properly calculate a valid signature they cannot spoof it. As stated a bit earlier this type of fingerprinting, where on top of the checksum one also provides a secret key, is called a Message Authentication Code.
What about Sessions?
Sessions in their classical implementation are cookies that carry only an ID in the content field, the session_id. The purpose of sessions is exactly the same as signed cookies, i.e. to prevent cookie tampering. Classical sessions have a different approach though. Upon receiving a session cookie the server uses the ID to look up the session data in its own local storage, which could be a database, a file, or sometimes a cache in memory. The session cookie is typically set to expire when the browser is closed. Because of the local storage lookup step, this implementation of sessions typically incurs a performance hit. Signed cookies are becoming a preferred alternative and that's how Flask's sessions are implemented. In other words, Flask sessions are signed cookies, and to use signed cookies in Flask just use its Session API.
Why not also encrypt the cookies?
Sometimes the contents of cookies can be encrypted before also being signed. This is done if they're deemed too sensitive to be visible from the browser (encryption hides the contents). Simply signing cookies however, addresses a different need, one where there's a desire to maintain a degree of visibility and usability to cookies on the browser, while preventing that they'd be meddled with.
What happens if I change the SECRET_KEY?
By changing the SECRET_KEY you're invalidating all cookies signed with the previous key. When the application receives a request with a cookie that was signed with a previous SECRET_KEY, it will try to calculate the signature with the new SECRET_KEY, and both signatures won't match, this cookie and all its data will be rejected, it will be as if the browser is connecting to the server for the first time. Users will be logged out and their old cookie will be forgotten, along with anything stored inside. Note that this is different from the way an expired cookie is handled. An expired cookie may have its lease extended if its signature checks out. An invalid signature just implies a plain invalid cookie.
So unless you want to invalidate all signed cookies, try to keep the SECRET_KEY the same for extended periods.
What's a good SECRET_KEY?
A secret key should be hard to guess. The documentation on Sessions has a good recipe for random key generation:
>>> import os
>>> os.urandom(24)
'\xfd{H\xe5<\x95\xf9\xe3\x96.5\xd1\x01O<!\xd5\xa2\xa0\x9fR"\xa1\xa8'
You copy the key and paste it in your configuration file as the value of SECRET_KEY.
Short of using a key that was randomly generated, you could use a complex assortment of words, numbers, and symbols, perhaps arranged in a sentence known only to you, encoded in byte form.
Do not set the SECRET_KEY directly with a function that generates a different key each time it's called. For example, don't do this:
# this is not good
SECRET_KEY = random_key_generator()
Each time your application is restarted it will be given a new key, thus invalidating the previous.
Instead, open an interactive python shell and call the function to generate the key, then copy and paste it to the config.
Anything that requires encryption (for safe-keeping against tampering by attackers) requires the secret key to be set. For just Flask itself, that 'anything' is the Session object, but other extensions can make use of the same secret.
secret_key is merely the value set for the SECRET_KEY configuration key, or you can set it directly.
The Sessions section in the Quickstart has good, sane advice on what kind of server-side secret you should set.
Encryption relies on secrets; if you didn't set a server-side secret for the encryption to use, everyone would be able to break your encryption; it's like the password to your computer. The secret plus the data-to-sign are used to create a signature string, a hard-to-recreate value using a cryptographic hashing algorithm; only if you have the exact same secret and the original data can you recreate this value, letting Flask detect if anything has been altered without permission. Since the secret is never included with data Flask sends to the client, a client cannot tamper with session data and hope to produce a new, valid signature.
Flask uses the itsdangerous library to do all the hard work; sessions use the itsdangerous.URLSafeTimedSerializer class with a customized JSON serializer.
I'm building a Flash application to run on the web, where users can visit and create their own content in conjunction with my service (built with Python). Specifically: the user sends in some data; some transformation is performed on the server; then the finished content is sent back to the user, where it is rendered by the client app.
I want to be able to prevent the client from rendering bogus content, which I can do by passing a keyed hash along with the main content, generated by the server. The client would then use the same key to hash the content once again, and confirm that the hashes/signatures match. If there's a mismatch, it can be assumed that the content is inauthentic.
The problem I have is that keeping the key inside the SWF is insecure. I've considered a number of ways to obfuscate the key, but am learning that if an attacker wants it, they can get it quite easily. Once an attacker has that, they can start creating their own content to be unknowingly accepted by the client.
Is there another way that I can verify a file's signature on the client side, without exposing the method used to create that signature?
Is there another way that I can verify a file's signature on the client side, without exposing the method used to create that signature?
Public key crypto. You have only a public key at the client end, and require the private key on the server side to generate a signature for it to verify.
What is the attack you're trying to prevent? If you are concerned about a man-in-the-middle attack between an innocent user and your server, the sensible choice would be TLS (HTTPS). This is a pre-cooked, known-good implementation including public key cryptography. It's far preferable to rolling your own crypto, which is very easy to get wrong.