Develop Reference

Return multiple times from one api call in Flask Restful

I want to call a generate() function and send a user a message, but then continue executing a function.
#application.route("/api/v1.0/gen", methods=['POST'])
def generate():
return "Your id for getting the generated data is 'hgF8_dh4kdsRjdr'"
main() #generate a data
return "Successfully generated something. Use your id to get the data"
I understand that this is not a correct way of returning, but I hope you get the idea of what I am trying to accomplish. Maybe Flask has some build-in method to return multiple times from one api call?

Basically, what are you describing is called Server-Sent Events (aka SSE)
The difference of this format, that they returned an 'eventstream' Response type instead of usual JSON/plaintext
And if you want to use it with python/flask, you need generators.
Small code example (with GET request):
#application.route("/api/v1.0/gen", methods=['GET'])
def stream():
def eventStream():
text = "Your id for getting the generated data is 'hgF8_dh4kdsRjdr'"
yield str(Message(data = text, type="message"))
main()
text = "Successfully generated something. Use your id to get the data"
yield str(Message(data = text, type="message"))
resp.headers['Content-Type'] = 'text/event-stream'
resp.headers['Cache-Control'] = 'no-cache'
resp.headers['Connection'] = 'keep-alive'
return resp
Message class you can find here: https://gist.github.com/Alveona/b79c6583561a1d8c260de7ba944757a7
And of course, you need specific client that can properly read such responses.
postwoman.io supports SSE at Real-Time tab

Python way of polling longrunning operations from operation name in Google Cloud?

I'm calling a Google Cloud Function that returns an Operation object implementing the google.longrunning.Operations interface. I want to poll this operation from another Python process that will only receive the operation name (will not have access to the operation object itself). So I need something like:
operation = getOperation(operationName)
isdone = operation.done()
AFAIK, you can't do the first step above. I haven't found it here: https://google-cloud-python.readthedocs.io/en/stable/core/operation.html
I would like to do what is explained in the docs about the google.longrunning interface (https://cloud.google.com/speech-to-text/docs/reference/rpc/google.longrunning#google.longrunning.Operations.GetOperation):
rpc GetOperation(GetOperationRequest) returns (Operation)
Where the GetOperationRequest simply requires the operation name. Is there a way to "re-create" an operation using functions from the google-cloud-python library?

Update for more recent clients. You need to refresh the operation using the OperationClient:
For updating an existing operation you will need to pass the channel across to the OperationClient.
For example, backing up a Firestore datastore.
from google.cloud import firestore_admin_v1
from google.api_core import operations_v1, grpc_helpers
import time
def main():
client = firestore_admin_v1.FirestoreAdminClient()
channel = grpc_helpers.create_channel(client.SERVICE_ADDRESS)
api = operations_v1.OperationsClient(channel)
db_path = client.database_path('myproject', 'mydb')
operation = client.export_documents(db_path)
current_status = api.get_operation(operation.name)
while current_status.done == False:
time.sleep(5)
current_status = api.get_operation(operation.name)
print('waiting to complete')
print('operation done')

In my case, The AutoML Tables Client didn't have a SERVICE_ADDRESS or SCOPE properties, so I can't create a new gRPC channel.
But using the existing one in the client seems to work!
from google.api_core import operations_v1
from google.cloud.automl_v1beta1 import TablesClient
automl_tables_client = TablesClient(
credentials=...,
project=...,
region=...,
)
operation_name = ""
grpc_channel = automl_tables_client.auto_ml_client.transport._channel
api_client = operations_v1.OperationsClient(grpc_channel)
response = api_client.get_operation(operation_name)

You can use the get_operation method of the "Long-Running Operations Client":
from google.api_core import operations_v1
api = operations_v1.OperationsClient()
name = ...
response = api.get_operation(name)

Accessing JSON Attributes of Tweet using Twitter API

This is a pretty basic question, but regardless its had me stumped for a bit. I'm trying to access a specific attribute of a tweet (documentation found here), such as "text". I tried accessing it via data["text"], however this gives me the following error TypeError: string indices must be integers.
So I tried parsing the data using json.loads(data) thinking this would allow me to access each attribute of the tweet. However this instead returns solely the text portion of the tweet, meaning when I do print(newData), it prints out the text. Although this is useful, I need to be able to access other attributes of the tweet such as "created_at".
So my question is, how do I parse the tweet or access it which allows me to pluck out individual attributes I need. To reiterate, I'm sure this is pretty simple, however I'm new to handling JSON objects, and other solutions I found simply told me to use loads(), which isn't what I want.
class TwitterStreamer():
"""
Class for streaming and processing live tweets for a given list of hashtags
"""
def stream_tweets(selfself, hashtag_list):
listener = StdOutListener()
auth = OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
auth.set_access_token(ACCESS_TOKEN, ACCESS_TOKEN_SECRET)
stream = Stream(auth, listener)
stream.filter(track=hashtag_list)
class StdOutListener(StreamListener):
def on_data(self, data):
print(data)
newData = json.loads(data)
print(newData["text"])
return True
def on_error(self, status):
print(status)
def main():
hashtag_list = ['Chelsea']
fetched_tweets_filename = "tweets.json"
twitter_streamer = TwitterStreamer()
twitter_streamer.stream_tweets(hashtag_list)
main()

Try using "." operator to access attributes of the tweet. I used it in my code as follow:
tweet = follow_user.status.created_at
In this I got the user in the form of JSON data "status" is an attribute of that JSON object "follow_user"

Try using json.load() to load the JSON as a Python object. The method json.loads() load the JSON as a string, that's why it gives you a TypeError Exception since string objects indices can only be integers.

How to POST multiple FILES using Flask test client?

In order to test a Flask application, I got a flask test client POSTing request with files as attachment
def make_tst_client_service_call1(service_path, method, **kwargs):
_content_type = kwargs.get('content-type','multipart/form-data')
with app.test_client() as client:
return client.open(service_path, method=method,
content_type=_content_type, buffered=True,
follow_redirects=True,**kwargs)
def _publish_a_model(model_name, pom_env):
service_url = u'/publish/'
scc.data['modelname'] = model_name
scc.data['username'] = "BDD Script"
scc.data['instance'] = "BDD Stub Simulation"
scc.data['timestamp'] = datetime.now().strftime('%d-%m-%YT%H:%M')
scc.data['file'] = (open(file_path, 'rb'),file_name)
scc.response = make_tst_client_service_call1(service_url, method, data=scc.data)
Flask Server end point code which handles the above POST request is something like this
#app.route("/publish/", methods=['GET', 'POST'])
def publish():
if request.method == 'POST':
LOG.debug("Publish POST Service is called...")
upload_files = request.files.getlist("file[]")
print "Files :\n",request.files
print "Upload Files:\n",upload_files
return render_response_template()
I get this Output
Files:
ImmutableMultiDict([('file', <FileStorage: u'Single_XML.xml' ('application/xml')>)])
Upload Files:
[]
If I change
scc.data['file'] = (open(file_path, 'rb'),file_name)
into (thinking that it would handle multiple files)
scc.data['file'] = [(open(file_path, 'rb'),file_name),(open(file_path, 'rb'),file_name1)]
I still get similar Output:
Files:
ImmutableMultiDict([('file', <FileStorage: u'Single_XML.xml' ('application/xml')>), ('file', <FileStorage: u'Second_XML.xml' ('application/xml')>)])
Upload Files:
[]
Question:
Why request.files.getlist("file[]") is returning an empty list?
How can I post multiple files using flask test client, so that it can be retrieved using request.files.getlist("file[]") at flask server side ?
Note:
I would like to have flask client I dont want curl or any other client based solutions.
I dont want to post single file in multiple requests
Thanks
Referred these links already:
Flask and Werkzeug: Testing a post request with custom headers
Python - What type is flask.request.files.stream supposed to be?

You send the files as the parameter named file, so you can't look them up with the name file[]. If you want to get all the files named file as a list, you should use this:
upload_files = request.files.getlist("file")
On the other hand, if you really want to read them from file[], then you need to send them like that:
scc.data['file[]'] = # ...
(The file[] syntax is from PHP and it's used only on the client side. When you send the parameters named like that to the server, you still access them using $_FILES['file'].)

Lukas already addressed this,just providing these info as it may help someone
Werkzeug client is doing some clever stuff by storing requests data in MultiDict
#native_itermethods(['keys', 'values', 'items', 'lists', 'listvalues'])
class MultiDict(TypeConversionDict):
"""A :class:`MultiDict` is a dictionary subclass customized to deal with
multiple values for the same key which is for example used by the parsing
functions in the wrappers. This is necessary because some HTML form
elements pass multiple values for the same key.
:class:`MultiDict` implements all standard dictionary methods.
Internally, it saves all values for a key as a list, but the standard dict
access methods will only return the first value for a key. If you want to
gain access to the other values, too, you have to use the `list` methods as
explained below.
getList call looks for a given key in the "requests" dictionary. If the key doesn't exist, it returns empty list.
def getlist(self, key, type=None):
"""Return the list of items for a given key. If that key is not in the
`MultiDict`, the return value will be an empty list. Just as `get`
`getlist` accepts a `type` parameter. All items will be converted
with the callable defined there.
:param key: The key to be looked up.
:param type: A callable that is used to cast the value in the
:class:`MultiDict`. If a :exc:`ValueError` is raised
by this callable the value will be removed from the list.
:return: a :class:`list` of all the values for the key.
"""
try:
rv = dict.__getitem__(self, key)
except KeyError:
return []
if type is None:
return list(rv)
result = []
for item in rv:
try:
result.append(type(item))
except ValueError:
pass
return result

Get user id from user name for twitter using python

I am trying to get the user ID for a twitter user using a python
import twitter
twitter_api = twitter.Twitter(domain='api.twitter.com', api_version='1',auth = twitter.oauth.OAuth(ACCESS_KEY, ACCESS_SECRET,
CONSUMER_KEY, CONSUMER_SECRET))
usr = twitter_api.api.GetUser(SCREEN_NAME)
print usr.name
Does not work, and the debug information is not of much help, I did not find any more resources online.

The twitter_api object that you've instantiated implicitly converts all function calls performed on it into the path of the API URL, with any keyword arguments converted to API parameters. For example:
twitter_api.statuses.user_timeline()
# ^- this converts to "api.twitter.com/1/statuses/user_timeline.json"
twitter_api.statuses.user_timeline(user_id="a")
# ^- this converts to "api.twitter.com/1/statuses/user_timeline.json?user_id=a"
twitter_api.statuses.user_timeline(foo)
# ^- this breaks - "foo" is not a key/value pair, and cannot be sent
Hence, you're trying to call the URL 1/api/GetUser.json - and also passing it an argument, which it doesn't know how to handle. Try this instead:
import twitter
t = twitter.Twitter(auth=twitter.OAuth(ACCESS_KEY, ACCESS_SECRET, CONSUMER_KEY, CONSUMER_SECRET))
t.users.lookup(screen_name=SCREEN_NAME)
# returns the user object