I retrieved user information using api.followers in tweepy and am trying to store them in couchDB, but I keep getting this error message
"u'doc validation, u'Bad Special document member" _json".
def save_user(self, u):
temp = jsonpickle.encode(u)
temp_obj = json.loads(temp)
user_obj = temp_obj['py/state']
self.db.save(user_obj)
u is the user profile returned by the command
for user in api.followers(screen_name="sharonsanderso6"):
storage.save_user(user)
directly storing user to couchDB gives an error "string indices must be integers,not str". So tried decoding it using jsonpickle and json.loads. after doing this I get u'Bad character error. How else can I store it couchDB?
Bad Special document member _json
CouchDB reserves JSON properties beginning with an underscore for itself. Change the key of the property to something that doesn't starts with underscore.
Related
This is a pretty basic question, but regardless its had me stumped for a bit. I'm trying to access a specific attribute of a tweet (documentation found here), such as "text". I tried accessing it via data["text"], however this gives me the following error TypeError: string indices must be integers.
So I tried parsing the data using json.loads(data) thinking this would allow me to access each attribute of the tweet. However this instead returns solely the text portion of the tweet, meaning when I do print(newData), it prints out the text. Although this is useful, I need to be able to access other attributes of the tweet such as "created_at".
So my question is, how do I parse the tweet or access it which allows me to pluck out individual attributes I need. To reiterate, I'm sure this is pretty simple, however I'm new to handling JSON objects, and other solutions I found simply told me to use loads(), which isn't what I want.
class TwitterStreamer():
"""
Class for streaming and processing live tweets for a given list of hashtags
"""
def stream_tweets(selfself, hashtag_list):
listener = StdOutListener()
auth = OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
auth.set_access_token(ACCESS_TOKEN, ACCESS_TOKEN_SECRET)
stream = Stream(auth, listener)
stream.filter(track=hashtag_list)
class StdOutListener(StreamListener):
def on_data(self, data):
print(data)
newData = json.loads(data)
print(newData["text"])
return True
def on_error(self, status):
print(status)
def main():
hashtag_list = ['Chelsea']
fetched_tweets_filename = "tweets.json"
twitter_streamer = TwitterStreamer()
twitter_streamer.stream_tweets(hashtag_list)
main()
Try using "." operator to access attributes of the tweet. I used it in my code as follow:
tweet = follow_user.status.created_at
In this I got the user in the form of JSON data "status" is an attribute of that JSON object "follow_user"
Try using json.load() to load the JSON as a Python object. The method json.loads() load the JSON as a string, that's why it gives you a TypeError Exception since string objects indices can only be integers.
I am using Ajax to make some requests from client to server, I am using DJango and I have used some Raw Sql queries before, but all of my fields was Int, varchar and a Decimal, for the last one I had an enconding problem, but I overrided the "default" property of Json and everything worked.
But that was before, now I have a query wich gives me Decimal and DateTime fields, both of them gave me enconding errors, the overrided "default" doesn't work now, thats why with this new one I used DjangoJSONEncoder, but now I have another problem, and its not an encoding one, I am using dictfetchall(cursor) method, recomended on Django docs, to return a dictionary from the Sql query, because cursor.fetchall() gives me this error: 'tuple' object has no attribute '_meta'.
Before I just sended that dictionary to json.dumps(response_data,default=default) and everything was fine, but now for the encoding I have to use the following: json.dumps(response_data,cls=DjangoJSONEncoder) and if I send the dictionary in that way, I get this error:
SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data
And if I try to use the serializers, like this:
response_data2= serializers.serialize('json', list(response_data))
And later send response_data2 to dumps, I get this error:
'dict' object has no attribute '_meta'
This is the code for the MySql query:
def consulta_sql_personalizada(nombres,apellidos,puesto):
from django.db import connection, transaction
cursor = connection.cursor()
cursor.execute("""select E.idEmpleado as id,CONCAT(Per.nombres_persona,' ',Per.apellidos_persona) as nombre,P.nombre_puesto as puesto,E.motivo_baja_empleado as motivo_baja,E.fecha_contratacion_empleado AS fecha_contratacion,E.fecha_baja_empleado as fecha_baja, SUM(V.total_venta) AS ventas_mes,E.fotografia_empleado as ruta_fotografia from Empleado as E
inner join Puesto as P on E.Puesto_idPuesto=P.idPuesto
inner join Venta as V on V.vendedor_venta=E.idEmpleado
inner join Persona as Per on E.Persona_idPersona=Per.idPersona
where (Per.nombres_persona like %s OR Per.apellidos_persona like %s OR E.Puesto_idPuesto=%s)
AND E.estado_empleado=1 AND V.estado_venta=1
AND
(YEAR(V.fecha_venta) = YEAR(Now())
AND MONTH(V.fecha_venta) = MONTH(Now()))""",[nombres,apellidos,puesto])
row = dictfetchall(cursor)
return row
And this is the last part of the view that makes the query and send it to ajax using json:
response_data=consulta_sql_personalizada(rec_nombres,rec_apellidos,rec_puesto)
return HttpResponse(
json.dumps(response_data,cls=DjangoJSONEncoder),
content_type="application/json"
)
else:
return HttpResponse(
json.dumps({"nothing to see": "this isn't happening"}),
content_type="application/json"
)
What I want to know is, how can I parse the raw sql result to Json using that enconding?
Sorry, was my bad, i'm using JQuery ajax method, and in the "success" part I forgot to stop using json.parse to print the data in the console, the data was json already, that's why I had that line 1 column 1 error. My code worked exactly like it was posted here. If someone want to know how to make asynchronous requests, I followed this tutorial: Django form submissions using ajax
I'm trying to do API-request and I need API-key to different view. I'm trying to use session variable, but the key seems to be in some other format than trying to use variable from Sqlite database. API requests work with the key from database, but not with session variable
How I get API-key from database and from session:
key_session = request.session['key']
key_db = APIkey.objects.values_list('key', flat=True).get(pk=2)
Both of these return same values, when I print them. Key example:
3h3asdh-asdasd:oisf87sdf87a5df76asdf83jhjhasgd8
I'm using base64.encodestring function when trying to do authentication to API-service with my key:
query = request.GET.get('query')
url = urllib2.Request('https://api.someapiwebsite.com',
None, headers={'Content-Type':'application/json'})
base64string = base64.encodestring('%s' % (key_session)).replace('\n', '')
If I print base64string with session variable (key_session), I get:
MmoihjsdasdoihhaG5tbjpuq9876eq9asd98a7Nmd3dWYzN2JmbWZ2aW1nMGVw==
If I print base64string with session variable (key_db), only difference is the two last characters == is now 'IC', and I think that's why the authentication to API service is failing:
MmoihjsdasdoihhaG5tbjpuq9876eq9asd98a7Nmd3dWYzN2JmbWZ2aW1nMGVwIC
What is making this difference in the base64 encoded string?
Edit:
I can see difference when using print repr():
print repr(key_db)
3h3asdh-asdasd:oisf87sdf87a5df76asdf83jhjhasgd8
print repr(key_session)
3h3asdh-asdasd:oisf87sdf87a5df76asdf83jhjhasgd8\x02\x02
One of the strings probably contains some trailing characters that print isn't showing. If you use repr then you should be able to see what the difference is.
print(repr(key_session))
print(repr(key_db))
You can then strip any characters as necessary before encoding the string, for example:
key_session = key_session.rstrip('\x02')
So, I'm trying to use SQS to pass a Python object between two EC2 instances. Here's my failed attempt:
import boto.sqs
from boto.sqs.message import Message
class UserInput(Message):
def set_email(self, email):
self.email = email
def set_data(self, data):
self.data = data
def get_email(self):
return self.email
def get_data(self):
return self.data
conn = boto.sqs.connect_to_region('us-west-2')
q = conn.create_queue('user_input_queue')
q.set_message_class(UserInput)
m = UserInput()
m.set_email('something#something.com')
m.set_data({'foo': 'bar'})
q.write(m)
It returns an error message saying that The request must contain the parameter MessageBody. Indeed, the tutorial tells us to do m.set_body('something') before writing the message to the queue. But here I'm not passing a string, I want to pass an instance of my UserInput class. So, what should MessageBody be? I've read the docs and they say that
The constructor for the Message class must accept a keyword parameter “body” which represents the content or body of the message. The format of this parameter will depend on the behavior of the particular Message subclass. For example, if the Message subclass provides dictionary-like behavior to the user the body passed to the constructor should be a dict-like object that can be used to populate the initial state of the message.
I guess the answer to my question might be in that paragraph, but I can't make sense of it. Could anyone provide a concrete code example to illustrate what they are talking about?
For an arbitrary Python object the answer is to serialize the object into a string, use SQS to send that string to the other EC2 instance and deserialize the string back to an instance of the same class.
For example, you could use JSON with base64 encoding to serialize an object into a string and that would be the body of your message.
I am editing the Runwithfriends FB sample app to build one of my own. It was working fine and I was making a few changes here and there. I took a break from it for about a fortnight but now when I try to access the app, I get a strange python error:
C:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\ROOT\app\main.py in init_facebook(self=<main.RecentRunsHandler object>)
316 user_id=facebook.user_id, friends=friends,
317 access_token=facebook.access_token, name=me[u'name'],
=> 318 email=me.get(u'email'), picture=me[u'picture'])
319 user.put()
320 except KeyError, ex:
<..some portion clipped..>
class 'google.appengine.api.datastore_errors.BadValueError'>: Property picture must be a str or unicode instance, not a dict
args = ('Property picture must be a str or unicode instance, not a dict',)
message = 'Property picture must be a str or unicode instance, not a dict'"
I know this is a very generic error but its pointing to the following code. This code has always been there and I have never touched it. I really do not know where else to look now - I have searched exhaustively but couldnt find a clue. Sorry, if this is still too broad but I would be glad if you can tell me what other info can I provide to debug this :-(
def init_facebook(self):
"""Sets up the request specific Facebook and User instance"""
facebook = Facebook()
user = None
# initial facebook request comes in as a POST with a signed_request
if u'signed_request' in self.request.POST:
facebook.load_signed_request(self.request.get('signed_request'))
# we reset the method to GET because a request from facebook with a
# signed_request uses POST for security reasons, despite it
# actually being a GET. in webapp causes loss of request.POST data.
self.request.method = u'GET'
self.set_cookie(
'u', facebook.user_cookie, datetime.timedelta(minutes=1440))
elif 'u' in self.request.cookies:
facebook.load_signed_request(self.request.cookies.get('u'))
# try to load or create a user object
if facebook.user_id:
user = User.get_by_key_name(facebook.user_id)
if user:
# update stored access_token
if facebook.access_token and \
facebook.access_token != user.access_token:
user.access_token = facebook.access_token
user.put()
# refresh data if we failed in doing so after a realtime ping
if user.dirty:
user.refresh_data()
# restore stored access_token if necessary
if not facebook.access_token:
facebook.access_token = user.access_token
if not user and facebook.access_token:
me = facebook.api(u'/me', {u'fields': _USER_FIELDS})
try:
friends = [user[u'id'] for user in me[u'friends'][u'data']]
user = User(key_name=facebook.user_id,
user_id=facebook.user_id, friends=friends,
access_token=facebook.access_token, name=me[u'name'],
email=me.get(u'email'), picture=me[u'picture'])
user.put()
except KeyError, ex:
pass # ignore if can't get the minimum fields
self.facebook = facebook
self.user = user
Might have to do with the October 2012 Breaking Changes, quote:
/picture connection will return a dictionary when a callback is specified
We will start returning a dictionary containing the fields url, height, width, and is_silhouette when accessing the /picture connection for an object and specifying a callback property. Currently we just return the picture URL as a string.
So at this point in your code, where you are currently using picture=me[u'picture'], try accessing the url property of the picture dictionary instead. (If it has one; I can’t tell you for sure if this is applicable, since I don’t know if your code would be considered as having specified a callback property.)
If my assumption is correct, you could also enable the migration as described in the roadmap; but that will only make your app work in the old way until Oct. 3rd, so probably better to try and fix it right away.
This is the way to get the picture:
picture=me[u'picture'][u'data'][u'url']