finalKeywords = []
#after populating finalKeywords
query = {
"apiKey" : self.apikey,
"country" : country,
"currency" : currency,
"kw" : finalKeywords,
"t" : int(time.time())
}
uri = urllib.urlencode(query, True)
print uri
After decoding uri I get
country=&apiKey=5f0092f1777c3a5ed0de&kw=iphone&kw=android&t=1496924489¤cy=usd
While my expected output is
country=&apiKey=5f0092f1777c3a5ed0de&kw[0]=iphone&kw[1]=android&t=1496924489¤cy=usd
What to do..? I have been looking for solutions but unable to get.
Like this?
query = {
"apiKey" : self.apikey,
"country" : country,
"currency" : currency,
"kw[]" : finalKeywords,
"t" : int(time.time())
}
If you need to have the positional values in there as well for some reason, this is one way to do it:
query = {
"apiKey" : self.apikey,
"country" : country,
"currency" : currency,
"t" : int(time.time())
}
for p,q in enumerate(finalKeywords):
query["kw[{pos}]".format(pos=p)] = q
Based on the W3C spec for URLs, [ and ] aren't valid in the search string, and so will be escaped by urlencode. With that caveat, you can construct a dictionary with unique keys by using a dictionary comprehension:
kw_dict = { "kw[{}]".format(n): finalKeywords[n] for n in range(0, len(finalKeywords)) }
You can then add this to the other parts of your query as follows:
query = dict({
"apiKey" : self.apikey,
"country" : country,
"currency" : currency,
"t" : int(time.time())
}, **kw_dict)
Related
I have a structure like this:
{
"id" : 1,
"user" : "somebody",
"players" : [
{
"name" : "lala",
"surname" : "baba",
"player_place" : "1",
"start_num" : "123",
"results" : {
"1" : { ... }
"2" : { ... },
...
}
},
...
]
}
I am pretty new to MongoDB and I just cannot figure out how to extract results for a specific user (in this case "somebody", but there are many other users and each has an array of players and each player has many results) for a specific player with start_num.
I am using pymongo and this is the code I came up with:
record = collection.find(
{'user' : name}, {'players' : {'$elemMatch' : {'start_num' : start_num}}, '_id' : False}
)
This extracts players with specific player for a given user. That is good, but now I need to get specific result from results, something like this:
{ 'results' : { '2' : { ... } } }.
I tried:
record = collection.find(
{'user' : name}, {'players' : {'$elemMatch' : {'start_num' : start_num}}, 'results' : result_num, '_id' : False}
)
but that, of course, doesn't work. I could just turn that to list in Python and extract what I need, but I would like to do that with query in Mongo.
Also, what would I need to do to replace specific result in results for specific player for specific user? Let's say I have a new result with key 2 and I want to replace existing result that has key 2. Can I do it with same query as for find() (just replacing method find with method replace or find_and_replace)?
You can replace a specific result and the syntax for that should be something like this,
assuming you want to replace the result with key 1,
collection.updateOne({
"user": name,
"players.start_num": start_num
},
{ $set: { "players.$.results.1" : new_result }})
In the JSON below, I want to access the email-id and 'gamesplayed' field for each user.
"UserTable" : {
"abcd#gmailcom" : {
"gameHistory" : {
"G1" : [ {
"category" : "1",
"questiontext" : "What is the cube of 2 ?"
}, {
"category" : "2",
"questiontext" : "What is the cube of 4 ?"
} ]
},
"gamesplayed" : 2
},
"xyz#gmailcom" : {
"gameHistory" : {
"G1" : [ {
"category" : "1",
"questiontext" : "What is the cube of 2 ?"
}, {
"category" : "2",
"questiontext" : "What is the cube of 4 ?"
} ]
},
"gamesplayed" : 2
}
}
Following is the code that I using to try and access the users email-id:
for user in jp.match("$.UserTable[*].[0]", game_data):
print("User ID's {}".format(user_id))
This is the error I'm getting:
File "C:\ProgramData\Anaconda3\lib\site-packages\jsonpath_rw\jsonpath.py", line 444, in find
return [DatumInContext(datum.value[self.index], path=self, context=datum)]
KeyError: 0
And when I run the following line to and access the 'gamesplayed' field for each user, the IDE Crashes.
print (parser.ExtentedJsonPathParser().parse("$.*.gamesplayed").find(gd_info))
If you like to use JSONPath. Please try this.
Python code:
with open(json_file) as json_file:
raw_data = json.load(json_file)
jsonpath_expr = parse('$.UserTable')
players = [match.value for match in jsonpath_expr.find(raw_data)][0]
emails = players.keys()
result = [{'email': email, 'gamesplayed': players[email]['gamesplayed']} for email in emails ]
print (result)
Output:
[{'email': 'abcd#gmailcom', 'gamesplayed': 2}, {'email': 'xyz#gmailcom', 'gamesplayed': 2}]
Python can handle valid json's as dictionaries. Therefore you have to parse to json string to a python dictionary.
import json
dic = json.loads(json_str)
You can now access a value by using the specific key as an index value = dict[key].
for user in dic:
email = user
gamesplayed = dic[user][gamesplayed]
print("{} played {} game(s).".format(email, gamesplayed))
>>> abcd#gmailcom played 2 game(s).
xyz#gmailcom played 2 game(s).
I have an app where I am using flask, python, ajax, json, javascript, and leaflet. This app reads a csv file, puts it into json format, then returns it to an ajax call. My issue is that the geojson is not being returned. In the console, I am getting a 5000 NetworkError in the console log. The end result is to use the return geojson in a leaflet map layer. If I remove the jsonify, the return works fine, but it is a string of course, and this wont work for the layer.
As you can see, I have a simple alert("success") in the ajax success part. This is not being executed. Nor is the alert(data).
I do have jsonify in the from Flask import statement.
Thank you for the help
Ajax call
$.ajax({
type : "POST",
url : '/process',
data: {
chks: chks
}
})
.success(function(data){
alert("success"); // I am doing this just to get see if I get back here. I do not
alert(data);
python/flask
#app.route('/process', methods=['POST'])
def process():
data = request.form['chks']
rawData = csv.reader(open('static/csvfile.csv', 'r'), dialect='excel')
count = sum(1 for row in open('static/csvfile.csv))
template =\
''' \
{"type" : "Feature",
"geometry" : {
"type" : "Point",
"coordinates" : [%s, %s]},
"properties" : {"name" : "%s" }
}%s
'''
output = \
''' \
{"type" : "Feature Collection",
"features" : [
'''
iter = 0
separator = ","
lastrow = ""
for row in rawData:
iter += 1 // this is used to skip the first line of the csv file
if iter >=2:
id = row[0]
lat = row[1]
long = row[2]
if iter != count:
output += template % (row[2], row[1], row[0], separator)
else:
output += template % (row[2], row[1], row[0], lastrow)
output += \
''' \
]}
'''
return jsonify(output)
More Info - taking David Knipe's info into hand, If I remove the jsonify from my return statement, it returns what I expect, and I can output the return in an alert. It looks like this
{ "type" : "Feature Collection",
"features" : [
{"type" : "Feature",
"geometry" : {
"type" : "Point",
"coordinates" : [ -86.28, 32.36]},
"properties" : {"name" : "Montgomery"}
},
{ "type" : "Feature",
"geometry" : {
"type" : "Point",
"coordinates" : [ -105.42, 40.30]},
"properties" : {"name" : "Boulder"}
},
]}
If I take that data and hard code it into the ajax success, then pass it to the leaflet layer code like this - it will work, and my points will be displayed on my map
...
.success(function(data){
var pointsHC= { "type" : "Feature Collection",
"features" : [
{"type" : "Feature",
"geometry" : {
"type" : "Point",
"coordinates" : [ -86.28, 32.36]},
"properties" : {"name" : "Montgomery"}
},
{ "type" : "Feature",
"geometry" : {
"type" : "Point",
"coordinates" : [ -105.42, 40.30]},
"properties" : {"name" : "Boulder"}
},
]};
// leaflet part
var layer = L.geoJson(pointsHC, {
pointToLayer: function(feature, latlng){
return L.circleMarker( ...
If I do not hard code and pass the data via a variable, it does not work, and I get and invalid geoJson object. I have tried it with both the final semi-colon removed and not removed, and no love either way
...
.success(function(data){
// leaflet part
var layer = L.geoJson(data, {
pointToLayer: function(feature, latlng){
return L.circleMarker( ...
So it works if you don't try to parse the JSON, but if you do then it fails. Your JSON is invalid:
As loganbertram pointed out, you're missing a " on "Feature Collection".
You're missing a " on "properties".
output = template % ... should be output += template % ... - you're appending to output, not replacing it.
the features array will have a trailing comma (unless it is empty).
Although actually in your code features will always be empty anyway: you set iter = 0, never change its value, and then don't do the output = ... bit because iter < 2.
Are you sure you actually want to use jsonify? As I understand it, that turns any object into a JSON string. But output is already a JSON string - or should be, if you fix the various bugs loganbertram and I have spotted. In that case the client-side code will not fail trying to parse JSON. But if you jsonify something that's already JSON, you'll get something like this:
"{\"type\" : \"Feature\",
\"geometry\" : {
...
which the javascript will then convert back to the original JSON string, instead of a JSON object.
Actually, it would be better to rewrite the whole thing so it constructs an object instead of a string, and then calls jsonify on that object. But I don't know enough Python to give more details easily.
I am attempting to parse some JSON that I am receiving from a RESTful API, but I am having trouble accessing the data in Python because it appears that there is an empty property name.
A sample of the JSON returned:
{
"extractorData" : {
"url" : "RetreivedDataURL",
"resourceId" : "e38e1a7dd8f23dffbc77baf2d14ee500",
"data" : [ {
"group" : [ {
"CaseNumber" : [ {
"text" : "PO-1994-1350",
"href" : "http://www.referenceURL.net"
} ],
"DateFiled" : [ {
"text" : "03/11/1994"
} ],
"CaseDescription" : [ {
"text" : "Mary v. JONES"
} ],
"FoundParty" : [ {
"text" : "Lastname, MARY BETH (Plaintiff)"
} ]
}, {
"CaseNumber" : [ {
"text" : "NP-1998-2194",
"href" : "http://www.referenceURL.net"
}, {
"text" : "FD-1998-2310",
"href" : "http://www.referenceURL.net"
} ],
"DateFiled" : [ {
"text" : "08/13/1993"
}, {
"text" : "06/02/1998"
} ],
"CaseDescription" : [ {
"text" : "IN RE: NOTARY PUBLIC VS REDACTED"
}, {
"text" : "REDACTED"
} ],
"FoundParty" : [ {
"text" : "Lastname, MARY H (Plaintiff)"
}, {
"text" : "Lastname, MARY BETH (Defendant)"
} ]
} ]
} ]
And the Python code I am attempting to use
import requests
import json
FirstName = raw_input("Please Enter First name: ")
LastName = raw_input("Please Enter Last Name: ")
with requests.Session() as c:
url = ('https://www.requestURL.net/?name={}&lastname={}').format(LastName, FirstName)
page = c.get(url)
data = page.content
theJSON = json.loads(data)
def myprint(d):
stack = d.items()
while stack:
k, v = stack.pop()
if isinstance(v, dict):
stack.extend(v.iteritems())
else:
print("%s: %s" % (k, v))
print myprint(theJSON["extractorData"]["data"]["group"])
I get the error:
TypeError: list indices must be integers, not str
I am new to parsing Python and more than simple python in general so excuse my ignorance. But what leads me to believe that it is an empty property is that when I use a tool to view the JSON visually online, I get empty brackets, Like so:
Any help parsing this data into text would be of great help.
EDIT: Now I am able to reference a certain node with this code:
for d in group:
print group[0]['CaseNumber'][0]["text"]
But now how can I iterate over all the dictionaries listed in the group property to list all the nodes labeled "CaseNumber" because it should exist in every one of them. e.g
print group[0]['CaseNumber'][0]["text"]
then
for d in group:
print group[1]['CaseNumber'][0]["text"]
and so on and so forth. Perhaps incrementing some sort of integer until it reaches the end? I am not quite sure.
If you look at json carefully the data key that you are accessing is actually a list, but data['group'] is trying to access it as if it were a dictionary, which is raising the TypeError.
To minify your json it is something like this
{
"extractorData": {
"url": "string",
"resourceId": "string",
"data": [{
"group": []
}]
}
}
So if you want to access group, you should first retrieve data which is a list.
data = sample['extractorData']['data']
then you can iterate over data and get group within it
for d in data:
group = d['group']
I hope this clarifies things a bit for you.
this is my schema
class Url_per_date(EmbeddedDocument):
date = DateTimeField()
count = IntField(default=0)
class Daily_visit(Document):
domain = StringField()
count = IntField(default=0)
per_date = ListField(EmbeddedDocumentField('Url_per_date'))
i have collection like this:
{
"_id" : ObjectId("51c97e685aa3b3414c7e406a"),
"_types" : "Daily_visit",
"count" : 1,
"domain" : "yahoo.com",
"per_date" : {
"count" : 1,
"date" : ISODate("2013-05-20T00:00:00Z")
}
}
i need to update yahoo.com by date range. if per_date not have ISODate("2013-05-20T00:00:00Z") i need to create it, if i have it inc__count=1.
In MongoEngine, embedded fields are referenced by replacing dot-notation with double underscores:
Fields on embedded documents may also be referred to using field lookup syntax by using a double-underscore in place of the dot in object attribute access syntax:
http://mongoengine-odm.readthedocs.org/en/v0.6.8/guide/querying.html#filtering-queries
at first your result must be like this
{
"_id" : ObjectId("51c97e685aa3b3414c7e406a"),
"_types" : "Daily_visit",
"count" : 1,
"domain" : "yahoo.com",
"per_date" : [{
"count" : 1,
"date" : ISODate("2013-05-20T00:00:00Z")
}]
}
how create date? create new one for today and select database like this
today = datetime.today()
try:
yahoo_obj = Daily_visit.objects.get(domain="yahoo.com", per_date__date=date)
yahoo_obj.per_date[-1].count += 1
except:
yahoo_obj = Daily_visit.objects.get(domain="yahoo.com")
yahoo_obj.per_date = .... # just append new list