I have a small mongo DB and I am trying to write code that will reverence the value of parameters (keys) imbedded in a document.
I have a mongo DB server on localhost and am able to successfully input the json format data structure into my DB. However, when I try to reference the values of this data within, I get the errors as shown.
import pymongo
import json
import time
from datetime import datetime
server = pymongo.MongoClient('localhost')
database = server['testData']
collection = database['testCollection']
test_entry1 = '{"Level1" : {"level2_1" : {"param1" : "1.6","param2" : "32.3","param3" : "11.0"}, "level2_2" : {"param1" : "2.6","param2" : "9.3","param3" : "112.0"}}}'
mongo_import = json.loads(test_entry1)
collection.insert_one(mongo_import)
The above works fine and when I query the database I get the following response (as expected):
{ "_id" : ObjectId("5d0081e931775cbc28cf7704"), "Level1" : { "level2_1" : { "param1" : "1.6", "param2" : "32.3", "param3" : "11.0" }, "level2_2" : { "param1" : "2.6", "param2" : "9.3", "param3" : "112.0" } } }
Now, I would like to reference the data in these parameters. I would like to get a response for all of what is in "level 2_1". The attempt #I made is below...and the error received is below that.
level_1_param_1 = collection.find("level1" : "level2_1")
error:
File "Desktop/Scripts/StackOverflowSnippit.py", line 29, in <module>
level_1_param_1 = collection.find('"level1" : "level2_1"')
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pymongo/collection.py", line 1456, in find
return Cursor(self, *args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pymongo/cursor.py", line 146, in __init__
validate_is_mapping("filter", spec)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pymongo/common.py", line 452, in validate_is_mapping
"collections.Mapping" % (option,))
TypeError: filter must be an instance of dict, bson.son.SON, or other type that inherits from collections.Mapping
I would also like to be able to get the value of a parameter that is one layer lower like "level1 --> level2_1 --> param2" for instance but so far have been unable to do so.
My hope is to be able to reference the data in this structure as needed.
Your filter should have brackets:
level_1_param_1 = collection.find({"level1": "level2_1"})
Related
Current output is of format, dictionary of lists
{
"majestic-service-1.324.02070909": [
"/home/robotics/arm-services/FeaturesDir.yaml",
"/home/robotics/arm-service/majestic-service.tar.gz"
],
}
and I'm looking to change that output format to something like below.(dictionary of dictionary)
{
"majestic-service-1.324.02070909": {
"yaml_file": "/home/robotics/arm-services/FeaturesDir.yaml",
"tar_file": "/home/robotics/arm-services/majestic-service-1.324.02070909.tar.gz",
"kind": "FeaturesDir"
}
}
Corresponding code snippet that I've tried,
output_dict = {}
for file in application_files:
match = re.match(regex_pattern, os.path.basename(file))
if match:
if os.path.exists(os.path.join(os.path.dirname(file), "FeaturesDir.yaml")):
output_dict[file_without_extension(match.string)] = {os.path.join(os.path.dirname(file), "FeaturesDir.yaml")}
output_dict[file_without_extension(match.string)].append(file)
output_dict["Kind"] = "FeaturesDir"
elif os.path.exists(os.path.join(os.path.dirname(file), "output_path/Deviations.yaml")):
output_dict[file_without_extension(match.string)] = {os.path.join(os.path.dirname(file), "output_path/Deviations")}
output_dict[file_without_extension(match.string)].append(file)
output_dict["Kind"] = "Deviations"
# where the function file_without_extension, will return - majestic-service-1.324.02070909 from majestic-service-1.324.02070909.tar.gz
It reports the following error
AttributeError: 'set' object has no attribute 'append'
What am I doing wrong ?
The error occurs because in this line output_dict[file_without_extension(match.string)] = {os.path.join(os.path.dirname(file), "output_path/Deviations")} you're using {} brackets which for python are intended as the data type called set. Sets don't have append attribute, they have add attribute instead but it's a completely different data type. That's why it shows you that kind of error!
def cd():
person = {
'name' : 'bhavya',
'age' : 32,
'birth_date' : '18/10/1996'
}
print(person)
cd()
This is a dictionary I've saved as ey.py.
Now I want to import this whole data into another .py file with help of a module. So I can fetch whole data saved in ey.py.
Can anyone guide me?
If your goal is separating data from main python code, you have to make a new python code(ex. data.py)
data.py
def get_data():
data = { 'name' : 'bhavya', 'age' : 32, 'birth_date' : '18/10/1996' }
return data
main.py
from data import get_data
person = get_data()
print(person)
FYI
I am trying to save data in dynamodb using update_item for the first time. In another area of my project I have used put_item() successfully. For this new area of code I am saving only items that change, leaving items in the db that are unchanged. Thus, I need to use update_item(). However, I can't seem to figure out why my syntax is not correct for the API call. I am using this directly from the Amazon UI.
Here is my python code:
from __future__ import print_function
import json
import boto3
print('Loading function')
def saveScreenData(event, context):
dynamodb = boto3.client('dynamodb', region_name='us-east-1', endpoint_url="https://dynamodb.us-east-1.amazonaws.com")
print('The event: {}'.format(event))
key = {}
key['UID'] = event['uid']
key['screenId'] = event['screenid']
print('Key: {}'.format(key))
for item, val in event.items():
if item != 'uid' and item != 'screenid':
print("Saving!")
response = dynamodb.update_item({
"TableName" : "ScreenData",
"Key" : key,
"UpdateExpression" : "SET #attrName = :attrValue",
"ExpressionAttributeNames" : {
"#attrName" : item
},
"ExpressionAttributeValues" : {
":attrValue" : val
}
})
print('Response: {}'.format(response))
return response
Here is the output:
START RequestId: 2da9412a-b03d-11e7-9dc8-8fcb305833f6 Version: $LATEST
The event: {'article': '<p>↵ First!↵</p>', 'screenid': '13', 'uid': '0', 'section1': '<h1>↵ Second↵</h1>'}
Key: {'UID': '0', 'screenId': '13'}
Saving!
update_item() only accepts keyword arguments.: TypeError
Traceback (most recent call last):
File "/var/task/saveScreenData.py", line 30, in saveScreenData
":attrValue" : val
File "/var/runtime/botocore/client.py", line 310, in _api_call
"%s() only accepts keyword arguments." % py_operation_name)
TypeError: update_item() only accepts keyword arguments.
END RequestId: 2da9412a-b03d-11e7-9dc8-8fcb305833f6
I have researched the update_item docs (https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_UpdateItem.html) and have modeled my query after this SO q&a by mkobit (https://stackoverflow.com/users/627727/mkobit): https://stackoverflow.com/a/30604746/8027640
I have played with variations on the syntax, including adding the dictionary {"S" : "maybe this works"} instead of my variable val, and have also tried changing the variable to some static content to see if it works, but no luck.
Clearly this is a syntax issue, but I have been unable to track it down. Suggestions?
I think the example your are using is based on boto2 which has quite different interface comparing to boto3.
Instead, look into the boto3 documentation, you should use keyword arguments as the error states (and you are using the dictionary).
Your request should look approximately like this:
response = dynamodb.update_item(
TableName="ScreenData",
Key=key,
UpdateExpression="SET #attrName = :attrValue",
ExpressionAttributeNames={
"#attrName" : item
},
ExpressionAttributeValues={
":attrValue" : val
}
)
I have a .json.gz file that I wish to load into elastic search.
My first attempt involved using the json module to convert the JSON to a list of dicts.
import gzip
import json
from pprint import pprint
from elasticsearch import Elasticsearch
nodes_f = gzip.open("nodes.json.gz")
nodes = json.load(nodes_f)
Dict example:
pprint(nodes[0])
{u'index': 1,
u'point': [508163.122, 195316.627],
u'tax': u'fehwj39099'}
Using Elasticsearch:
es = Elasticsearch()
data = es.bulk(index="index",body=nodes)
However, this returns:
elasticsearch.exceptions.RequestError: TransportError(400, u'illegal_argument_exception', u'Malformed action/metadata line [1], expected START_OBJECT or END_OBJECT but found [VALUE_STRING]')
Beyond this, I wish to be able to find the tax for given point query, in case this has an impact on how I should be indexing the data with elasticsearch.
Alfe pointed me in the right direction, but I couldn't get his code to work.
I found two solutions:
Line by line with a for loop:
es = elasticsearch.Elasticsearch()
for node in nodes:
_id = node['index']
es.index(index='nodes',doc_type='external',id=_id,body=node)
In bulk, using helper:
actions = [
{
"_index" : "nodes_bulk",
"_type" : "external",
"_id" : str(node['index']),
"_source" : node
}
for node in nodes
]
helpers.bulk(es,actions)
Bulk was around 22 times faster for a list of 343724 dicts.
Here is my working code using bulk api:
Define a list of dicts:
from elasticsearch import Elasticsearch, helpers
es = Elasticsearch([{'host':'localhost', 'port': 9200}])
doc = [{'_id': 1,'price': 10, 'productID' : 'XHDK-A-1293-#fJ3'},
{'_id':2, "price" : 20, "productID" : "KDKE-B-9947-#kL5"},
{'_id':3, "price" : 30, "productID" : "JODL-X-1937-#pV7"},
{'_id':4, "price" : 30, "productID" : "QQPX-R-3956-#aD8"}]
helpers.bulk(es, doc, index='products',doc_type='_doc', request_timeout=200)
The ES bulk library showed several problems, including performance trouble, not being able to set specific _ids etc. But since the bulk API of ES is not very complicated, we did it ourselves:
import requests
headers = { 'Content-type': 'application/json',
'Accept': 'text/plain'}
jsons = []
for d in docs:
_id = d.pop('_id') # take _id out of dict
jsons.append('{"index":{"_id":"%s"}}\n%s\n' % (_id, json.dumps(d)))
data = ''.join(jsons)
response = requests.post(url, data=data, headers=headers)
We needed to set a specific _id but I guess you can skip this part in case you want a random _id set by ES automatically.
Hope that helps.
I am running into an issue that I can't seem to get past. Any insight would be great.
The script is supposed to get memory allocation information from a database, and return that information as a formatted JSON object. The script works fine when I give it a static JSON object will stack_ids (the information I would be passing) but it won't work when I try to pass the information via POST.
Although the current state of my code uses request.json("") to access the passed data, I have also tried request.POST.get("").
My HTML includes this post request, using D3's xhr post:
var stacks = [230323, 201100, 201108, 229390, 201106, 201114];
var stack_ids = {'stack_ids': stacks};
var my_request = d3.xhr('/pie_graph');
my_request.header("Content-Type", "application/json")
my_request.post(stack_ids, function(stuff){
stuff = JSON.parse(stuff);
var data1 = stuff['allocations'];
var data2 = stuff['allocated bytes'];
var data3 = stuff['frees'];
var data4 = stuff['freed bytes'];
...
...
}, "json");
while my server script has this route:
#views.webapp.route('/pie_graph', method='POST')
def server_pie_graph_json():
db = views.db
config = views.config
ret = {
'allocations' : [],
'allocated bytes' : [],
'frees' : [],
'freed bytes' : [],
'leaks' : [],
'leaked bytes' : []
}
stack_ids = request.json['stack_ids']
#for each unique stack trace
for pos, stack_id in stack_ids:
stack = db.stacks[stack_id]
nallocs = format(stack.nallocs(db, config))
nalloc_bytes = format(stack.nalloc_bytes(db, config))
nfrees = format(stack.nfrees(db, config))
nfree_bytes = format(stack.nfree_bytes(db, config))
nleaks = format(stack.nallocs(db, config) - stack.nfrees(db, config))
nleaked_bytes = format(stack.nalloc_bytes(db, config) - stack.nfree_bytes(db, config))
# create a dictionary representing the stack
ret['allocations'].append({'label' : stack_id, 'value' : nallocs})
ret['allocated bytes'].append({'label' : stack_id, 'value' : nalloc_bytes})
ret['frees'].append({'label' : stack_id, 'value' : nfrees})
ret['freed bytes'].append({'label' : stack_id, 'value' : nfree_bytes})
ret['leaks'].append({'label' : stack_id, 'value' : nleaks})
ret['leaked bytes'].append({'label' : stack_id, 'value' : nfree_bytes})
# return dictionary of allocation information
return ret
Most of that can be ignored, the script works when I give it a static JSON object full of data.
The request currently returns a 500 Internal Server Error: JSONDecodeError('Expecting value: line 1 column 2 (char 1)',).
Can anyone explain to me what I am doing wrong?
Also, if you need me to explain anything further, or include any other information, I am happy to do that. My brain is slightly fried after working on this for so long, so I may have missed something.
Here is what I do with POST and it works:
from bottle import *
#post('/')
def do_something():
comment = request.forms.get('comment')
sourcecode = request.forms.get('sourceCode')
Source
function saveTheSourceCodeToServer(comment) {
var path = saveLocation();
var params = { 'sourceCode' : getTheSourceCode() , 'comment' : comment};
post_to_url(path, params, 'post');
}
Source with credits to JavaScript post request like a form submit