def cd():
person = {
'name' : 'bhavya',
'age' : 32,
'birth_date' : '18/10/1996'
}
print(person)
cd()
This is a dictionary I've saved as ey.py.
Now I want to import this whole data into another .py file with help of a module. So I can fetch whole data saved in ey.py.
Can anyone guide me?
If your goal is separating data from main python code, you have to make a new python code(ex. data.py)
data.py
def get_data():
data = { 'name' : 'bhavya', 'age' : 32, 'birth_date' : '18/10/1996' }
return data
main.py
from data import get_data
person = get_data()
print(person)
FYI
Related
I am trying to ingest the data from one measurement (vulnerability) to another measurement (test1) using influxDB python client. Since i want to ingest only server, ID, route from vulnerability measurement into test1 measurement, i choose three columns. Any help would be appreciated on how to ingest the data from one measurement to another.
code:
from influxdb import InfluxDBClient
from datetime import datetime
client = InfluxDBClient('hostname', 8089, 'user', 'pwd', 'database')
results = client.query("SELECT server, ID, route from vulnerability")
for row in results:
influxJson = [
{
"measurement":"test1",
"time" : datetime.utcnow().isoformat() + "Z",
"tags": {
'ResiliencyTier':'targetResiliencyTier',
'lob' : 'technologyDivision'
},
"fields": {
columns[0][0] : str(row[1][0]),
columns[1][0] : str(row[1][1]),
columns[2][0] : str(row[1][2])
}
}
]
client.write_points(influxJson)
Sample Data of vulnerability measurement:
{'time': '2022-02-10T17:51:52.638000Z', 'server': '123123123', 'id': '351335', 'route': '37875'}, {'time': '2022-02-10T17:51:52.638000Z', 'server': '234', 'qid': '351343', 'route': '0037875'}
ERROR:
File "Vul_SUmmary_UTEP_data_PROD.py", line 29, in startprocess
columns[0][0] : str(row[1][0]),
NameError: name 'columns' is not defined
Thanks
Regarding the error (I know it's a little late), it clearly states that 'columns' is not defined. That's because it never is. 'row' is defined because it's the variable of your for-loop. If you want to fill another table with the data from row, you need to define it beforehand. You could also just write:
'server' : str(row[1][0]),
'ID' : str(row[1][1]),
'route' : str(row[1][2])
After all, you're just trying to create another dictionary for fields.
I hope you figured that out yourself in the last 5 months. I just stumbled across your question regarding another problem.
I have a small mongo DB and I am trying to write code that will reverence the value of parameters (keys) imbedded in a document.
I have a mongo DB server on localhost and am able to successfully input the json format data structure into my DB. However, when I try to reference the values of this data within, I get the errors as shown.
import pymongo
import json
import time
from datetime import datetime
server = pymongo.MongoClient('localhost')
database = server['testData']
collection = database['testCollection']
test_entry1 = '{"Level1" : {"level2_1" : {"param1" : "1.6","param2" : "32.3","param3" : "11.0"}, "level2_2" : {"param1" : "2.6","param2" : "9.3","param3" : "112.0"}}}'
mongo_import = json.loads(test_entry1)
collection.insert_one(mongo_import)
The above works fine and when I query the database I get the following response (as expected):
{ "_id" : ObjectId("5d0081e931775cbc28cf7704"), "Level1" : { "level2_1" : { "param1" : "1.6", "param2" : "32.3", "param3" : "11.0" }, "level2_2" : { "param1" : "2.6", "param2" : "9.3", "param3" : "112.0" } } }
Now, I would like to reference the data in these parameters. I would like to get a response for all of what is in "level 2_1". The attempt #I made is below...and the error received is below that.
level_1_param_1 = collection.find("level1" : "level2_1")
error:
File "Desktop/Scripts/StackOverflowSnippit.py", line 29, in <module>
level_1_param_1 = collection.find('"level1" : "level2_1"')
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pymongo/collection.py", line 1456, in find
return Cursor(self, *args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pymongo/cursor.py", line 146, in __init__
validate_is_mapping("filter", spec)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pymongo/common.py", line 452, in validate_is_mapping
"collections.Mapping" % (option,))
TypeError: filter must be an instance of dict, bson.son.SON, or other type that inherits from collections.Mapping
I would also like to be able to get the value of a parameter that is one layer lower like "level1 --> level2_1 --> param2" for instance but so far have been unable to do so.
My hope is to be able to reference the data in this structure as needed.
Your filter should have brackets:
level_1_param_1 = collection.find({"level1": "level2_1"})
I have a
".txt"
file which has JSON data in it. I want to read this file in python and convert it into a dataframe.
The Data in this text file looks like this:
{
"_id" : "116b244599862fd2200",
"met_id" : [
612019,
621295,
725,
622169,
640014,
250,
350,
640015,
613689,
650423
],
"id" : "104",
"name" : "Energy",
"label" : "Peer Group",
"display_type" : "Risky Peer Group",
"processed_time" : ISODate("2019-04-18T11:17:05Z")
}
I tried reading it using the
pd.read_json
function but it always shows me an error. I am quite new to JSON, how can I use this Text file and load it in Python?
Please check this link
Also, "processed_time" : ISODate("2019-04-18T11:17:05Z") is not JSON format.
We can check that in https://jsonlint.com/
I added python code.
import pandas as pd
import json
with open('rr.txt') as f:
string = f.read()
# Remove 'ISODate(', ')' For correct, we can use regex
string = string.replace('ISODate(', '')
string = string.replace(')', '')
jsonData = json.loads(string)
print (pd.DataFrame(jsonData))
I am trying to build a list/dict that will be converted to JSON later on. I am trying to write the code that builds and populates the multiple levels of the JSON format I ultimately need. I am having an issue wrapping my head around this. Thank you for the help.
What I ultimately need -> Populate this list/dict:
dataset_permission_json = []
with this format:
{
"projects":[
{
"project":"test-project-1",
"datasets":[
{
"dataset":"testing1",
"permissions":[
{
"role":"READER",
"google_group":"testing1#test.com"
}
]
},
{
"dataset":"testing2",
"permissions":[
{
"role":"OWNER",
"google_group":"testing2#test.com"
}
]
},
{
"dataset":"testing3",
"permissions":[
{
"role":"READER",
"google_group":"testing3#test.com"
}
]
},
{
"dataset":"testing4",
"permissions":[
{
"role":"WRITER",
"google_group":"testing4#test.com"
}
]
}
]
}
]
}
I have multiple for loops that successfully print out the information I am pulling from an external API but I to be able to enter that data into the list/dict. The dynamic values I am trying to input are:
'project' i.e. test-project-1
'dataset' i.e. testing1
'role' i.e. READER
'google_group' i.e. testing1#test.com
I have tried things like:
dataset_permission_json.update({'project': project})
but cannot figure out how not to overwrite the data during the multiple for loops.
for project in projects:
print(project) ## Need to add this variable to 'projects'
for bq_group in bq_groups:
delegated_credentials = credentials.create_delegated(bq_group)
http_auth = delegated_credentials.authorize(Http())
list_datasets_in_project = bigquery_service.datasets().list(projectId=project).execute()
datasets = list_datasets_in_project.get('datasets',[])
print(dataset['datasetReference']['datasetId']) ##Add the dataset to 'datasets' under the project
for dataset in datasets:
get_dataset_permissions_result = bigquery_service.datasets().get(projectId=project, datasetId=dataset['datasetReference']['datasetId']).execute()
dataset_permissions = get_dataset_permissions_result.get('access',[])
### ADD THE NEXT LEVEL 'permissions' level here?
for dataset_permission in dataset_permissions:
if 'groupByEmail' in dataset_permission:
if bq_group in dataset_permission['groupByEmail']:
print(dataset['datasetReference']['datasetId'] && dataset_permission['groupByEmail']) ##Add to each dataset
I appreciate the help.
EDIT: Updated Progress
Ok I have created the nested structure that I was looking for using StackOverflow
Things are great except for the last part. I am trying to append the role & group to each 'permission' nest, but after everything runs the data is only appended to the last 'permission' nest in the JSON structure. It seems like it is overwriting itself during the for loop. Thoughts?
Updated for loop:
for project in projects:
for bq_group in bq_groups:
delegated_credentials = credentials.create_delegated(bq_group)
http_auth = delegated_credentials.authorize(Http())
list_datasets_in_project = bigquery_service.datasets().list(projectId=project).execute()
datasets = list_datasets_in_project.get('datasets',[])
for dataset in datasets:
get_dataset_permissions_result = bigquery_service.datasets().get(projectId=project, datasetId=dataset['datasetReference']['datasetId']).execute()
dataset_permissions = get_dataset_permissions_result.get('access',[])
for dataset_permission in dataset_permissions:
if 'groupByEmail' in dataset_permission:
if bq_group in dataset_permission['groupByEmail']:
dataset_permission_json['projects'][project]['datasets'][dataset['datasetReference']['datasetId']]['permissions']
permission = {'group': dataset_permission['groupByEmail'],'role': dataset_permission['role']}
dataset_permission_json['permissions'] = permission
UPDATE: Solved.
dataset_permission_json['projects'][project]['datasets'][dataset['datasetReference']['datasetId']]['permissions']
permission = {'group': dataset_permission['groupByEmail'],'role': dataset_permission['role']}
dataset_permission_json['projects'][project]['datasets'][dataset['datasetReference']['datasetId']]['permissions'] = permission
a bit new to python and json.
i have this json file:
{ "hosts": {
"example1.lab.com" : ["mysql", "apache"],
"example2.lab.com" : ["sqlite", "nmap"],
"example3.lab.com" : ["vim", "bind9"]
}
}
what i want to do is use the hostname variable and extract the values of each hostname.
its a bit hard to explain but im using saltstack, which already iterates over hosts and i want it to be able to extract each host's values from the json file using the hostname variable.
hope im understood.
thanks
o.
You could do something along these lines:
import json
j='''{ "hosts": {
"example1.lab.com" : ["mysql", "apache"],
"example2.lab.com" : ["sqlite", "nmap"],
"example3.lab.com" : ["vim", "bind9"]
}
}'''
specific_key='example2'
found=False
for key,di in json.loads(j).iteritems(): # items on Py 3k
for k,v in di.items():
if k.startswith(specific_key):
found=True
print k,v
break
if found:
break
Or, you could do:
def pairs(args):
for arg in args:
if arg[0].startswith(specific_key):
k,v=arg
print k,v
json.loads(j,object_pairs_hook=pairs)
Either case, prints:
example2.lab.com [u'sqlite', u'nmap']
If you have the JSON in a string then just use Python's json.loads() function to load JSON parse the JSON and load its contents into your namespace by binding it to some local name
Example:
#!/bin/env python
import json
some_json = '''{ "hosts": {
"example1.lab.com" : ["mysql", "apache"],
"example2.lab.com" : ["sqlite", "nmap"],
"example3.lab.com" : ["vim", "bind9"]
}
}'''
some_stuff = json.loads(some_json)
print some_stuff['hosts'].keys()
---> [u'example1.lab.com', u'example3.lab.com', u'example2.lab.com']
As shown you then access the contents of some_stuff just as you would any other Python dictionary ... all the top level variable declaration/assignments which were serialized (encoded) in the JSON will be keys in that dictionary.
If the JSON contents are in a file you can open it like any other file in Python and pass the file object's name to the json.load() function:
#!/bin/python
import json
with open("some_file.json") as f:
some_stuff = json.load(f)
print ' '.join(some_stuff.keys())
If the above json file is stored as 'samplefile.json', you can write following in python:
import json
f = open('samplefile.json')
data = json.load(f)
value1 = data['hosts']['example1.lab.com']
value2 = data['hosts']['example2.lab.com']
value3 = data['hosts']['example3.lab.com']