I ve got a mongo database and I want to retrieve all documents with a cursor operation like in mongodb java api. I want to retrieve all username of database based on this cursor iteration. My code is like the above:
import pymongo
from pymongo import MongoClient
client = MongoClient('...', 27017)
db = client.test_database
db = client['...']
collection = db.test_collection
collection = db["..."]
result = collection.find()
obj = next(result, None)
if obj:
username= obj['username']
print username
I want for the collection to print all the usernames.
Just loop over the results and print the username. There's no reason to play with next()
.
for obj in collection.find():
print obj['username']
Related
I have created an account with Atlas and now I am trying to retrieve some information from the database. My native language is R, so I inserted the "iris" data set into the collection called "table".
To show that there is data in my new database "test", I use the mongolite command (this is the R pymongo):
#m is my client connection
m$count()
[1]5490
The problem is connecting with python. I am currently on Jupyter Notebook. Here is the code.
import pymongo as pm
import pprint
import requests
url= "mongodb://jordan:*************#jordandb-shard-00-00-ykcna.mongodb.net:27017,jordandb-shard-00-01-ykcna.mongodb.net:27017,jordandb-shard-00-02-ykcna.mongodb.net:27017/test?ssl=true&replicaSet=JordanDB-shard-0&authSource=admin&retryWrites=true"
client = pm.MongoClient(url)
print(client)
[out]Database(MongoClient(host=['jordandb-shard-00-02-ykcna.mongodb.net:27017', 'jordandb-shard-00-01-ykcna.mongodb.net:27017', 'jordandb-shard-00-00-ykcna.mongodb.net:27017'], document_class=dict, tz_aware=False, connect=True, ssl=True, replicaset='JordanDB-shard-0', authsource='admin', retrywrites=True), 'test')
#I am assuming this means I am connected
When I call any methods on the database I get the error.
db = client.test #test is name of collection
db.iris.find_one({})
ServerSelectionTimeoutError Traceback (most recent call last)
<ipython-input-15-ab74ef5f0195> in <module>()
----> 1 db.iris.find_one({})
/opt/conda/lib/python3.6/site-packages/pymongo/collection.py in find_one(self, filter, *args, **kwargs)
1260
1261 cursor = self.find(filter, *args, **kwargs)
-> 1262 for result in cursor.limit(-1):
1263 return result
1264 return None
I would like to be able to connect and start exploring the data in my "test" data set by using methods like, list_database_names(), list_collection_names() etc.. Thank you kindly
Try this: I am using a python kernel in a jupyter notebook though but the logic remains the same
from pymongo import MongoClient
# replace "USER" and "PASSWORD" with your user and password respectively.
client = MongoClient("mongodb+srv://USER:PASSWORD#cluster0-hhhbr.mongodb.net/test?retryWrites=true&w=majority")
# replace "local" with any database in your cluster
db = client.local
# replace "emps" with your collection's name
collection = db.emps
# you can use the collection e.g, but this will return a cursor
collection.find()
# To get the actual data do this or create a context
# Don't forget to delete the 3 previous lines of codes if you are going to
# create a context
with client as cl:
db = cl.local
collection = db.emps
for i in collection.find():
print(i)
To connect to mongodb using python from any editor
import pymongo
myclient = pymongo.MongoClient("mongodb://localhost:27017/")
mydb = myclient["pyramids"] #pyramids is the database
mycol = mydb["invoices"] #invoice is the collection
I´m pretty new learning Python and all the containing stuff.
I tried to make my first little steps, installing a MongoDB (working) and connecting to it.
from pymongo import MongoClient
from pprint import pprint
from random import randint
client = MongoClient('localhost', 27017)
db = client.test
collection = db.users
user = {"id": 1, "username": "Test"}
user_id = collection.insert_one(user).inserted_id
print(user_id)
This is the complete code.
pymongo Version: 3.7.2 checked with:
pip freeze | grep pymongo
Output: pymongo==3.7.2
Python Version: 3.7.1
If I try to execute my tiny script, the following error occurs:
'Collection' object is not callable.
If you meant to call the 'insert_one' method on a 'Collection'
object it is
failing because no such method exists.
Where is my Fault?
A little research showed that in pymongo v2 the ".insert_one" is ".insert", but the 3.7.2 version is installed so I should (and must) use the ".insert.one", not the ".insert"
insert_one according with pymongo documentation exists, for server version >= 3.2...
the use is:
user = {'x': 1}
result = db.test.insert_one(user)
result.inserted_id
more complete explanation about insert_one:
>>> db.test.count_documents({'x': 1})
0
>>> result = db.test.insert_one({'x': 1})
>>> result.inserted_id
ObjectId('54f112defba522406c9cc208')
>>> db.test.find_one({'x': 1})
{u'x': 1, u'_id': ObjectId('54f112defba522406c9cc208')}
the following below, I executed and works fine:
# importing client mongo to make the connection
from pymongo import MongoClient
print("--- Exemplo pymongo Connection ---")
# Connection to MongoDB
client = MongoClient('localhost', 27017)
# Selection the Database
db = client.python
# Select the collection
collection = db.users
# Set up a document
user = {"id": 1, "username": "Test"}
# insert one document into selected document
result = collection.insert_one(user)
# Selection just one document from collection
#result = collection.find_one()
# removing the document inserted
collection.delete_one(user)
# print the inserted_id
print("inserted_id: ", result.inserted_id)
Pymongo Documentation
In my mongodb I have around 20K records,
now From the same mongodb I am importing only 500 records in python and now I need to delete those 500 records which I have imported , is it possible to delete those 500 records using python?
my code for importing 500 records
from pymongo import MongoClient
con = MongoClient(local host, 27017)
db = con.rssfeeds_db
data = pd.DataFrame(list(db.restdata.find().limit(500)))
You would need to use the _id for each record so that you could tell pymongo which records to delete. For example:
from pymongo import MongoClient
con = MongoClient(localhost, 27017)
db = con.rssfeeds_db
// create a list of the records. each one will be a dict.
records = [x for x in db.restdata.find().limit(500)]
// create a list of the 500 ids so we only need to call the db once
record_ids = [record['_id'] for record in records]
//create your dataframe
data = pd.DataFrame(records)
// delete all 500 records at once
db.restdata.delete_many(
{'_id':
{'$in': record_ids}
},
)
As the title indicates, how does one use python to elegantly access an API and parse and save the JSON contents onto a relational database (MYSQL) for later access?
Here, I saved the data onto a pandas object. But how do I create a mysql database, save the json contents onto it, and access the contents for later use?
# Libraries
import json, requests
import pandas as pd
from pandas.io.json import json_normalize
# Set URL
url = 'https://api-v2.themuse.com/jobs'
# For loop to
for i in range(100):
data = json.loads(requests.get(
url=url,
params={'page': i}
).text)['results']
data_norm = pd.read_json(json.dumps(data))
You create your Mysql table on your server using something like Mysql Workbench CE. then in python you do this. I wasnt sure if you want to use data in for loop or data_norm so for ease of use, here some functions. insertDb() can be put in your for loop, since data will be overwriten by itself in every iteration.
import MySQLdb
def dbconnect():
try:
db = MySQLdb.connect(
host='localhost',
user='root',
passwd='password',
db='nameofdb'
)
except Exception as e:
sys.exit("Can't connect to database")
return db
def insertDb():
try:
db = dbconnect()
cursor = db.cursor()
cursor.execute("""
INSERT INTO nameoftable(nameofcolumn) \
VALUES (%s) """, (data))
cursor.close()
except Exception as e:
print e
If this is merely for storage for processing later, kind of like a cache, a varchar field is enough. If however you need to retrieve some structured jdata, JSON field is what you need.
I'm working with mongodb and python. So I have a huge JSON that is within the database in mongodb, and I wanted to make a list where he gave a "print" on the screen of the things I looked for.
db.getCollection('request').aggregate(
{$match : {"kind" :"qpxExpress#tripsSearch"}},
{$unwind:"$trips.tripOption"},
{$project: {"trips.tripOption.saleTotal":1}}
)
I use it to look inside the JSON, but I wanted one of the print on the screen with a code in python.
Here is a JSON exempo that I get the information:
http://codebeautify.org/jsonviewer/cb959272
I've tried to do something but doesn't work:
from pymongo import MongoClient
client = MongoClient()
cursor = db.getCollection('request').aggregate(
{$match : {"kind" :"qpxExpress#tripsSearch"}},
{$unwind:"$trips.tripOption"},
{$project:{"trips.tripOption.saleTotal":1}}
)
for saleTotal in cursor:
print(saleTotal)
Something like this should work (assuming your DB name is test):
from pymongo import MongoClient
client = MongoClient()
db = client.test
cursor = db.request.aggregate(
[
{$match : {"kind" :"qpxExpress#tripsSearch"}},
{$unwind:"$trips.tripOption"},
{$project:{"trips.tripOption.saleTotal":1}}
]
)
for saleTotal in cursor:
print(saleTotal)