In my mongodb I have around 20K records,
now From the same mongodb I am importing only 500 records in python and now I need to delete those 500 records which I have imported , is it possible to delete those 500 records using python?
my code for importing 500 records
from pymongo import MongoClient
con = MongoClient(local host, 27017)
db = con.rssfeeds_db
data = pd.DataFrame(list(db.restdata.find().limit(500)))
You would need to use the _id for each record so that you could tell pymongo which records to delete. For example:
from pymongo import MongoClient
con = MongoClient(localhost, 27017)
db = con.rssfeeds_db
// create a list of the records. each one will be a dict.
records = [x for x in db.restdata.find().limit(500)]
// create a list of the 500 ids so we only need to call the db once
record_ids = [record['_id'] for record in records]
//create your dataframe
data = pd.DataFrame(records)
// delete all 500 records at once
db.restdata.delete_many(
{'_id':
{'$in': record_ids}
},
)
Related
I am trying to read data from snowflake database using FASTAPI. I was able to create the connection which is able to pull data from snowflake.
The issue which I am facing right now is that I am only getting 1 record (instead of 10 records).
I suspect I am not using correct keyword while returning the data. appreciate any help.
Here is my code :-
from fastapi import FastAPI
import snowflake.connector as sf
import configparser
username='username_value'
password='password_value'
account= 'account_value'
warehouse= 'test_wh'
database= 'test_db'
ctx=sf.connect(user=username,password=password,account=account,warehouse=warehouse,database=database)
app = FastAPI()
#app.get('/test API')
async def fetchdata():
cursor = ctx.cursor()
cursor.execute("USE WAREHOUSE test_WH ")
cursor.execute("USE DATABASE test_db")
cursor.execute("USE SCHEMA test_schema")
sql = cursor.execute ("SELECT DISTINCT ID,NAME,AGE,CITY FROM TEST_TABLE WHERE AGE > 60")
for data in sql:
return data
You use return in your inner for-loop. This will return the first row encountered.
If you want to return all rows as a list, you can probably do (I'm not familiar with the snowflake connector):
return list(data)
instead of the for-loop, or sql.fetchall().
I'm totally new to AWS ElasticSearch Service.
First I tried to establish connection using python, It was successful.
Then I tried to get a single record from a specific index and bulk records from an index using query.
Like:
df = es.get(index="demo_data", doc_type="_doc", id="2") -- For getting single record
result_dict = es.search(index="demo_data", body={"query": {"match_all": {}}}) -- For getting Bulk Records
Now, I've been trying to fetch only newly inserted records i.e.(It can be a single record or a bulk of records)from that same index based on date using query, but I couldn't get an exact reference, I tried several ways like filter query with sort etc... Which either gets me the last record of the index or all the records in descending order.
I couldn't able to get the exact new records based on the date from my previous load.
from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth
from pandasticsearch import Select
from elasticsearch_dsl import Search
spark = SparkSession.builder.appName('abc').getOrCreate()
host = 'ENDPOINT' # ES hostname
region = 'REGION' # our region ID
service = 'es'
awsauth = AWS4Auth('Secret_Key', 'Secret_ID', region, service)
es = Elasticsearch(
hosts = [{'host': host, 'port': 443}],
http_auth = awsauth,
use_ssl = True,
verify_certs = True,
connection_class = RequestsHttpConnection
)
s = Search(index='demo_data').using(es).query("match", date= max(pandas_df.date))
pandas_df = Select.from_dict(s).to_pandas()
display(pandas_df)
Help is much appreciated. Thanks in Advance...
import pymongo
client=pymongo.MongoClient("mongodb+srv://Debaditya:<password>#pycluster.d6kvk.gcp.mongodb.net/<db>?retryWrites=true&w=majority")
db = client.test
db.command("createUser", "sample", roles=["read"])
Help me pls the code is given above
Here is the sample code for inserting a document in a collection.
from pymongo import MongoClient
client = MongoClient("mongodb+srv://Debaditya:<password>#pycluster.d6kvk.gcp.mongodb.net/<db>?retryWrites=true&w=majority")
db = client.test
db.students.insert_one({"name": "Test User", "rollNo": 37})
In above example students is the collection name where you have inserted the above row.
If you want to list the rows in the collection.
list(db.students.find())
Just got through the documention of pymongo here
I'm working with mongodb and python. So I have a huge JSON that is within the database in mongodb, and I wanted to make a list where he gave a "print" on the screen of the things I looked for.
db.getCollection('request').aggregate(
{$match : {"kind" :"qpxExpress#tripsSearch"}},
{$unwind:"$trips.tripOption"},
{$project: {"trips.tripOption.saleTotal":1}}
)
I use it to look inside the JSON, but I wanted one of the print on the screen with a code in python.
Here is a JSON exempo that I get the information:
http://codebeautify.org/jsonviewer/cb959272
I've tried to do something but doesn't work:
from pymongo import MongoClient
client = MongoClient()
cursor = db.getCollection('request').aggregate(
{$match : {"kind" :"qpxExpress#tripsSearch"}},
{$unwind:"$trips.tripOption"},
{$project:{"trips.tripOption.saleTotal":1}}
)
for saleTotal in cursor:
print(saleTotal)
Something like this should work (assuming your DB name is test):
from pymongo import MongoClient
client = MongoClient()
db = client.test
cursor = db.request.aggregate(
[
{$match : {"kind" :"qpxExpress#tripsSearch"}},
{$unwind:"$trips.tripOption"},
{$project:{"trips.tripOption.saleTotal":1}}
]
)
for saleTotal in cursor:
print(saleTotal)
I ve got a mongo database and I want to retrieve all documents with a cursor operation like in mongodb java api. I want to retrieve all username of database based on this cursor iteration. My code is like the above:
import pymongo
from pymongo import MongoClient
client = MongoClient('...', 27017)
db = client.test_database
db = client['...']
collection = db.test_collection
collection = db["..."]
result = collection.find()
obj = next(result, None)
if obj:
username= obj['username']
print username
I want for the collection to print all the usernames.
Just loop over the results and print the username. There's no reason to play with next()
.
for obj in collection.find():
print obj['username']