I'm attempting to clone a collection on a remote server to my localhost with indexs using Python (as I'll have to be automating multiple tasks with this function using Python later on). From what I've read, using db.command would be the best way to approach this, but I can't seem to get it working, or even find what I'm really doing wrong.. Here's my code and error, any help is appreciated.
client = MongoClient()
db = clientProd.tools
colc = db.newDump
db.command({"cloneCollection": "databaseName.dump", "from": "example.com:6001"})
Error:
Traceback (most recent call last):
File "/Users/dustin/Git/redbull_dev/test.py", line 14, in <module>
dbProd.command({"cloneCollection": "databaseName.dump", "from": "example.com:6001"})
File "/Library/Python/2.7/site-packages/pymongo-2.6.3-py2.7-macosx-10.8-intel.egg/pymongo/database.py", line 396, in command
File "/Library/Python/2.7/site-packages/pymongo-2.6.3-py2.7-macosx-10.8-intel.egg/pymongo/helpers.py", line 147, in _check_command_response
pymongo.errors.OperationFailure: command {'from': 'example.com:6001', 'cloneCollection': 'databaseName.dump'} failed: no such cmd: from
[Finished in 5.2s with exit code 1]
Apparently you looked at the doc from mongo, you should look at the doc from pymongo http://api.mongodb.org/python/current/api/pymongo/database.html
From that, the correct command should be:
db.command("cloneCollection", collection="databaseName.dump", from="example.com:6001")
Edit
Since from is a python keyword, we probably need to create a dict and unpack it:
db.command("cloneCollection", **{'collection': "databaseName.dump", 'from': "example.com:6001"})
I found that the selected solution didn't quite work (maybe version issue), this is my solution:
db_dest.command("cloneCollection", **{"cloneCollection": db_str + ".Sentence",
'collection': db_str + ".Sentence",
'from': mongodb_ip + ":27017"})
Related
Python Version = 3.9,
Foreman Version = 2.5.4
I'm using the Python Foreman (python-foreman) module to connect to Foreman to do a bunch of automation tasks, however as soon as i'm connecting to Foreman as below:
from foreman.client import Foreman
f = Foreman('http://foreman-url.com/api/hosts', ('admin', 'password'), api_version = 2)
I'm getting a bunch of conflicting statements as below and eventually erroring out.
There is a conflict trying to redefine a method for a foreign resource (foreman_tasks_index):
resource:
apipie_resource: foreman_tasks
new_api: <resource:, name:foreman_tasks_index>
new_url: /foreman_tasks/api/tasks/:parent_task_id/sub_tasks
old_api: <resource:, name:foreman_tasks_index>
old_url: /foreman_tasks/api/tasks
There is a conflict trying to redefine a method for a foreign resource (sync_plans_rganizations):
resource:
apipie_resource: sync_plans
new_api: <resource:organizations, name:sync_plans_rganizations>
new_url: /katello/api/organizations/:organization_id/sync_plans/:id/add_products
old_api: <resource:organizations, name:sync_plans_rganizations>
old_url: /katello/api/organizations/:organization_id/sync_plans/:id
Eventually the script is erroring out at this:
Traceback (most recent call last):
File "", line 1, in
File "/data/sfreport/python3.9/lib/python3.9/site-packages/foreman/client.py", line 619, in init
self._generate_api_defs(use_cache, strict_cache)
File "/data/sfreport/python3.9/lib/python3.9/site-packages/foreman/client.py", line 820, in generate_api_defs
new_resource, extra_foreign_methods = parse_resource_definition(
File "/data/sfreport/python3.9/lib/python3.9/site-packages/foreman/client.py", line 375, in parse_resource_definition
functions[api.name] = api.generate_func()
File "/data/sfreport/python3.9/lib/python3.9/site-packages/foreman/client.py", line 282, in generate_func
six.exec(code)
File "", line 1
def lifecycle_environments_update(self, id, organization_id=None, new_name=None, description=None, registry_name_pattern=None, registry_unauthenticated_pull=None, async=None):
^
SyntaxError: invalid syntax
Can you pls help with this or let me know the best way to connect to Foreman API via Python to carry out the automations?
The error message you are getting states that (sine Python 3.5) async is a reserved keyword, and cannot be a variable name. This means that the version of Foreman you are using is incompatible with Python 3.9. Try updating Foreman, downgrading Python, or (if you are comfortable doing it) try to fix the issue yourself (changing the variable name from async to something else, and hoping it works).
I am not sure whether the offending code comes directly from Foreman. Try to search your code (and libraries) for async=None pattern, as it may also not be Foreman's fault.
The whois.whois function always gets a timed out error.
At first, I thought it was because my project is written in Python 2.7 but I also checked in 3.7 and got the same error.
I checked the address on the online website that uses whois and the link worked and didn't get this error.
Anyone knows why this is happening?
import whois
w = whois.whois("https://stackoverflow.com")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Program Files\Python37\lib\site-packages\whois\__init__.py", line 43, in whois
text = nic_client.whois_lookup(None, domain.encode('idna'), flags)
File "C:\Program Files\Python37\lib\site-packages\whois\whois.py", line 264, in whois_lookup
result = self.whois(query_arg, nichost, flags)
File "C:\Program Files\Python37\lib\site-packages\whois\whois.py", line 142, in whois
s.connect((hostname, 43))
socket.timeout: timed out
Your code has at least two problems, and you may have a network problem also.
However, there is no reason for it not to work on Python2.
About the code
This works perfectly fine:
In [7]: import whois
In [8]: print whois.query('stackoverflow.com').expiration_date
2020-02-02 11:59:59
Note two things:
whois is about domain names, not URLs; so you should pass a domain name; note more generally that for new endeavors you should have a look at RDAP instead of whois since you will get a far better experience
you need to use whois.query not whois.whois (you are not saying which version of the library you use, but at its documentation page on https://pypi.org/project/whois/ you can clearly see it is whois.query so I do not know where your whois.whois` comes from).
About the network
You show a network error. It is not 100% clear but you may or may not have access to the whois servers you want to query.
Easy way to test: just use the command line whois from the same box as your code (but again use a domain name, not a URL as parameter) and you will see what is happening.
You can even do directly a telnet on port 43 as whois does nothing else.
$ echo 'stackoverflow.com' | nc whois.verisign-grs.com 43 | grep 'Expiry'
Registry Expiry Date: 2020-02-02T11:59:59Z
i'm a newbie in python and coding,i'm trying to use pyzabbix to add trigger dependecies,but some error occusrs.
When i run
zapi.trigger.addDependencies(triggerid, dependsOnTriggerid)
an error occurs
pyzabbix.ZabbixAPIException: ('Error -32500: Application error., No permissions to referred object or it does not exist!', -32500)
i get the "triggerid" and "dependsOnTriggerid" by trigger.get:
triggerid_info = zapi.trigger.get(filter={'host': 'xx','description': 'xx'},output=['triggerid'], selectDependencies=['description'])
triggerid = triggerid_info[0]['triggerid']
dependsOnTriggerid = trigger_info[0]['dependencies'][0]['triggerid']
The results are as follws:
Traceback (most recent call last): File "E:/10.python/2019-03-07/1.py", line 14, in zapi.trigger.addDependencies(triggerid, dependsOnTriggerid) File "D:\Program Files\Python37\lib\site-packages\pyzabbix__init__.py", line 166, in fn args or kwargs File "D:\Program Files\Python37\lib\site-packages\pyzabbix__init__.py", line 143, in do_request raise ZabbixAPIException(msg, response_json['error']['code']) pyzabbix.ZabbixAPIException: ('Error -32500: Application error., No permissions to referred object or it does not exist!', -32500)
Did i get the wrong triggerid? or the i use the method in a wrong way? Thanks so much
To add a dependancy means that you need to link two different triggers (from the same host or from another one) with a master-dependent logic.
You are trying to add the dependancy triggerid -> dependsOnTriggerid, which is obtained from a supposed existing dependancy (trigger_info[0]['dependencies'][0]['triggerid']), and this makes little sense and I suppose it's the cause of the error.
You need to get both trigger's triggerid and then add the dependancy:
masterTriggerObj = zapi.trigger.get( /* filter to get your master trigger here */ )
dependentTriggerObj = zapi.trigger.get( /* filter to get your dependent trigger here */)
result = zapi.trigger.adddependencies(triggerid=dependentTriggerObj[0]['triggerid'], dependsOnTriggerid=masterTriggerObj[0]['triggerid'])
The method "trigger.addDependencies" need only one parameter，and it should be a dict or some other object/array.The following code solves the problem.
trigger_info = zapi.trigger.get(filter={xx},output=['triggerid'])
trigger_depends_info_193 = zapi.trigger.get(filter={xx},output=['triggerid'])
trigger_dependson_193 = {"triggerid": trigger_info[0]['triggerid'], "dependsOnTriggerid": trigger_depends_info_193[0]['triggerid']}
zapi.trigger.adddependencies(trigger_dependson_193)
I am trying to run ElasticSearch python tests from official elasticsearch-py realease here : https://github.com/elastic/elasticsearch-py/tree/5.x
Actually, I have done all the steps mentionned into elasticsearch-py/test_elasticsearch/README.rst and it turns out that I can run Elasticsearch in a Container with success.
But so far running the tests is failing everytime.
I have got 12 errors out of 124 runned tests and it looks like test_elasticsearch.test_connection.TestRequestsConnection is always failing due to :
AttributeError: 'Session' object has no attribute
'merge_environment_settings'.
What I have done so far is downloading the official Repo, export ES_VERSION=5.4 and then running ElasticSearch with ./start_elasticsearch.sh. After that I am able to navigate to localhost:9200 into my Web Browser.
Then I've changed TEST_ES_SERVER into 'localhost:9200' into elasticsearch5/helpers/test.py file.
I am running the tests using python setup.py test or python run_tests.py but here's what the console returns in both case :
No elasticsearch repo found...
........................./home/user/bin/git/elasticsearch-py/elasticsearch5/client/utils.py:47:
UnicodeWarning: Unicode equal comparison failed to convert both
arguments to Unicode - interpreting them as being unequal quote_plus(_escape(p), b',*') for p in parts if p not in SKIP_IN_PATH)
.EE.EEE.E..EE.EE.EE................................................................................
ERROR: test_body_attached (test_elasticsearch.test_connection.TestRequestsConnection)
Traceback (most recent call last) :
File
"/home/user/bin/git/elasticsearch-py/test_elasticsearch/test_connection.py",
line 213, in test_body_attached
request = self._get_request(con, 'GET', '/', body='{"answer": 42}')
File "/home/user/bin/git/elasticsearch-py/test_elasticsearch/test_connection.py", line 75, in _get_request status, headers, data =
connection.perform_request(*args, **kwargs)
File
"/home/user/bin/git/elasticsearch-py/elasticsearch5/connection/http_requests.py",
line 72, in perform_request
settings =
self.session.merge_environment_settings(prepared_request.url, {},
None, None, None)
AttributeError: 'Session' object has no attribute
merge_environment_settings'
Do you have any idea how I can make this work ? Locally at first.
Thanks and regards.
I am migrating several hundred million tweets of the format {'id_str': , 'created_at': , 'text': } from text files into MongoDB using pymongo. A collection is created for each user to store his/her tweets. The insertion method I am using is insert_many(). It often runs into BulkWriteError.
Traceback (most recent call last):
File "pipeline.py", line 105, in <module>
timeline_db, meta_db, negative_db, log_col, dir_path)
File "/media/haitao/Storage/twitter_pipeline/migrate_old.py", line 134, in migrate_dir
timeline_db[user_id].insert_many(utility.temporal_sort(statuses))
File "/home/haitao/anaconda3/envs/py27/lib/python2.7/site-packages/pymongo/collection.py", line 711, in insert_many
blk.execute(self.write_concern.document)
File "/home/haitao/anaconda3/envs/py27/lib/python2.7/site-packages/pymongo/bulk.py", line 493, in execute
return self.execute_command(sock_info, generator, write_concern)
File "/home/haitao/anaconda3/envs/py27/lib/python2.7/site-packages/pymongo/bulk.py", line 331, in execute_command
raise BulkWriteError(full_result)
pymongo.errors.BulkWriteError: batch op errors occurred
This error seems to occur when there are duplicates of keys, which should not be the case for here. Are there other things that I can check to solve this issue?
Thanks in advance!
sorry for the delay.
1) I replicated the error. The following is close to the end of the mongod.log.
I - [ftdc] Assertion: 13538:couldn't open [/proc/5774/stat] errno:24 Too many open
files
W FTDC [ftdc] Uncaught exception in 'Location13538: couldn't open [/proc/5774/stat]
errno:24 Too many open files' in full-time diagnostic data capture subsystem. Shutting down
the full-time diagnostic data capture subsystem.
E STORAGE [conn2] WiredTiger (24) [1491020356:127332][5774:0x7f6f30e9d700], WT_SESSION
.create: /var/lib/mongodb/: directory-sync: open: Too many open files
I COMMAND [conn2] command timeline_db.231731006 command: insert { insert: "231731006",
ordered: true, documents: 1000 } ninserted:0 keyUpdates:0 writeConflicts:0 exception: 24:
Too many open files code:8 numYields:0 reslen:123 locks:{ Global: { acquireCount: { r: 2,
w: 2 } }, Database: { acquireCount: { w: 1, W: 1 } }, Collection: { acquireCount: { w: 1,
W: 1 } } } protocol:op_query 511ms```
2) Yes, only one instance of MongoClient() is passed around.
3) No multi-processing was run.
After I posted the initial question, I started to use insert_one() which explicitly raised open file limit error. I changed the design of the database (mainly, reduced the number of collections) and solved the issue about open file limit. I am not sure, but the log seems to suggest that the actual cause of the BulkWriteError is also open file limit.
For me, the error was occurring because of replication in _id during second time.
So if I used say items to add in Mongo, pymongo automatically inserts object_id aka _id and then if items are global, it will have the _id from the previous operation, which causes redundancy and ultimately BulkWriteError.
Clearing the cache solved it.
Do you can debug the error details with something like this:
try:
# pymongo process here
pass
except BulkWriteError as error
print(error.details)
Do you can also improve the code above using the pdb to debug on terminal:
try:
# pymongo process here
pass
except BulkWriteError as error
import pdb; pdb.set_trace()
Then print error.details or anything that you want:
(Pdb) error.details
This solution from here: https://github.com/microsoft/vscode-cosmosdb/issues/1296 is working for me
I had the same issue and I solved it by setting 'retrywrites=false' at the connection string like this:
mongodb://<username>:<password>#<host>:<port>/<db_name>?ssl=true&replicaSet=globaldb&retrywrites=false