I am having trouble with the parameter of an SNMP query in a python script. An SNMP query takes an OID as a parameter. The OID I use here is written in the code below and, if used alone in a query, should return a list of states for the interfaces of the IP addresses I am querying onto.
What I want is to use that OID with a variable appended to it in order to get a very precise information (if I use the OID alone I will only get a list of thing that would only complexify my problem).
The query goes like this:
oid = "1.3.6.1.4.1.2011.5.25.119.1.1.3.1.2."
variable = "84.79.84.79"
query = session.get(oid + variable)
Here, this query will return a corrupted SNMPObject, as in the process of configuration of the device I am querying on, another number is added, for some reason we do not really care about here, between these two elements of the parameter.
Below is a screenshot showing some examples of an SNMP request that only takes as a parameter the OID above, without the variable appended, on which you may see that my variable varies, and so does the highlighted additional number:
Basically what I am looking for here is the response, but unfortunately I cannot predict for each IP address I am querying what will that "random" number be.
I could use a loop that tries 20 or 50 queries and only saves the response of the only one that would have worked, but it's ugly. What would be better is some built-in function or library that would just say to the query:
"SNMP query on that OID, with any integer appended to it, and with my variable appended to that".
I definitely don't want to generate a random int, as it is already generated in the configuration of the device I am querying, I just want to avoid looping just to get a proper response to a precise query.
I hope that was clear enough.
Something like this should work:
from random import randint
variable = "84.79.84.79"
numbers = "1.3.6.1.4.1.2011.5.25.119.1.1.3.1.2"
query = session.get('.'.join([numbers, str(randint(1,100)), variable])
Related
I have a collection student and I want this collection as list in Python, but unfortunately I got the following error CursorNextError: [HTTP 404][ERR 1600] cursor not found. Is there an option to read a 'huge' collection without an error?
from arango import ArangoClient
# Initialize the ArangoDB client.
client = ArangoClient()
# Connect to database as user.
db = client.db(<db>, username=<username>, password=<password>)
print(db.collections())
students = db.collection('students')
#students.all()
students = db.collection('handlingUnits').all()
list(students)
[OUT] CursorNextError: [HTTP 404][ERR 1600] cursor not found
students = list(db.collection('students'))
[OUT] CursorNextError: [HTTP 404][ERR 1600] cursor not found
as suggested in my comment, if raising the ttl is not an option (what I wouldn't do either) I would get the data in chunks instead of all at once. In most cases you don't need the whole collection anyway, so maybe think of limiting that first. Do you really need all documents and all their fields?
That beeing said I have no experience with arango, but this is what I would do:
entries = db.collection('students').count() # get total amount of documents in collection
limit=100 # blocksize you want to request
yourlist = [] # final output
for x in range(int(entries/limit) + 1):
block = db.collection('students').all(skip=x*limit, limit=100)
yourlist.extend(block) # assuming block is of type list. Not sure what arango returns
something like this. (Based on the documentation here: https://python-driver-for-arangodb.readthedocs.io/_/downloads/en/dev/pdf/)
Limit your request to a reasonable amount and then skip this amount with your next request. You have to check if this "range()" thing works like that you might have to think of a better way of defining the number of iterations you need.
This also assumes arango sorts the all() function per default.
So what is the idea?
determin the number of entries in the collection.
based on that determin how many requests you need (f.e. size=1000 -> 10 blocks each containing 100 entries)
make x requests where you skip the blocks you already have. First iteration entries 1-100; second iteration 101-200, third iteration 201-300 etc.
By default, AQL queries generate the complete result, which is then held in memory, and provided batch by batch. So the cursor is simply fetching the next batch of the already calculated result. In most of the cases this is fine, but if your query produces a huge result set, then this can take a long time and will require a lot of memory.
As an alternative you can create a streaming cursor. See https://www.arangodb.com/docs/stable/http/aql-query-cursor-accessing-cursors.html and check the stream option.
Streaming cursors calculate the next batch on demand and are therefore better suited to iterate a large collection.
I'm having this function that communicates via pymysql to an SQL database stored to my localhost. I know there are similar posts about formatting an SQL section especially this one but could anyone suggest a solution?
Always getting TypeError: can't concat tuple to bytes. I suppose it's sth with the WHERE clause.
def likeMovement(pID):
print("Give a rating for the movement with #id:%s" %pID)
rate=input("Give from 0-5: ")
userID=str(1)
print(rate,type(rate))
print(pID,type(pID))
print(userID,type(userID))
cursor=con.cursor()
sqlquery='''UDPATE likesartmovement SET likesartmovement.rating=%s WHERE
likesartmovement.artisticID=? AND likesartmovement.userID=?''' % (rate,),
(pID,userID)
cursor.execute(sqlquery)
TypeError: not all arguments converted during string formatting
Thanks in advance!
The problem is that you're storing (pID,userID) as part of a tuple stored in sqlquery, instead of passing them as the arguments to execute:
sqlquery='''UDPATE likesartmovement SET likesartmovement.rating=%s WHERE
likesartmovement.artisticID=? AND likesartmovement.userID=?''' % (rate,)
cursor.execute(sqlquery, (pID,userID))
It may be clearer to see why these are different if you take a simpler example:
s = 'abc'
spam(s, 2)
s = 'abc', 2
spam(s)
Obviously those two don't do the same thing.
While we're at it:
You have to spell UPDATE right.
You usually want to use query parameters for SET clauses for exactly the same reasons you want to for WHERE clauses.
You don't need to include the table name in single-table operations, and you're not allowed to include the table name in SET clauses in single-table updates.
So:
sqlquery='''UDPATE likesartmovement SET rating=? WHERE
artisticID=? AND userID=?'''
cursor.execute(sqlquery, (rating, pID, userID))
I'm trying to extract email addresses from text in the column alltext and update the column email with the list of emails found in alltext. The datatype for email is a string array (i.e. text[]).
1) I'm getting the following error and can't seem to find a way around it:
psycopg2.ProgrammingError: syntax error at or near "["
LINE 1: UPDATE comments SET email=['person#email.com', 'other#email.com']
2) Is there a more efficient way to be doing this in the first place? I've experimented some with the PostgreSQL regex documentation but a lot of people seem to think it's not great for this purpose.
def getEmails():
'''Get emails from alltext.
'''
DB = psycopg2.connect("dbname=commentDB")
c = DB.cursor()
c.execute("SELECT id, alltext FROM comments WHERE id < 100")
for row in c:
match = re.findall(r'[\w\.-]+#[\w\.-]+', str(row[1]))
data = {'id':int(row[0]), 'email':match}
c.execute("UPDATE comments SET email=%(email)s WHERE id=%(id)s" % data)
DB.commit()
DB.close()
execute should be passed a list for unnamed arguments, or dict -- as in this case -- for named arguments, as a second argument to ensure that it is psycopg2 (via libpq) that is doing all the proper escaping. You are using native Python string interpolation, which is subject to SQL Injection, and leading to this error, since it isn't libpq doing the interpolation.
Also, as an aside, your regex won't capture various types of email addresses. One type that immediately comes to mind is the form foo+bar#loopback.edu. The + is technically allowed, and can be used, for example, for filtering email. See this link for more details as to issues that crop up with using regexes for validating/parsing email addresses.
In short, the above link recommends using this regex:
\b[A-Z0-9._%+-]+#[A-Z0-9.-]+\.[A-Z]{2,4}\b
With the caveat that it is valid for only what the author claims is valid email address. Still, it's probably a good jumping-off point, and can be adjusted if you have specific cases that differ.
Edit in response to comment from OP:
The execute line from above would become:
c.execute("UPDATE comments SET email=%(email)s WHERE id=%(id)s", data)
Note that data is now a second argument to execute as opposed to being part of an interpolation operation. This means that Psycopg2 will handle the interpolation and not only avoid the SQL Injection issue, but also properly interpret how the dict should be interpolated into the query string.
Edit in response to follow-up comment from OP:
Yes, the subsequent no results to fetch error is likely because you are using the same cursor. Since you are iterating over the current cursor, trying to use it again in the for loop to do an update interferes with the iteration.
I would declare a new cursor inside the for loop and use that.
Using Python with SPARQLWrapper, JSON, urlib2 & cgi. Had trouble passing a working SPARQL query with some NULL values to python so I populated the blanks with a literal and will try to filter at the output. I have this results section example:
for result in results["results"]["bindings"]:
project = result["project"]["value"].encode('utf-8')
filename = result["filename"]["value"].encode('utf-8')
url = result["url"]["value"].encode('utf-8')
...and I print the %s. Is there a way to filter a value, i.e., IF VALUE NE "string" then PRINT? Or is there another workaround? I'm at the tail-end of a small project, I know I need a better wrapper, I just need to get these results filtered before I can move on. T very much IA...
I'm one of the developers of the SPARQLWrapper library, and the question had been already answered at the mailing list.
Regarding optionals values on the original query, the result set will come with no values for those variables. The problems is that we'd need to parse the query to populate such missing entries, and we want to avoid such parsing; therefore you'd need to check it for avoiding runtime problems with KeyError.
Usually I use a code like:
for result in results["results"]["bindings"]:
party = result["party"]["value"] if ("party" in result) else None
I'm having a database (sqlite) of members of an organisation (less then 200 people). Now I'm trying to write an wx app that will search the database and return some contact information in a wx.grid. The app will have 2 TextCtrls, one for the first name and one for the last name. What I want to do here is make it possible to only write one or a few letters in the textctrls and that will start to return result. So, if I search "John Smith" I write "Jo" in the first TextCtrl and that will return every single John (or any one else having a name starting with those letters). It will not have an "search"-button, instead it will start searching whenever I press a key.
One way to solve this would be to search the database with like " SELECT * FROM contactlistview WHERE forname LIKE 'Jo%' " But that seems like a bad idea (very database heavy to do that for every keystroke?). Instead i thought of use fetchall() on a query like this " SELECT * FROM contactlistview " and then, for every keystroke, search the list of tuples that the query have returned. And that is my problem: Searching a list is not that difficult but how can I search a list of tuples with wildcards?
selected = [t for t in all_data if t[1].startswith('Jo')]
but, measure, don't guess. I think that in some cases, the query would be faster - specially if you have too many records. Maybe you should use a query on the first char, and then start using python-side filter, since you already have the results.
I think that generally, you shouldn't be afraid of giving tasks to a database. It's quite possible that the LIKE clause will be very fast. Sqlite is implemented in fairly robust C code, and will happily deal with queries like this.
If you're worried about sending too many requests, why not send a query once a user has entered a threshold of characters, such as three?
A list comprehension is probably the best way to return the result if you want to do added filtering.
If you are searching for a string matching the start using LIKE, eg 'abc%' (rather than anywhere in the string - '%abc%'), the search should be quite fast if you have an index on the field, as the db can use the index to help find the matches.