Develop Reference

server name reachability in dnspython

I am currently trying to find a way to check whether or not the name servers can respond to either TCP or UDP packets.
My idea behind that was, to get all the name servers from a website (for example google.com), store them in a list, and then try to send TCP and UDP messages to all of them.
Although I am getting the name servers, my interpreter shows a problem when I am trying to make a query on udp(check udpPacket on the code) saying:
"TypeError: coercing to Unicode: need string or buffer, NS found"
I am new in Python(coming from C and C++) and I am guessing this is just incompatible types.
I checked dnspython's documentation and could not find what kind of type NS is (probably it's a type by itself) and why it cannot be passed as an argument.
What do you think the problem is? Is there maybe a better way to solve that kind of problem?
def getNSResults(url):
#create an empty list where we can store all the nameservers we found
nameServers = []
nameServers = dns.resolver.query(url,dns.rdatatype.NS, raise_on_no_answer=False)
#create a dictionary where based on all the nameservers.
#1st label refers to the ns name of our url that we inserted.
#2nd label shows wether or not we received a UDP response or not.
#3rd label shows wether or not we received a TCP response or not.
results = {}
for nameServer in nameServers:
#make a dns ns query, acts as a dumb message since whatever we send we just care of what we get back
query = dns.message.make_query(dns.name.from_text(url), dns.rdatatype.ANY)
query.flags |= dns.flags.AD
query.find_rrset(query.additional, dns.name.root, 65535, dns.rdatatype.OPT, create=True, force_unique=True)
#try sending a udp packet to see if it's listening on UDP
udpPacket = dns.query.udp(query,nameServer)
#try sending a tcp packet to see if it's listening on TCP
tcpPacket = dns.query.tcp(None,nameServer)
#add the results in a dictionary and return it, to be checked later by the user.
results.update({"nsName" == nameServer, "receivedUDPPacket" == isNotNone(udpPacket),"receivedTCPPacket" == isNotNone(tcpPacket)})
Thanks in advance!

Looking at your code, I see some DNS problems, some Python problems, and some dnspython problems. Let's see if we can't learn something together.
DNS
First, the parameter to your function getNSResults is called url. When you send DNS queries, you query for a domain name. A URL is something totally different (e.g. https://example.com/index.html). I would rename url to something like domain_name, domain, or name. For more on the difference between URLs and domain names, see https://www.copahost.com/blog/domain-vs-url/.
Second, let's talk about what you're trying to do.
i am currently trying to find a way to check wether or not the name servers can respond to either tcp or udp packets.
My idea behind that was, to get all the name servers from a website (for example google.com), store them in a list, and then, try to send tcp and udp messages to all of them.
That sounds like a great approach. I think you might be missing a few details here. so let me explain the steps you can take to do this:
Do an NS query for a domain name. You already have this step in your code. What you'll actually get from that query is just another domain name (or multiple domain names). For example, if you run dig +short NS google.com, you'll get this output:
ns3.google.com.
ns1.google.com.
ns4.google.com.
ns2.google.com.
At this step, we have a list of one or more names of authoritative servers. Now we need an IP address to use to send them queries. So we'll do a type A query for each of the names we got from step 1.
Now we have a list of IP addresses. We can send a DNS query over UDP and one over TCP to see if they're supported.
Python
For the most part, your Python syntax is okay.
The biggest red flag I see is the following code:
results.update({"nsName" == nameServer,
"receivedUDPPacket" == isNotNone(udpPacket),
"receivedTCPPacket" == isNotNone(tcpPacket)})
Let's break this down a bit.
First, you have results, which is a dict.
Then you have this:
{"nsName" == nameServer,
"receivedUDPPacket" == isNotNone(udpPacket),
"receivedTCPPacket" == isNotNone(tcpPacket)}
which is a set of bools.
What I think you meant to do was something like this:
results.update({
"nsName": nameServer,
"receivedUDPPacket": true,
"receivedTCPPacket": true
})
Function and variables names in Python are usually written in lowercase, with words separated by underscores (e.g. my_variable, def my_function()). Class names are usually upper camel case (e.g. class MyClass).
None of this is required, you can name your stuff however you want, plenty of super popular libraries and builtins break this convention, just figured I'd throw it out there because it can be helpful when reading Python code.
dnspython
When you're not sure about the types of things, or what attributes things have, remember these four friends, all builtin to Python:
1. pdb
2. dir
3. type
4. print
pdb is a Python debugger. Just import pdb, and the put pdb.set_trace() where you want to break. Your code will stop there, and then you can check out the values of all the variables.
dir will return the attributes and methods of whatever you pass to it. Example: print(dir(udpPacket)).
type will return the type of an object.
print as you probably already know, will print out stuff so you can see it.
I'm going to leave this part for you to test out.
Run dir() on everything if you don't know what it is.
I also should probably mention help(), which is super useful for built-in stuff.
The summary for this section is that sometimes documentation isn't all there, or hard to find, especially when you're new to a language/library/whatever.
So you have to figure stuff out on your own, and that means using all the tools I've just mentioned, looking at the source code, things like that.
Summary
I hope this was helpful. I know it's a lot, it's probably too much, but just be patient and know that DNS and Python are some very useful and fun things to learn about.
I went ahead and wrote something up that is a start at what I think you're hoping to achieve.
I recommend walking through the whole thing and making sure you understand what's going on.
If you don't understand something, remember pdb and dir (and there's always Google, SO, etc).
import dns.resolver
import dns.message
import dns.rdatatype
import json
import sys
def check_tcp_and_udp_support(name):
# this will give me the first default system resolver from /etc/resolv.conf
# (or Windows registry)
where = dns.resolver.Resolver().nameservers[0]
q = dns.message.make_query(name, dns.rdatatype.NS)
ns_response = dns.query.udp(q, where)
ns_names = [t.target.to_text() for ans in ns_response.answer for t in ans]
# this code is the same as the one-liner above
# ns_names = []
# for ans in ns_response.answer:
# for t in ans:
# ns_names.append(t.target.to_text())
results = {}
for ns_name in ns_names:
# do type A lookup for nameserver
q = dns.message.make_query(ns_name, dns.rdatatype.A)
response = dns.query.udp(q, where)
nameserver_ips = [item.address for ans in response.answer for item in ans.items if ans.rdtype == dns.rdatatype.A]
# now send queries to the nameserver IPs
for nameserver_ip in nameserver_ips:
q = dns.message.make_query('example.com.', dns.rdatatype.A)
try:
udp_response = dns.query.udp(q, nameserver_ip)
supports_udp = True
except dns.exception.Timeout:
supports_udp = False
try:
tcp_response = dns.query.tcp(q, nameserver_ip)
supports_tcp = True
except dns.exception.Timeout:
supports_tcp = True
results[nameserver_ip] = {
'supports_udp': supports_udp,
'supports_tcp': supports_tcp
}
return results
def main():
results = check_tcp_and_udp_support('google.com')
# this is just fancy JSON printing
# you could do print(results) instead
json.dump(results, sys.stdout, indent=4)
if __name__ == '__main__':
main()
Again, I hope this is helpful. It's hard when I don't know exactly what's going on in your head, but this is what I've got for you.

Making all attributes and methods available for a socket server in Python

I use a Raspberry Pi to collect sensor data and set digital outputs, to make it easy for other applications to set and get values I'm using a socket server. But I am having some problems finding an elegant way of making all the data available on the socket server without having to write a function for each data type.
Some examples of values and methods I have that I would like to make available on the socket server:
do[2].set_low() # set digital output 2 low
do[2].value=0 # set digital output 2 low
do[2].toggle() # toggle digital output 2
di[0].value # read value for digital input 0
ai[0].value # read value for analog input 0
ai[0].average # get the average calculated value for analog input 0
ao[4].value=255 # set analog output 4 to byte value 255
ao[4].percent=100 # set analog output 4 to 100%
I've tried eval() and exec():
self.request.sendall(str.encode(str(eval('item.' + recv_string)) + '\n'))
eval() works unless I am using equal sign (=), but I'm not to happy about the solution because of dangers involved. exec() does the work but does not return any value, also dangerous.
I've also tried getattr():
recv_string = bytes.decode(self.data).lower().split(';')
values = getattr(item, recv_string[0])
self.request.sendall(str.encode(str(values[int(recv_string[1])].value) + '\n'))
^^^^^
This works for getting my attributes, and the above example works for getting the value of the attribute I am getting with getattr(). But I can not figure out how to use getattr() on the value attribute as well.
The semi-colon (;) is used to split the incoming command, I've experimented with multiple ways of formatting the commands:
# unit means that I want to talk to a I/O interface module,
# and the name specified which one
unit;unit_name;get;do;1
unit;unit_name;get;do[1]
unit;unit_name;do[1].value
I am free to choose the format since I am also writing the software that uses these commands. I have not yet found a good format which covers all my needs.
Any suggestions how I can write an elegant way of accessing and returning the data above? Preferably with having to add new methods to the socket server every time a new value type or method is added to my I/O ports.
Edit: This is not public, it's only available on my LAN.

Suggestions
Make your API all methods so that eval can always be used:
def value_m(self, newValue=None):
if newValue is not None:
self.value = newValue
return self.value
Then you can always do
result = str(eval(message))
self.request.sendall(str.encode(result + '\n'))
For your message, I would suggest that your messages are formatted to include the exact syntax of the command exactly so that it can be evaled as-is, e.g.
message = 'do[1].value_m()' # read a value, alternatively...
message = 'do[1].value_m(None)'
or to write
message = 'do[1].value_m(0)' # write a value
This will make it easy to keep your messages up-to-date with your API, because they must match exactly, you won't have a second DSL to deal with. You really don't want to have to maintain a second API, on top of your IO one.
This is a very simple scheme, suitable for a home project. I would suggest some error handling in evaluation, like so:
import traceback
try:
result = str(eval(message))
except Exception:
result = traceback.format_exc()
self.request.sendall(str.encode(result + '\n'))
This way your caller will receive a printout of the exception traceback in the returned message. This will make it much, much easier to debug bad calls.
NOTE If this is public-facing, you cannot do this. All input must be sanitised. You will have to parse each instruction and compare it to the list of available (and desirable) commands, and verify input validity and validity ranges for everything. For such a scenario you are better off simply using one of the input validation systems used for web services, where this problem receives a great deal of attention.

How do I assign variables within WMI calls?

Using WMI within Python to request data from other servers. I know my credentials are correct, as I can hard-code them in the file and connect without issues. However, my string formatting for variables doesn't seem to be working.
I've tried both of these with no luck:
wmi_sql = wmi.WMI(SQLServer_raw, user="%s\\%s", password="%s") % (sql_domain, sql_user, sql_pass)
and
wmi_sql = wmi.WMI(SQLServer_raw, user="{0}\\{1}", password="{2}").format(sql_domain, sql_user, sql_pass)
Tried moving the format() call inside, but it didn't work either:
wmi_sql = wmi.WMI(SQLServer_raw, user="{0}\\{1}", password="{2}".format(sql_domain, sql_user, sql_pass))

You've got the string formatters in the wrong place. They need to be used for each string, not the result of the call. As written, python thinks you want to call wmi.WMI and then apply the formatting to whatever is returned.
Try:
wmi_sql = wmi.WMI(SQLServer_raw, user="%s\\%s" % (sql_domain, sql_user),
password=sql_pass)

In one and two you are executing wmi.WMI() and then doing the formatting on the return type. In the third example you are formatting the value of the password but providing three parameters to it hoping it would build user variable. If you really want to build user and password dynamically do this:
usr ="{0}\\{1}".format(sql_domain,ql_user)
wmi.WMI(SQLServer_raw, user=usr, password=sql_pass)

Python object persistence

I'm seeking advice about methods of implementing object persistence in Python. To be more precise, I wish to be able to link a Python object to a file in such a way that any Python process that opens a representation of that file shares the same information, any process can change its object and the changes will propagate to the other processes, and even if all processes "storing" the object are closed, the file will remain and can be re-opened by another process.
I found three main candidates for this in my distribution of Python - anydbm, pickle, and shelve (dbm appeared to be perfect, but it is Unix-only, and I am on Windows). However, they all have flaws:
anydbm can only handle a dictionary of string values (I'm seeking to store a list of dictionaries, all of which have string keys and string values, though ideally I would seek a module with no type restrictions)
shelve requires that a file be re-opened before changes propagate - for instance, if two processes A and B load the same file (containing a shelved empty list), and A adds an item to the list and calls sync(), B will still see the list as being empty until it reloads the file.
pickle (the module I am currently using for my test implementation) has the same "reload requirement" as shelve, and also does not overwrite previous data - if process A dumps fifteen empty strings onto a file, and then the string 'hello', process B will have to load the file sixteen times in order to get the 'hello' string. I am currently dealing with this problem by preceding any write operation with repeated reads until end of file ("wiping the slate clean before writing on it"), and by making every read operation repeated until end of file, but I feel there must be a better way.
My ideal module would behave as follows (with "A>>>" representing code executed by process A, and "B>>>" code executed by process B):
A>>> import imaginary_perfect_module as mod
B>>> import imaginary_perfect_module as mod
A>>> d = mod.load('a_file')
B>>> d = mod.load('a_file')
A>>> d
{}
B>>> d
{}
A>>> d[1] = 'this string is one'
A>>> d['ones'] = 1 #anydbm would sulk here
A>>> d['ones'] = 11
A>>> d['a dict'] = {'this dictionary' : 'is arbitrary', 42 : 'the answer'}
B>>> d['ones'] #shelve would raise a KeyError here, unless A had called d.sync() and B had reloaded d
11 #pickle (with different syntax) would have returned 1 here, and then 11 on next call
(etc. for B)
I could achieve this behaviour by creating my own module that uses pickle, and editing the dump and load behaviour so that they use the repeated reads I mentioned above - but I find it hard to believe that this problem has never occurred to, and been fixed by, more talented programmers before. Moreover, these repeated reads seem inefficient to me (though I must admit that my knowledge of operation complexity is limited, and it's possible that these repeated reads are going on "behind the scenes" in otherwise apparently smoother modules like shelve). Therefore, I conclude that I must be missing some code module that would solve the problem for me. I'd be grateful if anyone could point me in the right direction, or give advice about implementation.

Use the ZODB (the Zope Object Database) instead. Backed with ZEO it fulfills your requirements:
Transparent persistence for Python objects
ZODB uses pickles underneath so anything that is pickle-able can be stored in a ZODB object store.
Full ACID-compatible transaction support (including savepoints)
This means changes from one process propagate to all the other processes when they are good and ready, and each process has a consistent view on the data throughout a transaction.
ZODB has been around for over a decade now, so you are right in surmising this problem has already been solved before. :-)
The ZODB let's you plug in storages; the most common format is the FileStorage, which stores everything in one Data.fs with an optional blob storage for large objects.
Some ZODB storages are wrappers around others to add functionality; DemoStorage for example keeps changes in memory to facilitate unit testing and demonstration setups (restart and you have clean slate again). BeforeStorage gives you a window in time, only returning data from transactions before a given point in time. The latter has been instrumental in recovering lost data for me.
ZEO is such a plugin that introduces a client-server architecture. Using ZEO lets you access a given storage from multiple processes at a time; you won't need this layer if all you need is multi-threaded access from one process only.
The same could be achieved with RelStorage, which stores ZODB data in a relational database such as PostgreSQL, MySQL or Oracle.

For beginners, You can port your shelve databases to ZODB databases like this:
#!/usr/bin/env python
import shelve
import ZODB, ZODB.FileStorage
import transaction
from optparse import OptionParser
import os
import sys
import re
reload(sys)
sys.setdefaultencoding("utf-8")
parser = OptionParser()
parser.add_option("-o", "--output", dest = "out_file", default = False, help ="original shelve database filename")
parser.add_option("-i", "--input", dest = "in_file", default = False, help ="new zodb database filename")
parser.set_defaults()
options, args = parser.parse_args()
if options.in_file == False or options.out_file == False :
print "Need input and output database filenames"
exit(1)
db = shelve.open(options.in_file, writeback=True)
zstorage = ZODB.FileStorage.FileStorage(options.out_file)
zdb = ZODB.DB(zstorage)
zconnection = zdb.open()
newdb = zconnection.root()
for key, value in db.iteritems() :
print "Copying key: " + str(key)
newdb[key] = value
transaction.commit()

I suggest using TinyDB, it's much much better and simple to use.
https://tinydb.readthedocs.io/en/stable/

Python c_types .dll functions (pari library)

Alright, so a couple days ago I decided to try and write a primitive wrapper for the PARI library. Ever since then I've been playing with ctypes library in loading the dll and accessing the functions contained using code similar to the following:
from ctypes import *
libcyg=CDLL("<path/cygwin1.dll") #It needs cygwin to be loaded. Not sure why.
pari=CDLL("<path>/libpari-gmp-2.4.dll")
print pari.fibo #fibonacci function
#prints something like "<_FuncPtr object at 0x00BA5828>"
So the functions are there and they can potentially be accessed, but I always receive an access violation no matter what I try. For example:
pari.fibo(5) #access violation
pari.fibo(c_int(5)) #access violation
pari.fibo.argtypes = [c_long] #setting arguments manually
pari.fibo.restype = long #set the return type
pari.fibo(byref(c_int(5))) #access violation reading 0x04 consistently
and any variation on that, including setting argtypes to receive pointers.
The Pari .dll is written in C and the fibonacci function's syntax within the library is GEN fibo(long x).
Could it be the return type that's causing these errors, as it is not a standard int or long but a GEN type, which is unique to the PARI library? Any help would be appreciated. If anyone is able to successfully load the library and use ANY function from within python, please tell; I've been at this for hours now.
EDIT: Seems as though I was simply forgetting to initialize the library. After a quick pari.pari_init(4000000,500000) it stopped erroring. Now my problem lies in the in the fact that it returns a GEN object; which is fine, but whenever I try to reference the address to which it points, it's always 33554435, which I presume is still an address. I'm trying further commands and I'll update if I succeed in getting the correct value of something.

You have two problems here, one give fibo the correct return type and two convert the GEN return type to the value you are looking for.
Poking around the source code a bit, you'll find that GEN is defined as a pointer to a long. Also, at looks like the library provides some converting/printing GENs. I focused in on GENtostr since it would probably be safer for all the pari functions.
import cytpes
pari = ctypes.CDLL("./libpari.so.2.3.5") #I did this under linux
pari.fibo.restype = ctypes.POINTER(ctypes.c_long)
pari.GENtostr.restype = ctypes.POINTER(ctypes.c_char)
pari.pari_init(4000000,500000)
x = pari.fibo(100)
y = pari.GENtostr(x)
ctypes.string_at(y)
Results in:
'354224848179261915075'