Develop Reference

Threading NNTP, how? (Newbie here)

I can't wrap my head around how I could possibly rewrite my code to be multi-threaded.
The code I'm writing is made to automatically archive every single article in a list of newsgroups that exist, but I wanna be able to utilize my newsgroup plan and make it up to 20 threads. I've never coded threading before and my attempts were in vein.
Here's my code, excluding the username and pass ( but you can get a free account with max 5 threads if you really want to at https://my.xsusenet.com )
Please don't judge me too hard :(
import nntplib
import sys
import datetime
import os
basetime = datetime.datetime.today()
#daysback = int(sys.argv[1])
#date_list = [basetime - datetime.timedelta(days=x) for x in range(daysback)]
s = nntplib.NNTP('free.xsusenet.com', user='USERNAME', password='PASSWORD') # I am only allowed 5 connections at a time, so try for 4.
groups = []
resp, groups_list_tuple = s.list()
def remove_non_ascii_2(string):
return string.encode('ascii', errors='ignore').decode()
for g_tuple in groups_list_tuple:
#print(g_tuple) # DEBUG_LINE
# Parse group_list info
group = g_tuple[0]
last = g_tuple[1]
first = g_tuple[2]
flag = g_tuple[3]
# Parse newsgroup info
resp, count, first, last, name = s.group(group)
for message_id in range(first, last):
resp, number, mes_id = s.next()
resp, info = s.article(mes_id)
if os.path.exists('.\\' + group):
pass
else:
os.mkdir('.\\' + group)
print(f"Downloading: {message_id}")
outfile = open('.\\' + group + '\\' + str(message_id), 'a', encoding="utf-8")
for line in info.lines:
outfile.write(remove_non_ascii_2(str(line)) + '\n')
outfile.close()
Tried threading using a ThreadPoolExecutor, to cause it to use 20 threads, and failed, caused it to repeat the same process to the same message id. The expected result was to download 20 different messages at a time.
Here's the code I tried with threading, mind you I did like 6-8 variations of it to try and get it to work, this was the last one before I gave up to ask on here.
import nntplib
import sys
import datetime
import os
import concurrent.futures
basetime = datetime.datetime.today()
#daysback = int(sys.argv[1])
#date_list = [basetime - datetime.timedelta(days=x) for x in range(daysback)]
s = nntplib.NNTP('free.xsusenet.com', user='USERNAME', password='PASSWORD') # I am only allowed 5 connections at a time, so try for 4.
groups = []
resp, groups_list_tuple = s.list()
def remove_non_ascii_2(string):
return string.encode('ascii', errors='ignore').decode()
def download_nntp_file(mess_id):
resp, count, first, last, name = s.group(group)
message_id = range(first, last)
resp, number, mes_id = s.next()
resp, info = s.article(mes_id)
if os.path.exists('.\\' + group):
pass
else:
os.mkdir('.\\' + group)
print(f"Downloading: {mess_id}")
outfile = open('.\\' + group + '\\' + str(mess_id), 'a', encoding="utf-8")
for line in info.lines:
outfile.write(remove_non_ascii_2(str(line)) + '\n')
outfile.close()
for g_tuple in groups_list_tuple:
#print(g_tuple) # DEBUG_LINE
# Parse group_list info
group = g_tuple[0]
last = g_tuple[1]
first = g_tuple[2]
flag = g_tuple[3]
# Parse newsgroup info
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
futures = executor.submit(download_nntp_file)

I can't test it with XSUseNet.
I wouldn't use global variables because when processes work at the same time then they may get the same values from these variables.
You should rather send values as parameters to functions.
Something like this:
def download_nntp_file(g_tuple):
# ... code which uses `g_tuple` instead of global variables ...
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
for g_tuple in groups_list_tuple:
executor.submit(download_nntp_file, g_tuple)
But I would be simpler to use map() instead of submit() because it gets list with arguments and it doesn't need for-loop
def download_nntp_file(g_tuple):
# ... code which uses `g_tuple` instead of global variables ...
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
executor.map(download_nntp_file, groups_list_tuple)

Find next available IP address in python

Using python I need to find the next IP address given a range of IP addresses I've already used. So if I have a list of IP address like...
IPs = ['10.220.1.1','10.220.1.2','10.220.1.3','10.220.1.5']
When I ask for the next IP address I need it to return '10.220.1.4'. The next request would return '10.220.1.6' and so on.

If you're using Python 3.3 (or newer), you can use the ipaddress module. Example for all hosts in the subnet 10.220.1.0/24 except for those in reserved:
from ipaddress import IPv4Network
network = IPv4Network('10.220.1.0/24')
reserved = {'10.220.1.1', '10.220.1.2', '10.220.1.3', '10.220.1.5'}
hosts_iterator = (host for host in network.hosts() if str(host) not in reserved)
# Using hosts_iterator:
print(next(hosts_iterator)) # prints 10.220.1.4
print(next(hosts_iterator)) # prints 10.220.1.6
print(next(hosts_iterator)) # prints 10.220.1.7
# Or you can iterate over hosts_iterator:
for host in hosts_iterator:
print(host)
So basically this can be done in a single line (+ imports and definition of network and reserved addresses).

If you're using Python 3 you could use ipaddress with generator:
import ipaddress
def gen_ip(start, reserved):
start = int(ipaddress.IPv4Address(start))
for i in range(start + 1, int(2 ** 32) + 1):
ip = str(ipaddress.IPv4Address(i))
if ip not in reserved:
yield ip
IPs = ['10.220.1.1','10.220.1.2','10.220.1.3','10.220.1.5']
j = 0
for ip in gen_ip(min(IPs), set(IPs)):
print(ip)
j += 1
if j == 5:
break
Output:
10.220.1.4
10.220.1.6
10.220.1.7
10.220.1.8
10.220.1.9

You can use an generator using the ipaddress module which is a bultin from python >= 3.3 or you can install with pip for earlier versions:
In [20]: from ipaddress import ip_network
In [21]: IPs = {'10.220.1.1','10.220.1.2','10.220.1.3','10.220.1.5'}
In [22]: net = ip_network(u"10.220.1.0/24")
In [23]: avail =(str(ip) for ip in net.hosts() if str(ip) not in IPs
....: )
In [24]: next(avail)
Out[24]: '10.220.1.4'
In [25]: next(avail)
Out[25]: '10.220.1.6'

Convert your list to a set, for performance:
used_ips = set(IPs)
Now generate your IP#'s however you would like, and check if they are contained in the set:
for next_ip in generate_ip_numbers():
if next_ip in used_ips:
continue
print("Next ip address is:", next_ip)

Create a basic ip object to hold a record of your current ip and to get the next ip
class IPObj(object):
def __init__(self, list_of_ips):
self.position = 0
self.ips = list_of_ips
self.current_ip = self.ips[self.position]
def next_ip(self, stop_iteration=False):
'''
return the next ip in the list
'''
if self.position >= len(self.ips)-1:
self.position = 0 # By default next_ip will do nothing when it reaches the end but it will throw an exception if stop_iteration==True
if stop_iteration:
raise StopIteration
self.position += 1
self.current_ip = self.ips[self.position]
return self.current_ip
def __repr__(self):
return repr(self.current_ip)
#Testing
ips = IPObj(['10.220.1.1','10.220.1.2','10.220.1.3','10.220.1.5'])
print ips
while True:
print ips.next_ip(True),
Output:
10.220.1.1,
10.220.1.2,
10.220.1.3,
10.220.1.5,
Traceback (most recent call last):
File "C:\Users\J10ey\workspace\SO_Help\src\ip's.py", line 32, in
print ips.next_ip(True)
File "C:\Users\J10ey\workspace\SO_Help\src\ip's.py", line 21, in next_ip
raise StopIteration
StopIteration

Find / Slicing Method issue

I was trying to use the find and string slicing method to extract the number at the end of the line. but I get this mismatch error because I came back with position 18 but from what I have read and research this position is suppose to be 18 am I missing something here?
str = ('X-DSPAM-Confidence:0.8475')
atpos = str.find(':')
print atpos
sppos = str.find(' ',atpos)
print sppos
host = float(str[atpos + 1:sppos])
print host

str = ('X-DSPAM-Confidence:0.8475')
atpos = str.find(':')
sppos = str.find(' ',atpos)
host = float(str[atpos + 1:])
print (host)

You should rather use the string.split method than looking for the specific position of the : delimiter,
_str = 'X-DSPAM-Confidence:0.8475'
host = float(_str.split(':')[1])
print(host)

str = ('X-DSPAM-Confidence:0.8475')
atpos = str.find(':')
sppos = str.find(' ',atpos)
host = float(str[atpos + 1:])
print host

text = "X-DSPAM-Confidence: 0.8475";
spacePos = text.find(" ")
number = text[spacePos::1]
#not really necessary but since we are just learning and playing
strippedNumber = number.lstrip();
result = float(strippedNumber)
def reprint(printed):
print printed
reprint(result)

wpa-handshake with python - hashing difficulties

I try to write a Python program which calculates the WPA-handshake, but I have problems with the hashes. For comparison I installed cowpatty (to see where I start beeing wrong).
My PMK-generation works fine, but the PTK-calculation alsways seems to be wrong. I am not sure if I have to format my input (macadresses and noces) or just give them into the function as a string.
I will give you my routerinformation, which is no problem since I just set it up for testing.
My program looks as follows:
import hmac,hashlib,binascii
passPhrase = "10zZz10ZZzZ"
ssid = "Netgear 2/158"
A = "Pairwise key expansion"
APmac = "001e2ae0bdd0"
Clientmac = "cc08e0620bc8"
ANonce = "61c9a3f5cdcdf5fae5fd760836b8008c863aa2317022c7a202434554fb38452b"
SNonce = "60eff10088077f8b03a0e2fc2fc37e1fe1f30f9f7cfbcfb2826f26f3379c4318"
B = min(APmac,Clientmac)+max(APmac,Clientmac)+min(ANonce,SNonce)+max(ANonce,SNonce)
data="0103005ffe010900200000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
def customPRF512(key,A,B):
blen = 64
i = 0
R = ''
while i<=((blen*8+159)/160):
hmacsha1 = hmac.new(key,A+chr(0x00)+B+chr(i),sha)
i+=1
R = R+hmacsha1.digest()
return R[:blen]
pmk = pbkdf2(passPhrase, ssid, 4096, 32) #no sourcecode, since b2a_p(pmk) output fits to those of cowpatty
ptk = customPRF512(pmk,A,B) #the prf-function fits the pseudocode in the ieee, but does not give me the correct output (like cowpatty does)
# and i have no idea why :(
print b2a_p(pmk),"\n\n\n"
print b2a_p(ptk),"\n\n\n"
mic1 = hmac.new(ptk[0:16],data)
print mic1.hexdigest() #should be the mic-calculation, not sure if this is correct...
the desired outputs (which cowpatty confirmed) are:
PMK is
01b8 09f9 ab2f b5dc 4798 4f52 fb2d 112e
13d8 4ccb 6b86 d4a7 193e c529 9f85 1c48
Calculated PTK for "10zZz10ZZzZ" is
bf49 a95f 0494 f444 2716 2f38 696e f8b6
428b cf8b a3c6 f0d7 245a d314 a14c 0d18
efd6 38aa e653 c908 a7ab c648 0a7f 4068
2479 c970 8aaa abc3 eb7e da28 9d06 d535
Calculated MIC with "10zZz10ZZzZ" is
4528 2522 bc67 07d6 a70a 0317 a3ed 48f0
Maybe someone of you could tell me, why my program simply doesn't work. Do the hmac-functions work correctly? Is my input formatted wrong? Do I have to regard endianess anywhere? Thanks for your time in advance, I would appreciate any help!

Alright, I figured it out by myself... more by desperate testing and some luck, than successful research, which lead to nothing long enough. Instead of using the MAC-adresses and nonces as the strings they were, I had to unhexlify them. I used
a2b_hex() #alternatively unhexlify()
My final code looks somewhat like this, defs excluded:
import hmac,hashlib,binascii
passPhrase="10zZz10ZZzZ"
ssid = "Netgear 2/158"
A = "Pairwise key expansion"
APmac = a2b_hex("001e2ae0bdd0")
Clientmac = a2b_hex("cc08e0620bc8")
ANonce = a2b_hex("61c9a3f5cdcdf5fae5fd760836b8008c863aa2317022c7a202434554fb38452b")
SNonce = a2b_hex("60eff10088077f8b03a0e2fc2fc37e1fe1f30f9f7cfbcfb2826f26f3379c4318")
B = min(APmac,Clientmac)+max(APmac,Clientmac)+min(ANonce,SNonce)+max(ANonce,SNonce)
data = a2b_hex("0103005ffe01090020000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
pmk = pbkdf2(passPhrase, ssid, 4096, 32)
ptk = customPRF512(pmk,A,B)
mic = hmac.new(ptk[0:16],data)
print "desiredpmk:\t","01b809f9ab2fb5dc47984f52fb2d112e13d84ccb6b86d4a7193ec5299f851c48"
print "pmk:\t\t",b2a_hex(pmk),"\n"
print "desired ptk:\t","bf49a95f0494f44427162f38696ef8b6"
print "ptk:\t\t",b2a_hex(ptk[0:16]),"\n"
print "desired mic:\t","45282522bc6707d6a70a0317a3ed48f0"
print "mic:\t\t",mic.hexdigest(),"\n"
So the answers to my questions were: yes, hashfunctions work correctly, yes, input is formatted wrong, no, no endianess-issues.

Thanks for posting. This helped me out, so posting my revisions:
#==========================================================================================
#
# Verify the MIC code in EAPoL Message #2 is valid, or not (WPA2)
#
#==========================================================================================
#
# The home for this code is (so check for updates):
#
# https://www.duckware.com/tech/verify-mic-in-four-way-handshake.py.txt
#
# and this code is fully public, as it was based on/derived from this public code:
#
# https://stackoverflow.com/questions/12018920/wpa-handshake-with-python-hashing-difficulties
#
# 1. PMK: 'Pairwise Master Key' (256-bit) is generated from SSID/PASS in WPA2:
#
# o https://www.wireshark.org/tools/wpa-psk.html (SSID/PASS to PMK)
# o http://anandam.name/pbkdf2/ (Password-Based Key Derivation Function 2)
#
# 2. PRF512: The PRF-512 function is used to compute four 128-bit keys (KCK,KEK,TK1,TK2).
# For details on this function, see:
#
# o http://etutorials.org/Networking/802.11+security.+wi-fi+protected+access+and+802.11i/Part+II+The+Design+of+Wi-Fi+Security/Chapter+10.+WPA+and+RSN+Key+Hierarchy/Computing+the+Temporal+Keys/
#
# 3. KCK: The KCK (first 128 bits of the PTK; see above) are used to generate the MIC:
#
# o https://tldp.org/HOWTO/8021X-HOWTO/intro.html
#
# RUN: Run the code below in an ONLINE Python 2.7 compiler. For example:
#
# o https://repl.it/languages/python
# o https://www.tutorialspoint.com/execute_python_online.php
#
# CUSTOMIZE: How to customize the code below:
#
# 1) PCAP the problematic handshake (TIP: use tcpdump with ether host xx:xx:xx:xx:xx:xx)
# 2) Update SSID/PASS vars below with the known Wi-Fi name/password
# 3) Copy entire Ethernet frames for EAPoL Message #1/#2 into EAPOL1/2 vars below.
# TIP: In Wireshark, right click on Ethernet frame, 'Copy' / '...as a Hex Stream' / paste below
# 4) Use first with a working 4-way handshake (to confirm proper usage; MIC match), then apply
# to non-working 4-way handshake to confirm that the MIC in Message #2 is good/bad.
# 5) The code below, unmodified, results in a MIC found/calculated 'match'
#
# See also:
#
# o https://www.wifi-professionals.com/2019/01/4-way-handshake
# o https://stackoverflow.com/questions/15133797/creating-wpa-message-integrity-code-mic-with-python
# o https://www.shellvoide.com/wifi/understanding-wpa-wpa2-hash-mic-cracking-process-python/
# o https://ww.ins1gn1a.com/understanding-wpa-psk-cracking/
# o https://docs.python.org/3/library/binascii.html
# o https://stackoverflow.com/questions/9020843/how-to-convert-a-mac-number-to-mac-string
#
#==========================================================================================
import hmac,hashlib,binascii
def to_mac(addr): return ':'.join(addr[i:i+2] for i in range(0,len(addr),2))
def PRF_512(key,A,B): return ''.join(hmac.new(key,A+chr(0)+B+chr(i),hashlib.sha1).digest() for i in range(4))[:64]
def a2b(s): return binascii.a2b_hex(s);
def b2a(by): return binascii.b2a_hex(by);
EAPOL1 = a2b("60f189052d94a00460216606888e0203005f02008a00100000000000000001141f7a3ebdc0b51712934bef6e43ea13f80cb460f121f35408aa607046e239980000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
EAPOL2 = a2b("a0046021660660f189052d94888e0103007502010a000000000000000000015b46c7165f504c664aed90b78f3b705e02b4b029a67e3189d1632479d7e7a4e6000000000000000000000000000000000000000000000000000000000000000056de18f5efa272a4663560b73c537a65001630140100000fac040100000fac040100000fac028000")
SSID = "your-wifi-ssid"
PASS = "your-wifi-password"
PMK = hashlib.pbkdf2_hmac('sha1', PASS, SSID, 4096, 32)
VER_WPA = 2 # WPA2 means use 'SHA1'
XAUTH = a2b("888E")
if EAPOL1[0:6]==EAPOL2[6:12] and EAPOL2[0:6]==EAPOL1[6:12] and EAPOL1[12:14]==XAUTH and EAPOL1[12:14]==XAUTH:
if ord(EAPOL1[20])%8==VER_WPA and ord(EAPOL2[20])%8==VER_WPA:
R1 = EAPOL1[31:63] # random 1 (AP nonce)
R2 = EAPOL2[31:63] # random 2 (STA nonce)
M1 = EAPOL2[0:6] # MAC 1 (AP MAC)
M2 = EAPOL1[0:6] # MAC 2 (STA MAC)
# Generate KCK, KEK, TK1, TK2 from the PMK (and AP/STA info)
PTK = PRF_512(PMK,"Pairwise key expansion",min(M1,M2)+max(M1,M2)+min(R1,R2)+max(R1,R2))
KCK = PTK[0:16];
# try to validate the MIC in EAPoL message #2 is correct
MICRAW = hmac.new(KCK,EAPOL2[14:95]+a2b("00000000000000000000000000000000")+EAPOL2[111:],hashlib.sha1)
MICFOUND = b2a(EAPOL2[95:111])
MICCALC = MICRAW.hexdigest()[0:32]
print "SSID/PASS: ",SSID,"/",PASS
print "PMK: ",b2a(PMK)
print "AP-MAC: ",to_mac(b2a(M1))
print "STA-MAC: ",to_mac(b2a(M2))
print "AP-NONCE: ",b2a(R1)
print "STA-NONCE: ",b2a(R2)
print "KCK: ",b2a(KCK)
print "MIC-found: ",MICFOUND
print "MIC-calc: ",MICCALC
print "Result: ",("OK: EAPoL message #2 validated" if MICFOUND==MICCALC else "ERROR: MIC does not match")
else:
print "***ERROR: Did not find expected 'WPA2' version in EAPoL messages"
else:
print "***ERROR: Problem validated Ethernet frames. Do EAPOL1 and EAPOL2 both include the Ethernet headers?"

XMODEM for python

I am writing a program that requires the use of XMODEM to transfer data from a sensor device. I'd like to avoid having to write my own XMODEM code, so I was wondering if anyone knew if there was a python XMODEM module available anywhere?

def xmodem_send(serial, file):
t, anim = 0, '|/-\\'
serial.setTimeout(1)
while 1:
if serial.read(1) != NAK:
t = t + 1
print anim[t%len(anim)],'\r',
if t == 60 : return False
else:
break
p = 1
s = file.read(128)
while s:
s = s + '\xFF'*(128 - len(s))
chk = 0
for c in s:
chk+=ord(c)
while 1:
serial.write(SOH)
serial.write(chr(p))
serial.write(chr(255 - p))
serial.write(s)
serial.write(chr(chk%256))
serial.flush()
answer = serial.read(1)
if answer == NAK: continue
if answer == ACK: break
return False
s = file.read(128)
p = (p + 1)%256
print '.',
serial.write(EOT)
return True

There is XMODEM module on PyPi. It handles both sending and receiving of data with XModem. Below is sample of its usage:
import serial
try:
from cStringIO import StringIO
except:
from StringIO import StringIO
from xmodem import XMODEM, NAK
from time import sleep
def readUntil(char = None):
def serialPortReader():
while True:
tmp = port.read(1)
if not tmp or (char and char == tmp):
break
yield tmp
return ''.join(serialPortReader())
def getc(size, timeout=1):
return port.read(size)
def putc(data, timeout=1):
port.write(data)
sleep(0.001) # give device time to prepare new buffer and start sending it
port = serial.Serial(port='COM5',parity=serial.PARITY_NONE,bytesize=serial.EIGHTBITS,stopbits=serial.STOPBITS_ONE,timeout=0,xonxoff=0,rtscts=0,dsrdtr=0,baudrate=115200)
port.write("command that initiates xmodem send from device\r\n")
sleep(0.02) # give device time to handle command and start sending response
readUntil(NAK)
buffer = StringIO()
XMODEM(getc, putc).recv(buffer, crc_mode = 0, quiet = 1)
contents = buffer.getvalue()
buffer.close()
readUntil()

I think you’re stuck with rolling your own.
You might be able to use sz, which implements X/Y/ZMODEM. You could call out to the binary, or port the necessary code to Python.

Here is a link to XMODEM documentation that will be useful if you have to write your own. It has detailed description of the original XMODEM, XMODEM-CRC and XMODEM-1K.
You might also find this c-code of interest.

You can try using SWIG to create Python bindings for the C libraries linked above (or any other C/C++ libraries you find online). That will allow you to use the same C API directly from Python.
The actual implementation will of course still be in C/C++, since SWIG merely creates bindings to the functions of interest.

There is a python module that you can use -> https://pypi.python.org/pypi/xmodem
You can see the transfer protocol in http://pythonhosted.org//xmodem/xmodem.html