Python String Slicing with Paramiko Output - python

Hi I'm building a python app that will simplify managing multiple Raspberry Pi's on my network. The idea is that it will speed up basic tasks like updates on the Pi's by issuing the commands to the Pi via the paramiko python module. I can save commands to the application and then run them via a simple shortcut rather than having to login and type them out.
I've hit a bit of a stumbling block in that my commands are running and I'm getting the output but because of the way I'm using paramiko every time a command runs I'm getting the full console output i.e. all the stuff the appears when you login over SSH in my output. To slim that down and only display the output I'm interested in (the result of the command I've run). I'm trying to use the string slicing method.
My code will check the output initially for the presence of :~$ using string.find() as the first intstance it will find will be just before the command runs I'm using that as the start point for the slice. At the minute I'm only looking to get rid of the output before the :~$. I'll probably then pass the result of the first slice to a new string and then look for user# in that string as the end point for the slice.
The issue I'm having is that once I've sliced the string I get no output whatsoever. Sometimes I'll get an output and that does appear to be correct in that its what I'm expecting to happen but most of the time I'm getting no output and sometime if I check the length of the string it will come back to say there are a number of characters in there but I get no output when I try to print the variable (rem1).
Below is my code, I'm issuing a command and then looping the script repeatedly until I either pick up on a second :~$ in the output indicating the command has finised running or until there is nothing in the output variable. On each loop the contents of output is transfered to final_output which I use as the full string to manipulate at the end.
This may be a long winded way of doing what I want to do but honestly I couldn't think of another way round it as I kept running into issues when trying to use interactive commands like apt full-upgrade
def interactive_command(connect, ip_addr, port, uname, passwd, admin, issue_cmd):
affirmative = {'yes', 'y', 'Y', 'Yes', 'ye', 'Ye', ''}
connect.connect(ip_addr, port, uname, passwd)
channel = connect.invoke_shell()
channel.set_combine_stderr(True)
stdin = channel.makefile('wb')
channel.send(issue_cmd + '\n')
if admin != -1:
print('Sudo detected. Attempting to elevate privileges...')
channel.send(passwd + '\n')
else:
pass
fin_check = {':~$', ':~ $'}
ser_check = {'[Y/n]', '(Y/I/N/O/D/Z) [default=N] ?'}
counter = 0
prompt_count = 0
running = True
monitoring = False
final_output = ''
while running:
stdin
counter = counter + 1
#print('Loop Counter: ' + str(counter))
# allow for 10 while loops to be completed before raising an error.
if counter == 10:
print('Unable to establish whether the command has finished running')
running = False
print('Counter: ' + str(counter))
print('Prompt Count: ' + str(prompt_count))
#print('Prompt Count: ' + str(prompt_count))
sleep(2)
if channel.recv_ready():
output = channel.recv(65535).decode('utf-8')
else:
running = False
monitoring = False
print('No output detected. Printing last output.')
break
final_output = final_output + output
#print(len(output))
search = str(output)
finish = -1
if search == -1 or finish == -1:
for y in ser_check:
search = str(output).find(y)
if search != -1:
print(output)
resp = input('Response?: ') + '\n'
channel.send(resp)
running = False
monitoring = True
next
for x in fin_check:
if x in output:
prompt_count = prompt_count + 1
#print('Prompt Count: ' + str(prompt_count))
if prompt_count == 2:
finish = 1
break
else:
finish = -1
if finish != -1:
print('Command Finished')
#print('Prompt Count: ' + str(prompt_count))
running = False
if monitoring:
pass
else:
# print(output)
break
next
sleep(3)
while monitoring:
output = channel.recv(65535).decode('utf-8')
final_output = final_output + output
for x in fin_check:
if x in output:
prompt_count = prompt_count + 1
#print('Prompt Count: ' + str(prompt_count))
if prompt_count == 2:
finish = 1
else:
finish = -1
if finish != -1:
print('Command Finished')
#print('Prompt Count: ' + str(prompt_count))
running = False
monitoring = False
#print(output)
next
if prompt_count == 2:
print('Prompt Count Reached. Command Finished')
break
sleep(3)
print()
#print(final_output)
for c in fin_check:
start_point = final_output.find(c)
if start_point == -1:
pass
else:
#print(len(final_output))
#print(start_point)
start_point = start_point + len(c)
#print(start_point)
next
rem1 = final_output[start_point:]
print(len(rem1))
print(rem1)
print()
complete = input('Are you finished with this connection? ')
if complete in affirmative:
complete = 'Y'
return complete

Related

Problem Importing and Writing a Float to a Text File on Interval

I am making a Python program that simulates a stock market for Zer0 Coin (ZRC). Basically, every 2 seconds the program will get a new "token" that is usually around 0.8 to 1.2. The token is written to Token.txt every time it generates a new one to save it. When the program starts, it should import it for use.
I am having problems writing and importing the text file because it needs to be written as a float and imported as a float. I am new to text files in Python, so I kind of threw together a lot of file stuff and it didn't work. I would like to know how to import a float from a text file. I also need to know how to replace the previous text file number with a new one every 2 seconds. I tried to do this, and it kept writing a ton of spaces before the number, preventing it from importing as a float the next time I ran it.
It should be able to run multiple times, using the last "token" from the previous run.
The Token.py file will start out as just 1.0.
ComputerInvest.py File
import time
import random
import decimal
userUSD = 0
tokenInterval = 2
def user():
print('Running as User...\n')
userUSD = input('USD Input: ')
ZRC = int(userUSD)
print('USD: ' + str(userUSD) + ' is now equal to ZRC: ' + str(ZRC))
invest(ZRC)
def invest(ZRCl):
global token
seconds = int(input('\nRepeat for time (seconds): '))
print('Running Investment...\n\n')
tokenFW = open(r"Token.txt","r+")
start = time.time()
while time.time() - start < seconds:
time.sleep(int(tokenInterval))
upordown = random.randint(0,5)
step = float(decimal.Decimal(random.randrange(0,10000))/1000000000)
if upordown == 1 or upordown == 2:
token = token + step
elif upordown == 4 or upordown == 5:
token = token - step
elif upordown == 3:
pass
ZRCl = ZRCl * token
ZRCf = format(ZRCl, ".4f")
tokenFW.truncate(0)
tokenFW.write(str(token))
print('Token: ' + str(token))
print('New ZRC Amount: ' + str(ZRCf) + '\n')
print('Investment End')
print('Final New ZRC Amount: ' + ZRCf)
ZRC = ZRCf
USD = ZRC
tokenFW.close()
tokenF = open(r"Token.txt","r")
token = float(tokenF.read())
tokenF.close()
user()
I am also kind of new to StackOverflow, so if I did something wrong please tell me.
The problem is that you're writing the token multiple times in a loop while the file is still open and in use by the program. I'd suggest making use of a with statement instead of using the file close() method (which commits the changes/writing). The with statement will keep the file open only until the commands complete, then close and save automatically.
def invest(ZRCl):
global token
seconds = int(input('\nRepeat for time (seconds): '))
print('Running Investment...\n\n')
tokenFW = 'Token.txt'
start = time.time()
while time.time() - start < seconds:
time.sleep(int(tokenInterval))
upordown = random.randint(0,5)
step = float(decimal.Decimal(random.randrange(0,10000))/1000000000)
if upordown == 1 or upordown == 2:
token = token + step
elif upordown == 4 or upordown == 5:
token = token - step
elif upordown == 3:
pass
ZRCl = ZRCl * token
ZRCf = format(ZRCl, ".4f")
with open(tokenFW, 'w') as f:
f.write(str(token))
print('Token: ' + str(token))
print('New ZRC Amount: ' + str(ZRCf) + '\n')
print('Investment End')
print('Final New ZRC Amount: ' + ZRCf)
ZRC = ZRCf
USD = ZRC
with open(r'Token.txt', 'r') as f:
token = float(f.read())
user()

Benchmarking cache dictionary leads to "Unexpected EOF while reading bytes"

I have Clickhouse version 20.8.3.18 and python3 installed on a vm stress testing Cache dictionaries. After a certain number of entries the query using clickhouse_driver, I'll get the error
Unexpected EOF while reading bytes
Is this an error due to the driver/python related or due to the cache being maxed on the system. For example this happens on a file size 203 columns and 10000 rows on a machine with 32Gb of RAM and 256Gb of SSD memory, a csv file of around 66Mb which seems quite small for such an error. The query I'm running is:
SELECT
dictGet('CacheDictionary', 'date', toUInt64(number)) AS date,
SUM(dictGet('CacheDictionary', 'filterColumn', toUInt64(number))) AS val,
AVG(dictGet('CacheDictionary', 'filterColumn', toUInt64(number))) AS avg
FROM numbers(1, 10000)
GROUP BY date
An example entry of the csv file is:
20000,2021-02-05,6867,0.5314826651111791,OA9SMRN54LC3MTDW,D6S8AYXZ3JVSHPCY,12UQV1JR87MT00EP,3WBT23MA2QN6URA7,YGKJR5577BP6S3AD,2T90WPW1REOZA0L9,JQG8Z6FXXIX2788M,OAOVV1YX3A6HKQV8,FISBMOAHEXHAAKEY,XAULW5F90T3VEMUL,RAAZ5TM5XL7GRC1F,B16JEGDHXUXFI2R9,DETSZ7BR45CRAIA7,Z2X53PAQYCSBHPU3,SRISC0ZLWXC2DP34,KO2M3044JX5JCB74,ML776REFIX3Z1L78,ND6PXBOR135SWFSB,ZF4K45N2AIGFAK0L,RFE3EHCKC5EPYE2V,NJKM5T8UUD5NRDPX,O57IQW0670LP00I9,F0EBZ3BXHPETCFSY,RUZ7VH2IM0DIZ4UC,08BP467WG7ROEHTJ,9LSTNLUA240T2K4D,5L4PIRKMK746QW5Q,2VX3SER8ULU93NZG,Z0MZ9C3TTPR6WFDV,KB32XWCR67AWGSIB,PDM8QJ34X4EOTVN1,P7TUVP8Q1YF9S746,YDFDBCG6S2EXYPNW,55RN0F4UMGF3ABQZ,RRF895J8LQSLI48U,54OQWCJODIEQLRQF,D5ZJPGAG7CCO4LWA,UQDWEXPI184UUJQD,3QF6QAS32ITRL8JH,FPQ324RO04LNVAMO,ZJ6QCWNQCBQOE7F5,6OWVEVWHNSZILC6E,GIUD29OIFF3LUCCX,VGBJHKW32BUNUSDH,908TDRODVZIIC5O8,UCIU38BXEREJMO4M,5LKJ23ER4CKUZ88J,A1GBKPPM10L8X5RM,BB3SAVWF3CNBDXHO,279MIC1OXTDS2PFP,J6UVFJE8RGFK4LDN,3CE12GT27GX0WVWU,PNNTRLDFVJQ0TCRK,MI7XOHWUQX3W938H,LKZPV4K0BA6OE3R0,YJMLI82UBLSZWP7U,JORNKD1MSVECXBRF,CO5KKJIL1FHEYA11,GXVXWDOI538WCLC0,OPODB2R2ITSX0E6J,3VE7SOJZL3DKIES7,5LPXB17GJ94S86HL,UQ0DZVUDMBD39LC3,KSSVOBUKMZC7T89M,P6YL0WW22NOM5A36,RA46SZF4ZLO5YWUM,TUTMJ34X4040USXX,09HPKJAD58P3FVMP,DM0NJVFYKR2653HH,HP869NM4Y2EBE3ND,RVKP40RPBOPB6RPQ,WI3QXYA5XIWJUFUK,770L6U5KAEPKKJC1,2H0XNUDM41QBAZWB,8AWJ2Y7RB9F2WTT0,Y6T3PIPLU3FCBZCU,CY8SCO15RNUWQU2B,DRC88XH21J9ADT6Z,MLZ2JN7F8MXVBHBI,2YSUVHRL4V0EVHXF,Y0U12EBQSEVE6W6X,A6RRJY191S0JOXJH,4F12P4K0SJ6EDKSD,THCRJ2ZEXGM1RUM4,PF0OUAULUNIW0W9X,EK1249WXC0C2KKY8,11WEDAAJL7BL4T4U,4K8OP1WXSN1MIXPF,8D0WNN1672A6WK07,5RLYH7K00ZSR1LL2,EKEXBG87U1X6UOLL,YWK3V1F7MTAF9T19,XZ8ZF0XO5V8TCBPS,A3RX8X8A8I11Z8X3,77P2Q5WRSTL4ERAI,00BGNPDYFSVG5F81,5KTUM76C42VTP4I7,TA933GZZN8OQ20QJ,612WNQ74RDHMBWX3,D41HNOBPX11GFYWO,OGR4A0EPCSS00XL6,QIOH165Y5JGKJMFC,TF2R9TFC5TJN2PER,TYNXWI46H7I83O77,JMD5DOEV4U628SDK,D7ECJH43FEC77UCJ,FKA9AT5J20QI3MQP,7QSU0I8VRRLUMD7R,6OJ1O2XI2QJXP6W2,UD2QVJXNUFRCAO43,GS3TZUW8U6Z8EWWQ,QD79GBSO6D6GCAZ1,GQ5TUY2FMJSNMTRK,OGOYL2PD64E2DOOQ,Q733OU5P7J7SAFS1,GBS7MV5QOMQ4E89N,SB8MIQ1P37HMQZBJ,Z6G96BM7FL4150H3,05PS81HW528971RM,6F3KFLYT0345GI43,G65CDWEORNH3OUCY,12F43L99AZ84PDWR,GQQVWMTMS471WAWD,F1DFWRJ1F9M9MUTT,1M734H07IQAW49Q3,OPSRG5J7370227XE,BIPNR22KFF71MKQN,PV7DWGCQF5551FKT,YPGQVGUP37MRJY2B,RILKP96QV69WBW2D,4RXDCJURAVCQEGLX,XGIPC0AK1K0I6KDP,HMSE306L5NAK62LC,YAZHMS2UHGMWIB44,RZCAVUM45YTNV23T,3B7K07XPRTE8OMW1,FTP48ED5DQ4K3DM8,WW419RRJ2WU1F15L,85FWD49J0ARSUGI9,4U4768ANPCJ46K5P,EJ24BNUA6OZMUDEL,6Z27W6BN36GO8QWU,5AMZ4UU819GSI454,KMNIEJ2V5PI83KGP,APT4CYG8M5FM0BSW,IME5VRP08W468DZE,6BT4W0ZAW6C7993L,DRD6Q4P8BZVDG37U,2R1OEWQFV5J597AF,CKS41A6PXKVYICAG,OQYZ9UOQRVS3LLTF,JA3PZSAXFCJVZVLB,J23BP73T6GNC0Z08,GWOJXMXDVHCRE51Y,I826DE6KEVQK2PFC,6FF5LWM61KCM4C9K,P16P80EIX2X87OZO,O5GEOEO72CDV4GAX,UMKFUKMV6U0L5PM5,U64YI4G53LR3SC6J,CLML8KPAL697KYYJ,LMH2W0STEJ5H2J2S,AL61EP61ZR3GOPN3,Z3AEUMZSX4MQJ6M6,IS5RFEWIJ8XHYNK0,TNE1BS4JYN280PIF,67IER2YS6N2XHEW1,63P3O4X42T2INRT4,XYV043108XRK7Y4S,RW0HN600K0GQXF4Y,BZ1ZE6IBB4B72A81,QHAINYDIZX7838YI,7FFCKG3XJSZ2DIHJ,DF6C1OMPC1ETFPDZ,1EJ3EW0TXKVBC88R,WX6HG8FD021VFZ2S,W4OB9NZRODSTM96M,6GDA3L5CLBPVTPWQ,1Y4U7BL9UHPBJVIX,Y31SUUZ0JF2AXZWO,PL2I18PA0SVXG85E,TEY1HC97QMZ5YXMI,T49EVLLM43AI4OG3,0SDNMLWY85Z7NENX,4446QKGO8UL6RERT,IMEAM22I51GT4ZHY,HUCLC93NIUG0C5R0,5VPBRUUVMBXP7HJY,XCOOPM3JU5VHQ94T,3LRZGAF451G9XDIN,Y6VIN1E31NYRLA2N,RAROO2EM5Q9NJRG9,NUQ2QJ9M6T5KRCHK,WQKKQK8UBB30GRWI,20SOMMKD08FYAENW,1G9K4UFWAI8Q7Z8K,XLG898A4MQXZHVYR,FPT67A7VDLVZEWYH,6DQ6417FF07FORXZ,10RUAPY5KGAYBZZD
I've posted part of the code trying to find the maximum number of cache items stored, along with the queries executed for each. In selectBenchmark the string correspond to the query above. The parameters for each are fairly self explanatory (the xmlFile is the dictionary created in /etc/lib/clickhouse-server).
def cacheMaxItems(csvRead, xmlFile, benchmarkType, columnStepSize, rowStepSize):
maxCache = []
os.system('rm -f ' + csvRead)
os.system('bash /root/restartCH.sh')
for j in range(1, 13):
outputCSV = '/root/results' + benchmarkType + '/cacheResults' + str(j*columnStepSize) + '.csv'
with open(outputCSV, 'w') as fp:
wr = csv.writer(fp)
wr.writerow([benchmarkType + ': Number of rows', 'Loading time', 'Mean', 'Variance', 'Skewness', 'Number of Columns: ' + str(j*columnStepSize)])
for i in range(1, 10000):
if i%5 == 0:
os.system('bash /root/restartCH.sh')
createCSV(10000, j*columnStepSize, csvRead)
try:
clickhouseDictionary(rowStepSize*i*j*columnStepSize, j*columnStepSize, xmlFile, csvRead, 'Cache')
if benchmarkType == 'Random':
results = selectBenchmark(i*rowStepSize, j*columnStepSize, 'Random', 'Cache')
elif benchmarkType == 'Consecutive':
results = selectBenchmark(i*rowStepSize, j*columnStepSize, 'Consecutive', 'Cache')
elif benchmarkType == 'CPU':
results = selectBenchmark(i*rowStepSize, j*columnStepSize, 'CPU', 'Cache')
results.insert(0, i*rowStepSize)
with open(outputCSV, 'a') as fp:
wr = csv.writer(fp)
wr.writerow(results)
print('Successfully loaded and queried cache of size ' + str(rowStepSize*i*j*columnStepSize) + '.')
except Exception as ex:
print(ex)
os.system('rm -f ' + csvRead)
os.system('bash /root/restartCH.sh')
maxCache.append([j*columnStepSize, (i-1)*rowStepSize])
print(maxCache)
break
return maxCache
def selectBenchmark(numberOfRows, numberOfColumns, benchmarkType, dictType):
client = Client('localhost', port=9000, database='system')
client.execute('SYSTEM RELOAD DICTIONARY ' + dictType + 'Dictionary')
loadingTime = client.last_query.elapsed
client.execute('SELECT dictGet(\'' + dictType + 'Dictionary\', \'random0\', toUInt64(1))', query_id=str(uuid.uuid4()))
loadingTime += client.last_query.elapsed
loop = True
counter = 0
j=0
while loop:
times = []
for i in range(0, 31):
query_id = str(uuid.uuid4())
string = stringGen(numberOfRows, numberOfColumns, benchmarkType, dictType)
client.execute(string, query_id = query_id)
times.append(client.last_query.elapsed)
if max(times) > loadingTime:
loadingTime = max(times)
stats = transformedMLE(times)
redactedTimes = [x for x in times if (stats[0]-3*np.sqrt(stats[1])) < x < (stats[0]+3*np.sqrt(stats[1]))]
if len(times) - len(redactedTimes) <= 3:
loop = False
elif j > 15:
print('High variance query')
loop = False
j+=1
result = transformedMLE(redactedTimes)
loadingTime = loadingTime - result[0]
result.insert(0, loadingTime)
client.disconnect()
return result
The restartCH.sh file is
service clickhouse-server forcerestart
as the cache overflow often blocks the restart command.
There is no output to the server error logs indicating that this is a problem with the python driver, perhaps reading the large amounts of data being returned. I also get the 'Killed' python output which also points towards cache issues, which is to be expected as I'm benchmarking cache dictionaries.
Unexpected EOF while reading bytes -- it's python driver error.
Check clickhouse-server.log for real error.
20.8.3.18 is out support , please upgrade to 20.8.12.2
I was running into a similar problem on Ubuntu when starting the server binary directly using "2>&1 /dev/null &" to suppress the output from stderr and stdout to /dev/null, Python driver was throwing the error but server would still be working when connecting via the clickhouse client binary command-line. Issue was resolved by tweaking the server startup script to just redirect stderr with " 2> /dev/null &" (referring to https://www.baeldung.com/linux/silencing-bash-output difference between using 2> and 2>&1).

python file based program taking time to execute

I have created below Script processing two files based on user input and generating third result file.
Scripts executes properly without any issue but when both file having high count then it is taking time. During my testing i have tested with InputFile-1 having 500000 records and InputFile-2 having 100 records.
So wanted to check if there is any possibility of optimization reducing overall execution time. Kindly share your thoughts.!
Thanks in advance.
import ipaddress
filePathName1 = raw_input('InputFile-1 : ')
filePathName2 = raw_input('InputFile-2: ')
ipLookupResultFileName = filePathName1 + ' - ResultFile.txt'
ipLookupResultFile = open(ipLookupResultFileName,'w+')
with open(filePathName1,'r') as ipFile:
with open(filePathName2,'r') as ipCidrRangeFile:
for everyIP in ipFile:
ipLookupFlag = 'NONE'
ipCidrRangeFile.seek(0)
for everyIpCidrRange in ipCidrRangeFile:
if (ipaddress.IPv4Address(unicode(everyIP.rstrip('\n'))) in ipaddress.ip_network(unicode(everyIpCidrRange.rstrip('\n')))) == True:
ipLookupFlag = 'True'
break
if ipLookupFlag == 'True':
ipLookupResultFile.write(everyIP.rstrip('\n') + ' - Valid_Operator_IP' + '\n')
else:
ipLookupResultFile.write(everyIP.rstrip('\n') + ' - Not_Valid_Operator_IP' + '\n')
ipFile.close()
ipCidrRangeFile.close()
ipLookupResultFile.close()
Sample records for InputFile-1:
192.169.0.1
192.169.0.6
192.169.0.7
Sample records for InputFile-2:
192.169.0.1/32
192.169.0.6/16
255.255.255.0/32
255.255.255.0/16
192.169.0.7/32
Sample records for ResultFile.txt:
192.169.0.1 - Not_Valid_Operator_IP
192.169.0.6 - Valid_Operator_IP
192.169.0.7 - Not_Valid_Operator_IP
A better approach is to load and process each file once, and then use this data to do the processing:
filePathName1 = raw_input('InputFile-1 : ')
filePathName2 = raw_input('InputFile-2: ')
ipLookupResultFileName = filePathName1 + ' - ResultFile2.txt'
with open(filePathName1) as ipFile:
ip_addresses = [unicode(ip.strip()) for ip in ipFile]
with open(filePathName2) as ipCidrRangeFile:
ip_cidr_ranges = [unicode(cidr.strip()) for cidr in ipCidrRangeFile]
with open(ipLookupResultFileName,'w+') as ipLookupResultFile:
for ip_address in ip_addresses:
ipLookupFlag = False
for cidr_range in ip_cidr_ranges:
if ipaddress.IPv4Address(ip_address) in ipaddress.ip_network(cidr_range):
ipLookupFlag = True
break
if ipLookupFlag:
ipLookupResultFile.write("{} - Valid_Operator_IP\n".format(ip_address))
else:
ipLookupResultFile.write("{} - Not_Valid_Operator_IP\n".format(ip_address))
Note, using with() means you do not need to explicitly close the file afterwards.
Depending on your needs, a further speed improvement could be made by removing any duplicate ip_addresses. This could be done by loading the data into a set() rather than a list, for example:
ip_addresses = set(unicode(ip.strip()) for ip in ipFile)
You could also sort your results by IP address before writing them to a file.
The starting point is to focus on is that for every line in ipFile you re-read ipCidrRangeFile. Instead read the ipCidrRangeFile into a list or some other collection and read from there in the loop.
with open(filePathName2,'r') as ipCidrRangeFile:
ipCidrRangeList = ipCidrRangeFile.readlines()
with open(filePathName1,'r') as ipFile:
with open(filePathName2,'r') as ipCidrRangeFile:
for everyIP in ipCidrRangeList :
ipLookupFlag = 'NONE'
ipCidrRangeFile.seek(0)
for everyIpCidrRange in ipCidrRangeFile:
if (ipaddress.IPv4Address(unicode(everyIP.rstrip('\n'))) in ipaddress.ip_network(unicode(everyIpCidrRange.rstrip('\n')))) == True:
ipLookupFlag = 'True'
break
if ipLookupFlag == 'True':
ipLookupResultFile.write(everyIP.rstrip('\n') + ' - Valid_Operator_IP' + '\n')
else:
ipLookupResultFile.write(everyIP.rstrip('\n') + ' - Not_Valid_Operator_IP' + '\n')

Python: For loop only loops over the first part of a txt file

Recently I had to make a script for my internship to check if a subnet occurs in a bunch of router/switch configs.
I've made a script that generates the output. Now I need a second script (I couldn't get it to work into one), that reads the output, if the subnet occurs write it to aanwezig.txt, if not write to nietAanwezig.txt.
A lot of other answers helped me to make this script and it works but it only executes for the first 48 IPs and there are over 2000...
The code of checkOutput.py:
def main():
file = open('../iprangesclean.txt', 'rb')
aanwezig = open('../aanwezig.txt', 'w')
nietAanwezig = open('../nietAanwezig.txt', 'w')
output = open('output.txt', 'rb')
for line in file:
originalLine = line
line.rstrip()
line = line.replace(' ', '')
line = line.replace('\n', '')
line = line.replace('\r', '')
one,two,three,four = line.split('.')
# 3Byte IP:
ipaddr = str(one) + "." + str(two) + "." + str(three)
counter = 1
found = 0
for lijn in output:
if re.search("\b{0}\b".format(ipaddr),lijn) and found == 0:
found = 1
else:
found = 2
print counter
counter= counter + 1
if found == 1:
aanwezig.write(originalLine)
print "Written to aanwezig"
elif found == 2:
nietAanwezig.write(originalLine)
print "Written to nietAanwezig"
found = 0
file.close()
aanwezig.close()
nietAanwezig.close()
main()
The format of iprangesclean.txt is like following:
10.35.6.0/24
10.132.42.0/24
10.143.26.0/24
10.143.30.0/24
10.143.31.0/24
10.143.32.0/24
10.35.7.0/24
10.143.35.0/24
10.143.44.0/24
10.143.96.0/24
10.142.224.0/24
10.142.185.0/24
10.142.32.0/24
10.142.208.0/24
10.142.70.0/24
and so on...
Part of output.txt (I can't give you everything because it has user information):
*name of device*.txt:logging 10.138.200.100
*name of device*.txt:access-list 37 permit 10.138.200.96 0.0.0.31
*name of device*.txt:access-list 38 permit 10.138.200.100
*name of device*.txt:snmp-server host 10.138.200.100 *someword*
*name of device*.txt:logging 10.138.200.100
Try this change:
for lijn in output:
found = 0 # put this here
if re.search("\b{0}\b".format(ipaddr),lijn) and found == 0:
found = 1
else:
found = 2
print counter
counter= counter + 1
"""Indent one level so it us in the for statement"""
if found == 1:
aanwezig.write(originalLine)
print "Written to aanwezig"
elif found == 2:
nietAanwezig.write(originalLine)
print "Written to nietAanwezig"
If I understand the problem correctly, this should guide you to the right direction. The if statement is currently not executed in the for statement. If this does solve your problem, then you don't need the found variable either. You can just have something like:
for counter, lijn in enumerate(output, 1):
if re.search("\b{0}\b".format(ipaddr),lijn):
aanwezig.write(originalLine)
print "Written to aanwezig"
else:
nietAanwezig.write(originalLine)
print "Written to nietAanwezig"
print counter
Please let me know if I have misunderstood the question.
Note I haven't tested the code above, try them out as a starting point.

Python program crashing

So I've designed a program that runs on a computer, looks for particular aspects of files that have been plaguing us, and deletes the files if a flag is passed. Unfortunately the program seems to be almost-randomly shutting down/crashing. I say almost-randomly, because the program always exits after it deletes a file, though it will commonly stay up after a success.
I've run a parallel Python program that counts upwards in the same intervals, but does nothing else. This program does not crash/exit, and stays open.
Is there perhaps a R/W access issue? I am running the program as administrator, so I'm not sure why that would be the case.
Here's the code:
import glob
import os
import time
import stat
#logging
import logging
logging.basicConfig(filename='disabledBots.log')
import datetime
runTimes = 0
currentPhp = 0
output = 0
output2 = 0
while runTimes >= 0:
#Cycles through .php files
openedProg = glob.glob('*.php')
openedProg = openedProg[currentPhp:currentPhp+1]
progInput = ''.join(openedProg)
if progInput != '':
theBot = open(progInput,'r')
#Singles out "$output" on this particular line and closes the process
readLines = theBot.readlines()
wholeLine = (readLines[-4])
output = wholeLine[4:11]
#Singles out "set_time_limit(0)"
wholeLine2 = (readLines[0])
output2 = wholeLine2[6:23]
theBot.close()
if progInput == '':
currentPhp = -1
#Kills the program if it matches the code
currentTime = datetime.datetime.now()
if output == '$output':
os.chmod(progInput, stat.S_IWRITE)
os.remove(progInput)
logging.warning(str(currentTime) +' ' + progInput + ' has been deleted. Please search for a faux httpd.exe process and kill it.')
currentPhp = 0
if output2 == 'set_time_limit(0)':
os.chmod(progInput, stat.S_IWRITE)
os.remove(progInput)
logging.warning(str(currentTime) +' ' + progInput + ' has been deleted. Please search for a faux httpd.exe process and kill it.')
currentPhp = 0
else:
currentPhp = currentPhp + 1
time.sleep(30)
#Prints the number of cycles
runTimes = runTimes + 1
logging.warning((str(currentTime) + ' botKiller2.0 has scanned '+ str(runTimes) + ' times.'))
print('botKiller3.0 has scanned ' + str(runTimes) + ' times.')
Firstly, it'll be hell of a lot easier to work out what's going on if you base your code around something like this...
for fname in glob.glob('*.php'):
with open(fname) as fin:
lines = fin.readlines()
if '$output' in lines[-4] or 'set_time_limit(0)' in lines[0]:
try:
os.remove(fname)
except IOError as e:
print "Couldn't remove:", fname
And err, that's not actually a secondly at the moment, your existing code is just too tricky to follow fullstop, let alone all the bits that could cause a strange error that we don't know yet!
if os.path.exists(progInput):
os.chmod(progInput, stat.S_IWRITE)
os.remove(progInput)
ALSO:
You never reset the output or output2 variables in the loop?
is this on purpose?

Categories

Resources