I'm parsing data from a text file ('placlog.txt') that is continuously being updated. As I run the code everything prints as expected, but if there are any updates to the placlog file while the code is running it is not printed.
The placlog file is being updated by a third-party program and I am using the above code to read the file and print any updates.
Once formatted, the text should be sent via a Telegram API. This part is also working initially.
import urllib.parse
import time
import requests
import os
def post_to_telegram(msg):
#print(msg)
base_url = 'https://api.telegram.org/bot&text="{}'.format(msg)
requests.get(base_url)
def check_url_inMsgList(stringToMatch, msgList):
for i in msgList:
if (stringToMatch in i):
return False
return True
try:
f = open("oldFile.txt", "r")
msgList = f.read().split("\n")
f.close()
except:
f = open("oldFile.txt", "w")
msgList = []
f.close()
selections = []
urr = ""
name = ""
pie = ""
ourLines = 2400
url_found = 0
name_found = 0
pie_found = 0
while (True):
file1 = open('placlog.txt', 'r')
Lines = file1.readlines()
file1.close()
while (True):
# print("-------------------------------")
if (ourLines == len(Lines)):
break
elif (ourLines > len(Lines)):
ourLines = 0
else:
txt = Lines[ourLines].strip()
tlist = txt.split("&")
ourLines = ourLines + 1
for subtxt in tlist:
if "eventurl=" in subtxt:
a = subtxt[9:len(subtxt) - 3]
url = "www.awebsite.com/%23" + a.replace("%23", "/")
#url = url.replace("%23", "#")
for i in range(10):
if "F" + str(i) + "/" in url:
url = url.split("F" + str(i) + "/")[0] + "F" + str(i) + "/"
urr = url
url_found = 1
elif "bit=" in subtxt:
name = urllib.parse.unquote(subtxt[4:len(subtxt)])
name_found = 1
elif "pie\":" in subtxt:
a = subtxt.split("price")[1]
pie = a.split("\"")[2]
pie = float(pie)
pie = round(pie, 1)
pie = str(pie)
pie_found = 1
selections.append(url + name + pie)
msg = (url + " " + name + " " + pie)
stringToFind = url + " " + name
if (check_url_inMsgList(stringToFind, msgList)):
post_to_telegram(msg)
msgList.append(msg)
print(msg)
f = open("oldFile.txt", "a+")
f.write(msg + "\n")
f.close()
time.sleep(0.5)
elif "minodds=" in subtxt:
a = subtxt.split("minodds=")[1]
pie = a.split("&")[0]
pie = float(pie)
rie = round(pie, 1)
pie = str(pie)
pie_found = 1
selections.append(url + name + pie)
msg = (url + " " + name + " " + pie)
stringToFind = url + " " + name
if (check_url_inMsgList(stringToFind, msgList)):
post_to_telegram(msg)
msgList.append(msg)
print(msg)
f = open("oldFile.txt", "a+")
f.write(msg + "\n")
f.close()
time.sleep(0.5)
time.sleep(1)
I would recommend using watchdog, and seeing if that helps your situation. It can monitor for file system changes, so you could define a function which is executed when the placlog.txt file is changed/updated.
A good guide can be found here: http://thepythoncorner.com/dev/how-to-create-a-watchdog-in-python-to-look-for-filesystem-changes/
From that guide, you can simply change the functions defined to suit your needs i.e.
def on_modified(event):
if event.src_path == "path/to/placlog.txt":
with open('placlog.txt', 'r') as placlog:
lines = file1.readlines()
Could you try this out and see if it helps? I still recommend the with statement for file i/o since you always want your file to close no matter what.
This link might also be useful since they are also monitoring a single .txt file: Python Watchdog - src_path inconsistent
watchdog documentation: https://pythonhosted.org/watchdog/
Note: Deleted the old answer since you clarified the question.
Related
I made a program which has one function. This function has a file as an input and function writes result to output. I need to test if the my result is the same as expected. Below you can find a code of a program:
import os
def username(input):
with open(input, 'r') as file:
if os.stat(input).st_size == 0:
print('File is empty')
else:
print('File is not empty')
for line in file:
count = 1
id, first, middle, surname, department = line.split(":")
first1 = first.lower()
middle1 = middle.lower()
surname1 = surname.lower()
username = first1[:1] + middle1[:1] + surname1
username1 = username[:8]
if username1 not in usernames:
usernames.append(username1)
data = id + ":" + username1 + ":" + first + ":" + middle + ":" + surname + ":" + department
else:
username2 = username1 + str(count)
usernames.append(username2)
data = id + ":" + username2 + ":" + first + ":" + middle + ":" + surname + ":" + department
count += 1
with open("output.txt", "a+") as username_file:
username_file.write(data)
usernames = []
if __name__ == '__main__':
username("input_file1.txt")
username("input_file2.txt")
username("input_file3.txt")
with open("output.txt", "a+") as username_file:
username_file.write("\n")
How do I write an unittest on this type of program? I tried this but it gave me this error "TypeError: expected str, bytes or os.PathLike object, not _io.TextIOWrapper" . Code of my test is below:
import unittest
import program.py
class TestProgram(unittest.TestCase):
def test_username(self):
i_f = open("input_file1.txt", 'r')
result = program.username(i_f)
o_f = open("expected_output.txt", 'r')
self.assertEqual(result, o_f)
if __name__ == '__main__':
unittest.main()
I would be really happy if you could help me!!!
You didn't read the file, just pass the IO object.
Edit the TestProgram and add .read()
class TestProgram(unittest.TestCase):
def test_username(self):
i_f = open("input_file1.txt", 'r').read() # read the file
result = program.username(i_f)
o_f = open("expected_output.txt", 'r').read()
self.assertEqual(result, o_f)
you can also use with-as for automatic closing the file.
Hi everyone I'm having a problem with my script.
Basically it creates no account and doesn't write them in my .txt file(when I open it,it's empty) but it prints emails and password.
Could you please help me?
I expect to have some working accounts and read them in my .txt file
Hope you can help me,thanks in advance
P.S: data_gen is a dict that contains the infos for the post request (registration)
times = int(input("[" + (time.strftime("%H:%M:%S") + "]" + " - How many accounts?: ")))
Nome_completo = "testname"
Nome = Nome_completo.split()[0]
email = "test+{}#gmail.com".format(getrandbits(8))
Giorno_compleanno = random.randint(10, 27)
Mese_compleanno = random.randint(0, 12)
Anno = random.randint(1982, 2003)
Data_completa = str(str(Giorno_compleanno) + "/" + str(Mese_compleanno) + "/" + str(Anno))
password_1 = "test_1234"
s = requests.session()
r_1 = s.get(awlab_url,headers=headers)
r = s.get(url, headers=headers)
text_file = open("awlabaccounts", "r+")
def create():
global email
global password_1
if str(200) in str(r.status_code):
text_file.write(email + ":" + password_1 + "\n")
print("Accounts successfully created!")
else:
print("ERROR!")
reg = s.post(url_post, data=data_gen, headers=headers)
for i in range(times):
create()
You need to close the text file at the end of the code.
for i in range(times):
create()
text_file.close()
Hello Community Members,
I am getting the error NameError: name 'f' is not defined. The code is as follows. Please help. Any sort of help is appreciated. I have been strucked onto this since 3 days. The code is all about to extract all the subcategories name of wikipedia category in Python 3.
I have tried both the relative and absolute paths.
The code is as follows:
import httplib2
from bs4 import BeautifulSoup
import subprocess
import time, wget
import os, os.path
#declarations
catRoot = "http://en.wikipedia.org/wiki/Category:"
MAX_DEPTH = 100
done = []
ignore = []
path = 'trivial'
#Removes all newline characters and replaces with spaces
def removeNewLines(in_text):
return in_text.replace('\n', ' ')
# Downloads a link into the destination
def download(link, dest):
# print link
if not os.path.exists(dest) or os.path.getsize(dest) == 0:
subprocess.getoutput('wget "' + link + '" -O "' + dest+ '"')
print ("Downloading")
def ensureDir(f):
if not os.path.exists(f):
os.mkdir(f)
# Cleans a text by removing tags
def clean(in_text):
s_list = list(in_text)
i,j = 0,0
while i < len(s_list):
#iterate until a left-angle bracket is found
if s_list[i] == '<':
if s_list[i+1] == 'b' and s_list[i+2] == 'r' and s_list[i+3] == '>':
i=i+1
print ("hello")
continue
while s_list[i] != '>':
#pop everything from the the left-angle bracket until the right-angle bracket
s_list.pop(i)
#pops the right-angle bracket, too
s_list.pop(i)
elif s_list[i] == '\n':
s_list.pop(i)
else:
i=i+1
#convert the list back into text
join_char=''
return (join_char.join(s_list))#.replace("<br>","\n")
def getBullets(content):
mainSoup = BeautifulSoup(contents, "html.parser")
# Gets empty bullets
def getAllBullets(content):
mainSoup = BeautifulSoup(str(content), "html.parser")
subcategories = mainSoup.findAll('div',attrs={"class" : "CategoryTreeItem"})
empty = []
full = []
for x in subcategories:
subSoup = BeautifulSoup(str(x))
link = str(subSoup.findAll('a')[0])
if (str(x)).count("CategoryTreeEmptyBullet") > 0:
empty.append(clean(link).replace(" ","_"))
elif (str(x)).count("CategoryTreeBullet") > 0:
full.append(clean(link).replace(" ","_"))
return((empty,full))
def printTree(catName, count):
catName = catName.replace("\\'","'")
if count == MAX_DEPTH : return
download(catRoot+catName, path)
filepath = "categories/Category:"+catName+".html"
print(filepath)
content = open('filepath', 'w+')
content.readlines()
(emptyBullets,fullBullets) = getAllBullets(content)
f.close()
for x in emptyBullets:
for i in range(count):
print (" "),
download(catRoot+x, "categories/Category:"+x+".html")
print (x)
for x in fullBullets:
for i in range(count):
print (" "),
print (x)
if x in done:
print ("Done... "+x)
continue
done.append(x)
try: printTree(x, count + 1)
except:
print ("ERROR: " + x)
name = "Cricket"
printTree(name, 0)
The error encountered is as follows.
I think f.close() should be content.close().
It's common to use a context manager for such cases, though, like this:
with open(filepath, 'w+') as content:
(emptyBullets,fullBullets) = getAllBullets(content)
Then Python will close the file for you, even in case of an exception.
(I also changed 'filepath' to filepath, which I assume is the intent here.)
The program is intended to display a map with pins showing the locations of institutions that make use of one of our facilities. The program takes a csv file, reads the postcodes, geocodes them and places the pins on the map. The size of the pins is relative to the number of times they have used the facility.
However, when the csv file is uploaded the program generates a map with all the pins over Nigeria. Looking through the output from the program, it seems to be geocoding correctly so I am not sure what is going on. The program geocodes using an offline database as python's urllib is not compatible with the proxy setup at my office.
The program is split into two separate modules, the map generation module and the geocoding module.
Here is the map generation part:
import folium
from bottle import route, run, template, static_file, request
import urllib.request
import urllib.parse
import json
import os
os.system('start geocoder.bat')
institutionList = []
map_osm = folium.Map(location=[55, -2], zoom_start=5)
#route('/spreadsheet.html')
def send_static():
return static_file('spreadsheet.html',root='')
#route('/upload', method='POST')
def do_upload():
category = request.forms.get('category')
upload = request.files.get('upload')
name, ext = os.path.splitext(upload.filename)
if ext not in ('.csv'):
return 'File extension not allowed.'
upload.save('uploads/' + upload.filename)
fileList = []
with open('spreadsheetList','r') as f:
while True:
line = f.readline()
if not line: break
print(line.strip())
print("line should have just printed")
fileList.append(line.strip())
f.close()
lengthFileList = len(fileList)
x = 0
while x < lengthFileList:
with open(('uploads/' + fileList[x]),'r') as spread:
while True:
line = spread.readline()
if not line: break
institutionDetails = line.split(',')
institutionList.append(institutionDetails)
spread.close()
x = x + 1
spreadsheetName = upload.filename
f = open('spreadsheetList','a')
f.write(spreadsheetName + '\n')
f.close()
with open('uploads/' + spreadsheetName, 'r') as f:
while True:
line = f.readline()
if not line: break
institutionDetails = line.split(',')
institutionList.append(institutionDetails)
print(institutionList)
f.close()
lengthOfList = len(institutionList)
x = 0
coords = []
while x < lengthOfList:
address = urllib.parse.quote_plus(institutionList[x][1])
response = urllib.request.urlopen('http://localhost:80/geoCodeRequest/' + address).read().decode('utf-8')
cleanResponse = str(response).replace('"','')
coords = cleanResponse
print(cleanResponse)
institutionList[x].append(coords)
x = x + 1
print("http sources successfully accessed")
print(institutionList)
x = 0
while x < lengthOfList:
try:
map_osm.circle_marker(location=institutionList[x][3], radius=(int(institutionList[x][2]) * 10),popup=institutionList[x][0], line_color='#3186cc',fill_color='#3186cc', fill_opacity=0.2)
print("marker added")
except:
print("marker could not be added")
x = x + 1
map_osm.create_map(path='osm.html')
return '<meta http-equiv="refresh" content="0; url=osm.html">'
#route('/osm.html')
def send_static():
return static_file('osm.html',root='')
run(host='localhost', port=8080)
A batch file is used to start the second module:
#echo off
python geocodeProxyBypass.py
Here is the second module of code, the geocoding module:
from bottle import route, run, template
import string
location = []
x = 0
#route('/geoCodeRequest/<name>')
def redir(name):
x = 0
print(name)
print(name[:4])
print(name[:3])
with open('ukPostcode.csv','r') as f:
while True:
line = f.readline()
print(line)
if not line: break
locationDetails = line.split(',')
location.append(locationDetails)
print(location[x][0])
if location[x][0] == ('"' + name[:4] + '"'):
coords = location[x][3] + ", " + location[x][4]
return coords
elif location[x][0] == ('"' + name[:3] + '"'):
coords = location[x][3] + ", " + location[x][4]
return ((coords.replace('"','')))
else:
print("no match found for " + name)
x = x + 1
f.close()
run(host='localhost', port=80)
Here is what an example pin generated by the program should look like:
var circle_1 = L.circle([51.74, -1.25
], 7460, {
color: '#3186cc',
fillColor: '#3186cc',
fillOpacity: 0.2
});
circle_1.bindPopup("University of Oxford");
circle_1._popup.options.maxWidth = 300;
map.addLayer(circle_1)
Here is what is actually being output:
var circle_1 = L.circle([5, 1
], 7460, {
color: '#3186cc',
fillColor: '#3186cc',
fillOpacity: 0.2
});
circle_1.bindPopup("University of Oxford");
circle_1._popup.options.maxWidth = 300;
map.addLayer(circle_1)
Apologies for the very long question, please help!
I successfully simplified a python module that imports data from a spectrometer
(I'm a total beginner, somebody else wrote the model of the code for me...)
I only have one problem: half of the output data (in a .csv file) is surrounded by brackets: []
I would like the file to contain a structure like this:
name, wavelength, measurement
i.e
a,400,0.34
a,410,0.65
...
but what I get is:
a,400,[0.34]
a,410,[0.65]
...
Is there any simple fix for this?
Is it because measurement is a string?
Thank you
import serial # requires pyserial library
ser = serial.Serial(0)
ofile = file( 'spectral_data.csv', 'ab')
while True:
name = raw_input("Pigment name [Q to finish]: ")
if name == "Q":
print "bye bye!"
ofile.close()
break
first = True
while True:
line = ser.readline()
if first:
print " Data incoming..."
first = False
split = line.split()
if 10 <= len(split):
try:
wavelength = int(split[0])
measurement = [float(split[i]) for i in [6]]
ofile.write(str(name) + "," + str(wavelength) + "," + str(measurement) + '\n')
except ValueError:
pass # handles the table heading
if line[:3] == "110":
break
print " Data gathered."
ofile.write('\n')
do this:
measurement = [float(split[i]) for i in [6]]
ofile.write(str(name) + "," + str(wavelength) + "," + ",".join(measurement) + '\n')
OR
ofile.write(str(name) + "," + str(wavelength) + "," + split[6] + '\n')