I have a python script that reads from a logfile and outputs certain data from it. The way it reads from it is
try:
with open(os.path.expandvars('Path/To/My/Log.txt', 'r') as f:
logContent = [line.rstrip() for line in f]
except Exception as e:
print(e)
Now I wanted to recreate that python script in C#. The main problem is, that the log file makes about 30.000 Lines in 30 minutes. While the program that handles that log isn't being executed, I can easily open the file and read from it, because it's not being used by that program. But when that program runs, I need to read from the file with a filestream, and so the reading of 30.000 lines takes ages:
private string GetLog(string path)
{
string log = "";
FileStream reader = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
StreamReader logFileReader = new StreamReader(reader);
while (!logFileReader.EndOfStream)
{
log += logFileReader.ReadLine();
// Your code here
}
// Clean up
logFileReader.Close();
reader.Close();
return log;
}
Is there a way to make my code read from the file in max 5 seconds?
I got it. When I use stream.ReadToEnd() it reads everything in about 2 seconds
As you have mentioned file is big, so better to use StringBuilder over string, you can use using also so no need to call close() explicitly.
StringBuilder sb = new StringBuilder();
string path = "some path";
using (FileStream logFileStream = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
using (StreamReader logFileReader = new StreamReader(logFileStream))
{
while (!logFileReader.EndOfStream)
{
sb.Append(logFileReader.ReadLine());
}
}
}
string log = sb.ToString();
I added a line in the python code “speedtest.py” that I found at pimylifeup.com. I hoped it would allow me to track the internet provider and IP address along with all the other speed information his code provides. But when I execute it, the code only grabs the next word after the find all call. I would also like it to return the IP address that appears after the provider. I have attached the code below. Can you help me modify it to return what I am looking for.
Here is an example what is returned by speedtest-cli
$ speedtest-cli
Retrieving speedtest.net configuration...
Testing from Biglobe (111.111.111.111)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by GLBB Japan (Naha) [51.24 km]: 118.566 ms
Testing download speed................................................................................
Download: 4.00 Mbit/s
Testing upload speed......................................................................................................
Upload: 13.19 Mbit/s
$
And this is an example of what it is being returned by speediest.py to my .csv file
Date,Time,Ping,Download (Mbit/s),Upload(Mbit/s),myip
05/30/20,12:47,76.391,12.28,19.43,Biglobe
This is what I want it to return.
Date,Time,Ping,Download (Mbit/s),Upload (Mbit/s),myip
05/30/20,12:31,75.158,14.29,19.54,Biglobe 111.111.111.111
Or may be,
05/30/20,12:31,75.158,14.29,19.54,Biglobe,111.111.111.111
Here is the code that I am using. And thank you for any help you can provide.
import os
import re
import subprocess
import time
response = subprocess.Popen(‘/usr/local/bin/speedtest-cli’, shell=True, stdout=subprocess.PIPE).stdout.read().decode(‘utf-8’)
ping = re.findall(‘km]:\s(.*?)\s’, response, re.MULTILINE)
download = re.findall(‘Download:\s(.*?)\s’, response, re.MULTILINE)
upload = re.findall(‘Upload:\s(.*?)\s’, response, re.MULTILINE)
myip = re.findall(‘from\s(.*?)\s’, response, re.MULTILINE)
ping = ping[0].replace(‘,’, ‘.’)
download = download[0].replace(‘,’, ‘.’)
upload = upload[0].replace(‘,’, ‘.’)
myip = myip[0]
try:
f = open(‘/home/pi/speedtest/speedtestz.csv’, ‘a+’)
if os.stat(‘/home/pi/speedtest/speedtestz.csv’).st_size == 0:
f.write(‘Date,Time,Ping,Download (Mbit/s),Upload (Mbit/s),myip\r\n’)
except:
pass
f.write(‘{},{},{},{},{},{}\r\n’.format(time.strftime(‘%m/%d/%y’), time.strftime(‘%H:%M’), ping, download, upload, myip))
Let me know if this works for you, it should do everything you're looking for
#!/usr/local/env python
import os
import csv
import time
import subprocess
from decimal import *
file_path = '/home/pi/speedtest/speedtestz.csv'
def format_speed(bits_string):
""" changes string bit/s to megabits/s and rounds to two decimal places """
return (Decimal(bits_string) / 1000000).quantize(Decimal('.01'), rounding=ROUND_UP)
def write_csv(row):
""" writes a header row if one does not exist and test result row """
# straight from csv man page
# see: https://docs.python.org/3/library/csv.html
with open(file_path, 'a+', newline='') as csvfile:
writer = csv.writer(csvfile, delimiter=',', quotechar='"')
if os.stat(file_path).st_size == 0:
writer.writerow(['Date','Time','Ping','Download (Mbit/s)','Upload (Mbit/s)','myip'])
writer.writerow(row)
response = subprocess.run(['/usr/local/bin/speedtest-cli', '--csv'], capture_output=True, encoding='utf-8')
# if speedtest-cli exited with no errors / ran successfully
if response.returncode == 0:
# from the csv man page
# "And while the module doesn’t directly support parsing strings, it can easily be done"
# this will remove quotes and spaces vs doing a string split on ','
# csv.reader returns an iterator, so we turn that into a list
cols = list(csv.reader([response.stdout]))[0]
# turns 13.45 ping to 13
ping = Decimal(cols[5]).quantize(Decimal('1.'))
# speedtest-cli --csv returns speed in bits/s, convert to bytes
download = format_speed(cols[6])
upload = format_speed(cols[7])
ip = cols[9]
date = time.strftime('%m/%d/%y')
time = time.strftime('%H:%M')
write_csv([date,time,ping,download,upload,ip])
else:
print('speedtest-cli returned error: %s' % response.stderr)
$/usr/local/bin/speedtest-cli --csv-header > speedtestz.csv
$/usr/local/bin/speedtest-cli --csv >> speedtestz.csv
output:
Server ID,Sponsor,Server Name,Timestamp,Distance,Ping,Download,Upload,Share,IP Address
Does that not get you what you're looking for? Run the first command once to create the csv with header row. Then subsequent runs are done with the append '>>` operator, and that'll add a test result row each time you run it
Doing all of those regexs will bite you if they or a library that they depend on decides to change their debugging output format
Plenty of ways to do it though. Hope this helps
I am generating a protocol for a tcpip socket between python and matlab. While trying to setup some sort of a protocol, I ran into a problem. It has to do with this set of code below
FPath = Path('c:/test/dogg.jpg')
HASH = Commons.get_file_md5_hash((FPath))
msg = ('IDINFO'+FPath.name+'HASH'+ HASH+'\n')
generates the message
IDINFOdogg.jpgHASH7ad1a930dab3c099939b66267b5c57f8
I have in the message: IDINFO which will tell the server the name of the file and HASH which will tell the file details.
After this I open up the file using
f = open(FPath,"rb")
chunk = f.read(1020)
and build a package with the tag DATA in front
msg = b`DATA` + chunk + b'\n'
Problem is that the b'\n' is not the same as in the first message. as Matlab cannot read the delimiter and won't continue grabbing data chunks.
Matlab code for Below reference. This isn't the entire object just the part that is potentially causing trouble.
To setup a callback.
set(gh.tcpipServer, 'BytesAvailableFcnMode','Terminator');
set(gh.tcpipServer, 'BytesAvailableFcn', #(h,e)gh.Serverpull(h,e));
The Function for looking at the bytes
function Serverpull(gh,h,e)
gh.msg = fread(gh.tcpipServer,gh.tcpipServer.BytesAvailable);
gh.msgdecode = char(transpose(gh.msg));
if strfind(gh.msgdecode,'IDINFO')
Hst = strfind(gh.msgdecode,'HASH');
gh.Fname = gh.msgdecode(7:Hst-1);
gh.HASH = gh.msgdecode(Hst+4:end);
fwrite(gh.tcpipServer,'GoodToGo');
gh.PrepareforDataAq()
elseif strfind(gh.msgdecode,'DATA')
fwrite(gh.fileID,gh.msg(5:end),'double');
elseif strfind(gh.msgdecode,'EOF')
fclose(gh.fileID);
display('File Transfer Complete')
end
end
function PrepareforDataAq(gh)
path = fullfile('c:\temp\',gh.Fname);
gh.fileID = fopen(path,'w');
end
For the TLDR,
How to make the string '\n' the same as b \n when building a tcp message from binary instead of strings before encoding?
I want to remove multiple lines in a file using regex.
I have a file with something like this :
host host_name {
# comment (optional)
hardware ethernet 01:22:85:EA:A8:5D;
fixed-address 192.168.107.210;
}
host another_host_name {
# comment (optional)
hardware ethernet 01:22:85:EA:A8:5D;
fixed-address 192.168.107.210;
}
Basically, when I choose the host name like host_name for example, it'll detect the line that has it and remove all the lines after it until it encounters the first { :
#before
host host_name {
# comment (optional)
hardware ethernet 01:22:85:EA:A8:5D;
fixed-address 192.168.107.210;
}
host another_host_name {
# comment (optional)
hardware ethernet 01:22:85:EA:A8:5D;
fixed-address 192.168.107.210;
}
#after
host another_host_name {
# comment (optional)
hardware ethernet 01:22:85:EA:A8:5D;
fixed-address 192.168.107.210;
}
I guess we would use something like m = search('r"^host.*}', line) but it works for line by line stuff not for multiple lines.
def remove(filename, hostname):
with open(os.path.abspath("app/static/DATA/{}".format(filename)), "a") as f:
for line in f:
m = search('r"^hostname.*}', line, re.MULTILIGNE)
if m:
#we delete the bloc, I don't know how to do it though
Starting like this?
I have 3 ideas for you.
Try MULTILINE mode. You can read more about it here: https://docs.python.org/3/library/re.html#re.MULTILINE which I think will do what you are asking.
When that just doesn't do the trick, I cheat. I'll run a pre regex to swap all \n to something strange like "this_is_my_special_holder". Now everything is on one line. I'll do the work I want like you have written. Then I'll run a post regex that swaps all "this_is_my_special_holder" back to \n. If you ever get stuck in a language that doesn't support multiline this should always get it done :)
You may just be able to run the regex, my example here does just that:
Here is how I would do this whole thing:
import re
def main(regex_part):
the_regex = re.compile("".join(["host ", regex_part, " {[^}]+}"]))
with open('test.txt', 'r') as myfile:
data=myfile.read()
data = re.sub(the_regex, "", data)
print (data)
with open('test2.txt', 'w') as newfile:
newfile.write(data)
main("host_name")
I open the file with 'with', that way you don't have to close the file handle later. 'r' is to read the file and 'w' is to write the file. The regex simply replaces:
host host_name { everything up to the next } and then the next }
with nothing.
http://regex101.com is a handy site to actually play with the regexs. good luck!
I have read alot about this but I just don't seem to figure it out... I should use Blueprint for this but the problem I am having right now is that I do not know how to pass my variable from my main file in my second file.
As an example :
/app
/runserver.py
/app
init.py
main.py
second.py
Now I do have a dictionairy in my main that I fill. And I want to use it in my second file to adjust it etc. How will I be able to do this? Since I tried to import the files and tried:
import main
dictMain = main.dictFromMain
I thought this would be enough since I read it on different question on Stack Overflow but it doesn't seem to work!
EDIT: To sketch the problem further
More background : I am making a client - server application, the client is receiving and sending data from the server. But there is a difference is the data the client is sending. On one hand you have files and paramters which I want to 'capture' with my second file with ReST. And on the other hand I got a incomming stream which I 'capture' in my main file.
Example second file:
#app.route('/uploads/', methods = ['GET', 'POST'])
def get_files():
if request.method == 'GET':
sendDict = []
for element in ctxList:
for fileCtx in element['file']:
d = { 'id' : element['id'], 'file': [ {'name': fileCtx['name'], 'uri' : fileCtx['uri'], 'path' : fileCtx['path'] } ] }
sendDict.append(d)
jsonString = jsonify(ctx=sendDict)
return jsonString
But this code uses a dictionairy from my first file (the dict ctxList) I have no idea to get it out of my first file. I used to get a error when I did : ctxList = mainFile.ctxList that the module did not have this variable, but now I am getting a error that the first file does not know the URL structure ( /uploads/ from the second file).