Develop Reference

I tried to read some values in a txt file with java and python. Finally I could not manage it with both. I cannot figure out where is the problem

I try to extract two kind of value from a txt file and write them to two separate txt files. I know that my functions work properly and I cannot figure out any mistake in my code. I realised that both two languages do not read the text file as it is. What I mean by that is for example normally the txt file has 10367 lines in it but when I count the lines in the code, there are 20735 lines in python. I cannot understand why this happens. I do not have an in-depth knowledge about how programming languages read the files. Please give me some information about the possible causes of this situation.
thanks in advance...
This is the pyhton code:
def main():
serverSpeedsList=list()
totalSpeedsList=list()
ssString=str()
stString=str()
with open("C:\\Users\\yusuf\\OneDrive\\Masaüstü\\SpeedTests\\Logs\\log100.txt",'r') as inFile:
for line in inFile:
i+=1
ss=speedOfServer(line)
st=speedOfTotal(line)
if ss!="":
ssString+=ss+"\n"
serverSpeedsList.append(ss)
if st!="":
stString+=st+"\n"
totalSpeedsList.append(st)
with open("C:\\Users\\yusuf\\OneDrive\\Masaüstü\\SpeedTests\\Results\\server100.txt",'w') as outFile:
outFile.write(ssString)
with open("C:\\Users\\yusuf\\OneDrive\\Masaüstü\\SpeedTests\\Results\\total100.txt",'w') as outFile:
outFile.write(ssString)
def speedOfServer(text):
startStr="time\":\""
endStr=" ms"
result=str()
startIx=text.find(startStr)
endIx=text.find(endStr)
if startIx>=0 and endIx>=0:
result=text[startIx+len(startStr) : endIx]
return result
def speedOfTotal(text):
startStr="showProfile.php ("
endStr="ms"
result=str()
startIx=text.find(startStr)
endIx=text.find(endStr)
if startIx>=0 and endIx>=0:
result=text[startIx+len(startStr) : endIx]
return result
main()
and this is the java code to do the same
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
public class Main {
private static String serverSpeedFileName="C:\\Users\\yusuf\\OneDrive\\Masaüstü\\JavaSpeedAnalyser\\Results\\Server\\serverSpeed100.txt";
private static String responseSpeedFileName="C:\\Users\\yusuf\\OneDrive\\Masaüstü\\JavaSpeedAnalyser\\Results\\Total\\responseSpeed100.txt";
private static String logFilePath="C:\\Users\\yusuf\\OneDrive\\Masaüstü\\JavaSpeedAnalyser\\Logs\\log100.txt";
public static void main(String[] args){
StringBuilder serverSpeeds=new StringBuilder();
StringBuilder responseSpeeds=new StringBuilder();
try{
File file = new File(logFilePath);
BufferedReader br = new BufferedReader(new FileReader(file));
String line;
while ((line = br.readLine()) != null){
String serverSpeed=speedOfServer(line);
if(!serverSpeed.isEmpty()){
System.out.println(serverSpeed);
serverSpeeds.append(serverSpeed+"\n");
}
String responseSpeed=speedOfResponses(line);
if(!responseSpeed.isEmpty()){
responseSpeeds.append(responseSpeed+"\n");
}
}
br.close();
}catch(IOException e){
e.printStackTrace();
}
writeToFile(serverSpeedFileName, serverSpeeds.toString());
writeToFile(responseSpeedFileName, responseSpeeds.toString());
}
private static String speedOfServer(String text){
String start="time\":\"";
String end=" ms";
String result="";
int startIndex=text.indexOf(start);
int endIndex=text.indexOf(end);
if(startIndex>=0 && endIndex>=0 ){
result=text.substring(startIndex+start.length(),endIndex);
}
return result;
}
private static String speedOfResponses(String text){
String start="%5bF%5dshowProfile.php (";
String end="ms)";
String result="";
int startIndex=text.indexOf(start);
int endIndex=text.indexOf(end);
if(startIndex>=0 && endIndex>=0){
result=text.substring(startIndex+start.length(),endIndex);
}
return result;
}
}
I try to analyse a logcat file from an android phone, this is why I try to do that but I cannot manage it. Please help me

extracting 'blob' python serialized tuple data in php

I'm trying to utilize a database from another program in a php based website tool, and apparently the original was built in python and puts some of it's data into a python tuple and serializes it to store it as a blob in the sql table.
I'm not a python programmer so I'm not sure how to even see what is in this blob, but I do know that some of the 'type' indicators for the data field are stored in there and I want to extract them and anything else useful.
Is there any way to 'unserialize' a python tuple in php?

The blob data turned out to be a pickled tuple (part of the reason I despise python - both data types that only python can read! Python programmers: 'standardized conventions? Who needs standardized conventions?!?!')
I came up with a cludgy way to 'unpickle' the data and json serialize it using a command line. To get the binary blob data into the command line, I base64 encode it. It's janky but it works for what I need:
/**
* use a python exec call to 'unpickle' the blob_data
* to get the binary blob into a command line argument, base64 encode it
* to get the data back out of python, json serialize it
* #param string $blob binary blob data
* #return mixed
*/
public static function unpickle($blob) {
$cmd = sprintf("import pickle; import base64; import json; print(json.dumps(pickle.loads(base64.b64decode('%s'))))", base64_encode($blob));
$pcmd = sprintf("python -c \"%s\"", $cmd);
$result = exec($pcmd);
$resdec = json_decode($result);
return $resdec;
}

With a little more playing on this concept, I gave myself a few more alternatives. First is, I took the command line version above and made it into a little more functional python script:
unpickle.py:
#!/usr/bin/env python3
import pickle
import json
import sys
import base64
import select
def isBase64(s):
try:
return s == base64.b64encode(base64.b64decode(s)).decode('ascii')
except Exception:
return False
bblob = None
if (len(sys.argv) > 1) and isBase64(sys.argv[1]):
bblob = base64.b64decode(sys.argv[1])
elif select.select([sys.stdin, ], [], [], 0.0)[0]:
try:
with open(0, 'rb') as f:
bblob = f.read()
except Exception as e:
err_unknown(e)
if bblob != None:
unpik = pickle.loads(bblob)
jsout = json.dumps(unpik)
print(jsout)
This script allows you to either specify the blob data from the pickled tuple 'byte' as a base64 encoded string on the command line, or you can pipe raw blob data into the script. Both variations will output json if the data is valid and formatted properly. (null if not)
You can convert this to a self-contained binary to plop on systems without python using pyinstaller -F if need be. To play with it in the event I am running it on systems with the pyinstaller binary vs one with the python script vs one with just python, I created the following static methods in my laravel model. (I'll eventually move it into a service module)
/**
* call either a pyinstaller binary or python script with raw blob data to be unpickled
*
* #param string $b binary data of blob
* #return false|mixed
*/
public static function unpickle($b)
{
$cmd = base_path(env('UNPICKLE_BINARY', 'bin/unpickle'));
if(!(is_file($cmd) && is_executable($cmd))) { // make sure unpickle cmd exists
// check for UNPICKLE_BINARY with .py after and python binary
$pyExe = env('PYTHON_EXE', '/usr/bin/python');
if (is_file($cmd.".py") && (is_file($pyExe) && is_executable($pyExe))) {
$cmd = sprintf("%s %s.py", $pyExe, $cmd);
} else
return static::unpyckle($b); // try direct python call
}
$descriptorspec = [
["pipe", "r"],
["pipe", "w"],
["pipe", "w"]
];
$cwd = dirname($cmd);
$env = [];
$process = proc_open($cmd, $descriptorspec, $pipes, $cwd, $env);
if (is_resource($process)) {
fwrite($pipes[0], $b);
fclose($pipes[0]);
$output = stream_get_contents($pipes[1]);
fclose($pipes[1]);
$return_value = proc_close($process);
if(static::isJson($output))
return json_decode($output);
else
return false;
}
return false;
}
/**
* use a python exec call to 'unpickle' the blob_data
* to get the binary blob into a command line argument, base64 encode it
* to get the data back out of python, json serialize it
* #param string $blob binary blob data
* #return mixed
*/
public static function unpyckle($blob) {
$pyExe = env('PYTHON_EXE', '/usr/bin/python');
if (!(is_file($pyExe) && is_executable($pyExe)))
throw new Exception('python executable not found!');
$bblob = base64_encode($blob);
$cmd = sprintf("import pickle; import base64; import json; print(json.dumps(pickle.loads(base64.b64decode('%s'))))", $bblob);
$pcmd = sprintf("%s -c \"%s\"", $pyExe, $cmd);
$result = exec($pcmd);
$resdec = json_decode($result);
return $resdec;
}
/**
* try to detect if a string is a json string
*
* #param $str
* #return bool
*/
public static function isJson($str) {
if(is_string($str) && !empty($str)) {
json_decode($str);
return (json_last_error() == JSON_ERROR_NONE);
}
return false;
}
example .env values:
UNPICKLE_BINARY=bin/unpickle
PYTHON_EXE=/usr/bin/python3
basically showing three different ways to call python to do essentially the same thing...

C#: Read fast from a file that is being used by another process

I have a python script that reads from a logfile and outputs certain data from it. The way it reads from it is
try:
with open(os.path.expandvars('Path/To/My/Log.txt', 'r') as f:
logContent = [line.rstrip() for line in f]
except Exception as e:
print(e)
Now I wanted to recreate that python script in C#. The main problem is, that the log file makes about 30.000 Lines in 30 minutes. While the program that handles that log isn't being executed, I can easily open the file and read from it, because it's not being used by that program. But when that program runs, I need to read from the file with a filestream, and so the reading of 30.000 lines takes ages:
private string GetLog(string path)
{
string log = "";
FileStream reader = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
StreamReader logFileReader = new StreamReader(reader);
while (!logFileReader.EndOfStream)
{
log += logFileReader.ReadLine();
// Your code here
}
// Clean up
logFileReader.Close();
reader.Close();
return log;
}
Is there a way to make my code read from the file in max 5 seconds?

I got it. When I use stream.ReadToEnd() it reads everything in about 2 seconds

As you have mentioned file is big, so better to use StringBuilder over string, you can use using also so no need to call close() explicitly.
StringBuilder sb = new StringBuilder();
string path = "some path";
using (FileStream logFileStream = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
using (StreamReader logFileReader = new StreamReader(logFileStream))
{
while (!logFileReader.EndOfStream)
{
sb.Append(logFileReader.ReadLine());
}
}
}
string log = sb.ToString();

PGP. Sign one element of an XML file, then insert the signature and public key into another element

I got a task to add a digital signature and the signer's public key to an XML file. The XML file in question would look like this:
<data>
<head>
<field1>foo</field2>
<fieldn>bar</fieldn>
</head>
<headSigner>John Doe</headSigner>
<signerPublicKey>dfgdgd...sdfgdgdsg</signerPublicKey>
<headSignature>sdafa...sfsafsasdfsafasd</headSignature>
</data>
Is this common or even feasible? I can write something in Python or Powershell that would:
1) Write the head XML and dump it to a file.
2) Run gpg to sign the file with the --clear-sign flag.
3) Parse the signed file that gpg makes for the signature string.
4) Add that string to the corresponding element in the XML.
Is there are an easier-or standard-way to do this? Maybe a Python or Powershell module that's already set up for that?

Use C# console app to sign the XML file using a certificate. Like this:
using System;
using System.Xml;
using System.Security.Cryptography.X509Certificates;
using System.Security.Cryptography;
using System.Security.Cryptography.Xml ;
var certStore = new X509Store(StoreLocation.CurrentUser);
certStore.Open(OpenFlags.ReadOnly);
// get cert rerquired
var certificateThumbPrint = "33eeededldodoijdlnkddzippy2e";
var certCollection = certStore.Certificates.Find(X509FindType.FindByThumbprint , certificateThumbPrint, true);
var myCert = certCollection[0];
// I can get the correct certificate but the following line throws "Invalid provider type specified." error
var SigningKey = myCert.GetRSAPrivateKey();
XmlDocument xmlDoc = new XmlDocument();
xmlDoc.Load(args[0] );
xmlDoc.PreserveWhitespace = true;
Sign file using this:
private static void SignXml2(XmlDocument xmlDoc, X509Certificate2 cert , string file1)
{
// Create a SignedXml object.
SignedXml signedXml = new SignedXml(xmlDoc);
// Add the key to the SignedXml document.
signedXml.SigningKey = cert.PrivateKey;
// Create a reference to be signed.
Reference reference = new Reference();
reference.Uri = "";
// Add an enveloped transformation to the reference.
var env = new XmlDsigEnvelopedSignatureTransform();
reference.AddTransform(env);
// Include the public key of the certificate in the assertion.
signedXml.KeyInfo = new KeyInfo();
signedXml.KeyInfo.AddClause(new KeyInfoX509Data(cert, X509IncludeOption.WholeChain));
// Add the reference to the SignedXml object.
signedXml.AddReference(reference);
// Compute the signature.
signedXml.ComputeSignature();
// Get the XML representation of the signature and save
// it to an XmlElement object.
XmlElement xmlDigitalSignature = signedXml.GetXml();
// Append the element to the XML document.
xmlDoc.DocumentElement.AppendChild(xmlDoc.ImportNode(xmlDigitalSignature, true));
xmlDoc.PreserveWhitespace = true;
xmlDoc.Save(file1);
}

How to compare hash of an image using python and unity3d C#?

I have a unity3d application that request a json string of image name including its hash in my django webserver. Then my unity app will check my existing image hash if its the same as the json requested. My problem is that unity hash result is different from my python hash result value. I also tried to hash string on both and it returns the same hash value.
Python Hash:
>>> image_file = open('C:/image.png').read()
>>> hashlib.md5(image_file).hexdigest()
'658e8dc0bf8b9a09b36994abf9242099'
Unity3d Hash:
public static string ComputeHash()
{
// Form hash
System.Security.Cryptography.MD5 h =System.Security.Cryptography.MD5.Create();
var myImage = File.OpenRead(PathByPlatform("image.png"));
var data = h.ComputeHash(myImage );
System.Text.StringBuilder sb = new System.Text.StringBuilder();
for (int i = 0; i < data.Length; ++i)
{
sb.Append(data[i].ToString("x2"));
}
return sb.ToString();
//This fucntion returns
//fac7f19792a696d81be77aca7dd499d0
}

Did you try open('C:/image.png', "rb").read() in order to read the file in binary mode?
Reading files without the "b" will change line ending characters on Windows from CR/LF to LF which has an impact on the hash. (at least for python2)