Develop Reference

Reading parquet files using Parquet.NET takes more time than pyarrow (python)

Usually when it comes to parquet file operations,Parquet.Net package takes less/equal time compared to python. But my initial set of experiments doesn't align with that. To read 5 million data points in parquet python takes around 1 second while the .NET package takes around 20 seconds. The time taken to read the parquet files using .NET is far far higher than python. I am uploading the sample code here, can anybody point me out the reason for this behavior?
In C#:
{
List<string> metadata = new List<string>();
List<double[]> dataValues = new List<double[]>();
var watch = Stopwatch.StartNew();
using (Stream fileStream = File.OpenRead(path))
{
using (var parquetReader = new ParquetReader(fileStream))
{
DataField[] dataFields = parquetReader.Schema.GetDataFields();
for (int currentRowGroup = 0; currentRowGroup < parquetReader.RowGroupCount; currentRowGroup++)
{
using (ParquetRowGroupReader groupReader = parquetReader.OpenRowGroupReader(currentRowGroup))
{
for (int i = 0; i < yColIndex.Count(); i++)
{
var dataColumn = parquetReader.OpenRowGroupReader(currentRowGroup).ReadColumn(dataFields[yColIndex[i]]);
Array reData = dataColumn.Data;
dataValues.Add((double[])reData);
}
}
}
}
}
}
In Python:
def read_column_data_v1(file_path, file_name, columns):
file_path = f"{file_path}\\{file_name}.parquet"
file_data = pq.ParquetFile(file_path)
for i in range(file_data.metadata.num_row_groups):
data = file_data.read_row_group(i, columns)

How to start a python program using java (Runtime.getRuntime().exec())

I am writing a program that requires the starting of a python script before the rest of the java code runs. However, I cannot find a solution to my issues. I would appreciate if someone could suggest a solution to the problem I am facing.
Code (I need help on the part under the comment "start python"):
import java.io.IOException;
//makes it easier for user to
//select game/start python
public class gameselect {
public static void main(String args[]) throws IOException {
//start python
try {
String cmd = "python ngramcount.py";
Process process = Runtime.getRuntime().exec(cmd);
process.getInputStream();
}
catch (IOException e) {
e.printStackTrace();
}
//select game
try {
Scanner in = new Scanner (System.in);
game1 g = new game1();
game2 f = new game2();
int choice = 0;
System.out.println("Welcome to TranslateGame!");
System.out.println("Type 1 for game1 (words) or 2 for game2 (phrases)");
while (choice != 1 && choice != 2) {
choice = in.nextInt();
if (choice != 1 && choice != 2) {
System.out.println("No game associated with that number.");
}
}
if (choice == 1) {
g.game1();
}
else if (choice == 2) {
f.game2();
}
}
catch(IOException e) {
System.out.println("No.");
}
}
}

Here is some code that you might be able to get to work. I also commented it and provided some reference links to help you understand what the code is doing.
public static void main(String[] args) throws IOException {
// I'm using the absolute path for my example.
String fileName = "C:\\Users\\yourname\\Desktop\\testing.py";
// Creates a ProcessBuilder
// doc: https://docs.oracle.com/javase/7/docs/api/java/lang/ProcessBuilder.html
ProcessBuilder pb = new ProcessBuilder("python", fileName);
pb.redirectErrorStream(true); // redirect error stream to a standard output stream
Process process = pb.start(); // Used to start the process
// Reads the output stream of the process.
BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line; // this will be used to read the output line by line. Helpful in troubleshooting.
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
}

Issue returning data from Python to Unity

This question is an evolution of this: Issue returning header byte from Python to Unity
Now the issue about header byte is solved, but I have a problem during the loop.
The data incoming good but after "n" times the loop break because receive a json string with an entire structure of data with open/close bracket PLUS the next structure of data with open bracket, some data and in the "image:" some data and many of the "\0" and unless the close bracket.. And the json parser obviously breaks. Why and what can I do? Thanks
Below the code revised:
IEnumerator Client()
{
while (!IsConnected(client))
{
try
{
client = new TcpClient(host, port);
s = client.GetStream();
byte[] byteBuffer = Encoding.UTF8.GetBytes("Connected to client");
s.Write(byteBuffer, 0, byteBuffer.Length);
while (true)
{
if (s.DataAvailable)
{
while (s.DataAvailable)
{
var header = new byte[4];
s.Read(header, 0, header.Length);
var fileSize = BitConverter.ToUInt32(header,0);
StringBuilder myCompleteMessage = new StringBuilder();
MemoryStream ms = new MemoryStream();
int increment = 0;
while (ms.Length < fileSize)
{
byte[] dataReceived = new byte[fileSize];
increment = s.Read(dataReceived, 0, dataReceived.Length);
ms.Write(dataReceived.Take(increment).ToArray(), 0, increment);
}
myCompleteMessage.AppendFormat("{0}", Encoding.UTF8.GetString(ms.GetBuffer()));
JSONNode data = JSONNode.Parse(myCompleteMessage.ToString());
....
// Do work with myCompleteMessage
yield return null;
}
}
}
}
}
}

I don't have your Unity environment here, so I can't have tested this.
Something like a loop (likely run in a coroutine or something that doesn't block Unity's default handling) like this should be enough to handle a stream of messages of the shape we established before (a 4-byte header signifying the message length, followed by that many bytes of JSON).
while (!IsConnected(client))
{
try
{
client = new TcpClient(host, port);
s = client.GetStream();
while (true)
{
var data = ReadPacket(s);
// Do work with data
}
}
}
ReadPacket should be something like
JSONNode ReadPacket(Stream s)
{
var buffer = new byte[131072];
// Read 4 bytes (we will assume we can always read that much)
var n = s.Read(buffer, 0, 4);
if(n < 4) throw new Exception("short read");
var fileSize = BitConverter.ToUInt32(buffer, 0);
Debug.Print("Reading a blob of length", fileSize);
// Memory stream to accumulate the reads into.
var ms = new MemoryStream();
while (ms.Length < fileSize)
{
// Figure out how much we can read -- if we're near the end,
// don't overread.
var maxRead = Math.Min(buffer.Length, fileSize - ms.Length);
var increment = s.Read(buffer, 0, maxRead);
// ... and write them into the stream.
ms.Write(buffer, 0, increment);
Debug.Print("Read", ms.Length, "of", fileSize);
}
// Decode the bytes to UTF-8 and parse.
return JSONNode.Parse(Encoding.UTF8.GetString(ms.GetBuffer()));
}

child_process in NodeJS not having an error when it's supposed to

I use nodeJS and express as a server and use child_process.spawn to run a python script.
After running the script, I get the result and send it as a json response.
When I get correct post response, the server responses the right response.
When I post wrong parameters, the servers responses with the error message (as expected)
But when I post the wrong parameters after the correct parameter post, it gives the previous post's response and I don't know why. Maybe this is because of the spawn and should I use exec?
I want to know why this is happening.
Let me know if you need more details on this.
const { json } = require('body-parser')
const express = require('express')
const { setTimeout } = require('timers')
const router = express.Router()
let dataToSend;
router.post('/', (req, res) => {
// console.log(req.body)
let resol = req.body.Resolution;
let url1 = req.body.FirstImageURI;
let url2 = req.body.SecondImageURI;
let roi = req.body.DetectionArea;
let sens = req.body.Threshold;
if (isNaN(sens)) sens = "50";
console.log("start ai model");
console.log(`params : ${url1} ${url2} ${roi} ${sens}`);
const spawn = require("child_process").spawn;
const pythonProcess = spawn('python', ["img_diff.py", url1, url2, roi, sens]);
pythonProcess.stdout.on('data', function (data) {
console.log('Pipe data from python script ...');
dataToSend = data.toString();
console.log(`data in stdout : ${dataToSend}`)
});
// in close event we are sure that stream from child process is closed
pythonProcess.on('close', (code) => {
try {
jsonObject = {}
console.log(`child process close all stdio with code ${code}`);
// format json
dtList = dataToSend.split('\n');
ssim = Number(dtList[0].split(":")[1]);
detected = "UnDetected"
console.log(dtList[1])
let addInfo = dtList[1].trim().slice(1, -1)
addInfo = JSON.parse("[" + addInfo + "]");
jsonObject.AdditionalInfo = []
for (let i = 0; i < addInfo.length; i++) {
let thisObject = {}
thisObject.DetectedRegion = []
for (let j = 0; j < addInfo[i].length; j++) {
coordObject = {}
coordObject.XCoord = addInfo[i][j][0]
coordObject.YCoord = addInfo[i][j][1]
thisObject.DetectedRegion.push(coordObject)
}
jsonObject.AdditionalInfo.push(thisObject)
}
if (jsonObject.AdditionalInfo.length > 0) {
detected = "Detected"
}
jsonObject.Result = detected
console.log(detected)
res.json(jsonObject)
} catch (e) {
console.log(`error -> ${e}`)
jsonObject.AdditionalInfo = [];
let thisObject = {};
thisObject.DetectedRegion = [];
jsonObject.AdditionalInfo.push(thisObject);
jsonObject.detected = "UnDetected";
res.json(jsonObject);
}
});
})
module.exports = router

Still don't know why this happens,
but my guess is that as the process crashes,
Node just gives a response from the previous successful memory.
Will come back and update if I have a better answer.

How to detect and convert progressive jpegs with python

I'd like to be able to detect progressive jpegs using python and convert them to non progressive.
(I'm writing a tool to manage images for android and progressive jpegs seem to break it.)

I apologise in advance for providing a php based answer, whereas the question was asked about python. Nevertheless, I think it adds value and can be useful.
Before attempting to convert a progressive image to non-progressive, it is good to have a detection method for progressive Jpeg.
Here is the php function that does it, could easily be rewritten in other languages (python would be a candidate), as it reads binary data and Jpeg markers (and therefore does not rely on language specific library)
public function checkProgressiveJPEG($filepath) {
$result = false;
// $this->log = 'started analysis...';
// http://en.wikipedia.org/wiki/Jpeg
// for more details on JPEG structure
// SOI [0xFF, 0xD8] = Start Of Image
// SOF0 [0xFF, 0xC0] = Start Of Frame (Baseline DCT)
// SOF2 [0xFF, 0xC2] = Start Of Frame (Progressive DCT)
// SOS [0xFF, 0xDA] = Start Of Scan
if(file_exists($filepath)) {
$fs = #fopen($filepath, "rb");
$bytecount = 0;
$byte_last = 0;
$buffer = 0;
$buffer_length = 4*1024;
$begins_with_SOI = false;
while($buffer = fread($fs, $buffer_length)) {
// always carry over previous ending byte
// just in case the buffer is read after a 0xFF marker
if($byte_last) {
$buffer = $byte_last.$buffer;
}
$byte_last = 0;
preg_match("/\.$/", $buffer, $matches);
if(count($matches)) {
$byte_last = $matches[0];
}
// check if it begins with SOI marker
if(!$begins_with_SOI) {
preg_match("/^\\xff\\xd8/", $buffer, $matches);
if(count($matches)) {
$begins_with_SOI = true;
} else {
// $this->log = 'does not start with SOI marker';
$result = false;
break;
}
}
// check if SOS or SOF2 is reached
preg_match("/\\xff(\\xda|\\xc2)/", $buffer, $matches);
if(count($matches)) {
if(bin2hex($matches[0]) == 'ffda') {
// $this->log = 'SOS is reached and SOF2 has not been detected, so image is not progressive.';
$result = false;
break;
} else if(bin2hex($matches[0]) == 'ffc2') {
// $this->log = 'SOF2 is reached, so image is progressive.';
$result = true;
break;
}
}
} // end while
fclose($fs);
} // end if
return $result;
}