Develop Reference

Execute python script through Java

I'm trying to run a very simple python script that clears and writes to a CSV file, from inside of java but I'm having a lot of trouble doing it.
The scripts don't require any input and the output is all written into a CSV file so all I need to do is get the python scripts to run through my java code.
Below is a bit of code that I've seen all over the internet but doesn't seem to be working for me. It seems like for both of the scripts, using this command does nothing to the csv. No errors are thrown and the java program simply exits presumably without executing the python scripts.
public static void main(String[] args) throws IOException
{
Process p = Runtime.getRuntime().exec("python Refresh.py");
}
here are the scripts I'm trying to run.
Script1:
file = open("products.csv","r+")
file.truncate(0)
file.close()
Script2:
from bs4 import BeautifulSoup as soup
from urllib.request import Request, urlopen
import time
filename = "products.csv"
f = open(filename, "a")
#connects to the page and reads and saves raw HTML
for i in (0,25,50,75):
my_url = 'https://www.adorama.com/l/Computers/Computer-Components/Video-and-Graphics-Cards?startAt='+ str(i) +'&sel=Expansion-Ports_HDMI'
hdr = {'User-Agent': 'Mozilla/5.0'}
client = Request(my_url,headers=hdr)
page = urlopen(client).read()
#parsing the HTML
page_soup = soup(page, "html.parser")
#print (page_soup.h1)
containers = page_soup.findAll("div",{"class":"item"})
#print (len(containers))
containers.pop()
for container in containers:
title_container = container.findAll("div",{"class":"item-details"})
title = title_container[0].h2.a.text.strip()
status_container = container.findAll("div",{"class":"item-actions"})
status = status_container[0].form.button.text.strip()
if (status == "Temporarily not available"):
status = "Out of stock"
else:
status = "In stock"
price = container.find("div","prices").input["value"]
link = container.a["href"]
f.write(title.replace(",", "|") + "," + price.replace(",", "") + "," + status + "," + link + "\n")
time.sleep(0.01)
f.close()
The java file, Python script, and the csv file are all in the same folder.

Use the newer ProcessBuilder class:
ProcessBuilder pb = new ProcessBuilder("python","Refresh.py");
Process p = pb.start();
Hope that works for you!

You are not checking for errors from the python script. You can achieve this simply by merging STDERR to STDOUT and reporting the content of STDOUT to console:
Process p = new ProcessBuilder("python", "Refresh.py")
.redirectErrorStream(true)
.start();
p.getInputStream().transferTo(System.out);
int rc = p.waitFor();
This should print out the error message from python and give you error code back. You may have problems with path to files, so you might need to adjust your arguments to explicit pathnames to "python" and/or "Refresh.py".

I managed to fix the issue by constantly reading the "print" and error outputs of the Python file. Whilst I still don't completely understand how this fixed the issue, my best guess is that with this, the Java code keeps the python script "running" until the script itself is finished doing its thing, instead of just opening the script and instantly moving on.
Here's the code:
import java.io.BufferedReader;
import java.io.InputStreamReader;
public class Test {
public static void main(String... args) throws Exception {
String[] callAndArgs = {"python3", "YourScript.py"};
Process p = Runtime.getRuntime().exec(callAndArgs);
BufferedReader stdInput = new BufferedReader(new InputStreamReader(p.getInputStream()));
BufferedReader stdError = new BufferedReader(new InputStreamReader(p.getErrorStream()));
String s;
while ((s = stdInput.readLine()) != null) {
//System.out.println(s);
}
while ((s = stdError.readLine()) != null) {
//System.out.println(s);
}
}
}
Another Notable Detail is that this code only seems to work when Compiled and run through the Terminal/Geany. If I run the same thing with IntelliJ it does not work. Once again, I'm not sure why this is but I'm suspecting that IntelliJ compiles and runs in a VM of some sorts.

How to run python script from C# code by using command

I have a script file(run_edr.py) in my local machine and when I run it by using "cmd" and the following command then the script works perfectly. The script takes fewer parameters, the first parameter is an input document folder path and the second parameter is the output folder path to store the output documents.
my python command,
python run_edr.py -input_path "C:\Users\aslamm5165\Downloads\EDRCODE_ArgParser\files\EDR" -output_path "C:\Users\aslamm5165\Downloads\test" -site_name "a" -site_address "b" -site_city "c" -site_county "d" -site_state "e" -site_type "1"
I have tried like below, but not working, where Did I go wrong?
ScriptRuntimeSetup setup = Python.CreateRuntimeSetup(null);
ScriptRuntime runtime = new ScriptRuntime(setup);
ScriptEngine engine = Python.GetEngine(runtime);
ScriptSource source = engine.CreateScriptSourceFromFile(#"C:\Users\aslamm5165\Downloads\EDRCODE_ArgParser\run_edr.py");
ScriptScope scope = engine.CreateScope();
List<String> argv = new List<String>();
//Do some stuff and fill argv
argv.Add("python"+#" C:\Users\aslamm5165\Downloads\EDRCODE_ArgParser\run_edr.py -input_path" + #"C:\Users\aslamm5165\Downloads\EDRCODE_ArgParser\files\EDR");
argv.Add("-output_path"+ #"C:\Users\aslamm5165\Downloads\test");
argv.Add("-site_name 'a' -site_address 'b' -site_city 'c' -site_county 'd' -site_state 'e' -site_type '1'");
engine.GetSysModule().SetVariable("argv", argv);
source.Execute(scope);
I have tried with the system process as well as shown below, no error in the code, but the script is not getting executed. So I don't know what is the correct way of doing this but I want to start my script from my .Net Core application.
ProcessStartInfo start = new ProcessStartInfo();
start.FileName = #"cmd.exe";
start.Arguments = string.Format("python run_edr.py -input_path {0} -output_path {1} -site_name 'a' -site_address 'b' -site_city 'c' -site_county 'd' -site_state 'e' -site_type '1'", #"C:\Users\aslamm5165\Downloads\EDRCODE_ArgParser\files\EDR", #"C:\Users\aslamm5165\Downloads\test");
start.UseShellExecute = true;
start.RedirectStandardOutput = false;
start.WindowStyle = System.Diagnostics.ProcessWindowStyle.Hidden;
Process.Start(start);

I solved a similar running Python scripts in .NET Core 3.1 problem by changing the executable file from cmd.exe or /bin/bash in Linux to a batch script (Windows) or shell script (Linux) file. Here's my approach:
1, for Windows OS, create a run.bat file which include the python.exe and the %* to pass all arguments to it:
C:\YOUR_PYTHON_PATH\python.exe %*
2, for LInux OS, create a run.sh file to execute python with arguments:
#!/bin/bash
/usr/bin/python3 "$#"
3, use Process and ProcessStartInfo (your second approach):
string fileName = null;
if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
{
fileName = "path_to_bat/run.bat"
}
else
{
fileName = "path_to_bat/run.sh"
}
ProcessStartInfo start = new ProcessStartInfo
{
FileName = fileName,
Arguments = string.Format("\"{0}\" \"{1}\"", script, args),
UseShellExecute = false,
CreateNoWindow = true,
RedirectStandardOutput = true,
RedirectStandardError = true
};
using Process process = Process.Start(start);
the .NET code is same to Windows except the FileName should be the shell script's name with path.

Calling R script from Python does not save log file in version 4

I am calling from python an R script that is very simple called R Code.R:
args <- commandArgs(TRUE)
print(args)
source(args[1])
setwd("<YOUR PATH>")
output <- head(mtcars, n = n)
write.table(output, "output.txt")
using the following script:
import subprocess
pth = "<YOUR PATH>"
subprocess.call(" ".join(["C:/R/R-3.6.0/bin/x64/R.exe", "-f", '"' + pth + '/R Code.R"', "--args",
'"' + pth + '/arguments.txt"',"1>", '"' + pth + '/log.txt"', "2>&1"]))
subprocess.call(" ".join(["C:/R/R-4.0.3/bin/x64/R.exe", "-f", '"' + pth + '/R Code.R"', "--args",
'"' + pth + '/arguments.txt"',"1>", '"' + pth + '/log.txt"', "2>&1"]))
where arguments.txt contain:
n <- 10
The problem is that when I am using R-4.0.3 the log.txt file is not generating and I need to dump a log file because it is automatically looking for it in a posterior process I have.
When I am executing in CMD (Windows) the following command:
C:/R/R-4.0.3/bin/x64/R.exe -f "<YOUR PATH>/R Code.R" --args "<YOUR PATH>/arguments.txt" 1> "<YOUR PATH>/log.txt" 2>&1'
It does work perfectly, it is only when embedded in another software.
Also, I have tried without white space in the name and calling the scripts from root folder without having to specify the path.
Any idea of why it doesn't work for R-4.* or even better, how to solve it?
Thank you!
PD: Thank you, Martin, for your tips and for making me formulate a better question

Rhelp people got this solved, thank you, Duncan Murdoch!
Solution 1:
import os
pth = "<YOUR PATH>"
os.system(" ".join(["C:/R/R-4.0.3/bin/x64/R.exe", "-f", '"' + pth + '/RCode.R"', "--args",
'"' + pth + '/arguments.txt"',"1>", '"' + pth + '/log.txt"']))
Solution 2:
import subprocess
pth = "<YOUR PATH>"
subprocess.call(" ".join(["1>", '"' + pth + '/log.txt"', "2>&1",
"C:/R/R-4.0.3/bin/x64/R.exe", "-f", '"' + pth + '/RCode.R"', "--args",
'"' + pth + '/arguments.txt"']), shell = True)

well, in one case (3.6.0) you use R.exe , in the other (4.0.3) Rscript.exe.
Both R and Rscript have existed for a long time, and they have always had slightly different behavior.
You really should not confuse them with each other (even though, on Windows, I see, they look like the same file .. they will not behave the same).
Ok, now you use R.exe for both.
Just to find out more / see more where the problem may happen, can you try
all of
using a minimal reproducible example, i.e. one we can use directly ourselves, i.e., not using "<YOUR PATH>" (nor setwd(.))
not using file names with a ' ' (space), i.e., e.g., use code.R
calling this from a "terminal"/shell instead of as python subprocess ?
Last but not least: Yes, for R 4.0.0, a completely updated toolset ("brandnew toolchain", e.g. much newer C compiler) was used to build R for windows, "Rtools 4.0" or rtools40: https://cran.r-project.org/bin/windows/Rtools/ . So changes are expected but should typically only have been to the better, not the worse ..

How can I use Python path in my Visual Studio Code extension?

I'm writing my first VSCode extension. In short, the extension opens a terminal (PowerShell) and executes a command:
term = vscode.window.activeTerminal;
term.sendText("ufs put C:\\\Users\\<userID>\\AppData\\Local\\Programs\\Python\\Python38-32\\Lib\\site-packages\\mymodule.py");
After selecting a Python environment, VSCode should know where Python is located (python.pythonPath). But the path to Python will obviously vary depending on the Python installation, version and so on. So I was hoping that I could do something like:
term.sendText("ufs put python.pythonPath\\Lib\\site-packages\\mymodule.py");
But how can I do this in my extension (TypeScript)? How do I refer to python.pythonPath?
My configuration:
Windows 10
Python 3.8.2
VSCode 1.43.2
Microsofts Python extension 2020.3.69010
Node.js 12.16.1
UPDATE:
Nathan, thank you for your comment. I used a child process as suggested. It executes a command to look for pip. Relying on the location of pip is not bullet proof, but it works for now:
var pwd = 'python.exe -m pip --version';
const cp = require('child_process');
cp.exec(pwd, (err, stdout, stderr) => {
console.log('Stdout: ' + stdout);
console.log('Stderr: ' + stderr);
if (err) {
console.log('error: ' + err);
}
});
Not sure where to go from here to process stdout, but I tried child_process.spawn using this accepted answer:
function getPath(cmd, callback) {
var spawn = require('child_process').spawn;
var command = spawn(cmd);
var result = '';
command.stdout.on('data', function (data) {
result += data.toString();
});
command.on('close', function (code) {
return callback(result);
});
}
let runCommand = vscode.commands.registerCommand('extension.getPath', function () {
var resultString = getPath("python.exe -m pip --version", function (result) { console.log(result) });
console.log(resultString);
});
Hopefully, this would give me stdout as a string. But the only thing I got was undefined. I'm way beyond my comfort zone now. Please advise me how to proceed.

Run a perl script with Python on multiple files at once in a folder

This is my perl script at the moment:
#!/usr/bin/perl
use open qw/:std :utf8/;
use strict;
use warnings;
if (defined $ARGV[0]){
my $filename = $ARGV[0];
my %count;
open (my $fh, $filename) or die "Can't open '$filename' $!";
while (<$fh>)
{
$count{ lc $1 }++ while /(\w+)/g;
}
close $fh;
my $array = 0;
foreach my $word ( sort { $count{$b} <=> $count{$a} } keys %count)
{
print "$count{$word} $word\n" if $array++ < 10;
}
}else{
print "Please enter the name of the file: ";
my $filename = ($_ = <STDIN>);
my %count;
open (my $fh, $filename) or die "Can't open '$filename' $!";
while (<$fh>)
{
$count{ lc $1 }++ while /(\w+)/g;
}
close $fh;
my $array = 0;
foreach my $word ( sort { $count{$b} <=> $count{$a} } keys %count)
{
print "$count{$word} $word\n" if $array++ < 10;
}
}
And this is my Python script at the moment:
#!/usr/bin/env python3
import os
perlscript = "perl " + " perlscript.pl " + " /home/user/Desktop/data/*.txt " + " >> " + "/home/user/Desktop/results/output.txt"
os.system(perlscript)
Problem: When there are multiple txt-files in the data folder the script only runs on one file and ignores all the other txt-files. Is there a way to run the perlscript on all the txt-files at once?
Another problem: I'm also trying to delete the txt-files with the os.remove after they have been executed but they get deleted before the perlscript has a chance to execute.
Any ideas? :)

That Perl script processes one file. Also, that string passed to shell via os.system doesn't get expanded into a valid command with a file list as intended with the * shell glob.
Instead, build the file list in Python, using os.listdir or glob.glob or os.walk. Then iterate over the list and call that Perl script on each file, if it must process only one file at a time. Or, modify the Perl script to process multiple files and run it once with the whole list.
To keep the current Perl script and run it on each file
import os
data_path = "/home/user/Desktop/data/"
output_path = "/home/user/Desktop/result/"
for file in os.listdir(data_path):
if not file.endswith(".txt"):
continue
print("Processing " + file) # better use subprocess
run_perlscript = "perl " + " perlscript.pl " + \
data_path + file + " >> " + output_path + "output.txt"
os.system(run_perlscript)
The Perl script need be rewritten to lose that unneeded code duplication.
However, it is better to use subprocess module to run and manage external commands. This is advised even in the os.system documentation itself.
For instance
import subprocess
with open(output_path + "output.txt", "a") as fout:
for file in os.listdir(path):
if not file.endswith(".txt"):
continue
subprocess.run(["perl", "script.pl", data_path + file], stdout=fout)
where the file is opened in the append mode ("a") following the question's >> redirection.
The recommended subprocess.run is available since python 3.5; otherwise use Popen.
Another, and arguably "right," option is to adjust the Perl script so that it can process multiple files. Then you only need run it once, with the whole file list.
use strict;
use warnings;
use feature 'say';
use open ':std', ':encoding(UTF-8)';
foreach my $filename (#ARGV) {
say "Processing $filename";
my %count;
open my $fh, '<', $filename or do {
warn "Can't open '$filename': $!";
next;
};
while (<$fh>) {
$count{ lc $1 }++ while /(\w+)/g;
}
close $fh;
my $prn_cnt = 0;
foreach my $word ( sort { $count{$b} <=> $count{$a} } keys %count) {
print "$count{$word} $word\n" if $prn_cnt++ < 10;
}
}
This prints a warning on a file that it can't open and skips to the next one. If you'd rather have the script exit on any unexpected file replace or do { ... }; with the original die.
Then, and using glob.glob as an example now
import subprocess
data_path = "/home/user/Desktop/data/"
output_path = "/home/user/Desktop/result/"
files = glob.glob(data_path + "*.txt")
with open(output_path + "output.txt", "a") as fout:
subprocess.run(["perl", "script.pl", files], stdout=fout)
Since this passes the whole list as command arguments it assumes that there aren't (high) thousands of files, to exceed some length limits on pipes or command-line.