I am trying to create a website where a user can submit python code, it gets sent to my server to be executed and I send back the results to the client. Currently I am using a NodeJs server and need to run the python code from there. To do this, I am using Python-shell like so:
const runPy = async (code) => {
const options = {
mode: 'text',
pythonOptions: ['-u'],
scriptPath: path.join(__dirname, '../'),
args: [code],
};
const result = await PythonShell.run('script.py', options, (err, results) => {
if (err) throw err;
return results; <----- HOW DO I RETURN THIS
});
console.log(result.stdout);
return result;
};
I understand I can console.log() the results in the PythonShell.run() but is there a way to return the results from my runPy function to then be manipulated and sent back to the client?
It looks from the python-shell documentation that the PythonShell.run method doesn't have an async mode. So, one option is to wrap it in a promise:
const runPy = async (code) => {
const options = {
mode: 'text',
pythonOptions: ['-u'],
scriptPath: path.join(__dirname, '../'),
args: [code],
};
// wrap it in a promise, and `await` the result
const result = await new Promise((resolve, reject) => {
PythonShell.run('script.py', options, (err, results) => {
if (err) return reject(err);
return resolve(results);
});
});
console.log(result.stdout);
return result;
};
Related
I want to call API functions in python that return strings through typescript.
I have tried PythonShell, with no luck. I didn't get any errors, but I wasn't getting any values back from python and I was not getting any evidence to suggest that the python was getting the value sent from typescript.
Dummy.py
print("Hello World", flush=True, end='')
hello-world.ts
const { spawn } = require('child_process');
const sensor = spawn('python', ['dummy.py']);
sensor.stdout.on('data', function(data) {
sensor.stdout.on('data', (data) => {
console.log(`stdout: ${data}`);
});
sensor.stderr.on('data', (data) => {
console.log(`stderr: ${data}`);
});
sensor.on('exit', (code) => {
console.log(`Python process ended with code: ${code}`);
});
});
I only get "Python process ended with code: 0" when I run this.
I would recommend to use the libary "subprocess"
import subprocess
def run_python_script(script, args):
result = subprocess.run(["python", script] + args, capture_output=True, text=True)
return result.stdout
And in TypeScript, you can use this Python function to run your script:
import { run_python_script } from "./your-python-module";
async function runScript(script: string, args: string[]): Promise<string> {
return run_python_script(script, args);
}
I am wondering how to use node.js to run a python file that prints until it stops.
Right now when I run it it does not print anything, is there a way I can make it work properly?
Node.js
let {PythonShell} = require('python-shell')
var options = {
pythonOptions: ['-u']
};
PythonShell.run('main.py', options, function (err, result){
if (err) throw err;
// result is an array consisting of messages collected
//during execution of script.
if (result !== null){
console.log(result.toString());
}
});
PythonShell.run('main.py', options, function (err, result){
if (err) throw err;
// result is an array consisting of messages collected
//during execution of script.
if (result !== null){
console.log(result.toString());
}
});
A function similar to mine
main.py
num = 1
while True:
print(num)
num += 1
I'm not familiar with python-shell package but you can easily spawn a new process to run python programs by using spawn method from child_process package that comes with node.
Here is how you can use it:
const { spawn } = require("child_process");
const cmd = spawn("python3", [__dirname + "/main.py"]);
cmd.stdout.on("data", (data) => {
console.log(`stdout: ${data}`);
});
cmd.stderr.on("data", (data) => {
console.error(`stderr: ${data}`);
});
cmd.on("close", (code) => {
console.log(`child process exited with code ${code}`);
});
Read the documentation for more info [link]
I am working on an ethereum application that uses react, next-routing and a basic node.js server. I want to run a python script on a specific route, claimAnalysis to be specific, in order to perform some predictions. I want the script to be executed every time I visit the claimAnalysis route.
server.js
const express = require("express")();
const next = require("next");
const app = next({
dev: process.env.NODE_ENV !== "production"
});
const routes = require("./routes");
const handler = routes.getRequestHandler(app);
app.prepare().then(() => {
const server = express.use(handler);
server.listen(3000, (err) => {
if(err) throw err;
console.log("Server ready on localhost:3000");
});
});
routes.js
const routes = require("next-routes")();
routes
.add("/insurance/new", "/insurance/new")
.add("/insurance/:address", "/insurance/show")
.add("/insurance/:address/claims", "/insurance/claims/index")
.add("/insurance/:address/claims/new", "/insurance/claims/new")
.add("/insurance/:address/claims/:id/claimAnalysis", "/insurance/claims/claimAnalysis");
module.exports = routes;
Please guide me if I can call a function from the claimAnalysis.js file that runs the python script.
If you're using Node 10+, you can use util.promisify to execute your python scriptwhich returns a Promise<{ stdout, stderr }> object. See an example below:
const util = require('util');
const exec = util.promisify(require('child_process').exec);
// function to execute python script
async function executeScript(path) {
try {
const { stdout, stderr } = await exec(`python3 ${path}`);
console.log('stdout:', stdout);
console.log('stderr:', stderr);
} catch (e) {
console.error(e);
}
}
Now let's use the function in your route:
app.get('/insurance/claims/claimAnalysis', async function (req, res) {
await executeScript();
res.send('claimAnalysis request completed');
...
})
You can use "child_process" to finish function.
You can see this example:
const path = require('path')
const {spawn} = require('child_process')
/**
* Run python myscript, pass in `-u` to not buffer console output
* #return {ChildProcess}
*/
function runScript(){
return spawn('python', [
"-u",
path.join(__dirname, 'myscript.py'),
"--foo", "some value for foo",
]);
}
const subprocess = runScript()
// print output of script
subprocess.stdout.on('data', (data) => {
console.log(`data:${data}`);
});
subprocess.stderr.on('data', (data) => {
console.log(`error:${data}`);
});
subprocess.stderr.on('close', () => {
console.log("Closed");
});
Here i want to send the next() when my data is received from the python script but i am getting Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client.
const {PythonShell} = require('python-shell');
module.exports = (req, res, next)=>{
let myPythonScriptPath = 'encrypt.py';
const pyshell = new PythonShell(myPythonScriptPath);
let path = req.file.path;
pyshell.send(path);
pyshell.on("message", function(data){
console.log(data);
req.filepath = data;
next();
})
// end the input stream and allow the process to exit
pyshell.end(function (err) {
if (err){
throw err;
}
});
}
but it is working when i put next() at the end of code but then my req.filepath is undefined
const {PythonShell} = require('python-shell');
module.exports = (req, res, next)=>{
let myPythonScriptPath = 'encrypt.py';
const pyshell = new PythonShell(myPythonScriptPath);
let path = req.file.path;
pyshell.send(path);
pyshell.on("message", function(data){
console.log(data);
req.filepath = data;
})
// end the input stream and allow the process to exit
pyshell.end(function (err) {
if (err){
throw err;
}
});
next();
}
What i want is to store the data coming from python script in req.filepath which is to be sent to next middleware. Can anyone help me with this?
You're seeing this behavior due to the order of asynchronous tasks, which I've enumerated below in the comments. The execution doesn't necessarily happen in the same sequence that the code is written, so, you're either trying to modify a request/response after the PyShell command has already been sent (case #1 above), or Express has already moved on to the next middleware before req.filepath has been set (case #2 above).
const {PythonShell} = require('python-shell');
module.exports = (req, res, next) => {
let myPythonScriptPath = 'encrypt.py';
const pyshell = new PythonShell(myPythonScriptPath);
let path = req.file.path;
pyshell.send(path); // 1) Send the command
// 3) This anonymous function is invoked, but it's
// too late -- the command has already been sent
// and has continued on to the next middleware.
pyshell.on("message", function(data){
console.log(data);
// This will be undefined in the next middleware
// because it moved on before it was actually set.
req.filepath = data;
})
// 4) This anonymous function gets run last.
pyshell.end(function (err) {
if (err){
throw err;
}
});
next() // 2) Tell Express to continue on to the next middleware.
}
Do you have other middleware before or after this one that is trying to modify or send an HTTP response?
Using callback methodolgy this can be acheived by sending the result of the function in callback().(Here callback is next)
The code is given below:
const {PythonShell} = require('python-shell');
module.exports = function (req, res, next){
try{
let myPythonScriptPath = '/home/anmolmiddha/Projects/coreshare/server/api/python/encrypt.py';
const pyshell = new PythonShell(myPythonScriptPath);
let path = req.file.path;
pyshell.send(path);
pyshell.on("message", function(data){
});
pyshell.end(function (err, rslt) {
if(err){
res.status(500).json(err);
}
else{
req.filepath = JSON.stringify(path).split('"')[1];
next(rslt);
}
});
}
catch(error) {
return res.status(401).json({
message: "Invalid token Auth failed"
})
}
}
It doesn't look to me that using python to combine all the json files is convenient, and the combined file would be 30G.
I am using mongoDB and nodejs. The way how I populate a sample json is:
var data = require('./data1.json')
var populateDB = function() {
db.collection('temp', function(err, collection) {
collection.insert(data, {safe:true}, function(err, result) {});
});
};
This only adds one json file. How should I populate the collection with the 10000+ json files from here? any suggestion is highly appreciated!
The easiest way would be to write a Node program that processes one JSON file and then run it multiple time from the shell:
for i in *.json; do node program.js $i; done
Your Node program would just need to access the name from process.argv instead of having it hardcoded but the logic would be the same.
If you want to do everything in node then you will have to read the directory, get all .json files, read every one of them in sequence and then run a code similar to the one you posted. If it's a one off task then you can even get away with using the "Sync" functions to simplify your code if it's a sequential task to do one thing at a time and you don't care about adding the data in parallel.
Something like this would work
npm i glob-fs mongodb async --save
const async = require('async');
const fs = require('fs');
const glob = require('glob-fs')({ gitignore: true });
const MongoClient = require('mongodb').MongoClient;
const files = './files/data*.json';
const collection = 'test';
const url = 'mongodb://localhost:27017/test';
// Connect to db
MongoClient.connect(url, function (err, db) {
if (err) {
console.log(err);
}
// Get the collection
const col = db.collection(collection);
glob.readdirPromise(files)
.then(function (f) {
return async.eachSeries(f, (item, callback) => {
fs.readFile(item, 'utf8', function (err, data) {
if (err) {
return console.log(err);
}
// Insert into mongo
col.insertMany(JSON.parse(data)).then((r) => {
console.log(r);
return callback(r);
}).catch(function (fail) {
console.log(fail)
});
});
}, err => {
console.log(err);
});
})
.then(err => {
if (err) {
db.close();
}
})
.catch(err => {
console.log(err);
});
});