How to run a Python script from Node.js

calendar icon
29 Mar
2024
12 Apr
2023
scroll

Quite often we hear disputes about what is better to use Node.js or Python. But what if one day, you come up with a great idea to combine those, for instance to use some Python libraries in your application, but you just do not have any idea how to integrate it with your Node.js application. Of course you can always build API on top of Python backend (Flack, etc), but in that case you need to build, host and manage one more application, when you just need to run a single Python script. That's why I want to give you step by step instructions on how to achieve this.

To begin with, let's briefly define what are both languages.

Node.js is a server-side platform developed on Google Chrome’s JavaScript Engine. Node.js allows to develop data-intensive real-time web applications. It is written on JavaScript and allows to run applications on various operating systems such as Windows, Linux, and Mac OS.

Python is an object-oriented programming language used to create dynamic web applications. The syntax of Python and dynamic typing make it an ideal programming language for scripting.

Why you might need to run Python scripts from Node.js

  • Node js can't run heavy multitask as it works on a single thread basis, making it difficult for Node Js to execute CPU-bound tasks. Whenever any request is received, the system will complete the request once and only accept the other request. This process slows down the system and ends up in a massive delay.
  • Python is much more suitable for that back-end applications, that use numerical computations, big data solutions, and machine learning.
  • Python has one of the largest ecosystems out of any programming community and a lot of open-source libraries. For instance some of built-in AI libraries (Pylearn2, Tensorflow for neural networks, and Scikit-learn for data analysis).

Сall your Python script from Node.js using Child Process module

You will find several libraries that can help you with this issue. As for me, I prefer to use built in child_process. The node:child_process module provides the ability to spawn subprocesses. This capability is primarily provided by the <mark>child_process.spawn() function.</mark>

First, we need to create a server.js file as the server for our app. Where we suppose that <mark>x</mark> and <mark>y</mark> are large arrays.


const express = require('express');
const { spawnSync } = require('child_process');
const { readFile } = require('fs/promises');
const { appendFile } = require('fs/promises');
const { join } = require('path');
const app = express();

app.get("/", async (req, res, next)=>{
   const X = [1,2, 5] // large array
   const y = [[1,2], [2,3], [1,2]] // large array

   await appendFile(
            join(`scripts/args.json`),
            JSON.stringify({ X, y }),
            {
                encoding: 'utf-8',
                flag: 'w',
            },
        );
   const pythonProcess = await spawnSync('python3', [
     '/usr/src/app/scripts/python-script.py',
     'first_function',
     '/usr/src/app/scripts/args.json',
     '/usr/src/app/scripts/results.json'
   ]);
  const result = pythonProcess.stdout?.toString()?.trim();
  const error = pythonProcess.stderr?.toString()?.trim();

   const status = result === 'OK';
  if (status) {
    const buffer = await readFile('/usr/src/app/scripts/results.json');
    const resultParsed = JSON.parse(buffer?.toString());
    res.send(resultParsed.toString())
  } else {
    console.log(error)
    res.send(JSON.stringify({ status: 500, message: 'Server error' }))
  }
});

//Creates the server on default port 8000 and can be accessed through localhost:8000
const port=8000;
app.listen(port, () => console.log(`Server is listening on PORT ${port}`));

The first parameter, that was passed to the <mark>spawnSync</mark> is <mark>command &lt;string&gt;</mark> (the command to run, in this case is <mark>python3</mark> - the version of python that is required to run your libraries), the second is <mark>args &lt;string[]&gt;</mark> (list of string arguments). You can also pass the third parameter <mark>options</mark>, the more detailed information can be found by the link.

I propose to consider in more detail the list I am passing as the second argument:

  • '/usr/src/app/scripts/python-script.py' - is the path of the script I need to execute
  • 'first_function' - is the name of the function I want to run inside that file
  • '/usr/src/app/scripts/args.json' - is the path to the file with arguments to this function. Actually you can directly pass those arguments using <mark>JSON.stringify()</mark>function in Node.js, but as for me it's better to write those arguments down into the file and then read that from Python script. One of the reasons is that you might need to pass as arguments large arrays of data and this will cause errors
  • '/usr/src/app/scripts/results.json' - is the path, where I want my Python script to write results, after it is finished, so I can read that from Node.js.

Passed arguments can be found in <mark>sys.argv</mark>array in Python script. The <mark>sys.argv</mark> list is a list that contains command line arguments in a python program. Whenever we run a python program from a command line interface, we can pass different arguments to the program. The program stores all the arguments and the file name of the python file in the <mark>sys.argv</mark> list. This can be seen in more detail by writing a simple example script:

import json
import sys

def first_function():
    json_obj = open(sys.argv[2])
    data = json.load(json_obj)
    calculatedResults = [1,2,4,3] # it is just example of data to return, in fact you will calculate it bellow
    X = data["X"]
    y = data["y"]
    # do some your calculations based on X and y and put the result to the calculatedResults
    # print(make_pipeline)
    json_object_result = json.dumps(calculatedResults, indent=4)

    with open(sys.argv[3], "w") as outfile:
        outfile.write(json_object_result)
    print("OK")


if sys.argv[1] == 'first_function':
    first_function()

sys.stdout.flush()

In two words, in Node.js script we write down to the file all required arguments, run <mark>spawnSync</mark> passing list of arguments and after Python script reads passed arguments from the file, makes all calculations and writes down to the file all results. At the moment all this results can be read in Node.js from file.

Run Python script, that uses specific Python libraries

One of the main goals is to have ability to use built-in AI or other libraries. It's always not a good idea to install dependencies and libraries globally in a system, it can lead to version conflicts, deployment difficulties, and problems with multiple developers working on a project at the same time, for example. That's why the perfect way is to use Docker. You can build the image yourself, but the easiest solution is to use an existing one.

FROM nikolaik/python-nodejs:latest as development
WORKDIR /usr/src/app
RUN pip install scikit-learn numpy pandas simplejson
COPY ./package.json ./yarn.lock ./
RUN yarn
RUN yarn install
COPY ./ ./
RUN yarn build
ENV PORT 4000
EXPOSE $PORT


# Production mode
FROM nikolaik/python-nodejs:latest as production
RUN pip install scikit-learn numpy pandas simplejson
ARG NODE_ENV=production
ENV NODE_ENV=${NODE_ENV}
WORKDIR /usr/src/app
COPY --from=development /usr/src/app ./
ENV PORT 4000
EXPOSE $PORT
CMD ["node", "dist/main"]

For now we can use <mark>scikit-learn</mark>, <mark>numpy</mark>, <mark>pandas</mark> and <mark>simplejson</mark> libraries on Python script, as they are available in the created docker container.

By following the steps above, you can create a new process and execute a Python script from your Node.js application. This can be useful in cases where you need to integrate Python code into your Node.js application or when you need to perform certain tasks that are better suited for Python. Described simple app can be downloaded from the Github, there you can also find instructions how to run it.

Happy coding and wish you to have a nice day!

Writing team:
No items found.
Have a project
in your mind?
Let’s communicate.
Get expert estimation
Get expert estimation
expert postexpert photo

Frequently Asked Questions

No items found.

Frequently Asked Questions

copy iconcopy icon

Ready to discuss
your project with us?

Please, enter a valid email
By sending this form I confirm that I have read and accept the Privacy Policy

Thank you!
We will contact you ASAP!

error imageclose icon
Hmm...something went wrong. Please try again 🙏
SEND AGAIN
SEND AGAIN
error icon
clutch icon

Our clients say

The site developed by Halo Lab projected a very premium experience, successfully delivering the client’s messaging to customers. Despite external challenges, the team’s performance was exceptional.
Aaron Nwabuoku avatar
Aaron Nwabuoku
Founder, ChatKitty
Thanks to Halo Lab's work, the client scored 95 points on the PageSpeed insights test and increased their CR by 7.5%. They frequently communicated via Slack and Google Meet, ensuring an effective workflow.
Viktor Rovkach avatar
Viktor Rovkach
Brand Manager at felyx
The client is thrilled with the new site and excited to deploy it soon. Halo Lab manages tasks well and communicates regularly to ensure both sides are always on the same page and all of the client’s needs are addressed promptly.
Rahil Sachak Patwa avatar
Rahil Sachak Patwa
Founder, TutorChase

Join Halo Lab’s newsletter!

Get weekly updates on the newest stories, posts and case studies right in your mailbox.

Thank you for subscribing!
Please, enter a valid email
Thank you for subscribing!
Hmm...something went wrong.