Get subprocess.Popen stdout and stderr in real time

After processing the contents of stdout and stderr of the child process with python's subprocess.Popen, I would like to output them to the parent's stdout and stderr in real time and in order, respectively.
For example, I would like to run hoge.sh as shown in Example.

hoge.sh

 echoes1
echo no1>&2
echoes21
echoes22
echo no2>&2

fuga.py:

import subprocess as sp
p=sp.Popen(['bash', 'hoge.sh'], stdout=sp.PIPE, stderr=sp.PIPE,...)

~~  p.stdout starts with "out:" and p.stderr outputs with "err:"~

p.wait()

Example Execution:

$python fuga.py 加工Processed stdout stderr is output in order
out —yes1
err —No1
out —Yes 21
out —Yes 22
err —No2
$ python fuga.py>/dev/null 加工Only processed stderr will be printed
err —No1
err —No2

The following code I found online is the closest to my purpose at the moment, but this code does not guarantee the order of stdout and stderr.

#!/usr/bin/python3
import subprocess
import sys
from subprocess import PIPE
from threading import Thread


default_stream(in_file, out_file):
    For line in in_file:
        print(line.strip(), file=out_file, flush=True)

p=subprocess.Popen("."/test_stream.py", stdin=sys.stdin, stdout=PIPE, stderr=PIPE, universal_newlines=True)
Thread(target=read_stream, args=(p.stdout, sys.stdout)) .start()
Thread(target=read_stream, args=(p.stderr, sys.stderr)) .start()

Output: (stdout and stderr may change order)

 out:yes1
err —No1
out —Yes 21
err —No2
out —Yes 22

Thank you for your help.

python python3 linux posix python-multiprocessing

2022-09-30 19:35

1 Answers

Stdout and stderr may change order

Application (bash) input and output may be buffered
See if there is a boot option that allows the app to explicitly flush the buffer.
https://gist.github.com/riywo/874011

There seems to be no such option for bash itself, so
How to Reduce the Standard Output Buffer Size for Linux

stdbuf-i0-o0-e0<command>

https://stackoverflow.com/questions/36847897/in-bash-how-can-i-force-a-flush-of-an-incomplete-line-printed-to-the-terminal

Try the .

Also, I think that the string processing will be done in multiple threads in read_stream, so
There is a possibility that the order will be changed at this time, so the app will take action.

and so on.

I haven't tried it, so it might be different.
For your information...

2022-09-30 19:35

If you have any answers or tips

Popular Tags

python x 4647

android x 1593

java x 1494

javascript x 1427

c x 927

c++ x 878

ruby-on-rails x 696

php x 692

python3 x 685

html x 656