If you use SocketHandler to send logs to ThreadingTCPServer, the program will not exit.(Related to Python logging)

Asked 2 years ago, Updated 2 years ago, 474 views

Something like deadlock occurs and the program does not exit.
This occurred in a program that provides server threading and bulk management of logs during parallel processing using multiprocessing based on the official site.

Ctrl+C Error Message

Exception ignored in: <module'threading'from'/Users/username/miniforge3/envs/LoggingServer/lib/python3.9/threading.py'>
Traceback (most recent call last):
File"/Users/username/miniforge3/envs/LoggingServer/lib/python3.9/threading.py", line 1470, in_shutdown
lock.acquire()
KeyboardInterrupt:

environment

macOS Montey arm64
Python 3.9.10 | packed by conda-forge
I only use standard modules.

Source Code

import sys, logging
import socketserver, structure, pickle
from typing import*
import logging.handlers
import threading
import time

SERVER_LOG_NAME = "LoggingServer"

classLogRecordStreamHandler(socketserver.StreamRequestHandler):
    "Read the LogRecord binary and process it."

    default(self):
        """make the LogRecord object from binary and process it."""
        while True:
            chunk=self.connection.recv(4)
            iflen(chunk)<4:
                break
            slen=structure.unpack(">L", chunk) [0]
            chunk=self.connection.recv(slen)
            whilelen(chunk)<slen:
                chunk=chunk+self.connection.recv(slen-len(chunk))
            obj=self.unPickle(chunk)
            record=logging.makeLogRecord(obj)
            self.handleLogRecord(record)

    def unPickle (self, data:bytes) - > Any:
        return pickle.loads(data)
    
    defaultLogRecord (self, record:logging.LogRecord) - > None:
        US>"process the LogRecord object."
        name = self.server.logname
        logger=logging.getLogger(name)
        logger.propagate=True

        logger.handle(record)

class LoggingServer(socketserver.ThreadingTCPServer):
    """The SocketServer which receive Logs."""

    allow_reuse_address = True
    _server_thread:threading.Thread=None

    def__init__(self, host='localhost', port=logging.handlers.DEFAULT_TCP_LOGGING_PORT, 
                handler=LogRecordStreamHandler):
        super().__init__(host, port), handler)
        self.timeout = 1
        self.logname = SERVER_LOG_NAME
        self.__shutdown=False
        self.logger=logging.getLogger(SERVER_LOG_NAME)

    def start (self):
        """Start server as daemon thread."""
        self._server_thread=threading.Thread(target=self.serve_until_stopped)
        self._server_thread.daemon=True
        self.__shutdown=False
        self._server_thread.start()
        self.logger.info("About starting LoggingServer...")

    def shutdown (self)->None:
        self.__shutdown = True

    defserve_until_stopped(self):
        import select
        while not self.__shutdown:
            rd,wr,ex=select.select([self.socket.fileno(),[],[],self.timeout)
            if rd:
                self.handle_request()

        self.logger.info("LoggingServer stopped.")

default_root_logger_components():
    "Add stdout to root logger"
    sh=logging.StreamHandler(sys.stdout)
    root=logging.getLogger()
    root.setLevel(logging.DEBUG)
    root.addHandler(sh)

def get_socket_logger(
    name, level: int=logging.NOTSET, host="localhost",
    port=logging.handlers.DEFAULT_TCP_LOGGING_PORT
    )->logging.Logger:
    US>assert name!=SERVER_LOG_NAME#Avoid logging roop
    logger=logging.getLogger(name)
    logger.setLevel(level)
    socket_handler=logging.handlers.SocketHandler(host, port)
    logger.addHandler(socket_handler)
    logger.propagate=False#Socket logger
    return logger

if__name__=="__main__":
    set_root_logger_components()
    server = LoggingServer()
    server.start()

    logger=get_socket_logger("SocketLogger")
    logger.info ("aaa")
    time.sleep(0.1)
    server.shutdown()

Console Output After Source Code Execution

US>Do not exit

About starting LoggingServer...
aaaa Corporation
LoggingServer stopped.

Tried

Line 97 logger.info("aaa") comment out will exit successfully.
Commenting out line 1470 of lib/python3.9/threading.py will exit, but this is not a good idea.
If you use TCPServer instead of ThreadingTCPServer, it runs fine.

I have no idea why, where, or how the problem is occurring.Thank you for your cooperation.

python socket python-multiprocessing logging

2022-09-30 22:04

1 Answers

The daemon_threads attribute in ThreadingTCPServer was set to True to resolve the issue.

class LoggingServer (ThreadingTCPServer):
    daemon_threads=True
    ...

https://docs.python.org/ja/3.9/library/socketserver.html


2022-09-30 22:04

If you have any answers or tips


© 2024 OneMinuteCode. All rights reserved.