We are migrating our previously deployed blog system to the AWS EKS cluster.On the EC2 of the existing system, two containers operate: a web server (nginx) container and an AP server (django+gunicorn) container, which can be accessed successfully through a browser.So when I deployed to node (EC2) on AWS EKS in the same way, I couldn't access it from my browser and it says 502 Bad Gateway, and the gunicorn log shows the message [CRITICAL] WORKER TIMEOUT (pid:18294) at the time of browser access.We are currently investigating the cause of this, but we do not know the current situation.If anyone has any idea, please let me know.
root@blogsystem-appserver01:/#systemctl status unicorn
● gunicorn.service-gunicorn daemon
Loaded: loaded (/etc/systemd/system/gunicorn.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2021-05-0908:57:19 UTC; 5 daysago
Main PID: 18291 (gunicorn)
Tasks:4 (limit:4636)
Memory: 95.8M
CGgroup: /kubepods/bestefort/podd270872c-cc5b-4a3b-92ed-f463ee5f5d77/1eafc79ffd656ff1c39175ee06c7a5ca8692715c5e2bfe2f979d8718411ba/system.slice/gunicorn.service
--18291/home/ubuntu/python3/bin/python/home/ubuntu/python3/bin/gunicorn --access-logfile---workers3 --bindunix:/home/ubuntu/socket/myproject.sock myproject.wsgi:a
application
--18295/home/ubuntu/python3/bin/python/home/ubuntu/python3/bin/gunicorn --access-logfile---workers3 --bindunix:/home/ubuntu/socket/myproject.sock myproject.wsgi:a
application
--18299/home/ubuntu/python3/bin/python/home/ubuntu/python3/bin/gunicorn --access-logfile---workers3 --bindunix:/home/ubuntu/socket/myproject.sock myproject.wsgi:a
application
--18300/home/ubuntu/python3/bin/python/home/ubuntu/python3/bin/gunicorn --access-logfile---workers3--bindunix:/home/ubuntu/socket/myproject.sockmyproject.wsi:a
application
May 0908:57:20 blogsystem-appserver01gunicorn [18291]: [2021-05-0908:57:20+0000][18291][INFO] Starting gunicorn 20.0.4
May 0908:57:20 blogsystem-appserver01gunicorn [18291]: [2021-05-0908:57:20+0000][18291][INFO] Listening at: unix:/home/ubuntu/socket/myproject.sock (18291)
May 0908:57:20 blogsystem-appserver01gunicorn [18291]: [2021-05-0908:57:20+0000][18291][INFO] Using worker:sync
May 0908:57:20 blogsystem-appserver01gunicorn [18293]: [2021-05-0908:57:20+0000][18293][INFO] Booting worker with pid:18293
May 0908:57:20 blogsystem-appserver01gunicorn [18294]: [2021-05-0908:57:20+0000][18294][INFO] Booting worker with pid:18294
May 0908:57:20 blogsystem-appserver01gunicorn [18295]: [2021-05-0908:57:20+0000][18295][INFO] Booting worker with pid:18295
May 0908:57:59 blogsystem-apserver01gunicorn [18291]: [2021-05-0908:57:59+0000] [18291] [CRITICAL] WORKER TIMEOUT (pid:18293)
May 0908:58:00 blogsystem-appserver01gunicorn [18299]: [2021-05-0908:58:00+0000][18299][INFO] Booting worker with pid:18299
May 0908:58:01 blogsystem-apserver01gunicorn [18291]: [2021-05-0908:58:01+0000] [18291] [CRITICAL] WORKER TIMEOUT (pid:18294)
May 0908:58:02 blogsystem-appserver01gunicorn [18300]: [2021-05-0908:58:02+0000][18300][INFO] Booting worker with pid:18300
root@blogsystem-appserver01:/#
I've looked into many things, but I can't say for sure, but it seems that changing the "sync" worker in Gunicorn to "give" worker may solve the problem.
reference:
https://github.com/benoitc/gunicorn/issues/1194
I tried to edit the communicorn configuration file and change it to "give" worker as below, but when I restart the communicorn and look at the status, it says "RuntimeError: gevent worker requirements gevent 1.4 or higher" and I can't start the communicorn.Then, I installed 1.4 or higher version of gevent with python3-mpip install gevent, and it says RuntimeError: gevent worker require gevent 1.4 or higher again."I think there is a possibility that this matter may be related to the above mentioned ""Worker Timeout"", so I would appreciate it if you could let me know if anyone knows how to solve it."
·gunicorn configuration file
(python3)ubuntu@blogsystem-appserver01:/etc/systemd/system$moregunicorn.service
Unit
Description=gunicorn daemon
After=network.target
Service
User=ubuntu
Group = www-data
WorkingDirectory=/home/ubuntu/myproject
ExecStart=/home/ubuntu/python3/bin/gunicorn --access-logfile---workers3 --worker-class gevent -- bind unix:/home/ubuntu/socket/myproject.sock myproject.wsi:application
Install
WantedBy=multi-user.target
(python3)ubuntu@blogsystem-apserver01:/etc/systemd/system$
·gunicorn status
root@blogsystem-apserver01:/etc/systemd/system#systemctl status unicorn
● gunicorn.service-gunicorn daemon
Loaded: loaded (/etc/systemd/system/gunicorn.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2021-05-1502:30:08 UTC; 1sago
Process: 19182ExecStart=/home/ubuntu/python3/bin/gunicorn --access-logfile---workers3 --worker-class gevent -- bind unix:/home/ubuntu/socket/myproject.sock myproject.wsi:a
application(code=exited, status=1/FAILURE)
Main PID: 19182 (code=exited, status=1/FAILURE)
May 1502:30:08 blogsystem-apserver01gunicorn [19182]: File "<frozen importlib._bootstrap>", line 975, in_find_and_load_unlocked
May 1502:30:08 blogsystem-apserver01gunicorn [19182]: File "<frozen importlib._bootstrap>", line 671, in_load_unlocked
May 1502:30:08 blogsystem-apserver01gunicorn [19182]: File "<frozen importlib._bootstrap_external>", line 783, in exec_module
May 1502:30:08 blogsystem-apserver01gunicorn [19182]: File "<frozen importlib._bootstrap>", line 219, in_call_with_frames_removed
May 1502:30:08 blogsystem-apserver01gunicorn [19182]: File "/home/ubuntu/python3/lib/python 3.8/site-packages/gunicorn/workers/ggevent.py", line 16, in <module>
May 1502:30:08 blogsystem-appserver01gunicorn [19182]: raise RuntimeError("gevent worker requirements gevent 1.4 or higher")
May 1502:30:08 blogsystem-appserver01gunicorn [19182]: RuntimeError: gevent worker requirements gevent 1.4 or higher
May 1502:30:08 blogsystem-appserver01gunicorn [19182]:]
May 1502:30:08 blogsystem-appserver01systemd[1]:gunicorn.service:Main process expired, code=exited, status=1/FAILURE
May 1502:30:08 blogsystem-appserver01systemd[1]:gunicorn.service: Failed with result 'exit-code'.
root@blogsystem-appserver01:/etc/systemd/system#
·Install gevent workers
root@blogsystem-appserver01:/etc/systemd/system#python3-mpip install gevent
Requirement already satified: gevent in /usr/local/lib/python 3.8/dist-packages (1.4.0)
Requirement already satified: greenlet> = 0.4.14 in/usr/local/lib/python 3.8/dist-packages (from event) (1.1.0)
WARNING: Running pip as root will break packages and permissions. You should install packages relatively by using venv: https://pip.pypa.io/warnings/venv
root@blogsystem-appserver01:/etc/systemd/system#
·Gunicorn status after installing the gevent worker and restarting it
root@blogsystem-apserver01:/etc/systemd/system#systemctl restartgunicorn
root@blogsystem-appserver01:/etc/systemd/system#
root@blogsystem-apserver01:/etc/systemd/system#systemctl status unicorn
● gunicorn.service-gunicorn daemon
Loaded: loaded (/etc/systemd/system/gunicorn.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2021-05-1503:08:42 UTC; 1sago
Process: 19196 ExecStart=/home/ubuntu/python3/bin/gunicorn --access-logfile---workers3 --worker-class gevent --bindunix:/home/ubuntu/socket/myproject.sock myproject.wsi:a
application(code=exited, status=1/FAILURE)
Main PID: 19196 (code=exited, status=1/FAILURE)
May 1503:08:42 blogsystem-apserver01gunicorn [19196]: File "<frozen importlib._bootstrap>", line 975, in_find_and_load_unlocked
May 1503:08:42 blogsystem-apserver01gunicorn [19196]: File "<frozen importlib._bootstrap>", line 671, in_load_unlocked
May 1503:08:42 blogsystem-apserver01gunicorn [19196]: File "<frozen importlib._bootstrap_external>", line 783, in exec_module
May 1503:08:42 blogsystem-apserver01gunicorn [19196]: File "<frozen importlib._bootstrap>", line 219, in_call_with_frames_removed
May 1503:08:42 blogsystem-apserver01gunicorn [19196]: File "/home/ubuntu/python3/lib/python 3.8/site-packages/gunicorn/workers/ggevent.py", line 16, in <module>
May 1503:08:42 blogsystem-appserver01gunicorn [19196]: raise RuntimeError("gevent worker requirements gevent 1.4 or higher")
May 1503:08:42 blogsystem-appserver01gunicorn [19196]: RuntimeError: gevent worker requirements gevent 1.4 or higher
May 1503:08:42 blogsystem-appserver01gunicorn [19196]:]
May 1503:08:42 blogsystem-appserver01systemd[1]:gunicorn.service:Main process expired, code=exited, status=1/FAILURE
May 1503:08:42 blogsystem-appserver01systemd[1]:gunicorn.service: Failed with result 'exit-code'.
root@blogsystem-appserver01:/etc/systemd/system#
I solved myself.Sorry for the trouble.
The reason was that the VPC of the RDS that was originally used in the existing environment (AWS) and the VPC of the newly built AWS EKS cluster were different.Therefore, the AP server was unable to connect to the RDS and the communicorn timed out.After installing ESK and RDS in the same VPC, the problem was resolved by providing browser access.
© 2024 OneMinuteCode. All rights reserved.