A Web 502 Error Caused By A Docker Upgrade

发表于 更新于 阅读次数： 本文字数： 821 阅读时长 ≈ 3 分钟

💡 This post is insightful for the following scenarios.

  • Handle Docker or DevOps Error
  • Server Down
  • Rookie DevOps

Background

The docker image is upgraded. And I need to deal with the error it causes.

Framework

  • Amazon Cloud: EC2 Instance
  • Gitlab: Pipeline
  • Dockerhub

Operation & Problem

1. Upgrade the docker image

Update the template in EC2 instance. Deploy a new environment by GitLab.

2. 502 Error

Yeah…it causes 502 error obviously.

3. Check the docker container

In fact, it took quite a bit of time to jump from the previous step to this one.

At first, I mistakenly thought it was due to a network configuration error, so I kept checking the template and also the configuration file. But in fact, the 502 error can be interpreted as a service provisioning failure, and therefore incoming requests cannot be accessed.

So the first step of troubleshooting should be to check whether the service is started properly, and then start the rest of the troubleshooting later.

1
2
# check all the containers
docker ps -a

As we can see from the following picture, the containers started and exited many times in about 3min. It means there are some error when waking up the image

4. Track error information - exitcode 126

Get the basic information from container.

1
docker inspect ContainerID

There are some information we may be interested in.

  • ExitCode: The reason why a container exit. You can check this page for more specific exitcode. It is always a error summary. For 126, it means that a command used in the container specification could not be invoked.
  • LogPath: The location where docker containers save their log

Print out the log shown on the inspect, which is excatly a great example for exitcode-126.

5. Upgrade docker image - to use user “odoo” to install pyenv

As you can see from the previous post, the error is introduced at that time.

The following dockerfile sentence will install pyenv into /root/, which results in that user “odoo” can not call python3.8.

1
2
3
4
5
6
7
8
USER root

# omit some sentences

# install pyenv, install python3.8.15 and make it global
RUN curl -L https://raw.githubusercontent.com/yyuu/pyenv-installer/master/bin/pyenv-installer | bash && printf "export PATH=\"/root/.pyenv/bin:\$PATH\"\neval \"\$(pyenv init -)\"\neval \"\$(pyenv virtualenv-init -)\"\n\nexport LC_ALL=C.UTF-8\nexport LANG=C.UTF-8" >> ~/.bashrc && source ~/.bashrc && pyenv install 3.8.15 && pyenv global 3.8.15

# omit some sentences

Therefore, after lots of unsuccessful trial, I choose to let odoo install the pyenv.

1
2
3
4
5
6
7
8
9
10
11
USER root

# omit some sentences

# use user "odoo" install pyenv, install python3.8.15 and make it global
USER odoo
RUN curl -L https://raw.githubusercontent.com/yyuu/pyenv-installer/master/bin/pyenv-installer | bash
RUN printf "export PYENV_ROOT=\"\$HOME/.pyenv\"\ncommand -v pyenv >/dev/null || export PATH=\"\$PYENV_ROOT/bin:\$PATH\"\neval \"\$(pyenv init -)\"\neval \"\$(pyenv virtualenv-init -)\"\n" >> ~/.bashrc && source ~/.bashrc && pyenv install 3.8.15 && pyenv global 3.8.15

USER root
# omit some sentences

6. Upload docker image to docker hub

After downloading lots of third-party libraries, the size of image increased to almost 10GB.

There’s not much more to say. In the future, I will only recommend using the command line for uploading docker images that are too large, which took me 5 hours to learn this lesson.

7. 502 Error

🧘‍♀️ Take it easy.

8. Fail to download docker image in server

Though I click the “Run” button in the pipeline, I still can not find any containers on the server.

1
2
3
4
5
# check new version container
docker ps -a

# check new version image
docker image ls

After waiting for a long time, I still can not see the new version image. At this time, you can download it by yourself.

BTW, The way for the server to download a docker image:

  • check if the docker image exist locally
    • if exist, check the docker hub if the version updated (imageID)
      • if updated, pull the new version into local
      • if not updated, just use current one
    • if not exist, pull the new version into local

9. 502 Error

🧘‍♀️

10. Track error information - exitcode 127

Here is the log

Here is some python feature. Some .py files will put “#!/usr/bin/env python3” in their first line. Therefore, the file can be run by system python3 parser.

According to the log, there seems to be no python in this image, which is completely different from my local environment.

As the docker is run by the command “/entrypoint.sh odoo”, I need to debug by putting some output in the entrypoint.sh to see what is going on.

Here comes the result.

Fxck, it takes me about 4 hours. And the reason is that the .yaml file of this image change the folder.

Then just need to update the docker to change the pyenv location.

After reading the pyenv-installer script, I set up the env variable $PYENV_ROOT by dockerfile. And finally, everything works well!

11. Winner Winner Chicken Dinner

Reference