原创, 服务器

Gitlab 备份恢复升级管理界面操作出现500错误

因为一些原因需要给Gitlab做迁移，顺便升级，比如从 13.1.11升级到16.9.2

给原Gitlab服务做备份，打包成tar文件, 生成的文件名可能类似这样的 1710355231_2024_03_13_13.1.11_gitlab_backup.tar

docker exec -i gitlabcontainer gitlab-rake gitlab:backup:create

在目标服务器上创建Gitlab服务， Gitlab版本要跟原Gitlab版本保持一致。

docker run -d -p 8880:80 -p 222:22 --name=gitlab13-1-11 -v /data/dockers/gitlab/config:/etc/gitlab -v /data/dockers/gitlab/data:/var/opt/gitlab -v /data/logs/gitlab:/var/log/gitlab -v /etc/localtime:/etc/localtime:ro --restart=always gitlab/gitlab-ce:13.1.11-ce.0

将tar文件传送到目标服务器的 /data/dockers/gitlab/data/backup目录下，并调用恢复命令

docker exec -i gitlab13-1-11 gitlab-rake gitlab:backup:restore BACKUP=1710355231_2024_03_13_13.1.11

当我们成功恢复时，访问新的Gitlab地址是正常的，但当进入管理中心，设置一些信息保存时都会提示500错误， 查看Gitlab运行log,发现错误代码如下(在Gitlab15.11.13下报的)：

==> /var/log/gitlab/gitlab-rails/production.log <==

OpenSSL::Cipher::CipherError ():

lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `public_send'
lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `block in write_using_load_balancer'
lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'
lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'
lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'
lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `write_using_load_balancer'
lib/gitlab/database/load_balancing/connection_proxy.rb:72:in `transaction'
app/services/application_settings/update_service.rb:50:in `update_settings'
app/services/application_settings/update_service.rb:12:in `execute'
app/controllers/admin/application_settings_controller.rb:296:in `perform_update'
app/controllers/admin/application_settings_controller.rb:59:in `block (2 levels) in <class:ApplicationSettingsController>'
...

解决 OpenSSL::Cipher::CipherError (): 的方法不尽相同，网上的很多方法使用后并不起作用，最终在Gitlab的论坛上找到了一种简单实用的方法， 操作如下：

# 打开控制台
docker exec -i gitlab15-11-13 gitlab-rails console

# 执行如下操作，重置ApplicationSetting
ApplicationSetting.first.delete
ApplicationSetting.first

这时候，我们在设置管理中心的数据时都能成功了。

如果保存设置都是成功的， 但是进入某个项目下的设置，点击 CI/CD 页面500错误时 

#打开控制台
docker exec -i gitlab15-11-13 gitlab-psql
#执行下面命令
UPDATE projects SET runners_token = null, runners_token_encrypted = null;

下面是Docker 升级Gitlab的流程

13.1.11 -> 13.8.8 -> 13.12.15 -> 14.0.12 -> 14.3.6 -> 14.9.5 -> 14.10.5 -> 15.0.2 -> 15.1.2 -> 15.4.6 -> 15.9.3 -> 15.11.13 -> 16.1.0 -> 16.3.5 -> 16.7.7 -> 16.9.2 -> 16.11.6 -> 17.0.4


docker run -d  -p 8880:80 -p 222:22 --name gitlab13 -v /data/gitlab/config:/etc/gitlab -v /data/gitlab/data:/var/opt/gitlab -v /data/gitlab/logs:/var/log/gitlab gitlab/gitlab-ce:13.1.11-ce.0

docker run -d  -p 8880:80 -p 222:22 --name gitlab13.8 -v /data/gitlab/config:/etc/gitlab -v /data/gitlab/data:/var/opt/gitlab -v /data/gitlab/logs:/var/log/gitlab gitlab/gitlab-ce:13.8.8-ce.0

docker run -d  -p 8880:80 -p 222:22 --name gitlab13.12 -v /data/gitlab/config:/etc/gitlab -v /data/gitlab/data:/var/opt/gitlab -v /data/gitlab/logs:/var/log/gitlab gitlab/gitlab-ce:13.12.15-ce.0

docker run -d  -p 8880:80 -p 222:22 --name gitlab14.0 -v /data/gitlab/config:/etc/gitlab -v /data/gitlab/data:/var/opt/gitlab -v /data/gitlab/logs:/var/log/gitlab gitlab/gitlab-ce:14.0.12-ce.0

docker run -d  -p 8880:80 -p 222:22 --name gitlab14.3 -v /data/gitlab/config:/etc/gitlab -v /data/gitlab/data:/var/opt/gitlab -v /data/gitlab/logs:/var/log/gitlab gitlab/gitlab-ce:14.3.6-ce.0

docker start gitlab14.3
docker exec -it gitlab14.3 /bin/bash

gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds,id,'[["id"\, "stage_id"]\, ["id_convert_to_bigint"\, "stage_id_convert_to_bigint"]]']
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_job_artifacts,id,'[["id"\, "job_id"]\, ["id_convert_to_bigint"\, "job_id_convert_to_bigint"]]']
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_stages,id,'[["id"]\, ["id_convert_to_bigint"]]']
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds_metadata,id,'[["id"]\, ["id_convert_to_bigint"]]']
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds_metadata,id,'[["build_id"]\, ["build_id_convert_to_bigint"]]']
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,push_event_payloads,event_id,'[["event_id"]\, ["event_id_convert_to_bigint"]]']
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,taggings,id,'[["id"\, "taggable_id"]\, ["id_convert_to_bigint"\, "taggable_id_convert_to_bigint"]]']


docker run -d  -p 8880:80 -p 222:22 --name gitlab14.9 -v /data/gitlab/config:/etc/gitlab -v /data/gitlab/data:/var/opt/gitlab -v /data/gitlab/logs:/var/log/gitlab gitlab/gitlab-ce:14.9.5-ce.0

docker run -d  -p 8880:80 -p 222:22 --name gitlab14.10 -v /data/gitlab/config:/etc/gitlab -v /data/gitlab/data:/var/opt/gitlab -v /data/gitlab/logs:/var/log/gitlab gitlab/gitlab-ce:14.10.5-ce.0

docker run -d  -p 8880:80 -p 222:22 --name gitlab15.0 -v /data/gitlab/config:/etc/gitlab -v /data/gitlab/data:/var/opt/gitlab -v /data/gitlab/logs:/var/log/gitlab gitlab/gitlab-ce:15.0.2-ce.0

docker run -d  -p 8880:80 -p 222:22 --name gitlab15.1 -v /data/gitlab/config:/etc/gitlab -v /data/gitlab/data:/var/opt/gitlab -v /data/gitlab/logs:/var/log/gitlab gitlab/gitlab-ce:15.1.2-ce.0

docker run -d  -p 8880:80 -p 222:22 --name gitlab15.4 -v /data/gitlab/config:/etc/gitlab -v /data/gitlab/data:/var/opt/gitlab -v /data/gitlab/logs:/var/log/gitlab gitlab/gitlab-ce:15.4.6-ce.0

docker run -d  -p 8880:80 -p 222:22 --name gitlab15.9 -v /data/gitlab/config:/etc/gitlab -v /data/gitlab/data:/var/opt/gitlab -v /data/gitlab/logs:/var/log/gitlab gitlab/gitlab-ce:15.9.3-ce.0

docker run -d  -p 8880:80 -p 222:22 --name gitlab15.11.13 -v /data/dockers/gitlab/config:/etc/gitlab -v /data/dockers/gitlab/data:/var/opt/gitlab -v /data/logs/gitlab:/var/log/gitlab gitlab/gitlab-ce:15.11.13-ce.0

# 给16的版本做准备（未验证是否必须）
docker exec -i gitlab15.11.13 gitlab-rake gitlab:check SANITIZE=true

gitlab-rake db:migrate:down VERSION=20230411153310
gitlab-psql -c "INSERT INTO schema_migrations (version) VALUES ('20230411153310');"

gitlab-rake db:migrate:down VERSION=20230321163547
gitlab-psql -c "INSERT INTO schema_migrations (version) VALUES ('20230321163547');"

docker run -d -p 8880:80 -p 222:22 --name=gitlab16-1-0 -v /data/dockers/gitlab/config:/etc/gitlab -v /data/dockers/gitlab/data:/var/opt/gitlab -v /data/logs/gitlab:/var/log/gitlab -v /etc/localtime:/etc/localtime:ro --restart=always gitlab/gitlab-ce:16.1.0-ce.0


docker run -d -p 8880:80 -p 222:22 --name=gitlab16.3.5 -v /data/dockers/gitlab/config:/etc/gitlab -v /data/dockers/gitlab/data:/var/opt/gitlab -v /data/logs/gitlab:/var/log/gitlab -v /etc/localtime:/etc/localtime:ro --restart=always gitlab/gitlab-ce:16.3.5-ce.0


docker run -d -p 8880:80 -p 222:22 --name=gitlab16.7.7 -v /data/dockers/gitlab/config:/etc/gitlab -v /data/dockers/gitlab/data:/var/opt/gitlab -v /data/logs/gitlab:/var/log/gitlab -v /etc/localtime:/etc/localtime:ro --restart=always gitlab/gitlab-ce:16.7.7-ce.0

docker run -d -p 8880:80 -p 222:22 --name=gitlab16-9-2 -v /data/dockers/gitlab/config:/etc/gitlab -v /data/dockers/gitlab/data:/var/opt/gitlab -v /data/logs/gitlab:/var/log/gitlab -v /etc/localtime:/etc/localtime:ro --network=mynet --restart=always gitlab/gitlab-ce:16.9.2-ce.0

docker run -d -p 8880:80 -p 222:22 --name=gitlab16-11-6 -v /data/dockers/gitlab/config:/etc/gitlab -v /data/dockers/gitlab/data:/var/opt/gitlab -v /data/logs/gitlab:/var/log/gitlab -v /etc/localtime:/etc/localtime:ro --network=mynet --restart=always gitlab/gitlab-ce:16.11.6-ce.0

docker run -d -p 8880:80 -p 222:22 --name=gitlab17-0-4 -v /data/dockers/gitlab/config:/etc/gitlab -v /data/dockers/gitlab/data:/var/opt/gitlab -v /data/logs/gitlab:/var/log/gitlab -v /etc/localtime:/etc/localtime:ro --network=mynet --restart=always gitlab/gitlab-ce:17.0.4-ce.0

按上面的步骤成功升级。

但是发现项目提供的地址是docker容器的ID，我们需要替换为实际的网址。进入Docker修改下面的文件

### 进入 容器
docker exec -it gitlab16-9-2 bash
### 域名替换
vi /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml

...
  gitlab:
    ## Web server settings (note: host is the FQDN, do not include http://)
    host: yourdomain.com
    port: 8880
    https: false
...
  gitlab_shell:
...
    # If you use non-standard ssh port you need to specify it
    ssh_port: 222

### 重启服务
docker exec -i gitlab16-9-2 gitlab-ctl restart

如果拉取镜像有问题 建议修改 /etc/docker/daemon.json

{
    "data-root": "/data/var/lib/docker", # 修改docker 数据存放位置的地方
    "registry-mirrors": [
        "https://docker.m.daocloud.io",
        "https://noohub.ru",
        "https://huecker.io",
        "https://dockerhub.timeweb.cloud",
        "https://ccr.ccs.tencentyun.com/",
        "https://docker.mirrors.ustc.edu.cn"
    ]
}

其他 500错误

### 检查错误数据
gitlab-rake gitlab:doctor:secrets VERBOSE=1 # 查看问题
### 比如结果：
Gitlab::BackgroundMigration::BackfillIntegrationsEnableSslVerification::Integration[74]: properties

### 进入控制台
gitlab-rails console
### 查询并删除 find(xx), delete(xx)
Gitlab::BackgroundMigration::BackfillIntegrationsEnableSslVerification::Integration.find(74)

如果有 Project failures: Project[xx]: runners_token 这种错误 参考前面的CI/CD 页面500错误

以上总结，仅供参考

(130)

Related Post