[ovirt-users] Ovirt node 4.4.5 failure to upgrade to 4.4.6

[Guillaume Pavese]

Maybe my problem is in part linked to an issue seen by Jayme earlier, but
then the resolution that worked for him did not succeed for me :

I first upgraded my Self Hosted Engine from 4.4.5 to 4.4.6 and then
upgraded it to Centos-Stream and rebooted

Then I tried to upgrade the cluster (3 ovirt-nodes on 4.4.5) but it failed
at the first host.
They are all ovir-node hosts, originally first installed in 4.4.5

In Host Event Logs I saw :

...
Update of host ps-inf-prd-kvm-fr-510.hostics.fr.
Upgrade packages
Update of host ps-inf-prd-kvm-fr-510.hostics.fr.
Check if image was updated.
Update of host ps-inf-prd-kvm-fr-510.hostics.fr.
Check if image was updated.
Update of host ps-inf-prd-kvm-fr-510.hostics.fr.
Check if image-updated file exists.
Failed to upgrade Host ps-inf-prd-kvm-fr-510.hostics.fr (User:
***@hostics.fr).



ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch was installed according to
yum,
I tried reinstalling it but got errors: "Error in POSTIN scriptlet" :

Downloading Packages:
[SKIPPED] ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch.rpm: Already
downloaded
...
Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch

Reinstalling : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch

Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch

warning: %post(ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch) scriptlet
failed, exit status 1

Error in POSTIN scriptlet in rpm package ovirt-node-ng-image-update
---
Reinstalled:
ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch



nodectl still showed it was on 4.4.5 :

[***@ps-inf-prd-kvm-fr-510 ~]# nodectl info
bootloader:
default: ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64)
...
current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1



I tried to upgrade the Host again from oVirt and this time there was no
error, and the host rebooted.
However, it did not pass active after rebooting and nodectl still shows
that it's 4.4.5 installed. Similar symptoms as OP

So I removed ovirt-node-ng-image-update, then reinstalled it and got no
error this time.
nodectl info seemed to show that it was installed :


[***@ps-inf-prd-kvm-fr-510 yum.repos.d]# nodectl info
bootloader:
default: ovirt-node-ng-4.4.6.3-0.20210518.0 (4.18.0-301.1.el8.x86_64)
...
current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1


However, after reboot the Host was still shown as "unresponsive"
After Marking it as "Manually rebooted", passing it in maintenance mode and
trying to activate it, the Host was automatically fenced. And still
unresponsive after this new reboot.

I passed it in maintenance mode again, And tried to reinstall it with
"Deploy Hosted Engine" selected
However if failed : "Task Stop services failed to execute."

In
/var/log/ovirt-engine/host-deploy/ovirt-host-deploy-ansible-20210602082519-ps-inf-prd-kvm-fr-510.hostics.fr-0565d681-9406-4fa7-a444-7ee34804579c.log
:

"msg" : "Unable to stop service vdsmd.service: Job for vdsmd.service
canceled.\n", "failed" : true,

"msg" : "Unable to stop service supervdsmd.service: Job for
supervdsmd.service canceled.\n", failed" : true,

"stderr" : "Error: ServiceOperationError: _systemctlStop failed\nb'Job for
vdsmd.service canceled.\\n' ",

"stderr_lines" : [ "Error: ServiceOperationError: _systemctlStop failed",
"b'Job for vdsmd.service canceled.\\n' " ],


If I try on the Host I get :

[***@ps-inf-prd-kvm-fr-510 ~]# systemctl stop vdsmd
Job for vdsmd.service canceled.

[***@ps-inf-prd-kvm-fr-510 ~]# systemctl status vdsmd
● vdsmd.service - Virtual Desktop Server Manager
Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor
preset: disabled)
Active: deactivating (stop-sigterm) since Wed 2021-06-02 08:49:21 CEST;
7s ago
Process: 54037 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh
--pre-start (code=exited, status=0/SUCCESS)
...

Jun 02 08:47:34 ps-inf-prd-kvm-fr-510.hostics.fr vdsm[54100]: WARN Failed
to retrieve Hosted Engine HA info, is Hosted Engine setup finished?
...
Jun 02 08:48:31 ps-inf-prd-kvm-fr-510.hostics.fr vdsm[54100]: WARN Worker
blocked: <Worker name=jsonrpc/4 running <Task <JsonRpcTask {'jsonrpc':
'2.0', 'method': 'StoragePool.connectStorageServer', 'params': {'storage>
File:
"/usr/lib64/python3.6/threading.py", line 884, in _bootstrap

self._bootstrap_inner()



Retrying to manually stop vdsmd a second time then seems to work...
I tried rebooting again, restarting the install always fail at the the same
spot

What should I try to get this host back up?



Guillaume Pavese
Ingénieur SystÚme et Réseau
Interactiv-Group
--
Ce message et toutes les piÚces jointes (ci-aprÚs le “message”) sont
établis à l’intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur, merci de le détruire et d’en avertir
immédiatement l’expéditeur. Toute utilisation de ce message non conforme a
sa destination, toute diffusion ou toute publication, totale ou partielle,
est interdite, sauf autorisation expresse. L’internet ne permettant pas
d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales)
décline(nt) toute responsabilité au titre de ce message, dans l’hypothÚse
ou il aurait été modifié. IT, ES, UK. 
<https://interactiv-group.com/disclaimer.html>

[wodel youchi]

Hi,

Is this an hci deployment?

If yes :

- try to boot using the old version 4.4.5
- verify that your network configuration is still intact
- verify that the gluster part is working properly
# gluster peer status
# gluster volume status

- verify the your engine and your host can resolve each other hostname

If all is ok try to bring the host at available state


Regards.

Le mer. 2 juin 2021 08:21, Guillaume Pavese <

Post by Guillaume Pavese
Maybe my problem is in part linked to an issue seen by Jayme earlier, but
I first upgraded my Self Hosted Engine from 4.4.5 to 4.4.6 and then
upgraded it to Centos-Stream and rebooted
Then I tried to upgrade the cluster (3 ovirt-nodes on 4.4.5) but it failed
at the first host.
They are all ovir-node hosts, originally first installed in 4.4.5
...
Update of host ps-inf-prd-kvm-fr-510.hostics.fr.
Upgrade packages
Update of host ps-inf-prd-kvm-fr-510.hostics.fr.
Check if image was updated.
Update of host ps-inf-prd-kvm-fr-510.hostics.fr.
Check if image was updated.
Update of host ps-inf-prd-kvm-fr-510.hostics.fr.
Check if image-updated file exists.
ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch was installed according to
yum,
[SKIPPED] ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch.rpm: Already
downloaded
...
Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Reinstalling : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
warning: %post(ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch) scriptlet
failed, exit status 1
Error in POSTIN scriptlet in rpm package ovirt-node-ng-image-update
---
ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
default: ovirt-node-ng-4.4.5.1-0.20210323.0
(4.18.0-240.15.1.el8_3.x86_64)
...
current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1
I tried to upgrade the Host again from oVirt and this time there was no
error, and the host rebooted.
However, it did not pass active after rebooting and nodectl still shows
that it's 4.4.5 installed. Similar symptoms as OP
So I removed ovirt-node-ng-image-update, then reinstalled it and got no
error this time.
default: ovirt-node-ng-4.4.6.3-0.20210518.0 (4.18.0-301.1.el8.x86_64)
...
current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1
However, after reboot the Host was still shown as "unresponsive"
After Marking it as "Manually rebooted", passing it in maintenance mode
and trying to activate it, the Host was automatically fenced. And still
unresponsive after this new reboot.
I passed it in maintenance mode again, And tried to reinstall it with
"Deploy Hosted Engine" selected
However if failed : "Task Stop services failed to execute."
In
/var/log/ovirt-engine/host-deploy/ovirt-host-deploy-ansible-20210602082519-ps-inf-prd-kvm-fr-510.hostics.fr-0565d681-9406-4fa7-a444-7ee34804579c.log
"msg" : "Unable to stop service vdsmd.service: Job for vdsmd.service
canceled.\n", "failed" : true,
"msg" : "Unable to stop service supervdsmd.service: Job for
supervdsmd.service canceled.\n", failed" : true,
"stderr" : "Error: ServiceOperationError: _systemctlStop failed\nb'Job
for vdsmd.service canceled.\\n' ",
"stderr_lines" : [ "Error: ServiceOperationError: _systemctlStop failed",
"b'Job for vdsmd.service canceled.\\n' " ],
Job for vdsmd.service canceled.
● vdsmd.service - Virtual Desktop Server Manager
Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor
preset: disabled)
Active: deactivating (stop-sigterm) since Wed 2021-06-02 08:49:21 CEST;
7s ago
Process: 54037 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh
--pre-start (code=exited, status=0/SUCCESS)
...
Jun 02 08:47:34 ps-inf-prd-kvm-fr-510.hostics.fr vdsm[54100]: WARN Failed
to retrieve Hosted Engine HA info, is Hosted Engine setup finished?
...
Jun 02 08:48:31 ps-inf-prd-kvm-fr-510.hostics.fr vdsm[54100]: WARN Worker
'2.0', 'method': 'StoragePool.connectStorageServer', 'params': {'storage>
"/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
Retrying to manually stop vdsmd a second time then seems to work...
I tried rebooting again, restarting the install always fail at the the
same spot
What should I try to get this host back up?
Guillaume Pavese
Ingénieur SystÚme et Réseau
Interactiv-Group
Ce message et toutes les piÚces jointes (ci-aprÚs le “message”) sont
établis à l’intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur, merci de le détruire et d’en avertir
immédiatement l’expéditeur. Toute utilisation de ce message non conforme a
sa destination, toute diffusion ou toute publication, totale ou partielle,
est interdite, sauf autorisation expresse. L’internet ne permettant pas
d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales)
décline(nt) toute responsabilité au titre de ce message, dans l’hypothÚse
ou il aurait été modifié. IT, ES, UK.
<https://interactiv-group.com/disclaimer.html>
_______________________________________________
Privacy Statement: https://www.ovirt.org/privacy-policy.html
https://www.ovirt.org/community/about/community-guidelines/

[Lev Veyde]

Hi Guillaume,

Have you moved the host to the maintenance before the upgrade (making sure
that Gluster related options are unchecked)?

Or you started the upgrade directly?

Thanks in advance,

Post by wodel youchi
Hi,
Is this an hci deployment?
- try to boot using the old version 4.4.5
- verify that your network configuration is still intact
- verify that the gluster part is working properly
# gluster peer status
# gluster volume status
- verify the your engine and your host can resolve each other hostname
If all is ok try to bring the host at available state
Regards.
Le mer. 2 juin 2021 08:21, Guillaume Pavese <

Post by Guillaume Pavese
Maybe my problem is in part linked to an issue seen by Jayme earlier, but
I first upgraded my Self Hosted Engine from 4.4.5 to 4.4.6 and then
upgraded it to Centos-Stream and rebooted
Then I tried to upgrade the cluster (3 ovirt-nodes on 4.4.5) but it
failed at the first host.
They are all ovir-node hosts, originally first installed in 4.4.5
...
Update of host ps-inf-prd-kvm-fr-510.hostics.fr.
Upgrade packages
Update of host ps-inf-prd-kvm-fr-510.hostics.fr.
Check if image was updated.
Update of host ps-inf-prd-kvm-fr-510.hostics.fr.
Check if image was updated.
Update of host ps-inf-prd-kvm-fr-510.hostics.fr.
Check if image-updated file exists.
ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch was installed according
to yum,
[SKIPPED] ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch.rpm: Already
downloaded
...
Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Reinstalling : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
warning: %post(ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch) scriptlet
failed, exit status 1
Error in POSTIN scriptlet in rpm package ovirt-node-ng-image-update
---
ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
default: ovirt-node-ng-4.4.5.1-0.20210323.0
(4.18.0-240.15.1.el8_3.x86_64)
...
current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1
I tried to upgrade the Host again from oVirt and this time there was no
error, and the host rebooted.
However, it did not pass active after rebooting and nodectl still shows
that it's 4.4.5 installed. Similar symptoms as OP
So I removed ovirt-node-ng-image-update, then reinstalled it and got no
error this time.
default: ovirt-node-ng-4.4.6.3-0.20210518.0 (4.18.0-301.1.el8.x86_64)
...
current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1
However, after reboot the Host was still shown as "unresponsive"
After Marking it as "Manually rebooted", passing it in maintenance mode
and trying to activate it, the Host was automatically fenced. And still
unresponsive after this new reboot.
I passed it in maintenance mode again, And tried to reinstall it with
"Deploy Hosted Engine" selected
However if failed : "Task Stop services failed to execute."
In
/var/log/ovirt-engine/host-deploy/ovirt-host-deploy-ansible-20210602082519-ps-inf-prd-kvm-fr-510.hostics.fr-0565d681-9406-4fa7-a444-7ee34804579c.log
"msg" : "Unable to stop service vdsmd.service: Job for vdsmd.service
canceled.\n", "failed" : true,
"msg" : "Unable to stop service supervdsmd.service: Job for
supervdsmd.service canceled.\n", failed" : true,
"stderr" : "Error: ServiceOperationError: _systemctlStop failed\nb'Job
for vdsmd.service canceled.\\n' ",
"stderr_lines" : [ "Error: ServiceOperationError: _systemctlStop
failed", "b'Job for vdsmd.service canceled.\\n' " ],
Job for vdsmd.service canceled.
● vdsmd.service - Virtual Desktop Server Manager
Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor
preset: disabled)
Active: deactivating (stop-sigterm) since Wed 2021-06-02 08:49:21
CEST; 7s ago
Process: 54037 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh
--pre-start (code=exited, status=0/SUCCESS)
...
Jun 02 08:47:34 ps-inf-prd-kvm-fr-510.hostics.fr vdsm[54100]: WARN
Failed to retrieve Hosted Engine HA info, is Hosted Engine setup finished?
...
Jun 02 08:48:31 ps-inf-prd-kvm-fr-510.hostics.fr vdsm[54100]: WARN
Worker blocked: <Worker name=jsonrpc/4 running <Task <JsonRpcTask
{'storage>
"/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
Retrying to manually stop vdsmd a second time then seems to work...
I tried rebooting again, restarting the install always fail at the the
same spot
What should I try to get this host back up?
Guillaume Pavese
Ingénieur SystÚme et Réseau
Interactiv-Group
Ce message et toutes les piÚces jointes (ci-aprÚs le “message”) sont
établis à l’intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur, merci de le détruire et d’en avertir
immédiatement l’expéditeur. Toute utilisation de ce message non conforme a
sa destination, toute diffusion ou toute publication, totale ou partielle,
est interdite, sauf autorisation expresse. L’internet ne permettant pas
d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales)
décline(nt) toute responsabilité au titre de ce message, dans l’hypothÚse
ou il aurait été modifié. IT, ES, UK.
<https://interactiv-group.com/disclaimer.html>
_______________________________________________
Privacy Statement: https://www.ovirt.org/privacy-policy.html
https://www.ovirt.org/community/about/community-guidelines/

_______________________________________________
Privacy Statement: https://www.ovirt.org/privacy-policy.html
https://www.ovirt.org/community/about/community-guidelines/

--
Lev Veyde

Senior Software Engineer, RHCE | RHCVA | MCITP

Red Hat Israel

<https://www.redhat.com>

***@redhat.com | ***@redhat.com
<https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

[Guillaume Pavese]

Hello,

I used the cluster upgrade feature that moves hosts in maintenance one by
one.
This is not a HCI cluster, my storage is on iSCSI multipath

I managed to fully upgrade the 1st hosts after rebooting and fixing some
network/iSCSI errors.
However, now the second one is stuck at upgrading the ovirt-node layers too
but I can not succeed in upgrading that one.
On this 2nd host, the workaround of removing and reinstalling
ovirt-node-ng-image-update doesn't work. I only get the following error :

[***@ps-inf-prd-kvm-fr-511 ~]# nodectl check
Status: OK
Bootloader ... OK
Layer boot entries ... OK
Valid boot entries ... OK
Mount points ... OK
Separate /var ... OK
Discard is used ... OK
Basic storage ... OK
Initialized VG ... OK
Initialized Thin Pool ... OK
Initialized LVs ... OK
Thin storage ... OK
Checking available space in thinpool ... OK
Checking thinpool auto-extend ... OK
vdsmd ... OK


[***@ps-inf-prd-kvm-fr-511 ~]# nodectl info
bootloader:
default: ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64)
entries:
ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64):
index: 0
kernel:
/boot//ovirt-node-ng-4.4.5.1-0.20210323.0+1/vmlinuz-4.18.0-240.15.1.el8_3.x86_64
args: resume=/dev/mapper/onn-swap
rd.lvm.lv=onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1
rd.lvm.lv=onn/swap rhgb quiet
boot=UUID=a676b18f-0f1b-4ad4-88e1-533fe61ff063 rootflags=discard
img.bootid=ovirt-node-ng-4.4.5.1-0.20210323.0+1 intel_iommu=on
root: /dev/onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1
initrd:
/boot//ovirt-node-ng-4.4.5.1-0.20210323.0+1/initramfs-4.18.0-240.15.1.el8_3.x86_64.img
title: ovirt-node-ng-4.4.5.1-0.20210323.0
(4.18.0-240.15.1.el8_3.x86_64)
blsid:
ovirt-node-ng-4.4.5.1-0.20210323.0+1-4.18.0-240.15.1.el8_3.x86_64
layers:
ovirt-node-ng-4.4.5.1-0.20210323.0:
ovirt-node-ng-4.4.5.1-0.20210323.0+1
current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1


[***@ps-inf-prd-kvm-fr-511 ~]# yum remove ovirt-node-ng-image-update
[...]
Removing:
ovirt-node-ng-image-update noarch
4.4.6.3-1.el8
@ovirt-4.4 886 M
Erasing : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Verifying : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Unpersisting: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch.rpm

Removed:
ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Complete!
[***@ps-inf-prd-kvm-fr-511 ~]#


[***@ps-inf-prd-kvm-fr-511 ~]# yum install ovirt-node-ng-image-update
[...]
Installing:
ovirt-node-ng-image-update noarch
4.4.6.3-1.el8
ovirt-4.4 887 M
[...]
ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch.rpm

23 MB/s | 887 MB 00:39
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing :
Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Installing : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
*warning: %post(ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch) scriptlet
failed, exit status 1*

*Error in POSTIN scriptlet in rpm package ovirt-node-ng-image-update *
Verifying : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch


Installed:


ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Complete!
[***@ps-inf-prd-kvm-fr-511 ~]#

[***@ps-inf-prd-kvm-fr-511 ~]# nodectl info
bootloader:
default: ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64)
entries:
ovirt-node-ng-4.4.5.1-0.20210323.0 (4.18.0-240.15.1.el8_3.x86_64):
index: 0
kernel:
/boot//ovirt-node-ng-4.4.5.1-0.20210323.0+1/vmlinuz-4.18.0-240.15.1.el8_3.x86_64
args: resume=/dev/mapper/onn-swap
rd.lvm.lv=onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1
rd.lvm.lv=onn/swap rhgb quiet
boot=UUID=a676b18f-0f1b-4ad4-88e1-533fe61ff063 rootflags=discard
img.bootid=ovirt-node-ng-4.4.5.1-0.20210323.0+1 intel_iommu=on
root: /dev/onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1
initrd:
/boot//ovirt-node-ng-4.4.5.1-0.20210323.0+1/initramfs-4.18.0-240.15.1.el8_3.x86_64.img
title: ovirt-node-ng-4.4.5.1-0.20210323.0
(4.18.0-240.15.1.el8_3.x86_64)
blsid:
ovirt-node-ng-4.4.5.1-0.20210323.0+1-4.18.0-240.15.1.el8_3.x86_64
layers:
ovirt-node-ng-4.4.5.1-0.20210323.0:
ovirt-node-ng-4.4.5.1-0.20210323.0+1
current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1
[***@ps-inf-prd-kvm-fr-511 ~]#

Is there any way to see where in the POSTIN scriplet the installation fails?

Guillaume Pavese
Ingénieur SystÚme et Réseau
Interactiv-Group

Post by Lev Veyde
Hi Guillaume,
Have you moved the host to the maintenance before the upgrade (making sure
that Gluster related options are unchecked)?
Or you started the upgrade directly?
Thanks in advance,

Post by wodel youchi
Hi,
Is this an hci deployment?
- try to boot using the old version 4.4.5
- verify that your network configuration is still intact
- verify that the gluster part is working properly
# gluster peer status
# gluster volume status
- verify the your engine and your host can resolve each other hostname
If all is ok try to bring the host at available state
Regards.
Le mer. 2 juin 2021 08:21, Guillaume Pavese <

Post by Guillaume Pavese
Maybe my problem is in part linked to an issue seen by Jayme earlier,
I first upgraded my Self Hosted Engine from 4.4.5 to 4.4.6 and then
upgraded it to Centos-Stream and rebooted
Then I tried to upgrade the cluster (3 ovirt-nodes on 4.4.5) but it
failed at the first host.
They are all ovir-node hosts, originally first installed in 4.4.5
...
Update of host ps-inf-prd-kvm-fr-510.hostics.fr.
Upgrade packages
Update of host ps-inf-prd-kvm-fr-510.hostics.fr.
Check if image was updated.
Update of host ps-inf-prd-kvm-fr-510.hostics.fr.
Check if image was updated.
Update of host ps-inf-prd-kvm-fr-510.hostics.fr.
Check if image-updated file exists.
ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch was installed according
to yum,
[SKIPPED] ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch.rpm: Already
downloaded
...
Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Reinstalling : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
warning: %post(ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch)
scriptlet failed, exit status 1
Error in POSTIN scriptlet in rpm package ovirt-node-ng-image-update
---
ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
default: ovirt-node-ng-4.4.5.1-0.20210323.0
(4.18.0-240.15.1.el8_3.x86_64)
...
current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1
I tried to upgrade the Host again from oVirt and this time there was no
error, and the host rebooted.
However, it did not pass active after rebooting and nodectl still shows
that it's 4.4.5 installed. Similar symptoms as OP
So I removed ovirt-node-ng-image-update, then reinstalled it and got no
error this time.
default: ovirt-node-ng-4.4.6.3-0.20210518.0 (4.18.0-301.1.el8.x86_64)
...
current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1
However, after reboot the Host was still shown as "unresponsive"
After Marking it as "Manually rebooted", passing it in maintenance mode
and trying to activate it, the Host was automatically fenced. And still
unresponsive after this new reboot.
I passed it in maintenance mode again, And tried to reinstall it with
"Deploy Hosted Engine" selected
However if failed : "Task Stop services failed to execute."
In
/var/log/ovirt-engine/host-deploy/ovirt-host-deploy-ansible-20210602082519-ps-inf-prd-kvm-fr-510.hostics.fr-0565d681-9406-4fa7-a444-7ee34804579c.log
"msg" : "Unable to stop service vdsmd.service: Job for vdsmd.service
canceled.\n", "failed" : true,
"msg" : "Unable to stop service supervdsmd.service: Job for
supervdsmd.service canceled.\n", failed" : true,
"stderr" : "Error: ServiceOperationError: _systemctlStop failed\nb'Job
for vdsmd.service canceled.\\n' ",
"stderr_lines" : [ "Error: ServiceOperationError: _systemctlStop
failed", "b'Job for vdsmd.service canceled.\\n' " ],
Job for vdsmd.service canceled.
● vdsmd.service - Virtual Desktop Server Manager
Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled;
vendor preset: disabled)
Active: deactivating (stop-sigterm) since Wed 2021-06-02 08:49:21
CEST; 7s ago
Process: 54037 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh
--pre-start (code=exited, status=0/SUCCESS)
...
Jun 02 08:47:34 ps-inf-prd-kvm-fr-510.hostics.fr vdsm[54100]: WARN
Failed to retrieve Hosted Engine HA info, is Hosted Engine setup finished?
...
Jun 02 08:48:31 ps-inf-prd-kvm-fr-510.hostics.fr vdsm[54100]: WARN
Worker blocked: <Worker name=jsonrpc/4 running <Task <JsonRpcTask
{'storage>
"/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
Retrying to manually stop vdsmd a second time then seems to work...
I tried rebooting again, restarting the install always fail at the the
same spot
What should I try to get this host back up?
Guillaume Pavese
Ingénieur SystÚme et Réseau
Interactiv-Group
Ce message et toutes les piÚces jointes (ci-aprÚs le “message”) sont
établis à l’intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur, merci de le détruire et d’en avertir
immédiatement l’expéditeur. Toute utilisation de ce message non conforme a
sa destination, toute diffusion ou toute publication, totale ou partielle,
est interdite, sauf autorisation expresse. L’internet ne permettant pas
d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales)
décline(nt) toute responsabilité au titre de ce message, dans l’hypothÚse
ou il aurait été modifié. IT, ES, UK.
<https://interactiv-group.com/disclaimer.html>
_______________________________________________
Privacy Statement: https://www.ovirt.org/privacy-policy.html
https://www.ovirt.org/community/about/community-guidelines/

_______________________________________________
Privacy Statement: https://www.ovirt.org/privacy-policy.html
https://www.ovirt.org/community/about/community-guidelines/

--
Lev Veyde
Senior Software Engineer, RHCE | RHCVA | MCITP
Red Hat Israel
<https://www.redhat.com>
<https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

--
Ce message et toutes les piÚces jointes (ci-aprÚs le “message”) sont
établis à l’intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur, merci de le détruire et d’en avertir
immédiatement l’expéditeur. Toute utilisation de ce message non conforme a
sa destination, toute diffusion ou toute publication, totale ou partielle,
est interdite, sauf autorisation expresse. L’internet ne permettant pas
d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales)
décline(nt) toute responsabilité au titre de ce message, dans l’hypothÚse
ou il aurait été modifié. IT, ES, UK. 
<https://interactiv-group.com/disclaimer.html>

[Yedidyah Bar David]

On Tue, Jun 8, 2021 at 8:01 AM Guillaume Pavese <

Post by Guillaume Pavese
Hello,
I used the cluster upgrade feature that moves hosts in maintenance one by
one.
This is not a HCI cluster, my storage is on iSCSI multipath
I managed to fully upgrade the 1st hosts after rebooting and fixing some
network/iSCSI errors.
However, now the second one is stuck at upgrading the ovirt-node layers
too but I can not succeed in upgrading that one.
On this 2nd host, the workaround of removing and reinstalling
Status: OK
Bootloader ... OK
Layer boot entries ... OK
Valid boot entries ... OK
Mount points ... OK
Separate /var ... OK
Discard is used ... OK
Basic storage ... OK
Initialized VG ... OK
Initialized Thin Pool ... OK
Initialized LVs ... OK
Thin storage ... OK
Checking available space in thinpool ... OK
Checking thinpool auto-extend ... OK
vdsmd ... OK
default: ovirt-node-ng-4.4.5.1-0.20210323.0
(4.18.0-240.15.1.el8_3.x86_64)
index: 0
/boot//ovirt-node-ng-4.4.5.1-0.20210323.0+1/vmlinuz-4.18.0-240.15.1.el8_3.x86_64
args: resume=/dev/mapper/onn-swap rd.lvm.lv=onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1
rd.lvm.lv=onn/swap rhgb quiet
boot=UUID=a676b18f-0f1b-4ad4-88e1-533fe61ff063 rootflags=discard
img.bootid=ovirt-node-ng-4.4.5.1-0.20210323.0+1 intel_iommu=on
root: /dev/onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1
/boot//ovirt-node-ng-4.4.5.1-0.20210323.0+1/initramfs-4.18.0-240.15.1.el8_3.x86_64.img
title: ovirt-node-ng-4.4.5.1-0.20210323.0
(4.18.0-240.15.1.el8_3.x86_64)
ovirt-node-ng-4.4.5.1-0.20210323.0+1-4.18.0-240.15.1.el8_3.x86_64
ovirt-node-ng-4.4.5.1-0.20210323.0+1
current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1
[...]
ovirt-node-ng-image-update noarch
4.4.6.3-1.el8
@ovirt-4.4 886 M
Erasing : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Verifying : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Unpersisting: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch.rpm
ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Complete!
[...]
ovirt-node-ng-image-update noarch
4.4.6.3-1.el8
ovirt-4.4 887 M
[...]
ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch.rpm
23 MB/s | 887 MB 00:39
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Installing : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
*warning: %post(ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch) scriptlet
failed, exit status 1*
*Error in POSTIN scriptlet in rpm package ovirt-node-ng-image-update *
Verifying : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Complete!
default: ovirt-node-ng-4.4.5.1-0.20210323.0
(4.18.0-240.15.1.el8_3.x86_64)
index: 0
/boot//ovirt-node-ng-4.4.5.1-0.20210323.0+1/vmlinuz-4.18.0-240.15.1.el8_3.x86_64
args: resume=/dev/mapper/onn-swap rd.lvm.lv=onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1
rd.lvm.lv=onn/swap rhgb quiet
boot=UUID=a676b18f-0f1b-4ad4-88e1-533fe61ff063 rootflags=discard
img.bootid=ovirt-node-ng-4.4.5.1-0.20210323.0+1 intel_iommu=on
root: /dev/onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1
/boot//ovirt-node-ng-4.4.5.1-0.20210323.0+1/initramfs-4.18.0-240.15.1.el8_3.x86_64.img
title: ovirt-node-ng-4.4.5.1-0.20210323.0
(4.18.0-240.15.1.el8_3.x86_64)
ovirt-node-ng-4.4.5.1-0.20210323.0+1-4.18.0-240.15.1.el8_3.x86_64
ovirt-node-ng-4.4.5.1-0.20210323.0+1
current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1
Is there any way to see where in the POSTIN scriplet the installation fails?

You can try using dnf's option '--rpmverbosity'.

Best regards,

Post by Guillaume Pavese
Guillaume Pavese
Ingénieur SystÚme et Réseau
Interactiv-Group

Post by Lev Veyde
Hi Guillaume,
Have you moved the host to the maintenance before the upgrade (making
sure that Gluster related options are unchecked)?
Or you started the upgrade directly?
Thanks in advance,

Post by wodel youchi
Hi,
Is this an hci deployment?
- try to boot using the old version 4.4.5
- verify that your network configuration is still intact
- verify that the gluster part is working properly
# gluster peer status
# gluster volume status
- verify the your engine and your host can resolve each other hostname
If all is ok try to bring the host at available state
Regards.
Le mer. 2 juin 2021 08:21, Guillaume Pavese <

Post by Guillaume Pavese
Maybe my problem is in part linked to an issue seen by Jayme earlier,
I first upgraded my Self Hosted Engine from 4.4.5 to 4.4.6 and then
upgraded it to Centos-Stream and rebooted
Then I tried to upgrade the cluster (3 ovirt-nodes on 4.4.5) but it
failed at the first host.
They are all ovir-node hosts, originally first installed in 4.4.5
...
Update of host ps-inf-prd-kvm-fr-510.hostics.fr.
Upgrade packages
Update of host ps-inf-prd-kvm-fr-510.hostics.fr.
Check if image was updated.
Update of host ps-inf-prd-kvm-fr-510.hostics.fr.
Check if image was updated.
Update of host ps-inf-prd-kvm-fr-510.hostics.fr.
Check if image-updated file exists.
ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch was installed according
to yum,
[SKIPPED] ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch.rpm: Already
downloaded
...
Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Reinstalling : ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
Running scriptlet: ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
warning: %post(ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch)
scriptlet failed, exit status 1
Error in POSTIN scriptlet in rpm package ovirt-node-ng-image-update
---
ovirt-node-ng-image-update-4.4.6.3-1.el8.noarch
default: ovirt-node-ng-4.4.5.1-0.20210323.0
(4.18.0-240.15.1.el8_3.x86_64)
...
current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1
I tried to upgrade the Host again from oVirt and this time there was no
error, and the host rebooted.
However, it did not pass active after rebooting and nodectl still shows
that it's 4.4.5 installed. Similar symptoms as OP
So I removed ovirt-node-ng-image-update, then reinstalled it and got no
error this time.
default: ovirt-node-ng-4.4.6.3-0.20210518.0 (4.18.0-301.1.el8.x86_64)
...
current_layer: ovirt-node-ng-4.4.5.1-0.20210323.0+1
However, after reboot the Host was still shown as "unresponsive"
After Marking it as "Manually rebooted", passing it in maintenance mode
and trying to activate it, the Host was automatically fenced. And still
unresponsive after this new reboot.
I passed it in maintenance mode again, And tried to reinstall it with
"Deploy Hosted Engine" selected
However if failed : "Task Stop services failed to execute."
In
/var/log/ovirt-engine/host-deploy/ovirt-host-deploy-ansible-20210602082519-ps-inf-prd-kvm-fr-510.hostics.fr-0565d681-9406-4fa7-a444-7ee34804579c.log
"msg" : "Unable to stop service vdsmd.service: Job for vdsmd.service
canceled.\n", "failed" : true,
"msg" : "Unable to stop service supervdsmd.service: Job for
supervdsmd.service canceled.\n", failed" : true,
"stderr" : "Error: ServiceOperationError: _systemctlStop failed\nb'Job
for vdsmd.service canceled.\\n' ",
"stderr_lines" : [ "Error: ServiceOperationError: _systemctlStop
failed", "b'Job for vdsmd.service canceled.\\n' " ],
Job for vdsmd.service canceled.
● vdsmd.service - Virtual Desktop Server Manager
Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled;
vendor preset: disabled)
Active: deactivating (stop-sigterm) since Wed 2021-06-02 08:49:21
CEST; 7s ago
Process: 54037 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh
--pre-start (code=exited, status=0/SUCCESS)
...
Jun 02 08:47:34 ps-inf-prd-kvm-fr-510.hostics.fr vdsm[54100]: WARN
Failed to retrieve Hosted Engine HA info, is Hosted Engine setup finished?
...
Jun 02 08:48:31 ps-inf-prd-kvm-fr-510.hostics.fr vdsm[54100]: WARN
Worker blocked: <Worker name=jsonrpc/4 running <Task <JsonRpcTask
{'storage>
"/usr/lib64/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
Retrying to manually stop vdsmd a second time then seems to work...
I tried rebooting again, restarting the install always fail at the the
same spot
What should I try to get this host back up?
Guillaume Pavese
Ingénieur SystÚme et Réseau
Interactiv-Group
Ce message et toutes les piÚces jointes (ci-aprÚs le “message”) sont
établis à l’intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur, merci de le détruire et d’en avertir
immédiatement l’expéditeur. Toute utilisation de ce message non conforme a
sa destination, toute diffusion ou toute publication, totale ou partielle,
est interdite, sauf autorisation expresse. L’internet ne permettant pas
d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales)
décline(nt) toute responsabilité au titre de ce message, dans l’hypothÚse
ou il aurait été modifié. IT, ES, UK.
<https://interactiv-group.com/disclaimer.html>
_______________________________________________
Privacy Statement: https://www.ovirt.org/privacy-policy.html
https://www.ovirt.org/community/about/community-guidelines/

_______________________________________________
Privacy Statement: https://www.ovirt.org/privacy-policy.html
https://www.ovirt.org/community/about/community-guidelines/

--
Lev Veyde
Senior Software Engineer, RHCE | RHCVA | MCITP
Red Hat Israel
<https://www.redhat.com>
<https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

Ce message et toutes les piÚces jointes (ci-aprÚs le “message”) sont
établis à l’intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur, merci de le détruire et d’en avertir
immédiatement l’expéditeur. Toute utilisation de ce message non conforme a
sa destination, toute diffusion ou toute publication, totale ou partielle,
est interdite, sauf autorisation expresse. L’internet ne permettant pas
d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales)
décline(nt) toute responsabilité au titre de ce message, dans l’hypothÚse
ou il aurait été modifié. IT, ES, UK.
<https://interactiv-group.com/disclaimer.html>
_______________________________________________
Privacy Statement: https://www.ovirt.org/privacy-policy.html
https://www.ovirt.org/community/about/community-guidelines/