[ovirt-users] Correct recovery procedure of the oVirt Hosted Engine 4.0

[a***@it-kb.ru]

Hello oVirt guru`s!


My Hosted Engine VM located on a dedicated LUN FC Storage.

I do daily data backups (on NFS share) with the command:

/usr/bin/engine-backup --mode=backup --scope=all --file=$BcpFileName.xz --log=$BcpFileName.log --archive-compressor=xz --files-compressor=None

However, I don't know what would be the correct procedure to recover, because in different manuals outline the various steps.

For example, there is information that I have to do configure postgresql (with password from file files/etc/ovirt-engine/engine.conf.d/10-setup-database.conf) before restoring (engine-backup --mode=restore):
https://www.ovirt.org/documentation/admin-guide/hosted-engine-backup-and-restore/

And at the same time, in another document, there are no such steps:
https://access.redhat.com/documentation/en/red-hat-virtualization/4.0/single/self-hosted-engine-guide/#sect-Restoring_SHE_bkup

What should be the correct procedure for the recovery of Hosted Engine 4.0 ?

[Simone Tiraboschi]

Post by a***@it-kb.ru
Hello oVirt guru`s!
My Hosted Engine VM located on a dedicated LUN FC Storage.
/usr/bin/engine-backup --mode=backup --scope=all --file=$BcpFileName.xz
--log=$BcpFileName.log --archive-compressor=xz --files-compressor=None
However, I don't know what would be the correct procedure to recover,
because in different manuals outline the various steps.
For example, there is information that I have to do configure postgresql
(with password from file files/etc/ovirt-engine/engine.conf.d/10-setup-database.conf)
https://www.ovirt.org/documentation/admin-guide/hosted-engine-backup-and-
restore/

The recent releases of engine-backup can do that for you.
Adding Didi here.

Post by a***@it-kb.ru
https://access.redhat.com/documentation/en/red-hat-
virtualization/4.0/single/self-hosted-engine-guide/#
sect-Restoring_SHE_bkup
What should be the correct procedure for the recovery of Hosted Engine 4.0 ?
_______________________________________________
Users mailing list
http://lists.ovirt.org/mailman/listinfo/users

[a***@it-kb.ru]

Well.
Then, in the case of conditions:

1) the vm is not available anymore due to storage corruption
2) an empty shared storage is available
3) engine backup exists
4) all VMs still running on the hosts in the cluster


The recovery plan will be like this (as I understand it):


1) On all the hosts (if they are still available):

# service ovirt-ha-broker stop
# service ovirt-ha-agent stop
# chkconfig --del ovirt-ha-broker
# chkconfig --del ovirt-ha-agent


2) On first host (if the original host is not available anymore, provision a new host from scratch and proceed on this new host):

  2.1) # hosted-engine --deploy

 ◾use same fqdn you had previously in the HE VM.
 ◾point to the new shared storage
 ◾provide the same admin password you used in previous setup
 ◾install the OS on the vm
 ◾confirm it has been installed

 on Hosted Engine VM:

  a) Install the ovirt-engine rpms on the vm but don't run engine-setup:
  # yum install http://resources.ovirt.org/pub/yum-repo/ovirt-release40.rpm
  # yum install epel-release
  # yum install ovirt-engine
  b) Restore the backup:
  # engine-backup --mode=restore --file=file_name --log=log_file_name --provision-db --provision-dwh-db --restore-permissions
  c) Run "engine-setup"

   2.2) Open Administration Portal and remove the all old hosts used for Hosted Engine

   2.3) Confirm that the engine has been installed (Return to the host and continue the hosted-engine deployment script by selecting option 1) and then finish the deploy.

   2.4) In Administration Portal activate new host


3) On all additional hosts run "hosted-engine --deploy".


Right?

[Simone Tiraboschi]

Post by a***@it-kb.ru
Well.
1) the vm is not available anymore due to storage corruption
2) an empty shared storage is available
3) engine backup exists
4) all VMs still running on the hosts in the cluster
# service ovirt-ha-broker stop
# service ovirt-ha-agent stop
# chkconfig --del ovirt-ha-broker
# chkconfig --del ovirt-ha-agent
2) On first host (if the original host is not available anymore, provision
2.1) # hosted-engine --deploy
â—Ÿuse same fqdn you had previously in the HE VM.
â—Ÿpoint to the new shared storage
â—Ÿprovide the same admin password you used in previous setup
â—Ÿinstall the OS on the vm

I'd suggest to use the engine appliance also for this.
You can just say No when it asks about automatically running engine-setup.

Post by a***@it-kb.ru
â—Ÿconfirm it has been installed
# yum install http://resources.ovirt.org/pub/yum-repo/ovirt-release40.
rpm
# yum install epel-release
# yum install ovirt-engine
# engine-backup --mode=restore --file=file_name --log=log_file_name
--provision-db --provision-dwh-db --restore-permissions

In order to let the engine auto-import the new hosted-engine storage
domain, you have to remove the old one.
The same for the engine VM. Unfortunately you cannot do that from the
engine since they are somehow protected to avoid unintentional damages.
The easiest way is to remove them from the DB before running engine-setup.
I'm working on a helper utility to make it easiser:
https://gerrit.ovirt.org/#/c/64966/
I think I'll integrate it with engine-backup to simply do it with an
additional CLI flag.

Post by a***@it-kb.ru
c) Run "engine-setup"
2.2) Open Administration Portal and remove the all old hosts used for
Hosted Engine

Right, we can also integrate this step in the HE cleaning helper.

Post by a***@it-kb.ru
2.3) Confirm that the engine has been installed (Return to the host and
continue the hosted-engine deployment script by selecting option 1) and
then finish the deploy.
2.4) In Administration Portal activate new host
3) On all additional hosts run "hosted-engine --deploy".

I strongly suggest to deploy them from the engine and not from CLI.
CLI deploy support for additional HE host is deprecated an it will be
removed in 4.1.

Post by a***@it-kb.ru
Right?

[a***@it-kb.ru]

_______________________________________________
Users mailing list
***@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[Simone Tiraboschi]

Weird. The RHV4 guides not contain the information that we need to clean
the database from old storage domain before running the command
engine-setup.
What specific actions do we need?

You can check this as a reference but take care because it's still not
fully tested:
https://gerrit.ovirt.org/#/c/64966/3/packaging/setup/dbutils/hecleaner_sp.sql

Eventually I want to get a full recovery plan at the moment for oVirt 4.0.
Well.
1) the vm is not available anymore due to storage corruption
2) an empty shared storage is available
3) engine backup exists
4) all VMs still running on the hosts in the cluster
# service ovirt-ha-broker stop
# service ovirt-ha-agent stop
# chkconfig --del ovirt-ha-broker
# chkconfig --del ovirt-ha-agent
2) On first host (if the original host is not available anymore, provision
2.1) # hosted-engine --deploy
â—Ÿuse same fqdn you had previously in the HE VM.
â—Ÿpoint to the new shared storage
â—Ÿprovide the same admin password you used in previous setup
â—Ÿinstall the OS on the vm
I'd suggest to use the engine appliance also for this.
You can just say No when it asks about automatically running engine-setup.
â—Ÿconfirm it has been installed
# yum install http://resources.ovirt.org/pub/yum-repo/ovirt-release40.
rpm
# yum install epel-release
# yum install ovirt-engine
# engine-backup --mode=restore --file=file_name --log=log_file_name
--provision-db --provision-dwh-db --restore-permissions
In order to let the engine auto-import the new hosted-engine storage
domain, you have to remove the old one.
The same for the engine VM. Unfortunately you cannot do that from the
engine since they are somehow protected to avoid unintentional damages.
The easiest way is to remove them from the DB before running engine-setup.
https://gerrit.ovirt.org/#/c/64966/
I think I'll integrate it with engine-backup to simply do it with an
additional CLI flag.
c) Run "engine-setup"
2.2) Open Administration Portal and remove the all old hosts used for
Hosted Engine
Right, we can also integrate this step in the HE cleaning helper.
2.3) Confirm that the engine has been installed (Return to the host and
continue the hosted-engine deployment script by selecting option 1) and
then finish the deploy.
2.4) In Administration Portal activate new host
3) On all additional hosts run "hosted-engine --deploy".
I strongly suggest to deploy them from the engine and not from CLI.
CLI deploy support for additional HE host is deprecated an it will be
removed in 4.1.
Right?

[a***@it-kb.ru]

Ouch. It is beyond my understanding.

Thus, it appears that described in the RHV4 guide (https://access.redhat.com/documentation/en/red-hat-virtualization/4.0/single/self-hosted-engine-guide/#sect-Restoring_SHE_bkup) recovery procedure in fact incomplete?

[Simone Tiraboschi]

Post by a***@it-kb.ru
Ouch. It is beyond my understanding.
Thus, it appears that described in the RHV4 guide (
https://access.redhat.com/documentation/en/red-hat-
virtualization/4.0/single/self-hosted-engine-guide/#
sect-Restoring_SHE_bkup) recovery procedure in fact incomplete?

Yes, you are right although this is a kind of special case since we are
moving/restoring to a different storage domain while you are not asked to
remove the old storage if you are restoring in place.

[a***@it-kb.ru]

Hi Simone.
When can we expect a new version of the engine-backup with built-in cleaning helper?

Post by a***@it-kb.ru
Ouch. It is beyond my understanding.
Thus, it appears that described in the RHV4 guide (https://access.redhat.com/documentation/en/red-hat-virtualization/4.0/single/self-hosted-engine-guide/#sect-Restoring_SHE_bkup) recovery procedure in fact incomplete?

Yes, you are right although this is a kind of special case since we are moving/restoring to a different storage domain while you are not asked to remove the old storage if you are restoring in place.

[Simone Tiraboschi]

Post by a***@it-kb.ru
Hi Simone.
When can we expect a new version of the engine-backup with built-in cleaning helper?

That bug is targeted to 4.1

Post by a***@it-kb.ru

Post by Simone Tiraboschi

Post by a***@it-kb.ru
Ouch. It is beyond my understanding.
Thus, it appears that described in the RHV4 guide (

https://access.redhat.com/documentation/en/red-hat-
virtualization/4.0/single/self-hosted-engine-guide/#
sect-Restoring_SHE_bkup) recovery procedure in fact incomplete?

Post by Simone Tiraboschi
Yes, you are right although this is a kind of special case since we are

moving/restoring to a different storage domain while you are not asked to
remove the old storage if you are restoring in place.

[a***@it-kb.ru]

Simone, thank you for your work.
We will wait for the updates.

Post by Simone Tiraboschi

Post by a***@it-kb.ru
Hi Simone.
When can we expect a new version of the engine-backup with built-in cleaning helper?

That bug is targeted to 4.1