- Error encountered when UNC02 is made active, preventing testing of devices.

Specific error message seen in UNC GUI: `class com.powerup.common.CMSystemException: class com.powerup.common.CMSystemException: java.security.InvalidKeyException: No key given`.

Below errors seen in  /opt/ionix-ncm/logs/commmgr.log 

Mar 27 08:29:39 1441511552#1: Error while processing infrastructure config trying again...
Mar 27 08:29:39 1441511552#2: InfrastructureCfgMgr::processConfig ... reset devices
Mar 27 08:29:39 1441511552#4: Restart commands wait time is configured as 30 seconds 
Mar 27 08:29:39 1441511552#2: InfrastructureCfgMgr::processDevList ... start updating devices
Mar 27 08:29:39 1441511552#1: setupKeyStore: Failed to get eKey from FileKeyStore, trying lockbox 
Mar 27 08:29:39 1441511552#1: Lockbox error: Lockbox Access Error: The primary internal decryption routine failed.
Mar 27 08:29:39 1441511552#1: Resolution:Lockbox Access Error: Please check the lockbox access on this server, Please refer to the NCM troubleshooting guide for possible resolution.

Unified Network Configurator (UNC)

DSR

The issue was caused by a lockbox synchronization problem between UNC01 and UNC02.  

When UNC02 is enabled and device credential validation or pull is executed, the following error is seen in /opt/ionix-ncm/logs/commmgr.log 

Mar 27 08:29:40 1441511552#1: Error while processing infrastructure config trying again...
Mar 27 08:29:40 1441511552#2: InfrastructureCfgMgr::processConfig ... reset devices
Mar 27 08:29:40 1441511552#4: Restart commands wait time is configured as 30 seconds 
Mar 27 08:29:40 1441511552#2: InfrastructureCfgMgr::processDevList ... start updating devices
Mar 27 08:29:40 1441511552#1: setupKeyStore: Failed to get eKey from FileKeyStore, trying lockbox 
Mar 27 08:29:40 1441511552#1: Lockbox error: Lockbox Access Error: The primary internal decryption routine failed.
Mar 27 08:29:40 1441511552#1: Resolution:Lockbox Access Error: Please check the lockbox access on this server, Please refer to the NCM troubleshooting guide for possible resolution.

The following steps were executed to resolve the issue:

1. UNC02 was rebuilt with a clean configuration, database was not restored.

2. The database from UNC01 was synchronized with UNC02.

3. Testing credential validation to multiple devices failed and Lockbox synchronization error occurred in commmgr.log.

4. UNC01 was made active and following procedure executed to perform manual lockbox synchronization:

-Obtain password using command below with out single quotes, this password will need to be entered in later steps when prompted:

'/opt/Motorola/bin/pwvault_wrapper -u ncm-db-user -a unc'

-Execute to the configuration file to get environment setup:

`./etc/voyence.conf`

- Copy the lockbox file:

`cp $VOYENCE_HOME/data/lockb.clb $VOYENCE_HOME/lockbox/xml/csp.clb`

- Change directories:

`cd /opt/ionix-ncm/bin`

- Execute the following commands and provide password obtained earlier when prompted:

`./cstadmin list-hosts`

`./cstadmin remove-host ucs-unc02.ucs`

`./cstadmin list-hosts`

`./cstadmin add-host ucs-unc02.ucs`

`./cstadmin list-hosts`

5. Restore the lockbox file:

`cp $VOYENCE_HOME/lockbox/xml/csp.clb $VOYENCE_HOME/data/lockb.clb`

6. Use the passphrase to reinitialize the lockbox:

PASSPHRASE=$( /opt/Motorola/bin/pwvault_wrapper -u ncm-db-user -a unc )

/opt/ionix-ncm/bin/cstdriver -lockbox $VOYENCE_HOME/data/lockb.clb -passphrase $PASSPHRASE