Apg 40 Change

APG40, Node, Change, APG40C/4

OPERATIONAL INSTRUCTION

Page 1 of 16

9/154 31-CRZ 222 05 Uen A

© Ericsson AB 2006 - all rights reserved

APG40, Node, Change, APG40C/4 Contents 1 1.1

Introduction Scope

2 2.1 2.2

Procedure Prerequisites Actions

3 3.1 3.2 3.3

Additional Information Update AM FW Delete customer data on the faulty node Update RAID FW

4

Glossary

5 5.1 5.2

References Operational Instructions Printout Descriptions

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 2 of 16

1 Introduction 1.1 Scope This Operational Instruction describes the procedure to change a node in the Adjunct Processor (AP) cluster. This procedure must be performed on site.

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 3 of 16

2 Procedure 2.1 Prerequisites 2.1.1

Conditions

The following conditions must apply before this procedure can be completed: • • • •

The alarm AP FAULT is received or there is a work order received. A technician is available on site. The node to replace is faulty. The other node is the active working node. The user account names and passwords that were set in the system at the time the backup was taken are also available. • The user name and password valid for a user with administrative rights on the replaced node. • The new node (spare part) must contain a correct LBB software installation, to be able to restore the node. • A valid backup of the node to be replaced available on the APG or on DAT-Tape or on FTP site.

2.1.2

Data

The following data should be known before starting this procedure: • The user name and password valid for a user with administrative rights on the active node. • The hostname of the active node. • The public IP address of the active node. • The hostname of the faulty node. • The hostname of the replacement node. • The public IP address of the replacement node. • The subnet mask of the replacement node. • The gateway address of the replacement node. 2.1.3 • • • • •

Special Aids A VGA monitor. A keyboard with PS/2 or USB connector. A mouse with PS/2 or USB connector. An adapter to connect the keyboard and monitor to the same cable. A grounding strap. To be used when exchanging the equipment.

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 4 of 16

2.2 Actions Find out which Node to Replace 1. Find out which node to replace by examining the alarm printout from AP FAULT or the work order. Logon to Faulty Node 2. Connect to the faulty node by inserting the adapter to the mouse/keyboard input on the AP. Connect the keyboard and the mouse to the adapter and connect the monitor to the monitor input. Logon as a user with administrative rights. Note: If it is not possible to login to the faulty node then proceed with Step 6. In that case stopping the cluster server is not possible as in steps from Step 3 to Step 5. Stop Cluster Service 3. Stop ACS_PRC_EventAnalyser on the faulty node. Enter the following command in a Command Prompt window: cluster res ACS_PRC_EventAnalyser_<0/1> /off /wait Note: Select either '0' or '1' depending on the last letter of the faulty AP node's hostname. If the last letter is 'a' select '0' and if the last letter is 'b' select '1'. 4. Stop acs_prc_clustercontrol on the faulty node. Enter the following command in a Command Prompt window: net stop acs_prc_clustercontrol 5. Stop Cluster Service on the faulty node. Enter the following command in a Command Prompt window: net stop clussvc Note: Confirm the question, cluster service will also stop "ACS_FCH_Server" service with "yes". Logon to Active Node 6. Connect to the active node by inserting the adapter to the mouse/keyboard input on the AP. Connect the keyboard and the mouse to the adapter and connect the monitor to the monitor input. Logon as a user with administrative rights. Check Drive Status 7. Ensure that all the drives residing in the active node have the status optimal and also check their capacity. Enter the following command in a Command Prompt window: megarc -dispcfg -a0 Note:

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 5 of 16

All three of them should be in the state optimal. Under the column Capacity you can see the capacity. Take a note of this value. If the drives on the active node have a status other than optimal then this Operational Instruction cannot correct the problem. 8. Are all the drives in the active node optimal? Yes Go to Step 9. No Go to Step 69. 9. Check which firmware version that is installed. Enter the following command in a Command Prompt window: megarc -ctlrinfo -a0 Note: Under the column firmware version you can see the version. Note this down. Note down Heartbeat IP addresses 10. Enter the following command in a Command Prompt window: ipconfig 11. Note down the IP addresses of Heartbeat 1 and Heartbeat 2. Bring Cluster into Safe State 12. Set cluster's hardware and software in a state that allows replacement of the faulty node. Enter the following command in the active node: fcc_save_to_remove other Note: Confirm the question with "yes". Note: If you were unable to perform the steps from Step 3 to Step 5 the command may give an error message since the command will try to stop the cluster service on the faulty node. Note: If the fcc_save_to_remove command hang (for more than 10 minutes) or does not shut down the node properly then log on to the faulty (passive) node and shut it down with START->Shut Down and select Shut down. Disconnect Faulty Node 13.

Warning!

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 6 of 16

In case of a suspected hardware fault, the node shall be exchanged. It's not allowed to exchange boards since that will violate any warranties.

Warning! There is a special procedure for removing the SCSI cables. This procedure must be followed to ensure that the SCSI bus will be terminated correctly. Wait until the Manual Intervention Allowed (MIA) LED is lit and then disconnect the faulty node. Disconnect the cables from the faulty node in the following order: Table 1

1 2 3 4 5 6 7 8 9

Power cable Public ethernet cable IPN ethernet cable External alarms cable Alarm panel cable Both Heartbeat ethernet cables Both Serial cables (to CPU board and AM board) Monitor cable Keyboard/Mouse adapter cable

Note: The SCSI cables should be removed in the correct order. See next step. Disconnect SCSI Cables 14. Always disconnect the SCSI connectors of the active node before removing the SCSI connectors of the powered-down (faulty) node. Disconnect the SCSI cables in the following order: Table 2

1 2 3 4

Disconnect Disconnect Disconnect Disconnect

the the the the

upper SCSI connector on the active node lower SCSI connector on the active node upper SCSI connector on the faulty node lower SCSI connector on the faulty node

Replace Node 15. Replace the faulty node with the replacement node. 16. Connect the cables in the following order: Table 3

1 2 3 4 5

Both Heartbeat cables Public ethernet cable Monitor cable Keyboard/mouse adapter cable Power cable

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 7 of 16

Logon to Replaced Node 17. Logon as a user with administrative rights. Stop Cluster Service 18. Stop the Cluster Service on the replaced node. Enter the following commands in a Command Prompt window: net stop clussvc Check Heartbeat IP addresses 19. Check the IP addresses of Heartbeat 1 and Heartbeat 2. Enter the following command in a Command Prompt window: ipconfig 20. Were the Heartbeat IP addresses of the replaced node conflict with the Heartbeat IP addresses of the active node? Yes Go to Step 21. No Go to Step 22. 21. Change the IP addresses of the Heartbeat 1 to "192.168.202.3" and Heartbeat 2 to "192.168.201.3". Note: Go to Start->all programs->accessories->communications->network connections. Select the corresponding Heartbeat connection and double click on it. Click the properties button in the general tab. Select Internet Protocol (TCP/IP) in the general tab and double click on it. Check Disk Capacity 22. Check which capacity the drives have on the replaced node. Enter the following command in a Command Prompt window: megarc -LogPhysInfo -a0 Note: Under the column Capacity you can see the capacity. Take a note of this value. 23. Was the capacity of the disks equal to or bigger than the disks in the active node? Yes Go to Step 25. No Go to Step 24. Note: Refer to Step 7 for the Disk Capacity in the active node.

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 8 of 16

24. This spare part cannot be used. The disks in the spare part must have equal or bigger capacity than the disks in the active node. Go to Step 69. Unpack Backup to System Disk 25. On which media do you have a valid backup? DAT Tape Go to Step 26. FTP site Go to Step 34. The shared disk M:\Images\node(a/b) Go to Step 35. 26. In the active node, insert a tape containing the backup. 27. Check that the correct backup is inserted in the tape drive. Enter the following command in a Command Prompt window: emfls 28. Is the backup valid? YES Go to Step 29. NO Go to Step 25. 29. Copy the backup file to be restored from the tape containing the backup file to M:\Images. Enter the following command in a Command Prompt window: emfcopy -m FROMTAPE .zip M:\Images\node(a/b) Note: The backup should be valid for the node to be restored. Note: Select either a or b in (a/b) depending on the last letter of the replaced AP node's hostname. 30. Enter the following command in a Command Prompt window: emfinfo -n Note: is the name of the node on which the copy operation was initiated. 31. Is the copy operation finished? YES Go to Step 32. NO Go to Step 30. 32. When copying is finished, eject the tape.

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 9 of 16

33. Go to Step 35. 34. Use FTP to transfer the compressed backup. The backup should be transferred to the directory M:\Images\node(a/b). The backup should be valid for the node to be restored. Select either a or b in (a/b) depending on the last letter of the restored AP node's hostname. See AP, File, Transfer and then return to Step 35 in this Operational Instruction. 35. Access the Replaced Node. 36. Enter the following command : net use Z: \\\images/<user> <password> Note: is the IP address of either heartbeat 1 or Heartbeat 2 on the working AP node where the shared directory 'images' is located. When prompted, logon with a user with administrative rights. Note: Refer to Step 10 for the IP addresses of Heartbeat 1 and Heartbeat 2 on the active node. 37. Enter the following command in a Command Prompt window: cd /d c:\Program Files\pkware\pkzipc 38. Enter the following command: pkzipc -extract Z:\node(a/b)\.zip "program files\ap\acs\bin\burrestore.exe" Note: The backup should be valid for the replaced node. Note: Select either a or b in (a/b) depending on the last letter of the replaced AP node's hostname. 39. Enter the following command: burrestore -a Z:\node(a/b)\.zip -d D:\ Note: The backup should be valid for the node to be restored. Note: Select either a or b in (a/b) depending on the last letter of the replaced AP node's hostname. Note: Confirm the question with "yes". The prompt is not returned until the backup is restored. Check RAID FW 40. In a Command Prompt window, enter the following command in replaced node: megarc -ctlrInfo -a0

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 10 of 16

41. Was the RAID firmware (FW) version in the replaced node equal to the one in the active node? YES Go to Step 43. NO Go to Section 3.3 and then return to Step 42. Note: Refer to Step 9 for the RAID FW version in the active node. 42. In a Command Prompt window, enter the following command: shutdown /r /t 1 Check AM FW 43. Check which AM FW version is used on the active node. Enter the following command in the active node: fcc_amversion own 44. Check which AM FW version is used on the replaced node. Enter the following command in the replaced node: fcc_amversion own 45. Are the AM FW versions identical on the active and on the replaced node? YES Go to Step 50. NO Go to Step 46. Note: Compare the printouts received in Step 43 and Step 44. 46. Enter the following command: dir /d c:\Program Files\Force\am_services 47. Is the correct version available on the replaced node? YES Go to Section 3.1 and return to Step 50. NO Go to Step 48. Note: The file used when upgrading the AM FW should be named where 'x' and 'y' is the AM FW version and should correspond to the printout received in Step 43. 48. Copy the correct 'lbb_am_appl_x_y_r.bin' file from the active node. Enter the following command:

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 11 of 16

copy \\\C$\Program Files\Force\am_services\ d:\Program Files\Force\am_services 49. Go to Section 3.1 and return to Step 50. Restoring From Backup 50. Enter the following command in the replaced node: burrestore -r -s Note: The NAME is to recognize the present LBB OS. 51. Confirm the question with "yes". Shutdown Replaced Node 52. In a Command Prompt window, enter the following command: shutdown /s /t 1 Note: Wait until "It is now safe to turn off your computer" message appears. 53. Disconnect the power cable from the replaced node. Reconnect SCSI Cables 54. Always connect the SCSI connectors of the replaced (powered-down) node before connecting the SCSI connectors of the active node. Connect the SCSI cables in the following order: Table 4

1 2 3 4

Connect Connect Connect Connect

the the the the

upper SCSI connector on the replaced node lower SCSI connector on the replaced node upper SCSI connector on the active node lower SCSI connector on the active node

Note: The SCSI cables must be crossconnected (i.e. SCSI 1 on one node to SCSI 0 on the other. The same for SCSI 0 to SCSI 1). Reconnect Other Cables 55. Reconnect the remaining cables in the following order: Table 5

1 2 3 4 5

IPN ethernet cable External alarms cable Alarm panel cable Both Serial cables (to CPU board and AM board) Power cable

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 12 of 16

Logon to Replaced Node 56. Logon as a user with administrative rights. Setup Services 57. Enter the following command in a Command Prompt window: setupservices 58. Enter the following command in a Command Prompt window: prcboot Note: The node will reboot and start up again. 59. Logon as a user with administrative rights. Logon to Active Node 60. Connect to the active node by inserting the adapter to the mouse/keyboard input on the AP. Connect the keyboard and the mouse to the adapter and connect the monitor to the monitor input. Logon as a user with administrative rights. Integrate Shared Mirrored Disk Drive 61. Integrate the shared mirrored disk drive. Enter the following command in a Command Prompt window: fcc_integrate other 62. Confirm action by entering 'y' followed by <ENTER>. Note: The replaced node should now join the cluster and the mirroring of the data disk will start. The mirroring process will take around 1-2 hours. The lower figure is for idle disks with no traffic to them and the higher figure is for maximum specified disk traffic. Check RAID Mirroring 63. Ensure that the mirroring of the RAID is still ongoing. Enter the following command: megarc -dispcfg -a0 Note: One of the RAID-1 should be in the status "rebuild" and the other two should be in the status "rebuild/ pending". Log Off from Active Node 64. Log off from the active node using START-> Log Off.

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 13 of 16

Logon to Replaced Node 65. Connect to the replaced node by inserting the adapter to the mouse/keyboard input on the AP. Connect the keyboard and the mouse to the adapter and connect the monitor to the monitor input. Logon as a user with administrative rights. Wait for RAID Mirroring to Complete 66. Before proceeding with the next step, make sure that the mirroring of the RAID is completed i.e. the RAID is optimal. megarc -ctlrinfo -a0 Note: The mirroring process will take around 1-2 hours. The lower figure is for idle disks with no traffic to them and the higher figure is for maximum specified disk traffic. Backup System 67. Go to Operational Instruction AP, System Backup and Verify, Initiate. Perform the steps in that Operational Instruction, and return to Step 68 in this Operational Instruction. 68. Go to Step 70. Consult Next Level of Maintenance Support 69. Consult the next level of maintenance support. Further action is outside the scope of this Operational Instruction. Prepare Faulty Node 70. Perform the actions in Section 3.2 and then return to Step 71. Job Completed 71. Make a report. Note: See Operational Instruction Report of Finished Work. 72. The job is completed.

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 14 of 16

3 Additional Information 3.1 Update AM FW 1. Start the AM configuration tool by using START->Programs->AM Configuration->AM Configuration. 2. Select the tab Installation. 3. Press UpgradeAMgo.. button. 4. In the AMgo Upgrade window press the Browse button. 5. Select the file lbb_am_appl_x_y_r.bin (where 'x' and 'y' is the AM FW version). 6. Press the Open button. 7. Press the Start button. 8. Answer OK on the pop up menu and wait until the firmware is upgraded. 9. When the upgrade status has changed to Success the upgrade is finished. 10. Press OK to exit the AM configuration tool. 11. The AM firmware is now updated.

3.2 Delete customer data on the faulty node Before the faulty node is sent for repair, the customer data must be deleted from the data disks. If possible, perform these actions off site when this OPI has been completed successfully. 1. Connect only the keyboard and monitor to the faulty node, and then connect the power cable. 2. Wait until Ctrl+M appears. Press Ctrl+M to activate the Raid Bios Configuration menu. 3. Use down arrow key to reach Initialize and press <Enter> to open submenu to display all logical drives. 4. Use down arrow key to browse and select each drive by pressing <Space> key. 5. Press to initialize. 6. Select the "Yes" button with the tab key and press <ENTER> to confirm selection. Process will take few minutes to complete. 7. Confirm the action with OK, the system reboot.

3.3 Update RAID FW 1. Enter the command in the Command Prompt window: cd /d c:\program files\force\raid\firmware 2. Enter the command in the Command Prompt window: NTflash /r /f mflashw.exe

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 15 of 16

4 Glossary -

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 16 of 16

5 References 5.1 Operational Instructions AP, System Backup and Verify, Initiate Report of Finished Work

5.2 Manual Pages burrestore (1m) prcboot(1m)

5.3 Printout Descriptions AP FAULT

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016